Skip to content
Maintained by AxonOps — production-grade documentation from engineers who operate distributed databases at scale Get Cassandra Help Get Kafka Help

TLS Troubleshooting

This section covers common TLS configuration issues and their resolution.

Diagnostic Tools

OpenSSL Commands

# Test TLS connection
openssl s_client -connect cassandra-node:9042 -tls1_2

# Show server certificate
openssl s_client -connect cassandra-node:9042 2>/dev/null | \
    openssl x509 -noout -text

# Test specific cipher suite
openssl s_client -connect cassandra-node:9042 \
    -cipher ECDHE-RSA-AES256-GCM-SHA384

# Verify certificate chain
openssl verify -CAfile ca-chain.pem server-cert.pem

# Check certificate expiration
openssl x509 -in cert.pem -noout -dates

# View certificate SANs
openssl x509 -in cert.pem -noout -ext subjectAltName

Keytool Commands

# List keystore contents
keytool -list -v -keystore keystore.jks -storepass changeit

# Check specific alias
keytool -list -v -keystore keystore.jks -storepass changeit -alias node1

# Export certificate from keystore
keytool -exportcert -keystore keystore.jks -storepass changeit \
    -alias node1 -file exported-cert.pem -rfc

# Verify keystore password
keytool -list -keystore keystore.jks -storepass wrongpassword

Cassandra Logs

Check logs for TLS-related errors:

# Grep for SSL/TLS errors
grep -i "ssl\|tls\|certificate\|keystore\|truststore" /var/log/cassandra/system.log

# Recent errors
tail -1000 /var/log/cassandra/system.log | grep -i "error\|exception"

Common Errors

Keystore/Truststore Not Found

Error: 

java.io.FileNotFoundException: /etc/cassandra/certs/keystore.jks (No such file or directory)

Causes: - Incorrect file path in cassandra.yaml - File does not exist - Cassandra user lacks read permission

Resolution: 

# Verify file exists
ls -la /etc/cassandra/certs/keystore.jks

# Check permissions
stat /etc/cassandra/certs/keystore.jks

# Fix permissions
chown cassandra:cassandra /etc/cassandra/certs/keystore.jks
chmod 400 /etc/cassandra/certs/keystore.jks

Incorrect Keystore Password

Error: 

java.io.IOException: Keystore was tampered with, or password was incorrect

Causes: - Wrong password in cassandra.yaml - Keystore corrupted - Wrong keystore type specified

Resolution: 

# Test password
keytool -list -keystore keystore.jks -storepass your_password

# If password is correct but error persists, check keystore type
keytool -list -keystore keystore.jks -storetype PKCS12 -storepass your_password

Certificate Chain Validation Failed

Error: 

sun.security.validator.ValidatorException: PKIX path building failed
sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

Causes: - CA certificate missing from truststore - Intermediate CA certificate missing - Certificate chain incomplete

Resolution: 

# Verify certificate chain
openssl verify -CAfile ca-chain.pem server-cert.pem

# Check truststore contains CA
keytool -list -keystore truststore.jks -storepass changeit

# Add missing CA certificate
keytool -import -file ca-cert.pem -keystore truststore.jks \
    -storepass changeit -alias ca -noprompt

# If intermediate CA is missing, add it too
keytool -import -file intermediate-ca.pem -keystore truststore.jks \
    -storepass changeit -alias intermediate-ca -noprompt

Certificate Expired

Error: 

javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: validity check failed

Causes: - Server certificate expired - CA certificate expired

Resolution: 

# Check certificate expiration
openssl x509 -in cert.pem -noout -dates

# Check all certificates in chain
openssl x509 -in server-cert.pem -noout -dates
openssl x509 -in intermediate-ca.pem -noout -dates
openssl x509 -in root-ca.pem -noout -dates

# Generate and deploy new certificates

Hostname Verification Failed

Error: 

javax.net.ssl.SSLPeerUnverifiedException: Hostname cassandra-node-1 not verified
java.security.cert.CertificateException: No subject alternative names matching IP address 10.0.1.1 found

Causes: - Certificate CN/SAN does not match connection hostname - Connecting via IP but certificate only has DNS names - Connecting via hostname not in SAN list

Resolution: 

# Check certificate SANs
openssl x509 -in cert.pem -noout -ext subjectAltName

# Check what hostname is being used
nslookup 10.0.1.1

# Options:
# 1. Regenerate certificate with correct SANs
# 2. Connect using hostname that matches certificate
# 3. Disable hostname verification (not recommended for production)

Regenerate certificate with IP SAN: 

# In san.cnf
[alt_names]
DNS.1 = cassandra-node-1
DNS.2 = cassandra-node-1.example.com
IP.1 = 10.0.1.1
IP.2 = 192.168.1.1

No Cipher Suites in Common

Error: 

javax.net.ssl.SSLHandshakeException: no cipher suites in common

Causes: - Server and client have incompatible cipher suites - TLS version mismatch - Weak ciphers disabled on one side

Resolution: 

# Check server's supported ciphers
nmap --script ssl-enum-ciphers -p 9042 cassandra-node

# Check client's available ciphers
openssl ciphers -v

# Update cassandra.yaml to include compatible ciphers
# Or update client configuration

Example fix in cassandra.yaml: 

server_encryption_options:
    cipher_suites:
        - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
        - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
        - TLS_RSA_WITH_AES_256_GCM_SHA384  # Fallback if needed

Protocol Version Mismatch

Error: 

javax.net.ssl.SSLHandshakeException: No appropriate protocol (protocol is disabled or cipher suites are inappropriate)

Causes: - Client requires TLS 1.3 but server only supports TLS 1.2 - Server requires TLS 1.2 but client only supports TLS 1.0

Resolution: 

# Test supported protocols
openssl s_client -connect cassandra-node:9042 -tls1_2
openssl s_client -connect cassandra-node:9042 -tls1_3

# Update cassandra.yaml

server_encryption_options:
    accepted_protocols:
        - TLSv1.2
        - TLSv1.3

Private Key Does Not Match Certificate

Error: 

java.security.UnrecoverableKeyException: Cannot recover key

Causes: - Private key in keystore does not match certificate - Keystore corrupted during creation

Resolution: 

# Verify key matches certificate
openssl x509 -noout -modulus -in cert.pem | openssl md5
openssl rsa -noout -modulus -in key.pem | openssl md5
# Both should output identical hashes

# Recreate keystore with matching key and certificate
openssl pkcs12 -export \
    -in cert.pem \
    -inkey key.pem \
    -out keystore.p12 \
    -name alias \
    -password pass:changeit

Internode Connection Failures

Error in logs: 

ERROR [MessagingService-Outgoing] OutboundConnection.java: Failed to connect
javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown

Causes: - Peer node's certificate not trusted - Truststore missing CA certificate - Certificate CN/SAN mismatch

Resolution: 

# Test internode connection
openssl s_client -connect other-node:7000 -CAfile truststore.pem

# Verify both nodes have same truststore
md5sum /etc/cassandra/certs/truststore.jks  # Run on both nodes

# Check certificate is signed by trusted CA
openssl verify -CAfile ca-chain.pem peer-cert.pem

Client Connection Failures

cqlsh Error: 

Connection error: ('Unable to connect to any servers', {'cassandra-node': error(...)})

Resolution: 

# Test with OpenSSL first
openssl s_client -connect cassandra-node:9042 -CAfile ca-cert.pem

# Check cqlshrc configuration
cat ~/.cassandra/cqlshrc

# Try with explicit certificate
cqlsh --ssl --ssl-certfile=/path/to/ca-cert.pem cassandra-node

Node Startup Failures

Node Fails to Start After Enabling TLS

Diagnostic Steps:

  1. Check logs for specific error: 

    tail -100 /var/log/cassandra/system.log

  2. Verify keystore/truststore accessible: 

    sudo -u cassandra cat /etc/cassandra/certs/keystore.jks > /dev/null

  3. Test keystore password: 

    keytool -list -keystore /etc/cassandra/certs/keystore.jks -storepass $(grep keystore_password cassandra.yaml | awk '{print $2}')

  4. Validate configuration syntax: 

    python3 -c "import yaml; yaml.safe_load(open('/etc/cassandra/cassandra.yaml'))"

Rolling Back

If TLS configuration prevents cluster operation:

# Temporarily disable encryption
sed -i 's/internode_encryption: all/internode_encryption: none/' /etc/cassandra/cassandra.yaml
sed -i 's/enabled: true/enabled: false/' /etc/cassandra/cassandra.yaml

# Restart node
systemctl restart cassandra

# Fix certificates, then re-enable

Performance Issues

High CPU Usage

TLS adds CPU overhead. If experiencing performance degradation:

# Check if AES-NI is available
grep aes /proc/cpuinfo

# Use AES-GCM ciphers (hardware accelerated)
cipher_suites:
    - TLS_AES_256_GCM_SHA384
    - TLS_AES_128_GCM_SHA256

Connection Latency

TLS handshake adds latency. Mitigations:

  • Enable TLS session resumption
  • Use TLS 1.3 (faster handshake)
  • Monitor handshake times

Debug Logging

Enable SSL Debug Output

Add to jvm.options:

# Full SSL debug (verbose)
-Djavax.net.debug=ssl

# Handshake only
-Djavax.net.debug=ssl:handshake

# Certificate chain
-Djavax.net.debug=ssl:trustmanager

Warning: SSL debug output is extremely verbose. Enable temporarily for troubleshooting only.

Interpret Debug Output

Key messages to look for:

# Successful handshake
"ServerHello" with cipher suite
"Certificate chain" showing peer's certificates
"Finished" indicating successful handshake

# Failures
"alert: fatal" indicates handshake failure
"certificate_unknown" means cert not trusted
"handshake_failure" general failure