Kafka Authentication or Authorization Failure
Diagnose transport, authentication, and authorization separately. Test with the affected principal from the affected network.
Classify the Failure
Section titled “Classify the Failure”| Signal | Layer |
|---|---|
| Certificate or handshake exception | TLS/mTLS |
SaslAuthenticationException | SASL mechanism or credentials |
| OAuth claim/JWKS error | Token validation |
| Topic/group/cluster/transactional-ID denial | ACL authorization |
| Connection closes before Kafka error | Listener/security mismatch or authentication timeout |
TLS and mTLS
Section titled “TLS and mTLS”openssl s_client \ -connect broker.example.com:9093 \ -servername broker.example.com \ -CAfile ca.pem \ -verify_return_error </dev/nullCheck the full chain, SAN hostname, validity period, key usage, protocol/cipher compatibility, client certificate, and clock synchronization. Do not disable hostname verification as a permanent fix.
Run a Kafka CLI with the application's security properties:
kafka-cluster.sh list-endpoints \ --bootstrap-server broker.example.com:9093 \ --command-config client.propertiesVerify listener-specific mechanisms, JAAS context, principal, SCRAM credential, Kerberos DNS/keytab/ticket, and broker logs. Protect the properties file and redact secrets from output.
Check token expiry/not-before time, issuer, audience, scope, signature algorithm, JWKS key ID and reachability, TLS trust, allowed URL/file system properties, and clock skew.
Kafka 4.4
JWKS-backed OAUTHBEARER validation requires expected issuer and audience unless the explicit unverified opt-outs are configured. Treat those opt-outs as risk exceptions, not routine remediation.
kafka-acls.sh --bootstrap-server kafka:9093 \ --command-config admin.properties \ --list --principal User:orders-appMatch the denied operation to topic, group, transactional ID, cluster, delegation token, or share-group resources. Check literal versus prefixed patterns and host restrictions.
Recover and Verify
Section titled “Recover and Verify”Correct the narrow security mismatch. Avoid wildcard ACLs or plaintext listeners as diagnostic shortcuts. Repeat the real operation with the affected identity, confirm broker-side success, and verify no broader access was introduced.
Prevent Recurrence
Section titled “Prevent Recurrence”Monitor certificate/token expiry, authentication failures, ACL denials, JWKS availability, and time synchronization. Test rotations with overlap and rollback, and keep a secured break-glass administration path.