Cassandra Connectivity Issues

Nodes cannot reach each other, or clients cannot reach the cluster. The two cases present differently and use different checks, so they are handled separately below. A node showing DN (Down/Normal) to its peers is frequently a connectivity or gossip problem rather than a failed process; see Gossip Failures for the gossip-level diagnosis of a node marked down.

Prerequisites

• Firewall or network-administration access on the hosts and any intervening devices, to apply the resolutions that open ports or change routing.
• Access to edit cassandra.yaml and to restart the service on each affected node, for the resolutions that change listen or broadcast addresses.

---

Part A: Inter-Node Connectivity

Symptoms

• Nodes marking each other down (DN in nodetool status)
• Streaming failures during bootstrap, repair, or rebuild
• Intermittent timeouts between nodes under load

Diagnosis

# Test reachability on the inter-node ports
nc -zv <node> 7000    # storage (inter-node)
nc -zv <node> 7001    # SSL storage (inter-node, when encryption is enabled)
nc -zv <node> 7199    # JMX

# Check for TCP retransmits and timeouts
netstat -s | grep -i "retransmit\|timeout"

# Check for an MTU mismatch (1472 + 28 bytes of headers = 1500)
ping -M do -s 1472 <node>

A failing ping -M do at a payload size that should fit the path indicates an MTU mismatch: packets above the smallest MTU on the path are dropped rather than fragmented.

Resolution

Cause	Solution
Firewall blocking	Open ports 7000, 7001, and 9042 between nodes (see the caution below)
MTU mismatch	Set a consistent MTU across every host and network device on the path
Network saturation	Add bandwidth, or throttle streaming with nodetool setstreamthroughput
TCP timeouts	Tune kernel TCP settings for the link’s latency and loss

Opening a firewall port changes the host’s exposure and is a production, security-affecting change: open only the specific ports listed, scoped to the source addresses that need them (the other cluster nodes for 7000/7001, the client network for 9042), not to all sources. Immediately after applying the change, re-run the reachability check against the port that was opened to confirm the path now works and nothing wider was exposed:

nc -zv <node> 7000

To revert the change, remove the firewall rule that was added and re-run the same nc -zv check to confirm the port is closed again.

---

Part B: Client Connection Issues

Symptoms

• Clients cannot connect to the cluster
• Connection timeouts from application hosts
• Connections succeed from some hosts but not others

Diagnosis

# Native transport status on the node
nodetool statusbinary

# Reachability of the native port from a client host
nc -zv <node> 9042

# Current client connections
nodetool clientstats

# Native-transport and connection errors in the log
grep -i "native\|connect" /var/log/cassandra/system.log | grep -i error

Resolution

nodetool statusbinary reporting anything other than running means the native transport is not accepting client connections. Confirm it is enabled in cassandra.yaml:

start_native_transport: true

When the transport is running but clients still cannot connect, the addresses the node listens on and advertises must be reachable from the client network:

• rpc_address is the address the node binds the native transport to. Binding to a loopback or an unreachable interface prevents remote clients from connecting.
• broadcast_rpc_address is the address returned to clients for other nodes in the cluster. It must be an address clients can reach; an internal address advertised to external clients causes connections to the coordinator to succeed and subsequent connections to fail.

Changes to these settings take effect at the next restart. In a multi-node cluster, apply the change as a rolling restart, one node at a time, confirming each node returns to UN in nodetool status before restarting the next. Restarting several nodes at once can drop the number of available replicas below the consistency level in use and open an availability gap.

---

Recovery

• Confirm nc succeeds on the relevant ports (7000/7001 between nodes, 9042 from clients).
• Confirm every node shows UN in nodetool status.
• Confirm clients reconnect and nodetool clientstats shows the expected connections.

---

Related Commands

Command	Purpose
nodetool status	Node up/down state across the cluster
nodetool statusbinary	Native (CQL) transport status
nodetool clientstats	Current client connections
nodetool setstreamthroughput	Throttle streaming to relieve network saturation
nc -zv <node> <port>	Test reachability of a specific port

Related Documentation

• Gossip Failures - Node marked down at the gossip level
• Node Fails to Start - Startup failures, including port conflicts
• Repair Failures - Repair problems, including streaming failures