Driver Policies¶

Driver policies control how the application interacts with the Cassandra cluster during normal operation and failure scenarios. These policies are the primary mechanism through which developers configure failure handling behavior.

Developer Responsibility for Failure Handling¶

Unlike traditional databases where failure handling is largely abstracted away, Cassandra drivers expose failure scenarios directly to the application. The developer is responsible for configuring appropriate responses to failures.

This design is intentional: Cassandra's distributed architecture means that "failure" is nuanced. A node being slow is different from a node being down. A write timeout does not mean the write failed—it may have succeeded on some replicas. The driver cannot make assumptions about what the application considers acceptable behavior.

Failure Type	What Happened	Driver's Question	Developer Must Decide
Read timeout	Some replicas didn't respond in time	Retry or fail?	Is stale data acceptable? Retry elsewhere?
Write timeout	Coordinator didn't get enough acknowledgments	Retry or fail?	Is duplicate write acceptable? Is operation idempotent?
Unavailable	Not enough replicas alive to satisfy CL	Retry or fail?	Lower consistency acceptable? Wait and retry?
Node down	Node unreachable	Where to route? When to retry connection?	Failover strategy? Recovery timing?

Default policies exist but are generic. Production applications must evaluate each policy against their specific requirements for consistency, latency, and availability.

Policy Overview¶

Policy	Question It Answers	Default Behavior
Load Balancing	Which node should handle this request?	Round-robin across local datacenter, token-aware
Retry	Should a failed request be retried?	Retry read timeouts once, don't retry write timeouts
Reconnection	How quickly to reconnect after node failure?	Exponential backoff (driver-specific defaults)
Speculative Execution	Should redundant requests be sent?	Disabled

Default Policy Behavior¶

Understanding default behavior is essential before customizing policies.

Java Driver Defaults (v4.x)¶

Policy	Configuration	Behavior
Load Balancing	`basic.load-balancing-policy`	Token-aware, prefers local DC, round-robin within replicas
Retry	`DefaultRetryPolicy`	Retry read timeout if enough replicas responded; never retry write timeout
Reconnection	`ExponentialReconnectionPolicy`	Base: 1 second, Max: 60 seconds (verify for specific version)
Speculative Execution	None	Disabled—must explicitly enable

Python Driver Defaults¶

Policy	Default Implementation	Behavior
Load Balancing	`TokenAwarePolicy(DCAwareRoundRobinPolicy())`	Token-aware wrapping DC-aware round-robin
Retry	`RetryPolicy`	Retry read timeout once on same host; retry unavailable once on next host
Reconnection	`ExponentialReconnectionPolicy`	Base: 1 second, Max: 600 seconds
Speculative Execution	None	Disabled

Failure Scenarios¶

Understanding common failure scenarios helps in selecting appropriate policies.

Scenario 1: Single Node Failure¶

Policy involvement:

Load Balancing: Provides fallback nodes when primary fails
Retry: Determines if connection failure triggers retry
Reconnection: Schedules background reconnection to Node1

Scenario 2: Read Timeout (Partial Response)¶

Policy involvement:

Retry: Decides whether to retry based on how many replicas responded
Speculative Execution: Could have sent parallel request to avoid timeout

Scenario 3: Write Timeout (Dangerous)¶

Critical consideration: Write may have succeeded on R2 but acknowledgment was lost. Retrying non-idempotent writes risks data corruption.

Scenario 4: Network Partition¶

Policy involvement:

Load Balancing: Must route only to reachable nodes
Reconnection: Attempts to reconnect to partitioned nodes
Retry: Unavailable exceptions if CL cannot be met with reachable nodes

Multi-Datacenter Configuration¶

Multi-DC deployments require careful policy configuration to ensure correct behavior during normal operation and DC failures.

Local Datacenter Configuration¶

Always configure the local datacenter explicitly. This is the most critical setting for multi-DC deployments.

// Java - REQUIRED for multi-DC
CqlSession session = CqlSession.builder()
    .withLocalDatacenter("dc1")
    .build();

# Python - REQUIRED for multi-DC
from cassandra.policies import DCAwareRoundRobinPolicy
cluster = Cluster(
    load_balancing_policy=DCAwareRoundRobinPolicy(local_dc='dc1')
)

Multi-DC Request Routing¶

DC Failover Behavior¶

Configuration	Normal Operation	Local DC Down
`LOCAL_QUORUM` + local DC only	Routes to local DC	All requests fail
`LOCAL_QUORUM` + remote DC allowed	Routes to local DC	Still fails (`LOCAL_*` CLs only count local replicas)
`QUORUM` + remote DC allowed	May route anywhere	Continues if global quorum available

Multi-DC Policy Configuration¶

Configure multi-DC failover via application.conf:

datastax-java-driver {
  basic.load-balancing-policy {
    local-datacenter = "dc1"
  }
  advanced.load-balancing-policy {
    # Allow failover to remote DC (requires non-LOCAL consistency levels)
    dc-failover.max-nodes-per-remote-dc = 2
  }
}

LOCAL consistency levels

Allowing remote DC nodes in the load balancing policy does not enable failover for LOCAL_* consistency levels. These levels only count replicas in the coordinator's datacenter regardless of where the request originates.

Consistency Level Implications¶

Consistency Level	Multi-DC Behavior	DC Failure Impact
`LOCAL_ONE`	Local DC only	Fails if local DC down
`LOCAL_QUORUM`	Local DC only	Fails if local DC down
`QUORUM`	Global quorum	May succeed with one DC down
`EACH_QUORUM`	Quorum in every DC	Fails if any DC down
`ALL`	Every replica	Fails if any node down

Recommendation: Use LOCAL_QUORUM for most operations. Configure load balancer to allow remote DC failover only when acceptable for the use case.

Why Policies Matter¶

Default policies are designed for general use cases but may not match specific application requirements:

Load Balancing Examples¶

Scenario	Default Behavior	Problem
Multi-DC deployment	May route to remote DC	High latency if local DC not configured
Heterogeneous hardware	Equal distribution	Overloads weaker nodes
Batch analytics	Token-aware routing	Optimal for OLTP, but analytics may prefer round-robin

Retry Examples¶

Scenario	Default Behavior	Problem
Non-idempotent writes	May retry on timeout	Potential duplicate writes
Overloaded cluster	Retry immediately	Amplifies load, worsens situation
Read timeout	Retry same node	Node may still be slow

Reconnection Examples¶

Scenario	Default Behavior	Problem
Brief network blip	Exponential backoff	Slow recovery for transient issues
Node replacement	Standard reconnection	May attempt reconnection to decommissioned node
Rolling restart	Backoff after each node	Cascading delays

Policy Interactions¶

Policies do not operate in isolation—they interact during request execution:

If speculative execution is enabled, requests are sent concurrently:

Configuration Approach¶

Explicit Configuration¶

Do not rely on defaults for production deployments. Configure each policy explicitly via application.conf (preferred) or programmatically:

# application.conf - Recommended approach for Java Driver 4.x
datastax-java-driver {
  basic {
    load-balancing-policy.local-datacenter = "dc1"
  }
  advanced {
    reconnection-policy {
      class = ExponentialReconnectionPolicy
      base-delay = 1 second
      max-delay = 5 minutes
    }
    speculative-execution-policy {
      class = ConstantSpeculativeExecutionPolicy
      max-executions = 2
      delay = 100 milliseconds
    }
  }
}

// Load configuration from application.conf
CqlSession session = CqlSession.builder()
    .withConfigLoader(DriverConfigLoader.fromClasspath("application.conf"))
    .build();

Per-Statement Override¶

Some policies can be overridden per statement:

// Override retry policy for specific query
Statement statement = SimpleStatement.builder("SELECT * FROM users WHERE id = ?")
    .addPositionalValue(userId)
    .setRetryPolicy(FallthroughRetryPolicy.INSTANCE)  // No retries
    .build();

This allows different behavior for different query types (e.g., strict no-retry for non-idempotent writes).

Policy Recommendations by Use Case¶

Use Case	Load Balancing	Retry	Reconnection	Speculative Execution
OLTP (low latency)	Token-aware, local DC	Conservative (reads only)	Fast base (500ms)	Enable for reads
Batch/Analytics	Round-robin or token-aware	Aggressive retry	Standard	Disable
Multi-DC Active-Active	Token-aware, local DC, failover enabled	Per-DC retry	Standard	Local DC only
Write-heavy	Token-aware	No retry for writes	Standard	Disable
Read-heavy	Token-aware	Retry reads	Standard	Enable

OLTP Application Configuration¶

# application-oltp.conf
datastax-java-driver {
  basic.load-balancing-policy.local-datacenter = "dc1"
  advanced {
    load-balancing-policy.slow-replica-avoidance = true
    reconnection-policy {
      class = ExponentialReconnectionPolicy
      base-delay = 500 milliseconds
      max-delay = 2 minutes
    }
    speculative-execution-policy {
      class = ConstantSpeculativeExecutionPolicy
      max-executions = 2
      delay = 50 milliseconds
    }
  }
}

Multi-DC Active-Active Configuration¶

# application-multi-dc.conf
datastax-java-driver {
  basic.load-balancing-policy.local-datacenter = "dc1"
  advanced {
    load-balancing-policy {
      # Allow failover to remote DC (requires non-LOCAL consistency levels)
      dc-failover.max-nodes-per-remote-dc = 2
    }
    reconnection-policy {
      class = ExponentialReconnectionPolicy
      base-delay = 1 second
      max-delay = 5 minutes
    }
    # No speculative execution across DCs (latency difference too high)
  }
}

Section Contents¶

Load Balancing Policy — Node selection and request distribution
Retry Policy — Handling failed requests and error classification
Reconnection Policy — Recovery after node failures
Speculative Execution Policy — Reducing tail latency through redundant requests