Skip to content

AxonOps — AI-Native Control Plane for Open Source Data Platforms

Cassandra Driver Retry Policy

The retry policy determines whether to retry a failed request and on which node. This policy is critical for application reliability but must be configured carefully to avoid unintended consequences.


Not all errors are retryable. The retry policy classifies errors and decides appropriate action:

Error TypeTypical CauseRetryable?
ReadTimeoutExceptionReplica(s) did not respond in timeDepends on received/required
WriteTimeoutExceptionWrite coordinator timeoutUsually no (non-idempotent)
UnavailableExceptionNot enough replicas aliveUsually no (same CL will fail); consider fallback CL or circuit breaker
OverloadedExceptionCoordinator is overloadedRetry on different node
ServerErrorUnexpected server errorUsually no
QueryValidationExceptionSyntax or schema errorNo (not transient)

A read timeout includes information about how many replicas responded:

ReadTimeoutException Analysis:
Required replicas (based on CL): 2
Received responses: 1
Data received: false
Interpretation:
- Only 1 of 2 required replicas responded
- No replica sent actual data (only digest)
Retry decision:
- If 0 received: replica might be overloaded, retry elsewhere
- If received >= required but no data: coordinator issue, retry
- If received < required: may or may not help to retry

Write timeouts are more complex because writes may have partially succeeded:

WriteTimeoutException Analysis:
Write type: SIMPLE
Required acknowledgments: 2
Received acknowledgments: 1
Interpretation:
- Write reached coordinator
- At least 1 replica acknowledged
- Possibly 2+ replicas received it (timeout ≠ failure)
Danger:
- Retrying may cause duplicate write
- For non-idempotent operations (counters), this corrupts data

A statement is idempotent when applying it twice has the same effect as applying it once. The driver treats the idempotence flag on a statement as a gate rather than a hint. In Java driver 4.x the driver reads the flag before it consults the retry policy, and where the statement is not marked idempotent it rethrows the error to the application without asking the policy for a verdict.

Four events are gated on the flag:

EventStatement marked idempotentStatement not marked idempotent
Write timeoutThe retry policy decidesThe error is rethrown
Error response from the serverThe retry policy decidesThe error is rethrown
Request aborted before any response arrivedThe retry policy decidesThe error is rethrown
Slow response, with speculative execution configuredA second attempt is sent in parallelNo second attempt is sent

Two events are not gated. A read timeout and an unavailable error reach the retry policy whichever way the flag is set:

  • A read does not change data, so repeating it is safe.
  • An unavailable error means the coordinator rejected the request before contacting any replica. Nothing was applied, so there is nothing to duplicate.

The flag therefore decides one question: whether the driver may retry the failures in which the write may already have been applied. It has no effect on the failures in which nothing can have been applied.

The driver does not infer idempotence from the statement text. The application sets the flag, and the driver takes it at face value. See Marking Statements Idempotent.

StatementSafe to retryReason
SELECTyesA read does not change data.
INSERT or UPDATE of ordinary columns with USING TIMESTAMPyesEvery attempt writes the same cells with the same timestamp.
DELETE with USING TIMESTAMPyesEvery attempt writes the same tombstone with the same timestamp.
INSERT or UPDATE of ordinary columns without USING TIMESTAMPnoThe driver builds the request message once per attempt, and generates a timestamp then, so a retry is written at a later timestamp. That changes the outcome of last-write-wins resolution against a concurrent write.
INSERT ... USING TTLqualifiedThe expiry is computed from the time the write is applied, so a retried attempt expires later than the first. Safe unless the workload depends on the exact expiry instant.
Counter update, SET c = c + 1noEach attempt applies the increment again. See Counter Updates.
List append or prepend, SET l = l + [...] or SET l = [...] + lnoEach attempt appends again, so a retry leaves duplicate elements.
Set addition, SET s = s + {...}, or map put, SET m[k] = vyesWriting the same element or key twice produces the same collection.
A statement calling now(), uuid() or another non-deterministic functionnoThe function is evaluated again on each attempt and writes a different value.
Lightweight transaction, IF NOT EXISTS or IF <condition>noSee Lightweight Transactions.

Where a value must be both generated at write time and stable across retries, the application generates it before sending the statement and passes it as a bound value. A UUID produced in application code and bound as a parameter is written unchanged by every attempt; uuid() evaluated in the statement is not.

A counter update applies its increment once per attempt. UPDATE page_stats SET views = views + 1 WHERE page_id = ? adds one to the counter each time it reaches the replicas. Where the write times out after the increment was applied and the driver sends the statement again, the counter is incremented twice.

No form of the statement avoids this. Cassandra 5.0, 4.1 and 4.0 support only relative counter updates, so a counter update cannot be expressed as an absolute value that converges on repetition.

The damage also cannot be found afterwards. Reading the counter returns a number, and one increment and two increments both produce a plausible number. A read cannot establish which occurred, and there is no repair that restores the intended value.

Do not mark a counter update idempotent

Problem: A counter update adds its increment once per attempt. Marking it idempotent allows the driver to retry a write timeout, and the increment is applied a second time when the first attempt had already succeeded.

Symptoms: Counter values drift upward, in proportion to the write timeout rate. The drift is silent. Reading the counter cannot distinguish one increment from two, so the error is neither detectable nor repairable after the fact.

Instead: Leave the flag false on counter statements, and route them to an execution profile whose retry policy never retries. See Per-Statement Policy Override.

// BAD: a timed-out write is retried and the counter is double-counted
SimpleStatement.builder("UPDATE page_stats SET views = views + 1 WHERE page_id = ?")
.addPositionalValue(pageId)
.setIdempotent(true)
.build();
// GOOD: the driver rethrows the write timeout and the application decides
SimpleStatement.builder("UPDATE page_stats SET views = views + 1 WHERE page_id = ?")
.addPositionalValue(pageId)
.setIdempotent(false)
.build();

A counter update must not be marked idempotent.

A lightweight transaction that times out has an unknown outcome. The Paxos round may have committed or it may not have, and the error does not say which. The conditional result column, [applied], does not distinguish the two.

The coordinator reports a write type with a write timeout, saying what kind of write was attempted. For a lightweight transaction it is CAS. Neither policy shipped with Java driver 4.x retries it. DefaultRetryPolicy retries a write timeout only for write type BATCH_LOG. ConsistencyDowngradingRetryPolicy handles SIMPLE, BATCH, UNLOGGED_BATCH and BATCH_LOG, and rethrows every other write type. A timed-out lightweight transaction is therefore reported to the application whatever the idempotence flag says. ConsistencyDowngradingRetryPolicy also rethrows a read timeout at a serial level, because a read issued at SERIAL or LOCAL_SERIAL was issued at that level for a reason that downgrading would discard.

A lightweight transaction must not be marked idempotent.

Do not retry a timed-out lightweight transaction in application code

Problem: reissuing the conditional write after a timeout returns [applied] = false when the first attempt had in fact committed. That is the same response returned when another client won the race, so the application concludes it lost a race it had won, and takes the wrong branch.

Symptoms: an operation reports that the row already existed, or that a condition failed, when the application's own earlier attempt is what created the row or met the condition.

Instead: read the row at SERIAL or LOCAL_SERIAL, which returns the state the Paxos round committed, and branch on that. A second conditional write cannot supply the same answer.

// BAD: the outcome of the first attempt is unknown, and this cannot tell
// "someone else won" apart from "my own first attempt succeeded"
session.execute(SimpleStatement.builder(
"INSERT INTO accounts (id, owner) VALUES (?, ?) IF NOT EXISTS")
.addPositionalValues(id, owner).build());
// GOOD: establish what committed, then decide
Row row = session.execute(SimpleStatement.builder(
"SELECT owner FROM accounts WHERE id = ?")
.addPositionalValues(id)
.setConsistencyLevel(ConsistencyLevel.SERIAL).build()).one();

A retry policy is a set of callbacks, one for each class of failure. The driver classifies the failure and calls the callback for that class. It passes the callback the details of the failure: how many replicas the consistency level required, how many answered, and whether one of them returned data rather than a digest. For a write it also passes the write type (SIMPLE, BATCH, UNLOGGED_BATCH, BATCH_LOG, COUNTER and CAS for a lightweight transaction), which indicates how much of the write may already be durable. Those details are the whole basis for the decision.

Each callback returns a RetryVerdict. A verdict has two parts. The first is a RetryDecision:

DecisionMeaning
RETRY_SAMERetry on the same node
RETRY_NEXTRetry on the next node in the query plan
IGNOREReturn an empty result to the application, as if the request had succeeded
RETHROWPropagate the error to the application

The second is a getRetryRequest(previous) method that returns the request to send on the retry. Its default implementation returns the previous request unchanged, so a retry normally repeats the original request. Overriding it lets a policy modify the request first: its consistency level, its query timestamp, or its custom payload. That method is the mechanism by which consistency downgrading works.

Retry Policy Decision FlowRetry Policy Decision FlowRequest FailedClassify Error TypeError Type?Transient(timeout, unavailable)Permanent(syntax error, unauthorized)Check retry count/limitRetry count?Under limitExceeded limitRETRY_SAME or RETRY_NEXTRETHROWRETHROW(no point retrying)

Most drivers include a default policy that retries at most once and never lowers the consistency level. The Java driver's DefaultRetryPolicy is representative:

FailureDecisionCondition
Read timeoutRETRY_SAMEEnough replicas answered but none returned the data. A digest mismatch, where data was returned, is rethrown
Write timeoutRETRY_SAMEOnly for write type BATCH_LOG. Every other write type is rethrown
UnavailableRETRY_NEXTFirst attempt only

In the Java driver, DefaultRetryPolicy.onUnavailable returns RETRY_NEXT on the first attempt and RETHROW on every attempt after that. The coordinator refused the request based on its own view of which replicas were up, and that view may be stale. Another coordinator may hold a current view and accept the request. The retry does not change the consistency level, so it succeeds only if the second coordinator finds enough replicas alive.

This is a conservative policy suitable for mixed workloads.

Never retries any error; always propagates it to the application.

In Java driver 4.x this comes from a custom policy whose callbacks all return RETHROW, named under the advanced.retry-policy.class key in the driver's HOCON configuration file, typically application.conf:

datastax-java-driver.advanced.retry-policy {
class = com.example.FallthroughRetryPolicy
}

Java driver 3.x shipped this directly as FallthroughRetryPolicy.INSTANCE, set with withRetryPolicy on the cluster builder.

Use when:

  • Application handles retries itself
  • All operations are non-idempotent
  • Debugging (to see all errors)

Retries most errors multiple times:

Aggressive retry can cause cascading failures

Aggressive retry policies can cause cascading failures by amplifying load on an already struggling cluster.

Cascading Failure Scenario:
1. Node3 becomes slow (GC, disk issue)
2. Requests to Node3 timeout
3. Aggressive retry policy retries each request 3×
4. Node3 now receives 3× the requests
5. Node3 becomes slower
6. Timeouts increase, more retries triggered
7. Node3 overwhelmed, marks as DOWN
8. Load shifts to Node1, Node2
9. If they were near capacity, they may also degrade

A downgrading retry policy reissues a failed request at a lower consistency level than the one requested, instead of returning the error to the application. It does this by returning a verdict whose getRetryRequest method lowers the consistency level of the original request before the driver sends it again.

The trade-off is that the application gets a weaker guarantee than the one it asked for, and is not told. A read issued at LOCAL_QUORUM and retried at ONE may not see a write that was made at LOCAL_QUORUM before it. The same applies to a read issued at QUORUM.

ConsistencyDowngradingRetryPolicy ships with Java driver 4.10 and later. It downgrades a request at most once. If the downgraded request fails as well, the error is reported to the application.

On an unavailable error at a serial level (SERIAL or LOCAL_SERIAL), the verdict is RETRY_NEXT. A serial level means the failure happened in the Paxos phase of a lightweight transaction. The coordinator may be isolated from the rest of the cluster, so another coordinator may be able to complete the transaction. At any other level, the policy downgrades using the number of replicas the coordinator reported alive.

On a read timeout, the conditions are evaluated in this order:

ConditionVerdictReason
Consistency level is serialRETHROWDowngrading a CAS read is never correct
received is less than blockForDowngrade, based on receivedFewer replicas answered than the level required
Enough replicas answered but dataPresent is falseRETRY_SAMEThe replica the coordinator asked for the data had not yet been marked down; the same coordinator asks a different replica for it on the retry
Anything elseRETHROWThis is usually a digest mismatch. The read path's own reconciliation resolves it, and a retry at the same level does not

On a write timeout, the write type decides:

Write typeVerdictReason
SIMPLE, BATCHIGNORE if at least one replica acknowledged, otherwise RETHROWThe write is already durable on at least one replica
UNLOGGED_BATCHDowngrade, based on receivedAn unlogged batch is not atomic, so only part of it may have been persisted
BATCH_LOGRETRY_SAMEThe timeout occurred while writing the batch log, before the batch itself was applied
Any other typeRETHROW

The policy downgrades to a numeric level, THREE, TWO or ONE, chosen from the count of replicas that answered or were reported alive. It does not step down a ladder of named levels.

Downgrading discards the datacenter constraint

THREE, TWO and ONE count replicas cluster-wide. A request issued at LOCAL_QUORUM and downgraded to TWO or ONE may be satisfied by replicas in any datacenter. A write intended for the local datacenter can then be acknowledged only in a remote one, and a read can be answered by a datacenter that has not yet received the latest write. The Java driver documentation carries the same warning.

Replicas countedRetry level
3 or moreTHREE
Exactly 2TWO
Exactly 1ONE
0None; the verdict is RETHROW

A request issued at EACH_QUORUM is downgraded straight to ONE regardless of the count, because on Cassandra 5.0, 4.1 and 4.0 the error reports the blocked-for and live replica counts of the datacenter that failed, not of the cluster (JAVA-1005).

The write timeout case needs equal attention, because it does not surface as an error at all.

An ignored write timeout is reported as success

On a write timeout of type SIMPLE or BATCH where at least one replica acknowledged the write, ConsistencyDowngradingRetryPolicy returns IGNORE. The driver completes the request normally, so the application sees a successful write. The write reached at least one replica rather than the number the requested consistency level demanded. The application receives no indication that the guarantee it asked for was not met.

gocql takes the other approach. Its DowngradingConsistencyRetryPolicy holds a list of levels, ConsistencyLevelsToTry, and works through it: the initial attempt uses the level set on the query, the first retry uses the first entry in the list, the second retry uses the second entry, and the policy stops retrying once the attempt count exceeds the length of the list.

The operator supplies the sequence, so a ladder that preserves locality can be written directly. The Java policy cannot express one:

// gocql: a query issued at EACH_QUORUM steps down to QUORUM, then LOCAL_QUORUM
cluster.RetryPolicy = &gocql.DowngradingConsistencyRetryPolicy{
ConsistencyLevelsToTry: []gocql.Consistency{
gocql.Quorum,
gocql.LocalQuorum,
},
}

gocql applies the level change per attempt, independently of how it classifies the error. gocql classifies errors with four values: Retry repeats the request on the same connection, RetryNextHost sends it to another host, and Ignore and Rethrow match the Java decisions of the same name. For non-serial unavailable errors and for write timeouts, the classification otherwise reaches the same outcome as the Java policy. Three cases differ:

FailuregocqlJava policy
Write timeout of type COUNTER, at least one acknowledgmentIgnoreRETHROW
Read timeout with enough replicas and data present, that is, a digest mismatchRetryRETHROW
Read timeout with enough replicas but no data returnedRetry, at the next level in the listRETRY_SAME, at the original level

The third row follows from the ladder advancing on every attempt. Java retries that case at the level originally requested, because the coordinator had simply asked a replica it had not yet marked down. gocql lowers the level as well.

The COUNTER case is the one to weigh before adopting the policy. gocql reports a timed-out counter write as successful once at least one replica has acknowledged it, even though fewer replicas acknowledged than the requested level demanded. A counter increment cannot be replayed afterwards to establish what was applied.

DriverPolicyStatus
Java 4.10+ConsistencyDowngradingRetryPolicyShipped, opt-in and not the default (JAVA-2900)
Java 4.0 to 4.9NoneNot shipped
Java 3.xDowngradingConsistencyRetryPolicyDeprecated in 3.5.0 (JAVA-1752)
gocqlDowngradingConsistencyRetryPolicyPresent in the Apache gocql driver, and in the gocql/gocql releases that preceded it
Python 3.xDowngradingConsistencyRetryPolicyDocumented as deprecated and slated for removal in the next major release

An application that depends on the behaviour should therefore pin the driver version and confirm the policy is present before an upgrade, rather than assume it survives one.

A downgraded retry is still a retry, so the idempotency rules in Idempotency Consideration apply unchanged. A write timeout retried at ONE can still apply a counter increment twice.

Writing a custom retry policy is a supported alternative and is often the more precise one: a custom policy can downgrade only the statements for which a weaker guarantee is acceptable, refuse to downgrade non-idempotent writes, and record every downgraded request so that the affected rows can be re-read or repaired afterwards. See Custom Retry Policies.


For fine-grained control, implement the five callbacks directly:

CallbackCalled whenArguments beyond the request
onReadTimeoutVerdictThe coordinator reported a read timeoutconsistency level, blockFor, received, dataPresent, retryCount
onWriteTimeoutVerdictThe coordinator reported a write timeoutconsistency level, writeType, blockFor, received, retryCount
onUnavailableVerdictThe coordinator raised UnavailableExceptionconsistency level, required, alive, retryCount
onRequestAbortedVerdictThe request was aborted before any response arrivedthe error, retryCount
onErrorResponseVerdictThe server returned an error responsethe error, retryCount

These are the callback names of Java driver 4.10 and later, and the code below compiles against that shape. Before 4.10, the equivalent callbacks returned a RetryDecision directly.

The class must expose the public constructor (DriverContext context, String profileName), because the driver instantiates the policy by reflection from the advanced.retry-policy.class configuration key.

// Java driver 4.10+
public class IdempotentOnlyRetryPolicy implements RetryPolicy {
public IdempotentOnlyRetryPolicy(DriverContext context, String profileName) {
// Signature required by the driver
}
@Override
public RetryVerdict onReadTimeoutVerdict(
Request request,
ConsistencyLevel cl,
int blockFor,
int received,
boolean dataPresent,
int retryCount) {
// Reads are idempotent, so try another coordinator once
return retryCount == 0 ? RetryVerdict.RETRY_NEXT : RetryVerdict.RETHROW;
}
@Override
public RetryVerdict onWriteTimeoutVerdict(
Request request,
ConsistencyLevel cl,
WriteType writeType,
int blockFor,
int received,
int retryCount) {
// Retry only a statement the application marked idempotent
boolean idempotent =
request instanceof Statement
&& Boolean.TRUE.equals(((Statement<?>) request).isIdempotent());
return idempotent && retryCount == 0 ? RetryVerdict.RETRY_NEXT : RetryVerdict.RETHROW;
}
@Override
public RetryVerdict onUnavailableVerdict(
Request request, ConsistencyLevel cl, int required, int alive, int retryCount) {
// The coordinator's view of live replicas may be stale
return retryCount == 0 ? RetryVerdict.RETRY_NEXT : RetryVerdict.RETHROW;
}
@Override
public RetryVerdict onRequestAbortedVerdict(
Request request, Throwable error, int retryCount) {
return RetryVerdict.RETHROW;
}
@Override
public RetryVerdict onErrorResponseVerdict(
Request request, CoordinatorException error, int retryCount) {
return RetryVerdict.RETHROW;
}
@Override
public void close() {}
}

The policy is then named in the driver's HOCON configuration file, typically application.conf:

datastax-java-driver.advanced.retry-policy {
class = com.example.IdempotentOnlyRetryPolicy
}

Because a verdict can rewrite the request, a custom policy can perform the stepwise downgrade the shipped Java policy does not offer: EACH_QUORUM to QUORUM to LOCAL_QUORUM, which keeps the request answerable within the local datacenter instead of dropping straight to ONE.

// Java driver 4.10+: a verdict that retries the request at a lower consistency level
public class DowngradingVerdict implements RetryVerdict {
private final ConsistencyLevel level;
public DowngradingVerdict(ConsistencyLevel level) {
this.level = level;
}
@Override
public RetryDecision getRetryDecision() {
return RetryDecision.RETRY_SAME;
}
@Override
@SuppressWarnings("unchecked")
public <RequestT extends Request> RequestT getRetryRequest(RequestT previous) {
return previous instanceof Statement
? (RequestT) ((Statement<?>) previous).setConsistencyLevel(level)
: previous;
}
}

The two members below replace onUnavailableVerdict in the IdempotentOnlyRetryPolicy skeleton above. onReadTimeoutVerdict takes the same body, and the remaining callbacks are unchanged.

// Java driver 4.10+: one step down the ladder, or null at the bottom of it
private static final int MAX_DOWNGRADES = 2;
private static ConsistencyLevel downgrade(ConsistencyLevel current) {
if (current == DefaultConsistencyLevel.EACH_QUORUM) {
return DefaultConsistencyLevel.QUORUM;
}
if (current == DefaultConsistencyLevel.QUORUM) {
return DefaultConsistencyLevel.LOCAL_QUORUM;
}
return null;
}
@Override
public RetryVerdict onUnavailableVerdict(
Request request, ConsistencyLevel cl, int required, int alive, int retryCount) {
// Bound the ladder: at most MAX_DOWNGRADES retries, and no step below LOCAL_QUORUM
ConsistencyLevel next = retryCount < MAX_DOWNGRADES ? downgrade(cl) : null;
return next == null ? RetryVerdict.RETHROW : new DowngradingVerdict(next);
}

The bound is the part a custom policy must get right. retryCount is zero on the first decision for a request and rises with each retry, so comparing it against MAX_DOWNGRADES caps the attempts, and downgrade returning null stops the ladder at LOCAL_QUORUM. Without one of the two, a request that keeps failing is retried indefinitely. cl is the level the failed attempt used, so the second failure arrives with cl already set to QUORUM and steps to LOCAL_QUORUM.

The same structure allows a policy to refuse to downgrade a request that is not marked idempotent, and to record each downgraded request so that the affected rows can be re-read or repaired afterwards.

The flag is set per statement, on the statement itself or on its builder. Per-statement marking is the approach to use, because it records a decision about one query rather than about every query in the application.

// Java driver 4.x
SimpleStatement statement =
SimpleStatement.builder("UPDATE users SET name = ? WHERE id = ?")
.addPositionalValues("Alice", userId)
.setIdempotent(true)
.build();
// prepared: a PreparedStatement obtained from session.prepare(...)
BoundStatement bound = prepared.bind("Alice", userId).setIdempotent(true);

In Java driver 4.x the value has three states: true, false, and unset. Unset is the initial state of a statement that has never been marked, and it is not the same as false. A statement whose value is unset takes the driver-wide default at execution time; a statement marked false stays false.

The driver-wide default is the configuration option basic.request.default-idempotence, and it is false.

datastax-java-driver.basic.request {
# Leave this false. It applies to every statement that is not marked individually.
default-idempotence = false
}

Setting default-idempotence to true changes the safety of every unmarked statement

basic.request.default-idempotence = true marks every statement whose own value is unset as idempotent, including counter updates, list appends and lightweight transactions. One configuration line then allows the driver to retry write timeouts across the whole application, and there is no error or warning to indicate it. Leave the option at false unless every statement the application issues has been reviewed and marked individually.

Marking a statement false, rather than leaving it unset, keeps a counter update or a conditional write safe regardless of the driver-wide default.


In Java driver 4.x, the retry policy belongs to an execution profile rather than to a statement. A profile declares the policy, and a statement selects the profile:

datastax-java-driver.profiles.no-retry.advanced.retry-policy {
class = com.example.NeverRetryPolicy
}
// Java driver 4.x
SimpleStatement counterUpdate = SimpleStatement.builder(
"UPDATE page_stats SET views = views + 1 WHERE page_id = ?")
.addPositionalValue(pageId)
.setExecutionProfileName("no-retry") // Never retry a counter update
.build();

Java driver 3.x set the policy on the statement itself, with setRetryPolicy.


Monitor retry behavior in production:

MetricDescriptionWarning Sign
Retry rateRetries per secondSustained high rate indicates cluster issues
Retry success ratePercentage of retries that succeedLow success rate means retries are wasteful
Retry exhaustionRequests that failed after all retriesAny occurrence needs investigation

PracticeRationale
Default to conservativeBetter to fail fast than corrupt data
Mark idempotent operations explicitlyEnables safe retry for those operations
Monitor retry ratesHigh retry rates indicate underlying issues
Don't rely on retries for availabilityFix the root cause instead
Consider circuit breakersPrevent retry storms during outages