Skip to content
AxonOps Documentation

AxonOps — AI-Native Control Plane for Open Source Data Platforms

Secondary Index (2i) Query Execution

Legacy secondary indexes (2i) maintain per-column index data in hidden tables. Each node indexes only the partitions it owns. Queries on indexed columns that lack a partition key constraint are executed using a scatter-gather pattern across all token ranges.

Scatter-Gather Execution Model

Section titled “Scatter-Gather Execution Model”
Scatter-Gather Pattern: Legacy Secondary Index (2i)Scatter-Gather Pattern: Legacy Secondary Index (2i)Broadcast to all token ranges(one replica per range)Node Alocal index scanNode Blocal index scanNode Clocal index scanNode Dlocal index scanNode Elocal index scanNode Flocal index scanClientCoordinator1. query4. aggregated result2. scatter3. partial results

Scatter Phase

Section titled “Scatter Phase”
  1. The coordinator receives a query containing an indexed column predicate.
  2. The coordinator identifies target nodes:
    • Global query: one replica from each token range (all nodes in the cluster).
    • Partition-restricted query: replicas for the specified partition only.
  3. The query is dispatched to all identified nodes concurrently.

Gather Phase

Section titled “Gather Phase”
  1. Each replica node:
    • Executes a local index lookup against the hidden index table.
    • Retrieves matching partition keys.
    • Performs base table reads for matched keys.
    • Serializes and returns partial results.
  2. The coordinator:
    • Collects responses from all contacted nodes.
    • Merges and deduplicates result sets.
    • Applies post-filtering for non-indexed predicates.
    • Returns the final result to the client.

Latency Characteristics

Section titled “Latency Characteristics”

Query latency is dominated by the slowest responding node:

T_total = T_network + max(T_node_1, T_node_2, ..., T_node_n) + T_merge


Where T_node_i = T_index_lookup + T_base_read + T_serialization

This tail latency sensitivity is pronounced in heterogeneous clusters where node performance varies. As cluster size grows, the probability of encountering a slow node on any given query increases.

Implementation Characteristics

Section titled “Implementation Characteristics”
CharacteristicImplementation
Storage structureSeparate hidden table per indexed column
ReplicationInherits base table replication strategy
CompactionIndependent from base table compaction
Query coordinationStatic scatter-gather
ConsistencyIndex and base table may diverge temporarily
Index build visibilityNo gossip propagation; status requires manual inspection

The hidden table approach means the index is replicated separately from the data it references. This can result in temporary divergence between index entries and base table rows during compaction or after node failures.

Limitations

Section titled “Limitations”

Scatter-gather at cluster scale

Every global 2i query contacts one replica per token range, regardless of how many results match. In a 30-node cluster, every query contacts 30 nodes. Latency degrades as cluster size increases and is always bounded by the slowest node.

LimitationImplication
All-node contact for global queriesLatency degrades linearly with cluster size
Tail latency sensitivityOne slow node delays all results
Unbounded coordinator memoryLarge result sets accumulate before returning to client
No failed index detectionQueries silently skip nodes with failed index state
Section titled “Related Documentation”