Out-of-order data

Out-of-order data refers to events written to a stream that do not follow the expected ordering pattern. For timestamp-ordered streams, out-of-order data consists of events with timestamps that do not follow a strictly sequential pattern. For sequence-ordered streams, out-of-order data consists of events whose sequence numbers are not in ascending order. Stream Management must handle events arriving out of sequence by inserting them into their correct position in the stream index rather than simply appending them to the end. Understanding how out-of-order data is managed helps you optimize stream performance and prevent data quality issues.

Categories of out-of-order data

Out-of-order data scenarios fall into three primary categories:

• Highly out-of-order data

• Dual cursor data

• Parallel batch data

Each scenario presents different challenges and requires different prevention and resolution strategies. However, all three scenarios can be identified by the same symptom: write operations taking significantly longer than normal.

Note: Some degree of out-of-order data is normal and expected. For example, recovering from a temporary disconnection may result in events being written out of sequence. This does not indicate a problem with your configuration or usage. Out-of-order data only becomes a concern when it is severe enough to cause write performance degradation.

Highly out-of-order data

Highly out-of-order data occurs when events are written to a stream with completely randomized timestamps that follow no recognizable pattern or sequence. Unlike simple out-of-order data where timestamps arrive slightly out of sequence due to clock drift or network delays, highly out-of-order data has no predictable ordering logic. This makes it significantly more disruptive to indexing and causes severe write performance degradation. Because the timestamps are random, there is no consistent or easily identifiable pattern in the data itself that would indicate the issue. If you observe unexplained write performance degradation without a clear cause, highly out-of-order data may be a contributing factor.

Example

The following table illustrates how timestamps in a highly out-of-order stream have no recognizable sequence or pattern. In this example, a wind turbine stream is receiving events with randomized timestamps across multiple years:

Unlike simple out-of-order data where timestamps may be off by seconds or minutes, the timestamps here span years and follow no logical progression.

Dual cursor data

Dual cursor data refers to a condition where a stream is receiving two different time ranges of data at once, typically current realtime data and older historical backfill data. Dual cursor data occurs when these realtime and historical datasets are sent to the same stream simultaneously. Although each dataset is individually ordered by its own timestamps, the combined stream becomes out-of-order because events alternate between two distinct time periods. This interleaving disrupts indexing and can degrade write performance. In some cases, accessing live data before the backfill completes may still be necessary.

Example

The following table illustrates how a wind turbine stream appears when a real-time device and a backfilling device write to it simultaneously. The Source column is included for clarity and would not appear in the actual stream:

Each device writes in batches, so the stream receives blocks of real-time rows followed by blocks of backfill rows. While each batch is internally ordered, the alternating blocks create a continuously disordered index in the stream.

Prevention

To prevent out-of-order data from dual cursor writes, complete backfilling before enabling real-time writes. Finish writing all historical data before switching to real-time data collection.

Parallel batch data

Unanticipated system behavior or configuration issues may result in out-of-order data related to parallel batch data writes. Common causes include developer error (accidentally writing data in reverse order during a test or migration), transaction rollback followed by retry with different timestamps, or incorrect clock adjustments that cause a server to report times earlier than its previous writes. Additionally, applications that send multiple batches in parallel may experience out-of-order arrival if batches are not received in the order they were sent. This occurs when users assume their sorted batches will arrive in sequence, but network timing causes later batches to reach the server before earlier ones.

The following table illustrates a wind turbine stream where two batches were sent in chronological order, but network timing caused the second batch to arrive before the first. The Batch column is included for clarity and would not appear in the actual stream:

Batch 1 contained the earlier timestamps but arrived after Batch 2, causing the stream to receive data out of chronological order despite each batch being internally sorted.

Prevention

To prevent accidental out-of-order writes, serialize batch writes when sending multiple batches from a single source. Wait for each batch to complete before sending the next, ensuring batches arrive in order. Alternatively, send all data in a single batch to avoid coordination issues.

Resolution

If you encounter any of the scenarios listed above and need help repairing your streams, you can request assistance by opening a ticket with AVEVA support.