Historical Retrieval for the same time range

When the Historical Retrieval (HR) operation is executed over a time range that includes the same set of events multiple times, there is a chance of data duplication.

If the original time range contains events at timestamp T1, with values V1 and V2, the destination receives the following when Historical Retrieval is run:

• V1 with NoReplace

• V2 with InsertNoCompression

If Historical Retrieval is run again over the same time range, the destination contains T1, V1, V2, and a duplicate V2. This duplication occurs because InsertNoCompression allows additional identical values (V2 in this case) to be appended at the same timestamp (T1) without being removed.

Each subsequent HR pass over the same time range continues to add another V2 for the T1 timestamp, leading to potential unintended data duplication at the destination.

We recommend you avoid reprocessing the same time range with HR when using InsertNoCompression and the duplication of values is undesirable. If reprocessing is necessary, ensure the destination is cleared or removed beforehand.