Best practices for backfilling

During backfilling, data in files is sent to Data Archive at a rate that is much higher than the rate of the control system. To minimize the load on Data Archive, follow these recommendations.

• To avoid any burden on your main production server and risk to your real data, do backfilling jobs on an offline Data Archive server.

• To avoid archiving unnecessary data and ensure efficient backfilling, backfill with compression.

• Points for all backfilled data must be defined in the point database.

• Process data from all points in small batches, such as one day each.

• Backfill data in chronological order, from oldest to newest, within each point. If you backfill out-of-order data, it is not compressed.

For a large amount of data, follow these additional recommendations.

• Consider writing a custom application to do the backfilling.

• Before backfilling, run a backfill test with a smaller time range to ensure that the data imports properly. During the test, check the archive and snapshot statistics to see how the backfilling affects Data Archive performance, and troubleshoot any other issues as needed.