Archive reprocessing

Starting with Data Archive 2016, you can use the PI SMT Archives tool to reprocess archives while they are online.

Note: You still need to perform some reprocessing tasks when an archive is offline by using the command-line tool, piarchss. For example, use piarchss to combine archive files or divide a large archive file into smaller files. See Offline archive file management with piarchss for more information.

When to reprocess an archive

It is always necessary to evaluate whether reprocessing an archive is the best course of action. Reprocessing should be done on an as-needed basis, rather than be automatic or scheduled.

The following situations are recommended reasons to reprocess archives:

• To repair a corrupt archive to enable access to its data

• To compress an archive to recover space (offline reprocessing only)

  For example, if you have deleted many PI points, you can remove the data of the deleted points, or you might want to compress archives that are not full.

• To improve access time to an archive for certain queries

  Reprocessing an archive typically improves performance for so-called deep queries, which read single points over a long time range. However, reprocessing an archive can worsen performance of queries that read multiple points over a time range (for example, points associated with batches or event frames).

Outside these situations, there is no gain from reprocessing archives, and even potential degradation of performance due to unnecessary contention for system resources on a production system (RAM, disk space, CPU). In particular, it is not necessary to reprocess archives after a Data Archive upgrade, or to schedule a nightly, weekly, or monthly task to reprocess all archive files.

Archive reprocessing in the PI SMT Archives tool

When you open the Archives tool under Operations in PI SMT, corrupt archives are flagged so that you know you need to reprocess them.

Note: You are not prompted about corruption in archives that have not recently been read by Data Archive. When Data Archive next accesses that archive, for example to query for data or write new data, it uses that opportunity to check for corruption.

Select one or more archives to reprocess; they are reprocessed sequentially. The Archives tool shows the progress of the reprocessing job and at completion provides the status of the operation.

Note: If you need to reprocess the primary archive, you must first perform an archive shift. For more information, see Force an archive shift.

While the archive is being reprocessed, you can still read data from that archive. Also, new data is held in memory and so is available to be read. After reprocessing completes, the data in memory is written to the archive file.

The input archive is renamed using a suffix and a time stamp, and the output archive takes the input archive's original name, start time, and end time. Output archives are online and registered after reprocessing. All archives, both dynamic and fixed, are reprocessed to a fixed archive.

If Data Archive shuts down while a reprocessing job is running, at startup the current job is restarted and the existing queue of job requests is maintained.

For detailed steps on how to reprocess an archive, see the PI SMT topic Reprocess an archive.

Archive_ReprocessThreadCount

The Archive_ReprocessThreadCount tuning parameter is the number of worker threads dedicated to reprocessing an archive file. On a low memory machine, only 1 thread will be used regardless of the value of this tuning parameter. The parameter takes effect before archive reprocessing. The default setting is half the number of processors on the server, the min: 1, and max: processor count.