Scenario 3 workflow: eDNA to PI migration

The Scenario 3 workflow is for customers who have a pre-existing Data Archive that is receiving incoming, live data and out-of-order data. In this scenario, you will need to set up a second Data Archive to act as a temporary holding space for migrated data.

In step 18, you will merge the archives that have been transferred from the temporary Data Archive to the target Data Archive. Since the merge process is an offline process, the target Data Archive must be offline or shut down during the merge process.

Note: In order to properly migrate eDNA data to the temporary Data Archive, be sure there is a valid archive registered for the entire time range that the eDNA data covers.

1. Follow the instructions in this topic: Generate PI tags for eDNA point data.

2. Review and modify the mapping file as needed (change types and names).

3. Use PI Builder to create new PI tags on the target Data Archive based on the mapping file.

4. Communicate a tag freeze during the migration process. Advise administrators not to edit or create PI tags during the migration process.

5. PI tag edits can be disabled on the target Data Archive by limiting or restricting access permissions.

6. Force an archive shift to ensure the backfilling is performed on non-primary archive files.

7. Back up the target Data Archive. See Backup your source Data Archive for instructions.

8. Install Data Archive on the temporary Data Archive. Make sure to use the same version that is installed on the target Data Archive.

9. (Optional) Download the zip file that contains the license for the temporary Data Archive from the customer support portal to the host machine. The zip file contains the required license file (pilicense.dat).

10. Restore the backup to the temporary Data Archive. See Restore a Data Archive backup to a new computer.

11. (Optional) Unregister any archives that were transferred as part of the backup. See Unregister archives. (Required for Scenario 3 only.)

12. Create new historical archives for the time range being migrated. See Manage archive gaps

    Tip: Before creating empty archives, estimate the amount of data to migrate and then size accordingly. Size archives large enough to avoid auto-dynamic sizing, which significantly slows down the backfilling process. It is more efficient to create a smaller number of larger archive files for backfilling, than to create several smaller-duration archives. Any extra, empty space will be discarded during the merge process.

13. Run the eDNA to PI Migration Utility on the temporary Data Archive using the mapping file.

14. Ensure that all data has been flushed to disk on the temporary Data Archive and then stop the temporary Data Archive.

15. Copy and move archives from the temporary Data Archive to the target Data Archive server.

16. Stop the target Data Archive. See Start and stop Data Archive.

    Important: In the next step, you will record the start and end times for both the temporary and target archives. Recording these times helps identify which archives should be merged together. Because the temporary archives are empty and the target archives are not likely to be set to shift at regular intervals, the start and end times will not match.

17. To determine the start and end times of the temporary and target archives, run the -ahd <ARCHIVE PATH> command using the PI Diagnostic (pidiag) utility.

18. Merge archives using the offline archive utility (piarchss) for each input file. Make sure to use the same output file for all the input files. Begin with the oldest input file and continue in ascending time order (the offline archive utility will not work in descending or random time order). See Merge multiple archives into a single archive.

    Code example

    This example shows an example of merging archives with the target archive #1's start/end time is “1-Jan-02” to “1-Feb-02”, and target archive #2's is “1-Feb-02” to “1-Mar-02” and the temporary archive's start/end time is “1-Jan-02” and “1-Mar-02.”

    piarchss -if D:\pi\dat\temparc.dat -of D:\pi\dat\targetarc1.dat -filter_ex "1-jan-02" "1-Feb-02" -ost "1-jan-02" -oet "1-feb-02"

    piarchss -if D:\pi\dat\temparc.dat -of D:\pi\dat\targetarc12.dat -filter_ex "1-Feb-02" "1-Mar-02" -ost "1-Feb-02" -oet "1-Mar-02"

19. Bring the target Data Archive online. See Start and stop Data Archive.

20. On the target Data Archive, enable PI tag editing using the Database Security tool in PI System Management Tools (SMT). See Set access permissions with the Database Security tool.

21. Ensure that all archives are registered.

22. Finalize configuration of interfaces and tags to enable data collection for collected points.

23. Validate that all data has been transferred successfully.

24. Delete the temporary Data Archive and associated infrastructure.