Prepare and stage PI Archive files

Completed by: PI Server administrator

Use PI Archive data files maintained through regular archive and backup maintenance. As the PI Server administrator, identify the files that cover the historical range to import and confirm where they are stored. Create and run the piconfig_pointlist.inp script provided at the end of this topic on the server where the PI Archive files are located to generate metadata files that assist with the import, and then stage the following two file sets in the prepared Azure file share:

• PI Archive data files for the historical range to import.

• PI Server metadata files generated by the piconfig_pointlist.inp script.

Prerequisites

Note: PI bulk import supports any AVEVA PI Server version and is fully backward compatible.

• Access to the following AVEVA PI Server resources:

  • PI Archive data files (which use the .arc extension by default) for the historical range to import. Confirm their location and ensure you have read access to the files on the server where they are stored. For guidance, see Back up the Data Archive, Manage PI Archive files, and PI Data Archive Backup Best Practices.

  • For the archive data set, include only PI Archive files that cover the historical range; omit all other file types.

  • Stage every PI Archive data file that covers the historical range. Do not combine multiple archive files before staging them.

  • Access to run PI scripts on the server where the PI Archive files are located, with piconfig.exe available on that server.

  • Write access to the folder where you save the script and its generated CSV files.

• Artifacts and information provided by the Azure account holder:

  • The Azure subscription, storage account, and file share to use.

  • Confirmation that the root-level CONFIG directory exists.

  • Confirmation that the file share has enough capacity for the estimated total upload size.

  • For Azure CLI uploads using Microsoft Entra authentication, confirmation that the PI Server administrator's Microsoft SSO account has the Storage File Data Privileged Contributor role assigned directly at the target file share scope.

  • Any agreed historical range, staging method, or upload schedule.

To prepare and stage PI Archive data files

Complete the following tasks in order to successfully prepare and stage PI Archive data files.

1. Complete Prepare PI Archive data files below.

   Identify and prepare the PI Archive files that contain the historical data you want to import. Include only PI Archive files in this set. Do not include other file types.

2. Complete Generate PI Server metadata files below.

   Generates and reviews the metadata CONNECT needs to interpret the PI points and their values.

3. Complete Stage files below.

   Uploads both file sets to the prepared Azure file share so CONNECT can access them during the import.

1. Prepare PI Archive data files

Perform this task on the server where the PI Archive files are located, using the folder or folders that contain the files for your selected historical range. The server hosting these files may be different from the PI Server.

4. Log in to the server where the PI Archive files are located.

   Log in locally, or use Microsoft Remote Desktop Protocol (RDP) or another remote access method. Authenticate with an account that can read the PI Archive files and run PI scripts on this server.

5. Locate and access the PI Archive data files for the historical range to import.

   Open the folder or folders containing the files for the selected historical range. Use an account with read access to those files.

6. Confirm that the individual PI Archive data files cover the historical range to import.

   Stage only PI Archive files that cover the historical range. Omit all other file types from the archive data set. If the range spans multiple archive files, stage each one separately without combining them.

   As the PI Server administrator, manually review the selected archives for gaps and duplicates. Confirm that any gaps are expected, and avoid duplicate archive files covering the same time range. Some overlap is acceptable, but overlapping archives can reduce import performance and increase transfer time.

   For more information on PI archive file management best practices, see the following PI Server documentation:

   • Back up the Data Archive

   • Manage PI Archive files

   • Move PI Archive files

   • PI Data Archive Backup Best Practices

   Keep your remote access session open so that you can reuse it for the next task.

2. Generate PI Server metadata files

Continue on the same server where the PI Archive files are located. Save and run the script in a folder you can write to on that server; it does not have to be the folder containing the archive files. The generated CSV files are saved in the folder where you run the script.

7. Copy the script in piconfig_pointlist.inp at the end of this topic and save it as piconfig_pointlist.inp in a folder on the server where the PI Archive files are located.

   Open Command Prompt as an administrator and change directory to where you saved piconfig_pointlist.inp.

   cd "<path-to-folder-containing-piconfig_pointlist.inp>"

8. Run piconfig_pointlist.inp from the current working directory using piconfig.exe.

   Usage:

   "<full-path-to-piconfig.exe>" < piconfig_pointlist.inp

   For a typical installation:

   "C:\Program Files\PI\adm\piconfig.exe" < piconfig_pointlist.inp

   Note: This command can run for hours without printing any output. If the terminal appears idle, let the script continue running and do not stop it unless it returns an error. To reduce operational impact, consider starting this command overnight.

9. Verify the generated PI Server metadata files.

   The script generates the four metadata files listed in Resulting metadata files below. Confirm that each file is available before staging it.

10. Review the generated metadata files and edit them if needed.

    Review and edit the files as needed, following the guidance in Resulting metadata files below before staging them.

Resulting metadata files

Running the PI Server metadata script produces the following CSV files. Place all four metadata files in the root-level CONFIG directory of the customer-owned Azure file share. PI Archive data files can be placed anywhere in the file share. CONNECT uses the metadata files to interpret the PI points and their values during import.

Before staging, review the metadata files yourself or share them with AVEVA for review.

• pointlist.csv: Lists the PI points to import. CONNECT creates a stream for each PI point in this file and imports that point's data from the PI Archive files. The number of rows in this file is the number of streams the import creates.

  If you do not want all PI points in this file created as streams in the same bulk import, manually edit the file to keep only the subset of PI points you want to import. Delete the rows for the PI points you want to exclude.

• piserver.csv: Verify that the file contains only one entry. If it contains multiple entries, keep the row that matches the source PI Server and delete the others.

• picollective.csv: If the source PI Server is not part of a PI Collective, this file is blank. If the source PI Server is part of a collective, verify that the file contains only one entry. If it contains multiple entries, keep the row that matches the source PI Server's collective and delete the others.

  Important: Make sure piserver.csv and picollective.csv are correct. These files must identify only the source PI Server and its collective so that the data from this bulk import is compatible with PI to CONNECT when you later transition to ongoing updates.

• digitalsetslist.csv

3. Stage files

Perform this task from a local staging folder on a server suitable for uploading the files to Azure. You can use the server hosting the archive files or another server with access to the archive files and generated CSV files. Organize the files in this staging folder before uploading them to the prepared Azure file share.

11. Prepare a local staging directory with the PI Archive data files for the selected historical range and the four generated metadata CSV files. Put the metadata files in a CONFIG subdirectory. PI Archive files can be placed elsewhere in the directory.

    Example local staging directory

    <LocalStagingDirectory>/

    |-CONFIG/ # The four metadata CSV files go here.

    | |-pointlist.csv

    | |-piserver.csv

    | |-picollective.csv

    | `-digitalsetslist.csv

    `-ArchiveFiles/ # PI Archive files can be stored here.

    |-<PIArchiveFile1>

    `-- <PIArchiveFile2>

12. Estimate the combined size of all files in the local staging directory and send the estimate to the Azure account holder. Wait for confirmation that the file share has enough capacity before uploading; the Azure account holder can adjust the capacity if needed.

13. Choose a staging method and upload the files in the local staging directory to the prepared Azure file share, preserving the directory structure shown above.

    Use the same method for both file sets. Choose the method that best suits the file volume, available network bandwidth, and required upload time.

    Warning: Regardless of the method you choose, staging PI Archive data can involve a large volume of data and take up to one week. Allow the upload to complete, and do not interrupt or modify it while it is in progress.

    • Azure CLI (Recommended): Upload files by using the Azure CLI. This is the recommended method for most use cases because it supports repeatable uploads and is better suited to large file sets. For command reference, see az storage file upload.

    • Data Box Gateway virtual machine: Use a virtual machine in your network with a configured transfer bridge. This method requires additional setup, but can provide faster transfers when available. For setup instructions, see Provision Azure Data Box Gateway in Hyper-V.

    • Physical Azure Data Box: Use a physical device to transfer the files to Azure. This method is suitable for very large transfers, but requires advance planning and shipping time. For instructions, see Order Azure Data Box and Copy data to Azure Data Box.

    For general guidance about transferring data to Azure Storage, see Choose an Azure solution for data transfer.

14. After both file sets are staged, continue with Generate Azure file share SAS URL for PI bulk import.

piconfig_pointlist.inp

Copy the following text into a file named piconfig_pointlist.inp in a folder on the server where the PI Archive files are located.

@mode list

@tabl piserver

@ostr name,serverid,collective

@select name=*

@output piserver.csv

@ends

@tabl picollective

@ostr name, collectiveid

@select name=*

@output picollective.csv

@ends

@tabl pids

@ostr set,setno,state

@ostr ...

@select set=*

@output digitalsetslist.csv

@ends

@tabl pipoint

@ostr pointid,pointtype,step,tag,future,digitalset,pointsource,sourcetag,engunits,descriptor

@select tag=*

@delimiter ;

@output pointlist.csv

@ends

@exit

Handoff to the CONNECT operator

After staging is complete, provide the following information to the CONNECT operator through the next workflow step:

• The Azure file share location and the name of the storage account and file share.

• Confirmation that the PI Archive data files for the selected historical range are staged in the Azure file share, and the location of those files.

• Confirmation that pointlist.csv, piserver.csv, picollective.csv, and digitalsetslist.csv are staged in the root-level CONFIG directory.

• Any relevant notes about the historical range, archive files included, file volume, or staging method.

The SAS URL is generated separately in Generate Azure file share SAS URL for PI bulk import and must be provided to the CONNECT operator before import creation.