PI data replication

In a steady state, PI System Connector subscribes for changes on the source Data Archive, and collects snapshot and archived events. Archived events are collected for source PI points that have compression enabled/on as indicated by the PI point attribute compressing=1.

Snapshot events are collected for source PI points that have compression disabled/off as indicated by the PI point attribute compressing=0. Snapshot events are also collected for source PI points that match the global Send snapshot data for PI Points that match this query under the Global Settings of the data source configuration regardless of the PI point attribute compressing setting.

Note: PI System Connector does not replicate data collection for future PI points, nor data collection for Blob PI point types.

PI System Connector will replicate all data events from source to destination. In some situations, however, perfectly identical archives are not possible. For example, there is a possibility that snapshot values are returned and sent to the destination PI System when retrieving archive only data (such as a recorded values call). A call to retrieve recorded values (also known as compressed values) from the source PI System includes the snapshot value if the snapshot value timestamp falls within the range requested in the recorded values call. This is a standard PI System behavior and will occur regardless of the requested call used to retrieve recorded values.

During the start of data collection, the PI System Connector replicates PI point configuration from the source PI System to the destination PI System. By default, PI point compression is set to off, compressing=0, for all PI points on the destination PI System unless the PI point matches a defined global snapshot query for the source Data Archive. As a result, values transferred from the source PI System that have compressing=0 bypass the compression algorithm on the destination PI System and are archived.

History recovery

PI System Connector supports history recovery. The time of the initial backfill for any point is configurable through the data source property History Recovery Start Time in data source Global Settings configuration section.

If the PI System Connector Windows service is stopped and restarted, the connector will perform history recovery to obtain any of the events it missed during the downtime. In some cases, the history recovery will include/send the previous snapshot value the destination PI Server already received and archived, and this event may be resent by the real-time replication resulting in the same data event being stored twice. If this occurs, a [-109] Value at This Time Already Exists error could appear on the relay log.

Other data replication [-109] Value at This Time Already Exists errors may be generated as the result of changes to compression settings, to the gap in archive times indicated by archive markers, or to system shutdowns.

See also Show or hide archive gaps on our documentation portal.

Archived data collection

The PI System Connector supports the replication of archived data from the source PI System to the destination PI System for source PI points that have compression enabled/on as indicated by the PI point attribute compressing=1.

When the compressing attribute is set to compressing=1, archived data is transferred from the source PI System to the destination PI System based on the PI points associated with AF objects selected by the datasource configuration. By default, the connector sets the compressing attribute to compressing=0 on the destination Data Archive for PI points that have the compressing attribute set to compressing=1 on the source Data Archive. Archive data is transferred for all PI points associated with a Replicate Data Archive only data source configuration, or AF objects selected by a Replicate AF and Data Archive data source configuration. There is an exception regarding PI points that match the global Send snapshot data for PI Points that match this query under the Global Settings of the data source configuration. In that case, PI points that match the snapshot query replicate the PI Point attribute as it is defined on the source Data Archive through the compressing setting.

PI points associated with AF objects selected by the datasource configuration and, if globally configured, all the PI points associated with a Replicate Data Archive only data source configuration or AF objects selected by a Replicate AF and Data Archive data source configuration will have archive data transferred from the source PI System to the destination PI System with the compressing attribute set to compressing=0. The exception to this rule relates to PI points that match the global Send snapshot data for PI Points that match this query under the Global Settings of the data source configuration. In this case, PI points that match the snapshot query will replicate the PI Point attribute compressing setting as defined on the source Data Archive.

If desired, archive data replication for a PI point that currently sends snapshot data replication from the source PI System can be edited to send only archive data replication by setting the compressing attribute to compressing=1 on the source PI point to allow replication of archive only data from the source to destination PI System. In this scenario, the PI point must not match a defined global Send snapshot data for PI Points that match this query.

When a PI point is configured for archive only data replication, the destination PI System snapshot will contain the last archived value/timestamp contained in the source PI System. Therefore, the snapshot value/timestamp for a PI point on the destination PI System will be one archived value/timestamp behind the PI point snapshot value found on the source PI System. In other words, the source PI System will contain the latest snapshot value/timestamp of a PI point, whereas the destination will contain the last archived value/timestamp on the source PI System for the PI point.

Note: Slow changing archive only PI point data may result in the destination PI System not reflecting the desired value/timestamp for a PI point. This is a result of the destination PI point only containing the last archived value contained on the source PI System for the PI point.

Snapshot data collection

The PI System Connector supports the replication of snapshot data from the source PI System to the destination PI System under two scenarios.

Note: Setting the compressing attribute to compressing=0 will turn compression off for the PI point. This will result in increased archived events for the PI point and possibly increase required archive space. Please monitor archive disk usage when editing a significant amount of PI points to disable compression.

The first scenario relates to PI points that have compression disabled/off as indicated by the PI point attribute compressing=0. That is, PI points associated with a Replicate Data Archive only data source configuration or AF objects selected by a Replicate AF and Data Archive data source configuration will replicate snapshot data from the source PI System to the destination PI System. If needed, snapshot replication for a PI point that currently compresses data and only allows archive data replication from the source PI System can be edited to send snapshot data replication by setting the compressing attribute to compressing=0 on the source PI point to allow replication of snapshot data from the source to destination PI System.

The second scenario is associated with PI points that match the global Send snapshot data for PI Points that match this query under the Global Settings of the data source configuration. PI System Connector will replicate snapshot data from the source PI System to the destination PI System for source PI points returned by the query.

Removing points from data collection

The PI System Connector will continue data collection for PI Points associated with only Replicate Data Archive only data sources, until those PI Points are no longer associated with any Replicate Data Archive only data source. These PI Points are disassociated by removing the matching PI Point queries in Include PI Points that match this query fields or removing the matching Replicate Data Archive only data source. Once these PI Points are no longer associated with a Replicate Data Archive only data source, their data will no longer be collected.

PI Points associated with a Replicate AF and Data Archive data source will continue to have their data collected even after the replication option changes to Replicate AF only or the associated data source configuration is deleted. These PI Points will remain in the local point definition cache. To clear this point definition cache and the equivalent caching on the Relay, disconnect the connector and relay from one another while both are in a running state (collecting data). Clearing the Relay's cache always requires breaking the connection between the connector and relay while both are running (collecting data). This may also cause a resync of the source AF data sources, as well as trigger history recovery for all PI Points as the local point definition cache is rebuilt.

Note: These files should not be manually modified. If you attempt to manually modify them, it may result in severe data loss. If you require assistance with the cache files, please contact AVEVA Technical Support.

PI point selection using queries

To assist in selecting PI points for data replication, PI System Connector supports two options for defining PI point queries: Include PI Points that match this query associated with the Replicate Data Archive only data source replication option, and the Send snapshot data for PI Points that match this query option under the Global Settings for all data source configurations.

PI point selection queries can range from simple to complex. Therefore, it is recommended to use the PI System Explorer>Search>Tag Search tool to help develop the desired query or queries. Tag search syntax can be found using the PI System Explorer>Help>AF SDK Reference and searching for PIPoint Query Syntax Overview.

The following table lists the common PI point attributes and operator type used by the PI System Connector. Note: actual PI point attribute type may differ from how the connector interprets the operator type associated with the attribute. All attributes are case-insensitive.

The following table lists the supported query operations on a given attribute type.

Data archive PI point selection

PI System Connector supports the Replicate Data Archive only data source configuration option. This option requires the user to enter a query in the Include PI Points that match this query field to define the PI points desired for data collection for the Data Archive selected in the configuration. To reduce query complexity, multiple Replicate Data Archive only data source configurations can be defined for the same source Data Archive to achieve the desired PI points for data selection.

Note: PI point replication behavior for a data source instance with Replicate AF and Data Archive option does not change when a Data Archive referenced by AF is also specified in another data source instance with Replicate Data Archive only option.

PI point data references contained in the selected AF database associated with the Replicate AF and Data Archive option that match the selected source Data Archive selected in the Replicate Data Archive only option and match the Include PI Points that match this query shall have PI point data replicated to the destination Data Archive.

PI point data references contained in the selected AF database associated with the Replicate AF and Data Archive option that match the selected source Data Archive selected in the Replicate Data Archive only option and DO NOT match the Include PI Points that match this query shall also have PI point data replicated to the destination Data Archive.

Snapshot PI point selection

In addition to supporting snapshot replication for PI points that have compression disabled/off as indicated by the PI point attribute compressing=0, PI System Connector supports user defined queries to select PI points for snapshot replication. Query-defined snapshot replication is global relative to a selected Data Archive. PI points associated with any Replicate Data Archive only and Replicate AF and Data Archive configured data sources that match a defined snapshot query will have snapshot data replicated. Global snapshot data replication is defined using the Send snapshot data for PI Points that match this query option under the Global Settings section of the configuration page. To reduce query complexity, users can define multiple global Send snapshot data for PI Points that match this query configurations for the same source Data Archive to achieve the desired PI points for snapshot data replication.

PI point edits

PI System Connector supports PI point edits on the source PI System. As a result, an edited PI point that currently collects archive only data may have been modified to collect snapshot data or vice-versa. In this case, the PI System Connector will remove the PI point from its current data collection mode and place the PI point into the backfill mode. Once backfill is complete, the PI point will be placed into the appropriate data collection mode and resume data collection.

Note: It is not uncommon for PI users to make modifications to PI points, especially when determining appropriate compression settings. AVEVA recommends that data collection for the PI System Connector be stopped before making bulk edits to PI points, particularly when changing the compressing attribute. Once PI point edits are complete, restart data collection for the PI System Connector. Following this procedure will allow the PI System Connector to operate most efficiently during restart operations and PI point edit handling of replicating PI point configuration from source to destination PI System. Furthermore, the restart of data collection allows for more efficient configuration of the snapshot and archive data collection objects within the PI System Connector.

PI point value handling for source write options

To avoid data loss, before you make PointType edits in your source PI Server, stop data collection in the PI System Connector. After all PointType edits are made, it is then safe to restart the PI System Connector. This ensures I PointType edits are properly processed in the Connector and the source PI Server so that data is replicated as expected.

PI point value handling for source write options

The PI System Connector replicates PI point values from the source PI System to the destination PI System. PI point values can be written to the source PI System using the following write options: Insert, InsertNoCompression, Replace, NoReplace, ReplaceOnly, and Remove.

PI point value disposition of write options on the source and destination PI Systems

PI point replication handling for data source replication options

PI System Connector supports PI point replication using the Replicate Data Archive only and Replicate AF and Data Archive options. The following table describes the relationship between the configured data source replication options and the resulting archive and snapshot PI point data replication to the destination Data Archive.

1 PI point found in selected database configured in Replicate AF and Data Archive data source configuration.

2 PI point matches the Include PI Points that match this query defined in the Replicate Data Archive only data source configuration.

3 PI point matches the global Send snapshot data for PI Points that match this query under the Global Settings of the data source configuration.

4 No Replicate Data Archive only data source configured. References contained in Replicate AF and Data Archive data source configuration.

5 Change PI point compressing attribute to compressing=0.