Troubleshooting

AF Server RPC Metrics

Overall AF Server metrics.

If your AF Server is slow, this is a good place to start to get an idea which request is taking a long time. These metrics are for the whole server, not individual RPC(s).

To find AF Server RPC metrics, open PI System Explorer (PSE): File->Server Properties->RPC Metrics tab.

AFGetTrace

AFGetTrace Utility surfaces the time it takes a request to make a round trip. It provides an idea of how busy the AF Server is, how long it takes to process a specific request (ex: searching for Event Frames), etc.

Take a look at AFGetTrace Utility if you are experiencing slow performance reading or writing to your AF Server.

How do I use the AFGetTrace Utility?

Compare AF Database object counts

If you are missing items in your destination AF Server, take a look at AF Database object counts. These numbers will never be identical between the source and destination AF Servers, but it is a good indicator not all expected items were replicated if they vary widely. You would expect to see these counts similar to each other for replicated AF objects.

Look at every AF Database in PI System Explorer Library – the selected AF database in the tree has a page associated with it so you can see object counts (AF database properties and the Counts tab). Use this to compare between source and destination servers.

Slow replication

If your replication is running slowly, some things to check:

• Resource load on source and destination AF and SQL servers

  • For normal operations we recommend no more than 40% load overall CPU usage.

  • During peak processing (after restart or initial) we recommend it not exceed 60%.

  • Memory during normal operation should not exceed 25%.

  • During peak processing memory should not exceed 50%.

  • If you are experiencing loads greater than these recommendations, increase the available resources (CPU, Memory).

• Verify you are not seeing an error that SQL memory is not swapped out to disk often

  From Microsoft Documentation:

  A significant part of sql server process memory has been paged out. This may result in a performance degradation. Duration: #### seconds. Working set (KB): ####, committed (KB): ####, memory utilization: ##%.

  If you do see this error, your SQL Administrator needs to find out what else is running on that SQL Server causing the swap to disk, and mitigate that.

• AF replication can take time if there are a large number of changes to process.

  If you have a single parent Element with many children, any change on the parent element triggers the PI System Connector to resend the parent and all children. This is nearly the same amount of work as an initial replication and can take time. Even something as simple as editing a description or adding a child element will trigger this behavior.

If you see issues, turn on Information Logging in the Data Collection Manager on both your connector and relay. See Message logs for more information.