Change broker best practices

We recommend the following best practices when using change broker.

• Maximize the number of streams and data in a write operation:

  When you leverage Open Message Format (OMF) for data collection, you are in control of how much data is written to a stream and how many streams are included in a single write operation. For the PI to CONNECT Agent, this is already automatically optimized and you cannot adjust this.

  With minimal processing, change broker serves this write operation to the user or client querying a Signup. This means that if a stream has 10 timestamp-value pairs included in each write operation, then change broker will preserve this and serve the stream’s data packet (referred in the JSON format as an “event”) as a group of 10 timestamp-value pairs when a Signup is queried. Change broker is designed with minimal processing of the write operation to optimize for speed and throughput.

  For more information on streams, read Streams.

• Create, modify, and remove Signups iteratively if response times are too long:

  For most scenarios, creating, modifying, and removing Signups concurrently should complete in less than a second. Within a namespace, as the number of data streams in a Signup and the number of Signups grow very large, durations for these actions to complete could become long and not meet the needs of some use cases. In these scenarios, we recommend you perform a set of, or all these actions, iteratively to help achieve the desired performance in lieu of performing the actions simultaneously.

• Maximize the number of streams per Signup to optimize performance:

  Change broker was designed to handle a large number of streams in few Signups for each namespace. Tests showed that each namespace could support a few Signups of up to 100,000 streams per Signup. Although change broker is not yet explicitly designed to do so, many Signups (up to 100) with a few streams (up to 25) were also shown to be performant within a single namespace.

  With performance, there are many factors at play such as data frequency within streams, write operation design, and query frequency of the Signup. If you are approaching or near the aforementioned test scenarios, contact AVEVA customer support for further guidance and details.

• Query on a schedule that optimizes the number of events returned and retrieval time:

  Write operations for a stream are preserved by change broker. The grouping of data that is written to a stream in a single write operation is the same grouping when queried via a Signup. This is labeled in the Signup’s /Update response as an "event" for a stream.

  Change broker can return up to 100,000 events in a single query. To optimize performance, you should aim to have as many events as possible per query. You can increase your query cadence or adjust the Signup definition to include streams with higher data frequency so that your higher frequency queries have more events in each query.

• Ensure a Signup is queried within one hour of its previous query to ensure no change data is missed:

  A bookmark is only valid for an hour. A bookmark is either created when the Signup’s SignupState first becomes Active or when the Signup is queried via the /Update route. These two scenarios determine the start of the lifetime of a bookmark.

  Although the Signup is available beyond one hour of inactivity (no queries to a bookmark defines inactivity), the bookmark will expire and will not have any change data queued for querying. You can create a new bookmark to start queuing change data once again as long as the Signup has not expired.

  Your Signup is available for up to 24 hours of inactivity. After the 24-hour period, the Signup will not be retrievable and you will no longer be able to query for updates or modifying the Signup. The Signup’s definition will remain viewable in the portal for several days once expired, unless deleted.