Change broker

Change broker is an Enterprise-scale egress capability for CONNECT data services. Update streaming is crucial to enterprise-scale analytics, data science, visualization, and integration. Operations data is always changing. The change broker helps you navigate live data, delayed uploads, late manual measurements, recalculations, and even rare occurrences of incorrect readings that require change data.

Change data

This feature solves both new and changing data issues, and can provide near real-time data. More specifically, change broker monitors changes to the streams’ data values. The change data supported by the Signup includes the following capabilities: Update, Insert, Replace, Remove, and RemoveWindow. These are all write modes supported by Sequential Data Store (SDS). This capability serves both customers and partners for analytics, visualization/dashboarding, rules processing, evolving search indexes, anomaly detection, application providers, transaction processing, and digital service providers.

Change broker is designed to support the egress of millions of data events per second, retain the events for a period of time (1 hour), and serve that data to multiple, parallel consumers. A Signup is retained for 24 hours relative to the last query made. If a Signup is idle (without interaction) for 24 hours, it becomes Inactive from disuse and given an ExpiredDate. There is no known limitation on the size of data that can be held in a Signup during the one-hour expiration window.

REST API

A REST consumer creates a “Signup” for change data from a defined list of Stream IDs in a namespace (for example, data from Stream1 through Stream10). Signups have a lifecycle which includes states such as Active (ready for use) and Inactive (abandoned). Consumers poll an active Signup to pull the change data via GET Updates, and with each response, receive a Bookmark for the next request.

Reading is not destructive, as with a queue. Reading data from your Signup advances your bookmark, but does not immediately delete the data in the Signup. The data will expire based on the expiration assigned to it. For example, an hour expiration from the activation of a Signup or most recent query.

Bookmarks are unique to a Signup. You cannot use bookmarks for more than one Signup.

To receive change data from streams in different namespaces (for example, MyTenant/MyNamespace and CommunityFriend/TheirNamespace), a Signup is made for each namespace.

Applications for change broker

Some practical uses for change broker include:

• Data science

  • Improves accuracy of algorithms by providing recent data

  • Simplifies integration that sends recent data to an algorithm

  • Supports reference architectures

• Remote operations monitoring

  • Allows apps to retrieve near real-time data

  • Reduces components customers must manage

  • Standardizes how change data transfers to trusted business partners

• Maintenance and risk reduction

  • Near real-time data feeding alarm systems

Communities and change broker

While a community can contain multiple namespaces, change broker needs a Signup for each namespace in a community to get data updates for streams from more than one namespace.

When you create a Signup for a shared stream that is part of a community, you need to use the community Id along with the source tenant Id and namespace Id in your routes. For more information, see Manage change broker community data updates.

Disaster recovery

In the event of an CONNECT data services service failure, we will initiate a recovery process in accordance with the Recovery Point Objective (RPO) and Recovery Time Objective (RTO) as defined in the service description. This does not include change broker’s Signups and associated change data as part of the recovery effort.

The reasons for the exclusion of change broker from disaster recovery are as follows:

• Change data is only available within a Signup for one hour, which is shorter than CONNECT data services’s RPO. Therefore, even if a Signup were to be recovered, the change data would be expired and unavailable.

• During a service failure that also impacts data ingress and data storage, CONNECT data services will not capture change data.

• If data ingress was captured during change broker service failure, it will be available in Sequential Data Store.