Change Broker introduction

Change Broker is a service for viewing change data from your account's streams within a client application. Instead of polling each resource directly and comparing records yourself, your client application creates a signup, subscribes to the resources you want to watch, and queries Change Broker for updates.

This service provides a reliable, repeatable process for tracking what changed and when, including both newly written values and changing values over time, with near real-time delivery. Practical uses include:

• Data science: Improves the accuracy of algorithms by providing recent data, simplifies integration that sends recent data to an algorithm, and supports reference architectures.

• Remote operations monitoring: Allows client applications to retrieve near real-time data, reduces components customers must manage, and standardizes how change data transfers to trusted business partners.

• Maintenance and risk reduction: Supports near real-time data feeding into alarm systems.

How Change Broker works

This diagram gives an end-to-end view of how Change Broker works. The service continuously delivers stream change events to your application, keeping it current with what has changed and how, without re-reading all history each time.

Note: The Change Broker configuration workflow and Change Broker update workflow guides cover the same information shown in this diagram, but broken down step-by-step for specific use cases and implementation scenarios.

The following list explains each numbered point in the diagram in more detail. Where a step references a core concept, it links to an in-depth description in the headings below.

1. Your application creates a signup, optionally setting an inactivity period.

   This signup gives Change Broker a clear starting point for who is reading updates and which resources to include.

2. Change Broker returns the signup ID with an initial state of Activating.

   This response lets your application track one signup while Change Broker finishes setup in the background.

3. Your client application requests the signup by ID while waiting for activation.

   This polling request gives your client application a simple readiness check loop while activation completes.

4. Change Broker returns the current signup state.

   This signup-state response helps your application decide whether to keep waiting, continue, or recover.

5. After activation, your application asks for updates using the bookmark from the Get signup by ID response.

   This bookmark-based request is how your application begins incremental reads from a known cursor, using the bookmark in the Get signup by ID response payload (bookmarkInfo.bookmark) instead of re-reading all history.

6. Change Broker reads stream change events from Stream Management, as described in Change events below.

   This retrieval step pulls together the latest changes for the streams in your signup in a consistent format.

7. Stream Management returns the matching change events to Change Broker.

   This separation of responsibilities keeps stream access and update delivery separated, so your application has a simpler integration point.

8. Change Broker returns change events and the next bookmark to your client application.

   This update response gives your application both the data it needs now and the cursor it needs for the next read.

9. Your client application persists the latest bookmark for the next request.

   This bookmark persistence protects your progress between reads so your application doesn't reread duplicate data and can continue processing new updates.

10. Your client application requests updates again on its normal polling cadence.

    This polling cadence is how the sync cycle repeats, keeping your application continuously up to date with the latest stream changes.

Core concepts

These concepts define how Change Broker delivers updates and how your client application should read them safely over time.

Signups

A signup represents your client application and groups the streams you want to monitor. Signups have a signup state that describes lifecycle status:

• Activating: The signup is being set up and is not yet ready for update queries.

• Active: The signup is ready for use, but update queries can proceed only after a bookmark is available in the Get signup by ID response payload (bookmarkInfo.bookmark).

• Expired: The signup has reached its inactivity limit and is no longer valid. By default, this inactivity period is 24 hours, and a new signup must be created to resume monitoring.

• Failed: The signup encountered an internal error during activation.

Client applications can set an explicit inactivityPeriod value in the create-signup request when they need a threshold different from the default (24 hours). Setting a lower value (for example, 1 hour) can be useful when signups are short-lived or not expected to be reused, because unused signups are cleaned up sooner. This helps manage capacity against the 10,000-signup limit per Change Broker instance when applications create many temporary signups. If your workload permits it, deleting unused signups explicitly is also recommended. If inactivityPeriod is omitted, the default value is applied.

Client applications should wait for the signup to become Active, then retrieve the bookmark from Get signup by ID response payload (bookmarkInfo.bookmark) before calling Get updates. Only the signup owner can query updates for that signup.

Stream monitoring

Change Broker supports monitoring changes to streams from a single instance of Stream Management declared in a signup request. It does not span across multiple service instances.

Note: To monitor additional Stream Management service instances, create separate Change Broker instances that each target one service instance. A single Change Broker instance cannot monitor streams across multiple Stream Management service instances.

Each signup contains a collection of stream IDs called a subscription, which is a list of the streams included in the signup that are monitored for changes. The signup identifies who is reading, while the subscription identifies which streams that signup monitors. Resource accessibility indicates whether each stream in that set is currently accessible to the signup owner, who is the identity that created the signup. Together, subscription and accessibility determine the practical update set your client application receives, so client applications should validate stream IDs consistently and handle temporary access changes without interrupting ingestion.

Change events

A change event is a single recorded update for a stream. It carries operation metadata and the corresponding event data, representing the atomic unit of change that Change Broker delivers to your application.

Change events represent a consistent change data format across all streams in a signup. Supported change modes align with write modes supported by Stream Management:

• Update: Merges new values into existing events at matching index positions.

• Insert: Adds new events at index positions that do not already exist.

• Replace: Overwrites existing events at matching index positions with new values.

• Remove: Deletes events at specified index positions.

• RemoveWindow: Deletes all events within a specified index range.

Bookmarks

A bookmark is an encoded cursor token that marks the current read position for a signup and lets your client application resume from where it left off. This supports reliable incremental processing without scanning from the beginning on every request. Each successful update response includes a renewed bookmark for the next request, and advancing with the latest bookmark preserves ordered progression through change data and helps avoid gaps in your read sequence.

You can expect to find a bookmark in two places:

• In the Get signup by ID response payload, under bookmarkInfo.bookmark, after the signup is active and bookmarkInfo.bookmarkAvailable is true.

• In each Get updates response, as the next bookmark token to use in the following updates request.

Treat Get updates as cursor-based paging. A single Get updates response is capped at 100,000 data points (maximum page size per request). When your responses remain at or near that cap, immediately issue another Get updates request with the returned bookmark until response size drops significantly below that level. Otherwise, ingest can outpace reads and your client application can remain behind.

Example bookmark payload (placeholder value for illustration only):

{

"bookmarkInfo": {

"bookmark": "<bookmark-hash>",

"bookmarkAvailable": true

}

}

Reading update data is non-destructive, unlike a queue. Advancing a bookmark records your position but does not immediately delete the underlying change data. Data expires based on the one-hour retention window, not on whether it has been read. Bookmarks are scoped to a single signup and cannot be used across signups.

Bookmarks expire after one hour, with a small server-side buffer used to accommodate clock skew. If a bookmark expires, update queries fail and your client application must obtain a renewed bookmark from Get signup by ID. Poll at a cadence that keeps pace with incoming change volume, and persist the latest bookmark after every successful read.