Stream storage

Stream storage (formerly known as Sequential Data Store or SDS) is a service within CONNECT designed for storing, retrieving, and analyzing sequential data (usually time-series data). It provides efficient storage and retrieval of time-ordered data points from industrial sensors, IoT devices, and other data sources that generate continuous measurements. Stream storage maintains efficient query performance for both real-time monitoring and historical analysis.

The service organizes data using three core concepts that work together: service instances provide isolated workspaces for organizing resources, types define the structure and schema of your data, and streams store the actual data point sequences. When you create a stream, you associate it with a type that describes what properties each data point contains and which property serves as the index for ordering. This separation between schema definition and data storage promotes reusability, allowing multiple streams to share the same type definition while maintaining their own independent data sequences.

Service instances

A service instance is an isolated instance of Stream storage within a data store. Each service instance functions as a separate workspace where you create and manage streams and types. Service instances provide logical separation between different applications, departments, or environments, ensuring that resources in one instance remain independent from resources in another. Your CONNECT account can have multiple stream storage service instances, allowing you to organize resources across different projects or environments while maintaining complete isolation between them. Think of a service instance as a dedicated database or schema that contains all your sequential data and related configurations.

Streams

A stream is a sequential data container that stores sequences of data points ordered by an index. Each stream represents a single data source, such as a specific sensor or device. Every data point in the stream contains multiple property values recorded at a particular index point. For example, a wind turbine stream might store data points where each data point includes timestamp, power output, and operational status properties. Streams provide efficient storage and retrieval of sequential data while maintaining the chronological order of data points. Common use cases include storing industrial sensor data, tracking equipment performance metrics, recording system telemetry, maintaining audit logs, and capturing IoT device measurements.

The following example shows a stream storing power output data from a wind turbine. This stream is based on the type definition shown in the Types heading below. Each data point records the timestamp (which serves as the index), turbine power output in kilowatts, and operational status at a specific moment:

For more detailed information about streams and their full capabilities, see Streams.

Types

Types serve as immutable schema definitions that specify the structure of data stored in streams. A type defines the properties that each data point contains, the data type of each property, and which property serves as the index. Types ensure data consistency by enforcing a fixed structure for all data points written to streams that reference the type. Multiple streams can share the same type definition, promoting reusability and consistency across related data sources. Once created, a type cannot be modified, which ensures data integrity and prevents inconsistencies in stored data.

Note: Stream types are distinct from types in the Type Management service. Stream types work exclusively with streams for sequential data storage, while Type Management types work with Event Management and Entities. These are separate type systems that serve different purposes in CONNECT.

The following example shows the type definition for the wind turbine power output stream shown in the previous section. The type specifies three properties, with Timestamp serving as the index:

This type definition ensures that every data point written to any stream using this type must include all three properties with the correct data types, maintaining data consistency across multiple turbine streams.

For more detailed information about types and their full capabilities, see Types.

How streams and types work together

Streams and types have a fundamental relationship that shapes how data is organized and stored in streams. Before you can create a stream, you must first create a type that defines the structure of data points the stream will contain. When you create a stream, you specify an existing type that acts as a contract, ensuring that every data point written to the stream conforms to the same schema. After a stream is associated with a type, all data written to that stream must match the type's property definitions, including data types and required properties.

The separation between streams and types provides important benefits for data management. Multiple streams can reference the same type definition, allowing you to store similar data from different sources without duplicating schema definitions. For example, if you have temperature sensors across multiple facilities, you can create one TemperatureSensor type and then create individual streams for each physical sensor, all sharing the same type. This approach maintains consistency across related data sources while keeping each stream's data independent.

Because types are immutable, if you need to change the structure of your data, you must create a new type with the updated schema and then create new streams that reference this new type. You can migrate existing data from old streams to new streams as needed, but the original type definition remains unchanged. This design prevents scenarios where modifying a type could invalidate already-stored data points or create confusion about the structure of historical data.

Stream data points

Data points are the individual data points stored in streams. Each data point represents a measurement or observation at a specific point in time or sequence. When you write data to a stream, you send data points that conform to the stream's type definition. You can write individual data points for real-time scenarios or send batches of data points for efficient bulk ingestion. Reading data from streams allows you to retrieve specific data points by their index value, query ranges of data points between two index values, or filter data points based on property values. This flexibility supports both real-time monitoring applications that need the latest values and historical analysis applications that examine trends over time.

The following table shows the same wind turbine data from the Streams heading above, but the emphasis here is placed on the data points—the Power Output and Status values associated with a Timestamp:

Each row is a discrete data point containing values for all properties defined in the type. When you write data to a stream, you're creating new data points; when you read data from a stream, you're retrieving data points.

For more detailed information about stream data points and their full capabilities, see Stream data points.