Streams

A stream is a sequential data container that stores sequences of data points ordered by an index. Its the fundamental data container that holds sequential event data. Each stream represents a single data source, such as a specific sensor or device. Every data point in the stream contains multiple property values recorded at a particular index point. For example, a wind turbine stream might store data points where each data point includes timestamp, power output, and operational status properties.

Streams exist within a service instance and must reference a type that defines their structure. The type acts as a schema, ensuring that all data points written to the stream conform to the same property definitions and data types. This relationship between streams and types ensures data consistency while allowing you to reuse type definitions across multiple streams representing similar data sources.

Stream metadata and properties

Each stream contains metadata that describes its characteristics. This metadata includes:

• A unique ID for the stream within the service instance.

• A human-readable name and a detailed description of the streams purpose.

• A reference to the type definition that defines the properties permitted in the stream and describes the streams data structure.

• Tags and Metadata key-value pairs for categorization and search.

Stream indexes

The index is the stream property that determines the ordering of data points within a stream. Think of the index as the column that your stream is organized around—similar to how a spreadsheet might be sorted by a date column or an ID column. Every stream has an index property defined by its type, and this property serves as the key for organizing and retrieving data points. The index ensures that data points maintain their sequential order and enables efficient queries for specific time ranges or sequences. When data points are written to a stream, stream storage uses the index property to place each data point in the correct position, maintaining a sorted sequence that makes data retrieval fast and predictable.

The terms index and key are closely related in stream storage . When defining a type, you designate a property as the key in the type definition. Once that type is associated with a stream, that same property becomes the streams index. Both terms refer to the same property, but key is used when discussing the types schema definition, while index is used when discussing how data points are ordered and retrieved from the stream. Understanding this relationship helps clarify that the streams ordering mechanism is determined by the types key property.

Common data types used for indexes include DateTime for timestamp-based ordering (the most common pattern for time-series data), Int32 or Int64 for sequence-based ordering where data points are numbered sequentially, and TimeSpan for duration-based measurements. DateTime indexes are ideal for sensor data, telemetry, and any scenario where data points occur at specific points in time. Integer indexes work well for ordered sequences where the specific timestamp is less important than the data point order, such as processing steps, transaction sequences, or batch numbers in manufacturing processes. The index property comes from the type definition and cannot be changed after the stream is created. When you query data from a stream, you typically specify index values or ranges to retrieve the data points you need.

Tags and metadata

Effective stream organization helps you manage large numbers of streams and makes data discovery easier. Both tags and metadata help you categorize and describe streams, but they serve different purposes and work in different ways.

• Tags are simple string labels that you attach to streams for categorization. They work like hashtags, allowing you to group related streams together and filter streams quickly. Tags are ideal for broad categories like location (for example, building-A, floor-2), equipment type (temperature-sensor, pressure-gauge), or operational status (production, testing). The main advantage of tags is their simplicity—they enable fast searching and filtering across many streams without requiring specific key-value pairings.

• Metadata consists of key-value pairs that store additional properties and context about the stream. Unlike tags, metadata requires you to specify both a key (the property name) and a value (the property data). For example, you might use metadata to store manufacturer: 'Windtopia', modelNumber: 'WT-500', or calibrationDate: '2024-01-15'. Metadata is more structured than tags and better suited for storing detailed information that applications might query programmatically or that needs to be displayed in stream details.

Both tags and metadata support discovery and organization, but tags excel at quick categorization and filtering while metadata excels at storing structured, queryable information. You can use both together—for example, applying a temperature-sensor' tag for easy filtering while storing detailed specifications in metadata key-value pairs.

Best practices

Follow these practices to design and manage streams effectively:

• Type reusability: Create types that can be shared across multiple streams representing similar data sources. This promotes consistency and reduces schema duplication. For example, define a single temperature sensor type used by all temperature monitoring streams.

• Naming conventions: Establish clear naming patterns that include relevant identifying information. Use consistent separators and ordering of components in stream names to make browsing and searching more intuitive.

• Tagging strategies: Develop a consistent tagging taxonomy for your organization. Use tags to represent hierarchies, categories, and metadata that help users discover relevant streams. Apply tags consistently across related streams.

• Stream granularity: Create separate streams for distinct data sources rather than combining multiple sources into a single stream. This keeps data organized and makes it easier to manage permissions, retention policies, and queries. However, avoid creating unnecessary streams when data points could logically belong to the same source.