Read data

.NET and the Sequential Data Store (SDS) REST APIs provide programmatic access to read and write data. This topic identifies and describes the APIs used to read SdsStreams data. Results are affected by SdsTypes, SdsStreamViews, filter expressions, and table format.

If you are working in a .NET environment, convenient SDS Client Libraries are available. The ISdsDataService interface, which is accessed using the SdsService.GetDataService() helper, defines the functions that are available.

Reading data from streams

While SDS provides robust data storage, it performs best if you follow certain guidelines listed in the following subheadings.

Maximum limit for events in read data calls

The Read data API is limited to access less than 2,500,000 events per request. This limit includes events that are accessed but not returned, such as events that are filtered out of the response. An error message is returned when the maximum limit is reached. This maximum limit applies to List Values, List Summaries, List Sampled Values.

400 bad request error
{
    "Error": "The request is not valid.",
    "Reason": "Exceeded the maximum return count of 2500000 events.",
    "Resolution": "Reduce query size and resubmit the request."
}

Increase the Request-Timeout in the header

Read calls with a large amount of data may require more time than the default 30 second timeout. If the gateway responds with 408 - Operation timed out error, increase the Request-Timeout in the header to five minutes.

If multiple calls return 408 - Operation timed out error, even after increasing the timeout limit to five minutes, do one of the following:

• Reduce the range in the request calls of this type
• Retry with an exponential back-off policy

Maximum data read response

Responses are limited to 2³¹ bytes. The number of bytes written during response serialization cannot exceed that value. An error message suggesting a smaller read request is returned when the response limit is reached.

400 bad request error

{
    "Error": "Failed to serialize response.",
    "Reason": "The response size might be too large.",
    "Resolution": "Try again with a smaller read request."
}

Compression

Include Accept-Encoding: gzip, deflate in the HTTP header. This header enables compression. For more information, see Compression.

Use available read data APIs

Depending on the scenario, there are different read data APIs available. They return an overview of the values instead of reading all values at once. These APIs provide a good high-level view of the values without displaying them all at the same time:

• List Values with filters
• List Summaries
• List Sampled Values

Single stream reads

The following methods for reading a single value are available:

• Get First Value returns the first value in the stream.
• Get Last Value returns the last value in the stream.

In addition, the following methods support reading multiple values:

• List Values retrieves a collection of stored values based on the request parameters.
• List Interpolated Values retrieves a collection of stored or calculated values based on the request parameters.
• List Summaries retrieves a collection of evenly spaced summary intervals based on a count and specified start and end indexes.
• List Sampled Values retrieves a collection of sampled data based on the request parameters.

Bulk reads

SDS supports reading from multiple streams in one request. The following method for reading data from multiple streams is available:

• Create Bulk Access Job retrieves a collection of events across multiple streams and joins the results based on the request parameters.
• Join Values retrieves a collection of events across multiple streams and joins the results based on the request parameters.

Response format

Supported response formats include JSON, verbose JSON, and SDS. For more information on response formats, see Response format.

Indexes and reading data

Most read operations take at least one index as a parameter. Indexes may be specified as strings, or using the SDS Client libraries, the index may be passed as-is to read methods that take the index type as a generic argument. For more information, see Indexes. For information on compound indexes, see Compound indexes.

Read characteristics

When data is requested at an index for which no stored event exists, the read characteristics determine whether the result is an error, no event, interpolated event, or extrapolated event. The combination of the type of the index and the interpolation and extrapolation modes of the SdsType and the SdsStream determine the read characteristics.

Interpolation

Interpolation determines how a stream behaves when queried to return an event at an index between two existing events. InterpolationMode determines how the returned event is constructed. SDS provides multiple ways to set the interpolation mode to get the desired behavior. The table below lists InterpolationModes:

¹Continuous cannot return values for type properties that cannot be interpolated, such as when the type property is not numeric.

The table below describes how the Continuous InterpolationMode affects properties that occur between data in a stream: InterpolationMode = Continuous or Default.

¹When extreme values are involved in an interpolation (for example Decimal.MaxValue), the call may result in a BadRequest exception.

²For the Continuous interpolation mode, Nullable types are interpolated in the same manner as their non-nullable equivalents as long as the values surrounding the desired interpolation index are non-null. If either of the values are null, the interpolated value is null.

If the InterpolationMode is not assigned, the events are interpolated in the default manner, unless the interpolation mode is overridden in the SdsTypeProperty or the SdsStream. For more information on overriding the interpolation mode on a specific type property, see SdsTypeProperty. For more information on overriding the interpolation mode for a specific stream, see Sds Streams.

Extrapolation

Extrapolation defines how a stream responds to requests with indexes that precede or follow all data in the stream. ExtrapolationMode determines whether extrapolation occurs and at which end of the data.

ExtrapolationMode works with the InterpolationMode to determine how a stream responds. The following tables show how ExtrapolationMode affects returned values for each InterpolationMode value:

ExtrapolationMode with InterpolationMode = Default (or Continuous), StepwiseContinuousLeading, and StepwiseContinuousTrailing

ExtrapolationMode with InterpolationMode = Discrete

ExtrapolationMode with InterpolationMode = ContinuousNullableLeading and ContinuousNullableTrailing

For additional information about the effect of read characteristics, see the documentation on the read method you are using in Stream Data.

Filter expressions

You can apply filter expressions to any read operation that returns multiple values, including Get Values, Get Range Values, Get Window Values, and Get Intervals. The filter expression is applied to the collection events conditionally filtering events that do not meet the filter conditions.

For more information, see Filter expressions for SDS values.

Table format

Results of a query can be organized into tables by directing the form parameter to return a table. Two forms of table are available: table and header table.

When the form parameter is specified as table, ?form=table, events are returned in row column form. Results include a collection named Columns that lists column name and type and a collection named Rows containing a collection of rows matching the order of the columns.

Specifying a form of type table-headers, ?form=tableh, results in a collection where the rows collection contains a column header list.

For more information, see Table format.

SdsBoundaryType

The SdsBoundaryType enum defines how data on the boundary of queries is handled: around the start index for range value queries, and around the start and end index for window values. The following values are valid for SdsBoundaryType:

SdsSearchMode

The SdsSearchMode enum defines search behavior when seeking a stored event near a specified index. The following are valid values for SdsSearchMode:

Transforming data

SDS provides the ability to transform data upon reads. The supported data transformations are:

• Reading with SdsStreamViews: Changing the shape of the returned data.
• Unit of Measure Conversions: Converting the unit of measure of the data.

Data transformations are supported for all single stream reads, but transformations have specific endpoints. The following are the base URIs for the transformation endpoints:

api/v1/Tenants/{tenantId}/Namespaces/{namespaceId}/Streams/{streamId}/Data/Transform/First

api/v1/Tenants/{tenantId}/Namespaces/{namespaceId}/Streams/{streamId}/Data/Transform/Last

api/v1/Tenants/{tenantId}/Namespaces/{namespaceId}/Streams/{streamId}/Data/Transform

api/v1/Tenants/{tenantId}/Namespaces/{namespaceId}/Streams/{streamId}/Data/Transform/Interpolated

api/v1/Tenants/{tenantId}/Namespaces/{namespaceId}/Streams/{streamId}/Data/Transform/Summaries

api/v1/Tenants/{tenantId}/Namespaces/{namespaceId}/Streams/{streamId}/Data/Transform/Sampled

Reading with SdsStreamViews

When transforming data with an SdsStreamView, the data read is converted to the target type specified in the SdsStreamView. For more information on working with stream views, see Stream Views.

All stream view transformations are HTTP GET requests. Specify the stream view ID (streamViewId={streamViewId}) at the end of the transformation endpoint in the request as shown below. For example, the following request returns the first event of the stream transformed to the target type (per stream view definition specified by streamViewId):

GET api/v1/Tenants/{tenantId}/Namespaces/{namespaceId}/Streams/{streamId}/Data/Transform/First?streamViewId={streamViewId}

All single-stream data reads support stream view transformations.

# Get 100 events from start index
GET api/v1/Tenants/{tenantId}/Namespaces/{namespaceId}/Streams/{streamId}/Data/Transform?startIndex={startIndex}&count=100&streamViewId={streamViewId}

# Get last value
GET api/v1/Tenants/{tenantId}/Namespaces/{namespaceId}/Streams/{streamId}/Data/Transform/Last?streamViewId={streamViewId}

When data is requested with an SdsStreamView, the read characteristics defined by the target type of the SdsStreamView determine what is returned. The read characteristics are discussed in the code samples.

Unit conversion of data

SDS supports assigning Units of Measure (UOM) to stream data. If stream data has UOM information associated, SDS supports reading data with unit conversions applied. On each read data request, unit conversions are specified by a user-defined collection of SdsStreamPropertyOverride objects in read requests. The SdsStreamPropertyOverride object has the following structure:

This assignment is supported in the .NET client libraries methods using overloads that accept a collection of SdsStreamPropertyOverride objects, and in the REST API using HTTP POST calls with a request body containing a collection of SdsStreamPropertyOverride objects.

All unit conversions are POST HTTP requests. The unit conversion transformation URI is as follows:

POST api/v1/Tenants/{tenantId}/Namespaces/{namespaceId}/Streams/{streamId}/Data/Transform

Request body

The Request Body contains a collection of SdsStreamPropertyOverride objects.

The following code defines a Simple Type with one index, Time, and one additional property, Measurement. Measurement has an assigned unit of measure, meter.

public class SimpleType
{
   [SdsMember(IsKey = true, Order = 0) ]
   public DateTime Time { get; set; }
   [SdsMember(Uom = "meter")]
   public Double Measurement { get; set; }
}

This type is assigned to a stream, and the example request body below requests SDS to convert the Measurement property of the returned data from meter to centimeter.

[
  {
    "SdsTypePropertyId": "Measurement",
    "Uom": "centimeter"
  }
]

All single stream data reads with streams that have specified UOMs support UOM conversions.