Read data

The .NET and REST APIs provide programmatic access to read and write data. This section identifies and describes the APIs used to read Streams data. Results are influenced by Types, Stream views, Filter expressions, and Table format.

If you are working in a .NET environment, convenient Sequential Data Store (SDS) Client Libraries are available. The ISdsDataService interface, which is accessed using the SdsService.GetDataService() helper, defines the functions that are available.

Reading data from streams

While SDS is a robust data storage, it performs best if you follow certain guidelines:

• Maximum limit for events in read data calls

• Increase the Request-Timeout in the header

• Enable compression

• Use available read data APIs

Maximum limit for events in read data calls

Read data API is limited to retrieve less than 250,000 events per request. An error message is returned when the maximum limit is reached. This maximum limit applies to SdsStream API, SdsStream API, SdsStream API.

400 bad request error

{

"Error": "The request is not valid.",

"Reason": "Exceeded the maximum return count of 250000 events."

"Resolution": "Reduce query size and resubmit the request."

}

Increase the Request-Timeout in the header

Increase the Request-Timeout in the header to 5 minutes for large range calls that are requesting 250,000 events in a read call. The gateway will send 408 - Operation timed out error if the request needs more than 30 seconds.

The range of values that are held in memory can be large and be anywhere between 1GB and 2GB, so the system needs enough time to read and return the data.

If multiple calls return 408 - Operation timed out error even after increasing the timeout limit to 5 minutes, do one of the following:

• Reduce the range in the request calls of this type

• Retry with an exponential back-off policy

Compression

Include Accept-Encoding: gzip, deflate in the HTTP header. This enables compression. For more information, see Compression.

Use available read data APIs

Depending on the scenario, there are different read data APIs available. They return an overview of the values instead of reading all values at once. These APIs provide a good high-level view of the values without displaying them all at the same time:

• Read data API with filters

• Read data API

• Read data API

Single stream reads

The following methods for reading a single value are available:

• Read data API returns the first value in the stream.

• Read data API returns the last value in the stream.

• Read data API returns a value based on a starting index and search criteria.

In addition, the following methods support reading multiple values:

• Read data API retrieves a collection of stored values based on the request parameters.

• Read data API retrieves a collection of stored or calculated values based on the request parameters.

• Read data API retrieves a collection of evenly spaced summary intervals based on a count and specified start and end indexes.

• Read data API retrieves a collection of sampled data based on the request parameters.

Bulk reads

SDS supports reading from multiple streams in one request. The following method for reading data from multiple streams is available:

• Read data API retrieves a collection of events across multiple streams and joins the results based on the request parameters.

SdsBoundaryType

The SdsBoundaryType enum defines how data on the boundary of queries is handled—around the start index for range value queries and around the start and end index for window values. The following are valid values for SdsBoundaryType:

SdsSearchMode

The SdsSearchMode enum defines search behavior when seeking a stored event near a specified index. The following table shows valid values for SdsSearchMode.