Paging runtime behavior

Based on the MaxRowsToReadPerCallOnDataCuration application settings, the curator service creates a number of JSON records within a single curated row. Therefore, a single curated row in the staging_curate_log table can have as many JSON rows as the number of rows configured in this application setting. Also, every single curated row for a type may not have the equal number of json rows, it may differ in size with the number of JSON rows in a staging_curate_log row.

For the data to be paged, the type must have the is_paging_required flag set to true. If no continuation token is supplied, then for the supplied filters, the first row from the database is returned using the supplied filters. If the supplied filter contains more than one row from the staging_curate_log table, then the service returns a continuation token to navigate to other pages one by one until all the rows from the staging_curate_log table are returned. Note that when navigating pages, every single page may not have equal number of JSON rows in length.

If no rows are found from the supplied filters, or there is only one row found from the supplied filters, or the service retrieved the last row from the staging_curate_log table, then no continuation token will be supplied.

The continuation token key contain three parts. The first part is the filter key, the second part contains the row number to be retrieved, and the third part contains the number of rows. The filter key contains TypeId + Action + StartTime + EndTime to be unique. This is the filter value supplied through the Web API request. The filter keys are encrypted in the continuation token and that continuation token must match the filter values supplied through the Web API request to navigate to all the pages returned from the database.

If the supplied continuation token is invalid or if the supplied continuation token contains a filter key that is not the same as the supplied filters for the Web API request (i.e. filter parameters are different), then the supplied continuation token is ignored. The data is retrieved from the database using the supplied filters, and returns the first row from the database. A new continuation token is returned if there is more than one row from the supplied filters.

Overall, the idea is not to return millions of JSON rows from the staging_curate_log table on a single Web API request. Rather, the idea is to return one curated row from the staging_curate_log table at a time, and then make subsequent calls to the database to return further rows from the staging_curate_log table using the continuation token.

For example:

An MESEquipmentUtilizationHistory Web API request is made with the following parameters:

• typeId = MESEquipmentUtilizationHistory&Action = create&startTime = 2022-01-01&endTime = 2022-12-31

with a continuation token of:

• TUVTSk9CSE9VUkxZUkVTUE9OU0V8Q1JFQVRFfDIwMjMtMDEtMDEgMDA6MDA6MDB8MjAyNS0wMS0wMSAwMDowMDowMCMzIzQ=

The curator extracts the supplied parameters as follows:

• typeId = MESEquipmentUtilizationHistory&Action = create&startTime = 2022-01-01&endTime = 2022-12-31

and extracts the continuation token as:

• typeId = MESEquipmentUtilizationHistory&Action = create&startTime = 2022-01-01&endTime=2022-12-31

The curator checks whether the supplied filters and the decrypted continuation token contain the same filter parameters. If they match, then the next row from the database is returned; otherwise, the first row from the database is returned.