Missing data check

The check defined as the Missing data check ensures that contiguous measurement data is retrieved from the external systems or devices. The Missing Data Check applies to metering and gas quality (GQ) telemetry.

The Missing Data Check determines if there are gaps in the time series of data entered into the system. When data arrives in the system, the poll time is recorded in Historical Timestamp Override window. The system will ensure that all of the data between the last Missing Data Check and the poll time is there. If gaps are found, they are estimated using the configured estimation technique. Then the last Missing Data Check is updated to the poll time. You can modify the Missing Data Check times if you wishes to skip some time periods or recheck time periods.

Normally, the data will not be re-estimated to account for missing data. However, there is one exception: the data re-estimates when the linear interpolation is enabled. In this case, the normal estimation methods are ignored and a linear interpolation is performed for all the missing rows using the data that is before and after the gap. If some of the missing data arrives, then the linear interpolation is re-calculated and the estimate is updated.

When linear interpolation is enabled, it is considered to be a better estimate than the other techniques used to compensate for missing data. So if the late data arrives with missing data in the time series, linear interpolation is done to update the late data estimate with linear interpolated data.

The following steps detect missing data:

• Detect any missing rows after the expected start time, but before the observed set.

• Detect any missing rows in the expected range [start, end]. This is an iterative process.

• Detect any missing rows before the expected end, but after the observed set. Any data found to be missing will be estimated.

  For further information, see Historical timestamp override.