Data validation methods

Automatic Data Validation is the process of checking all new data against a series of configured, date time effective validation rules.

Data undergoes a series of comparison tests against their discrete value, Historical value, and station average/total, with some exceptions. In most cases, the system runs all the validation tests on a device, even if a particular test fails. The outcome of these tests determines the data quality of each measurement reading. You are able to view the results of all the validation tests using the Problem Summary window or the various object summaries and sort the results based on the severity. Where any problematic or erroneous readings may exist, the tests will identify them. If the data fails the validation check, the application flags the data or marks it invalid.

Notes: When configuring your data validation checks, it is possible to identify the Severity of the issue as Invalid or Flagged. Data that is Flagged is still considered valid, and it will be used for calculations.

Data validation is one of many tasks that are handled by the Processing Subsystem.

The Measurement system allows each validation check to be manually enabled or disabled. The system automatically performs validation checks on all measurement data entering the system, but only for each check that is enabled.

Measurement data that failed the validation checks is identified as suspect. You can review all suspect data in your Areas of Responsibility (AORs). System generated estimates are also identified.

Data Validation Checks

These checks are performed against data measurements from metering and gas quality (GQ) sources. Validation checks in this section are not specific to a particular type of data; rather, they are generic checks designed to ensure the integrity of measurement data items.

Validation checks are executed in the following order of priority:

1. Field-equipment-based checks that must be performed prior to performing any calculations.

2. All remaining validation checks.

The Measurement system generates problematic context records whenever a validation check fails. The record includes the type of check that failed, and which fields that failed.

The system assigns a configured severity to each of the validation checks as either invalid or flagged. This severity is used to determine the data quality of the record if the check fails. Flagged data is considered valid by all back-end processing, except closing.

Data validation checks fall into distinct categories. This module arranges these checks into subcategories that are functionally related from the perspective of the data on which they operate. Validation checks that require knowledge and/or algorithms specific to meters or GQ points, or that are specifically designed to account for Historical data, are categorized in the following subsections:

Meter Data Validation Checks

These checks are performed specifically against measurements taken from meters.

Gas Quality Data Validation Checks

These checks are performed specifically against GQ measurements.

Historical Data Validation Checks

These checks are performed against data measurements from metering and GQ sources. Historical data validation checks compare the row being validated against other Historical rows.