Data reconciliation algorithm

AVEVA Production Accounting is the main engine that optimally reconciles conflicting flow measurements within a plant. The algorithm is executed by a component known as Data Reconciliation Solver (DR Solver).

What is data reconciliation

Measurements in a chemical process are subject to errors, both random and systematic, so that the law of conservation of mass is not obeyed. Similarly, if the process lends itself to a volume balance, the volume may appear not to balance.

In order to record the performance of the process, these measurements are adjusted in order that they conform to the conservation laws and any other constraints imposed upon them. This procedure or methodology is known as data reconciliation.

Basic problem of flow management

Since measurements of process variables, such as flow rate, inventory, concentration and temperatures, are subject not only to measurement error, but also to process variability, we cannot expect that any set measurements will obey the laws of conservation. Such errors would usually be expected to be random – that is to have an expected value of zero – but occasionally we find biased errors with expected values other than zero. Moreover, there are also many possible forms of errors such as missing flow, balance model problem and etc. due to missing input data, process leaks and incorrect balance model.

Such gross errors can be of the following three types:

• Balance model error due to balance points consisting of only input flows or output flows

• Missing flow error due to missing input data and process leaks

• Faulty meters due to bias or malfunction of instruments, inadequate calibration or lack thereof, or poor sampling

Data reconciliation is the procedure of optimally adjusting measured data so that the adjusted values obey the conservation laws and other constraints, while staying as close to the actual measured values as possible to retain a maximum degree of accuracy. Fortunately, there is much redundancy of measurement in a typical plant, so the adjustments which bring measured values into harmony with each other result in greater overall accuracy.

By leveraging redundancy, we increase accuracy only if we are able to identify and ignore those measurements which are in conflict with related measurements so that we may safely conclude that they are best ignored. While most measured values are within accepted tolerances and thus are valid candidates for use as inputs in the reconciliation algorithm, if you fail to identify and ignore gross errors, all of the reconciliation adjustments are greatly affected by those gross errors. In such a case, the reconciled values would not be reliable indicators of the state of the process. Thus gross errors must, if possible, be identified and either corrected or eliminated before data reconciliation. In other words, since meaningful data adjustments can be obtained if and only if there is no gross error in the data, gross errors must be detected and then eliminated or corrected before valid data reconciliation can be achieved.

Furthermore, identification of measurements that are outside of accepted tolerances is one of the primary useful results of data reconciliation.

Features of AVEVA Production Accounting

For this reason, AVEVA Production Accounting shows you the set of measurements that were found to be gross errors. The gross error detection algorithm is one of the strong points of AVEVA Production Accounting because:

• The algorithm has been rigorously tested by industry and found to be very reliable at finding all gross errors.

• AVEVA Production Accounting excludes gross errors from the reconciliation algorithm automatically, rather than forcing you to rerun reconciliation after telling you that there were errors.

AVEVA Production Accounting finds all possible types of gross errors such as balance model problems, missing flows and faulty measurements with proprietary and unique algorithms before data reconciliation.

The following figure shows the overall step in data reconciliation, including gross error detection in AVEVA Production Accounting.

Figure: Steps in Data Reconciliation

The other important thing in data reconciliation is mathematically reducing the complexity of the problem to be solved, because brute force methods often result in algorithms that take impractically long to complete. The solvability analysis through exclusion of gross errors reduces the size of the reconciliation problem, while extracting and retaining the redundant flows which affect the data reconciliation.

In the data reconciliation modeling of chemical processes, measuring instruments are placed in some but not all of the streams. Fortunately, redundancy and the fact that the inputs and outputs of a balance point must equal each other often permit the algorithm to infer the values of unmeasured streams. However, such inference must take place after the reconciliation of measured quantities has taken place, because it depends on comparisons of balanced results. After reconciling measured quantities, AVEVA Production Accounting fills in all unmeasured stream values that can be inferred, which means they are solvable despite being unmeasured. Any unmeasured streams whose values cannot be inferred are clearly identified by AVEVA Production Accounting as non-solvable.

Data reconciliation requires redundancy. Reconciliation can only be done if all of the flows are measured around some part of the process, such that all input flow quantities and output flow quantities are known. The sum of the inputs must equal the outputs. Theoretically, one can infer (at least one of) the output quantities if all the input quantities are known. In this case, while AVEVA Production Accounting will perform the inference, the data is not truly reconciled because there was not adequate redundancy.

AVEVA Production Accounting will infer where necessary and reconcile where it can, exactly as a human Production Accountant would do – only with greater speed, accuracy, and repeatability. In practice, the final set of reconciled and inferred measurements may be referred to as the “reconciled” results, because they are the best set of values that mathematics can produce.

Why does redundancy produce accuracy? Any single meter measurement is guaranteed to be at least slightly biased, because a perfect meter does not exist. By measuring the same flow with multiple meters, we use the law of averages to cancel out bias. Using the data from multiple meters also allows you to use all of the information present in your plant, rather than relying only on a select few. It is the same principle that caused seamen to take three chronometers to sea rather than one.

Data reconciliation is a constrained minimization problem of constrained least squares. When balancing either mass or volume alone, the constraints are linear. However, when balancing mass and volume simultaneously, the constraints are nonlinear. The objective function is usually a quadratic form in the adjustments to the measurements, which is a general weighted sum of squares.