Statistically removing outliers (SigmaLimit)

This analog filter removes outliers from a set of analog points based on the assumption that the distribution of point values in the set is a normal distribution.

The following illustration shows an example of outliers.

You can filter outliers by specifying a filter called ‘SigmaLimit()’. This filter has one parameter defined for specifying the value of n. This parameter is of type double. If the parameter is omitted, then a default parameter of 2.0 is used.

When this filter is specified in any retrieval mode, a time weighted mean, ì (mu), and time weighted standard deviation, ó (sigma), are found for each analog tag for the entire query range including phantom cycles if any, and points falling outside of the range [ì - nó, ì + nó] are removed from the point set before the points are processed further. In other words, the value will be filtered out if value > ì + nó or value < ì – nó.

Time weighted standard deviation is calculated as:

Math.Sqrt( (integralOfSquares - 2 * timeWeightedAverage * integral + totalTime * timeWeightedAverage * timeWeightedAverage)/totalTime);

This is the single pass equivalent to the formula:

Ranges where the value is NULL are excluded from these calculations.

A cyclic query example using a ‘SigmaLimit()’ filter without specifying the n value would look like this:

SELECT DateTime, Value, wwFilter

FROM History

WHERE TagName = ('TankLevel')

AND DateTime >= '2008-01-15 15:00:00'

AND DateTime <= '2008-01-15 17:00:00'

AND wwRetrievalMode = 'Cyclic'

AND wwFilter = 'SigmaLimit()'

Not specifying the n-value as done here is the same as specifying ‘SigmaLimit(2)’. The result set might look like this:

If the first value would be filtered out by the SigmaLimit filter, the value will be replaced with the time weighted mean.