Guidelines for programming bulk validation

This section is designed to outline potential solutions in the event of problems running the bulk validation script.

If you have attempted to run the script, but are not seeing a change in the values, two possible problems could be the cause: The data could be skewed or the kurtosis could be too low. This section outlines a few possible solutions to the problem.

Note: It is important to ensure that the routine used to generate a change applies to the particular dataset being used. To do this, you must first understand your data.

To construct bulk validation routines

1. To gain an understanding of your data, simply return true in the routine and all of the calculated limits appears in the results.

   From the list that appears, you will be able to compare and contrast the calculated limits of the objects. The objective is to determine a set of rules that will apply to an entire list of validation objects.

2. Another option that can be used to gain an understanding of the data for each object is to right-click on the desired object in the summary window, and select Set Validation Limits from the action menu. This will provide insight into the calculated limits associated with an individual object. After viewing the calculated limits for a number of the objects that you want to group together, in order to create bulk validation limits, you will be able to determine a set of rules that will apply to a list of objects.

   Items for consideration:

   • Is the data skewed?

     • If skewed data is acceptable, remove the skewed data check.

   • Does the data have a high or low kurtosis value?

     • If it is flat (ie., with a low kurtosis value with not many outliers), consider using a higher standard deviation. A higher standard deviation will broaden the scope of the data that the bulk validation routine will assess.

   • Do the checks apply to all columns?

     • If the checks do not apply to all of the columns, the columns that do require checks can be defined.

   • Consider comparing the new limits to the current limits:

     • If there are too many instances of valid data being tagged as invalid, check to ensure that the new high is greater than the old high, which will increase the validation limits to accommodate a broader spectrum of data.

     • If bad data is not getting flagged, check to ensure that the new high is less than the old high. This will decrease the validation limits such that the room for error decreases.

       Note: Selecting Output > Verbose will provide information about what data passed to the validation procedure and the suggested high/low validation limits. The example provided does not use the calculated new limits unless they fall within the minimum and maximum range of the data.

       For example, if the new high limit will be greater than the maximum value of the data, than nothing will be caught in the Reasonability Limit Validation check.

   • If the data used in the selection criteria is all valid, this is not likely to be good selection criteria.

Validation Check Example

The example below showcases how to stop the bulk validation from updating any limits on temperature.

Visual Basic

If columnName = "temperature" Then

Return False

End If

C#

if (columnName == "temperature")

return false;