About historical data processing
- Last UpdatedApr 09, 2025
- 3 minute read
Historical data processing (HDP) is the ability to process data after the time to which the data related. Each piece of data in a historically processed system carries with it the time to which the data applied. Whenever data must be processed in conjunction with other data, the time information can be used to verify that the values used were current for the time that is being processed. This in effect gives Server the ability to process late arriving data. Data can be 1 second, 10 seconds or 10 days old and still be processed correctly and accurately.
Background
The requirement for HDP is most evident in a large multi-site environment. Some integrated mining and minerals processing operations for example can be spread out over thousands of kilometers of sparsely populated desert country. In this case, it is desirable to collect data locally, then send the data to a central location for processing. This data can often be slow to arrive, perhaps being delayed for days if there are communication problems.
With HDP, the data for each site can be delivered independently, some late, others not, and AVEVA™ Production Management processes data for the time to which the data is related when the data arrives.
Benefits of historical data processing
Benefits of the HDP approach include:
-
More robust data collection
-
Lower total cost of ownership (TCO)
-
Lower system complexity
-
More accurate, repeatable and traceable information (time-stamps)
An historically processed system waits for the required data to be available for a given period, and then processes the data within that period as a batch. This results in completely predictable and repeatable processing and 100% determinism of the results.
A real-time system does the required processing of data before the next data change is seen. If the computer gets busy and cannot complete its processing before a new value is seen, then the processing uses an incorrect value.
An historically processed system deals with data according to the time stamp associated with the data. This means that it doesn’t matter when data is processed, or how long it takes to arrive. The resulting data is accurate and relevant for the time in question, no matter how busy the server gets.
Effect on Applications modules
Instead of processing for a given instant in time, application modules rely on the Dependency Manager to inform them that there is a new period to process. When notified of a period to process, the application then steps through the stream data for that period.
This processing happens at each level of the hierarchy beneath a reporting point, with processing effectively bubbling up from the expressions on the lowest level items until the point has performed its required function.
Consider the basic structure of a AVEVA™ Production Management reporting point:
This structure effectively becomes a dependency hierarchy with the expressions in the Condition and Field items at the lowest level of that hierarchy. Each level of the hierarchy has dependencies on the level under it, which ultimately makes sure that required data for the point exists before the point is permitted to process for a period.
The general sequence of processing for a reporting point is as follows:
-
The expressions on the lowest level items, fields and conditions, are evaluated when their dependencies have data available. These in turn generate a resulting stream of data.
-
If necessary, these lowest level items can have internal dependencies which are then executed, which generates an aggregated result stream for use at higher level. An example of this is the Condition item, whose state is determined by a combination of the results of the condition expression, and the enabled when expression
-
The Dependency Manager identifies that the next level of the hierarchy, Fields and Conditions folders, can process for the period in question. These in turn generate a resulting stream of data.
-
Finally the Dependency Manager identifies that lower level processing is complete, and the Reporting Point itself can perform its processing for the required period.
System architectures
The implementation of HDP within AVEVA™ Production Management opens up new architecture possibilities to enhance data integrity.
As well as the current possibility of sourcing data directly from real-time data sources, including real-time data from historians, AVEVA™ Production Management permits data to be collected remotely and transferred to a central processing point independently of a real-time connection.