Key processes

Information loaded into the EDW Model is managed by four key processes:

• Extract is responsible for extracting information, which may either be a delta or full update as appropriate for the interface.

• Capture manages capturing information into the EDW and tracking changes at interface level.

• Interpret is responsible for interpreting (mapping) or re-interpreting information through the Active Standard and managing this state at interface level.

• Consolidation is responsible for consolidating (combining) records from all registers within a category (for example, Tags) to form a holistic view of an object and tracking changes to this state.

Additionally, the EDW supports processes which may be required:

• Withdrawing allows an interface to be removed from one or more of capture, interpret and consolidation processes, either temporarily or permanently.

• Patching information uses consolidation and interface priority to specify recurring patches to make to information, should this be necessary. These processes and associated concerns are described in more detail in the following sections.

Extract

The Extract process for an interface is responsible for pulling or pushing data from a source.

An Extract process may be deployed locally or remotely according to system and business requirement (a remote deployment might pass updates via a file system mechanism for example) and will be configured with any sensitive details such as passwords and database connections, which are not recommended to be managed within the Information Standard as a security best practice.

If no direct access route is available from which to extract data, extra must be implemented as a function in the source system and will deliver data via an agreed mechanism – either as a delta or in full as appropriate.

Information being extracted may be 1D, 2D, 3D, or some combination.

Capture

The Capture process manages capturing and tracking deltas of records and aggregates for each incoming source. New data for a load may be several files feeding into one interface. File processing failure (most commonly encountering invalid CSV format from a source system extract) may stop processing of that file at that point, dependent on severity, so any issued content up to that point is still captured; processing then moves to next file and warnings / errors are issued to help resolve data issues.

The EDW philosophy is to process and publish as much data as possible since this maximises usefulness, however, if a source system publishes information of very poor integrity this may not be possible.

Capture of information requires identification of an ID for each record – if an ID is not established the record will be omitted. Capture is greedy, recording as much information as is provided, and if a record with the same ID is encountered multiple times, the last encounter will 'win'. For Primary and Secondary Registers, this means overwriting the earlier record. Each record in an Attributes Register is an attribute value for an object, so the last attribute value encountered will be the result for that attribute. Since each record in an Associations Register is used to add or remove an association, add instructions will aggregate, but a delete instruction will cancel out an earlier add instruction if present.

Interpret

The Interpret process maps captured data according to the Active Standard. If the Active Standard is changed (for example, adding new rules that were not known previously, or changing class library mapping rules for interfaces), any captured information will be re-interpreted according to the new Active Standard when the EDW is next updated.

This facility importantly allows data to be visible and analysed rather than being discarded and potentially lost for not meeting a final set of rules.

Incoming data classifications and attributes are mapped according to rules in the Active Standard using class ID or a naming convention. If a class is not found, the information may still be interpreted using a default class for an interface, or as Unknown (the latter being of least preference since it represents the least understanding about an object).

As several primary interfaces have been configured and assigned different classifications, the subsequent Consolidation process will resolve this into a single result.

Information Interfaces specify an applicable corresponding identifiers. The EDW then uses corresponding identifier mappings for classes and attributes to map information into the Class Library defined within the Information Standard.

This mapping is designed to be managed within the Information Standard since managing changes within authoring systems will, in general, impact work processes and be required to be more complex adjustment than may be achieved within the Standard.

The Information Standard (and EDW in general) support concepts of class, sub-class and inheritance, allowing multiple levels of classification with rules to inherit details from more general classification to more detailed levels.

When classification structure and rules are altered in the Information Standard and issued as a new Active Standard to the EDW, the EDW will adjust to reflect the new standard upon next data update, which may result in a different class being assigned to an object and therefore a different set of expectations.

An update to the Information Standard will not require reloading of previously captured data since both captured and interpreted states are managed and maintained by the EDW.

Attributes

Each interface specifies several columns, or fields, which in a tabular data source are columns.

When data is captured, all available attributes are captured, however, when data is interpreted, only columns specified in the interface are processed. This allows data to be captured and interpreted, then reinterpreted with additional columns included without having to recapture existing data (unless the existing data does not include the required attributes).

Connecting information

Attributes may be defined as Reference Attributes to indicate an expected or desired relationship. This is used to create associations and form a connected network of information.

Reference attributes may be defined on an interface column or on an attribute as a validation rule. The latter case will publish an attribute value as both an object attribute and an association, whereas if an association type and class are specified on an interface column, this will publish only an association and not an explicit object attribute.

Consolidate

The Consolidation process combines all interpreted records with a matching ID in a single category (for example, Tags) into a consolidated record with a simple rule that a higher priority record (priority as defined by the information interface) will win over a lower priority, and a more recently updated record will win over an older record (when the priority is the same). Consolidation Process is illustrated in Fig. 4.2:

Consolidation enables information to be aggregated across multiple systems and from multiple organisations and be ‘flattened’ to form a simple consolidated view.

For example, engineering tag data will be managed in IEE, a commissioning system and an asset management system will also manage their appropriate concerns. The EDW then forms a view across the asset, understanding both the holistic (consolidated) state of an object as well as states in the different systems, without requiring impact on those systems.

Consolidation rules

When a category is consolidated, additional attribute rules may be specified to set or update attribute values. These rules may access special metadata such as the resolved classification for an object, as well as any other information within the class library, allowing a great degree of flexibility. It is recommended that each such rule is accompanied by a Remarks property explaining the intent and any specific concerns.

Rules may be set on an attribute itself, in which case it will be applied to all objects of a class where that attribute is permissible, or on a class permissible attribute, in which case it will apply to all objects of that class or sub class. Attribute consolidation rules take effect after consolidation establishes classification and may therefore use that information.

The following image shows an illustration of a Consolidation Attribute rule.

A name rule (displayed object name) may also be specified at consolidation. Use this to specify a default name based on known tag data, as shown in Fig. 4.4.

Withdrawing

In some circumstances, it may be required to pause or remove register information. An information interface may be updated to specify that any or all of capture, interpret and consolidation processes are not applicable to that interface.

Deactivating capture for an interface means that no new data will feed into the register, leaving it at the state of last capture.

Deactivating interpret for an interface will freeze the last interpreted state – if capture is still active then new data will continue to be captured but this will not flow into interpreted state.

Deactivating consolidation ignores the interface managed data when consolidating data.

If an interface is completely removed from the Information Standard, any existing captured and interpreted state will effectively be frozen, and consolidation will not include any data from this collected information.

An EDW administrator may archive or delete captured and interpreted states of information from the EDW if required.

Patching

Cases may arise where the IM team wants to override or supplement the incoming data and do not want to or are unable to effect changes in authoring systems.

When receiving data feeds from an external system that does not support strong integration of 1D, 2D, and 3D and instead requires information to be maintained and synchronised by manual processes across various tools and manual deliverables,

information may be delivered inconsistently over time with little ability to remediate the data without impacting schedule significantly. In these scenarios, it may be possible to establish and agree correct fragments of information and thus, override the inconsistent incoming data.

While you still have the option to receive such data into the EDW, and possibly raise an NCR, in some cases, this may hinder ability to report accurately on data, such as poor classification information (imagine a slightly different class code being provided for a functional object every month).

Patching is achieved using register priority by assigning a manual Excel interface, for example, with a higher priority than other interfaces. This allows effecting a change of consolidated information without changing the natural source.

As with other interfaces, a patch may be withdrawn at a later point in time (and reapplied, should that be necessary).