Understand EDW Information Model

The Engineering Data Warehousing (EDW) manages information in the form of asset records, which are identified typically by business identifiers such as tag and document numbers.

The result is an object-centric model with one or more associated asset records, as illustrated in Fig. 4.1:

Fig. 4.1: Object-Centric Model

An object must have at least one identifier that uniquely identifies the object and is used to map asset records to it. Additional identifiers may be applied, reflecting the need to identify the same information across different information systems.

An object may be optionally classified, providing a hierarchical understanding of the object. Classification may be changed over time – for example, an object may be classified as a Pump and later classified as a Centrifugal pump. Within the EDW, classification, along with any Information Standard expectations and rules, may be changed over time, even changing Centrifugal Pump to a Pump or Pressure Safety Valve, if this sho uld be required. Classification is optional since any object without a class assigned is assigned a classification of unknown, indicating the EDW has observed an object but is unable to understand what it is due to insufficient information. This might reflect old information that should be removed from a source system or document, or a valid object for which the EDW has insufficient information.

Note: As the classification hierarchy is treated as an ontology, a specific class (one with no child classes) is permitted.

Information is mapped to a model described within the Information Standard, according to rules within that standard.

Attribute rules may then be used to define associations between information (for example, a parent tag or an associated document). This then forms the premise for connecting information together, which primarily sho uld be from managed information (specified by information owners and contributors), but in the worst case may be discovered in document content by AVEVA Gateways.

Key Processes

Information loaded into the EDW Model is managed by four key processes:

Extract is responsible for extracting information, which may either be a delta or full update as appropriate for the interface.

Capture manages capturing information into the EDW and tracking changes at interface level.

Interpret is responsible for interpreting (mapping) or re-interpreting

information through the Active Standard and managing this state at interface level.

Consolidation is responsible for consolidating (combining) records from all registers within a category (for example, Tags) to form a holistic view of an object and tracking changes to this state.

Additionally, the EDW supports processes which may be required:

Withdrawing allows an interface to be removed from one or more of capture, interpret and consolidation processes, either temporarily or permanently.

Patching information uses consolidation and interface priority to specify recurring patches to make to information, should this be necessary. These processes and associated concerns are described in more detail in the following sections.

Extract

The Extract process for an interface is responsible for pulling or pushing data from a source.

An Extract process may be deployed locally or remotely according to system and business requirement (a remote deployment might pass updates via a file system mechanism for example) and will be configured with any sensitive details such as passwords and database connections, which are not recommended to be managed within the Information Standard as a security best practice.

If no direct access route is available from which to extract data, extra must be implemented as a function in the source system and will deliver data via an agreed mechanism – either as a delta or in full as appropriate.

Information being extracted may be 1D, 2D, 3D, or some combination.

Capture

The Capture process manages capturing and tracking deltas of records and aggregates for each incoming source. New data for a load may be several files feeding into one interface. File processing failure (most commonly encountering invalid CSV format from a source system extract) may stop processing of that file at that point, dependent on severity, so any issued content up to that point is still captured; processing then moves to next file and warnings / errors are issued to help resolve data issues.

The EDW philosophy is to process and publish as much data as possible since this maximises usefulness, however, if a source system publishes information of very poor integrity this may not be possible.

Capture of information requires identification of an ID for each record – if an ID is not established the record will be omitted. Capture is greedy, recording as much information as is provided, and if a record with the same ID is encountered multiple times, the last encounter will 'win'. For Primary and Secondary Registers, this means overwriting the earlier record. Each record in an Attributes Register is an attribute value for an object, so the last attribute value encountered will be the result for that attribute. Since each record in an Associations Register is used to add or remove an association, add instructions will aggregate, but a delete instruction will cancel out an earlier add instruction if present.

Interpret

The Interpret process maps captured data according to the Active Standard. If the Active Standard is changed (for example, adding new rules that were not known previously, or changing class library mapping rules for interfaces), any captured information will be re-interpreted according to the new Active Standard when the EDW is next updated.

This facility importantly allows data to be visible and analysed rather than being discarded and potentially lost for not meeting a final set of rules.

Incoming data classifications and attributes are mapped according to rules in the Active Standard using class ID or a naming convention. If a class is not found, the information may still be interpreted using a default class for an interface, or as Unknown (the latter being of least preference since it represents the least understanding about an object).

As several primary interfaces have been configured and assigned different classifications, the subsequent Consolidation process will resolve this into a single result.

Information Interfaces specify an applicable corresponding identifiers. The EDW then uses corresponding identifier mappings for classes and attributes to map information into the Class Library defined within the Information Standard.

This mapping is designed to be managed within the Information Standard since managing changes within authoring systems will, in general, impact work processes and be required to be more complex adjustment than may be achieved within the Standard.

The Information Standard (and EDW in general) support concepts of class, sub-class and inheritance, allowing multiple levels of classification with rules to inherit details from more general classification to more detailed levels.

When classification structure and rules are altered in the Information Standard and issued as a new Active Standard to the EDW, the EDW will adjust to reflect the new standard upon next data update, which may result in a different class being assigned to an object and therefore a different set of expectations.

An update to the Information Standard will not require reloading of previously captured data since both captured and interpreted states are managed and maintained by the EDW.

Attributes

Each interface specifies several columns, or fields, which in a tabular data source are columns.

When data is captured, all available attributes are captured, however, when data is interpreted, only columns specified in the interface are processed. This allows data to be captured and interpreted, then reinterpreted with additional columns included without having to recapture existing data (unless the existing data does not include the required attributes).

Connecting Information

Attributes may be defined as Reference Attributes to indicate an expected or desired relationship. This is used to create associations and form a connected network of

information.

Reference attributes may be defined on an interface column or on an attribute as a validation rule. The latter case will publish an attribute value as both an object attribute and an association, whereas if an association type and class are specified on an interface column, this will publish only an association and not an explicit object attribute.

Consolidate

The Consolidation process combines all interpreted records with a matching ID in a single category (for example, Tags) into a consolidated record with a simple rule that a higher priority record (priority as defined by the information interface) will win over a lower priority, and a more recently updated record will win over an older record (when the priority is the same). Consolidation Process is illustrated in Fig. 4.2:

Fig. 4.2: Illustration of Consolidation

Consolidation enables information to be aggregated across multiple systems and from multiple organisations and be ‘flattened’ to form a simple consolidated view.

For example, engineering tag data will be managed in IEE, a commissioning system and an asset management system will also manage their appropriate concerns. The EDW then forms a view across the asset, understanding both the holistic (consolidated) state of an object as well as states in the different systems, without requiring impact on those systems.

Consolidation Rules

When a category is consolidated, additional attribute rules may be specified to set or update attribute values. These rules may access special metadata such as the resolved classification for an object, as well as any other information within the class library, allowing a great degree of flexibility. It is recommended that each such rule is accompanied by a Remarks property explaining the intent and any specific concerns.

Rules may be set on an attribute itself, in which case it will be applied to all objects of a class where that attribute is permissible, or on a class permissible attribute, in which case it will apply to all objects of that class or sub class. Attribute consolidation rules take effect after consolidation establishes classification and may therefore use that

information. Fig. 4.3 shows an illustration of a Consolidation Attribute rule.

Fig. 4.3: Consolidation Attribute Rule

A name rule (displayed object name) may also be specified at consolidation. Use this to specify a default name based on known tag data, as shown in Fig. 4.4.

Fig. 4.4: Metadata Rule at Consolidation

Withdrawing

In some circumstances, it may be required to pause or remove register information. An information interface may be updated to specify that any or all of capture, interpret and consolidation processes are not applicable to that interface.

Deactivating capture for an interface means that no new data will feed into the register, leaving it at the state of last capture.

Deactivating interpret for an interface will freeze the last interpreted state – if capture is still active then new data will continue to be captured but this will not flow into interpreted state.

Deactivating consolidation ignores the interface managed data when consolidating data.

If an interface is completely removed from the Information Standard, any existing captured and interpreted state will effectively be frozen, and consolidation will not include any data from this collected information.

Fig. 4.5 shows an illustration of withdrawing a Register.

Fig. 4.5: Illustration of Withdrawing a Register

An EDW administrator may archive or delete captured and interpreted states of information from the EDW if required.

Patching

Cases may arise where the IM team wants to override or supplement the incoming data and do not want to or are unable to effect changes in authoring systems.

When receiving data feeds from an external system that does not support strong integration of 1D, 2D, and 3D and instead requires information to be maintained and synchronised by manual processes across various tools and manual deliverables,

information may be delivered inconsistently over time with little ability to remediate the data without impacting schedule significantly. In these scenarios, it may be possible to establish and agree correct fragments of information and thus, override the inconsistent incoming data.

While you still have the option to receive such data into the EDW, and possibly raise an NCR, in some cases, this may hinder ability to report accurately on data, such as poor classification information (imagine a slightly different class code being provided for a functional object every month).

Patching is achieved using register priority by assigning a manual Excel interface, for example, with a higher priority than other interfaces. This allows effecting a change of consolidated information without changing the natural source.

As with other interfaces, a patch may be withdrawn at a later point in time (and reapplied, should that be necessary).

Information Interfaces

Information interfaces are defined within the Information Standard to register a data contract with a simple set of controls used to manage capture, interpret and

consolidation of information within the EDW. These interfaces are also referred to as Registers. Registers manage the state of a set of object records, including metadata and attribute values.

Primary and secondary registers capture and interpret attributes. A Primary register may also assign classification and human readable names. Attributes are only mapped if the source (captured) field is specified in the interface and a nomenclature mapping is specified on an attribute.

An Attributes register expects each record to specify an ID with a single attribute, and must specify how to identify attribute name, value and optionally units. Attributes may be excluded by an Exclude rule and attributes are mapped if the attribute name for a record matches an attribute ID or a nomenclature mapping.

An Associations register is used to add or remove associations within an interface.

Note: If an association is deleted from one interface but included in another, the consolidated information will still include the association. An Associations register allows an association to be registered in one update and removed in a later update.

Within ISM, the Information Interface to control these processes is shown in Fig. 4.6.

Fig. 4.6: Information Interface within ISM

An Information Interface describes Columns, which are a list of record fields of interest from an interface. These may be a calculated field similar to an Excel formula, specifying an expression to derive a result. Calculated fields provide a flexible means to establish desired results from one or more fields in a record and are evaluated within the Interpret process (and therefore will be re-evaluated should the calculation be redefined).

Register/Interface Settings

An interface may set the following processing parameters to control how data is captured and interpreted. Capture-specific rules only affect newly captured data, whereas interpret-specific rules are applied whenever data is re-interpreted (typically on every update) (listed in Table 4.1).

Table 4.1: Register / Interface Settings

Register Prioritisation

Register prioritisation works as follows:

Register#1 updates Object P101 on Weds with A=1, B=2, C=3.

Register#2 updates Object P101 on Thurs with C=4, D=5.

Register#1 updates Object P101 on Fri with A=2 (other attributes remain but unchanged).

Merged object P101 has A=2, B=2, C=3, D=5.

C = 3 because register#1 was last updated.

If Register#1 is assigned Priority 2 and Register#1 is assigned Priority 1, the same scenario would produce object P101 with A = 2, B = 2, C = 4, D = 5.

Fig. 4.7 shows that multiple registers set up with prioritisation and different corresponding identifiers mapping:

Fig. 4.7: Register Prioritisation

Reporting Databases

The reporting database is a fixed schema (may be added to over time) that does not vary with different information standards. Instead, several tables are updated with information from a published Information Standard from ISM.