Appendix B: CSV Reporting

The extracted data and the way they are transformed can be inspected at any point during the processing by creating a CSV report from the selected data. The selection of data to be included in CSV reporting is defined by the Transform and Loader configuration extensions. CSV files may include attributes or associations, which are selected in columns keyword in the configuration file. Each transformation configuration extension represents a step in the transformation process, so you should position this CSV reporting configuration at the appropriate transformation step to access the state of the data due to all previous transformation steps. A CSV report of the final transformed data that is converted to EIWM format can also be generated in the loader configuration.

Note: If there are more than one million rows to be written then a warning is logged and the CSV output is segmented into multiple files, with an underscore plus index number added to the extra file names to keep them unique, for example: <filename>_1.csv, <filename>_2.csv, and so on.

• Reporting during transformation configured in Transform configuration file

  This feature generates the CSV report during transformation and customizes the type of the information written to the CSV file. Generate multiple reports by using different suffixes and output paths for CSV files.

Example 1: CSV Reporting section with basic attributes, from Transform configuration file

<extension name="CSVReporting">

<csvReport

suffix="_AfterTransformation"

addHeaderRow="true"

columns="ClassID, ObjectID, ClassName, Name, #Association:Referenced"

/>

</extension>

• Example 2: CSV Reporting section with tracking attributes, from Transform configuration file:

  <extension name="CSVReporting">

  <csvReport

  suffix="_AfterTransformation"

  addHeaderRow="true"

  columns="#InternalId#, ClassName">

  <column value="#Tracking:

  DateTime

  ModuleName

  EventType

  ChangedMember

  PrevAttributeName

  AttributeName

  PrevAttributeValue

  AttributeValue

  PrevAssociationType

  AssociationType

  ChangedMember

  PrevTargetInternalId

  TargetInternalId

  #Filter: EventType: AttributeChanged"

  />

  </csvReport>

  </extension>

  Notes:

• The optional 'annotations' feature described in Requirements for the Thermo Library Generation Tool provides additional information which also can be reported. In the above example, after keyword #Tracking: there are listed tracking attributes that store data about creation, modification and deletion operations and can be inspected via CSV reporting as any other information.

• Multiple attributes and keywords are separated by commas, but multiple parameters for a # keyword are part of the keyword definition and so are separated by spaces or end of line characters, see Example 2 above.

• Limitation: Tracking attributes are not generated for expand and interpolate transformation used under ObjectID node.

• Reporting after transformation configured from GUI

  This CSV report is generated when CSV value is selected in Data Output Type combo list in Load>CSV panel. This report reflects the extracted data state after all transformations. Report file with .CSV extension is generated in the location set in Output Path field.

• GUI Settings in the Load > CSV panel:

  

Settings

The following list details about the CSV Report features for the Transform and the Load configurations:

• columns = "<keywords, names...>": (Mandatory) Sets number of columns, each separated by a comma.

  • #Attribute: <name> - (Optional) Sets the name of the attribute to select the attribute values that will be shown in the CSV column for all extracted data entities. Name of the attribute can also be written without #Attribute. #Attribute is added to maintain consistency with other keywords. Besides standard objects' attributes, you collect internal globally unique identifier (GUID) of the objects by writing as name:

  • #InternalID# and internal source ID reflecting ID of the object in source data format by writing as name: #SourceID#.

  • GUID and SourceID cannot be changed but may be helpful in tracking the history of the objects.

  • • #Attributes – (Optional) adds all attributes of the listed objects to the CSV report.

  • #ObjectType# - Specifies the internal type of the object, for example: EngineeringObject, GraphicalObject2D.

  • #SourceDataName# - Specifies the name of source data like input file.

  • These attributes cannot be changed but may be helpful in tracking the history of the objects or filtering the CSV report by Object Type.

  • #Association: <type> - Checks if specified association type is available in objects. CSV cells indicate AVAILABLE status if the corresponding object contains selected association type.

  • #Associations - Checks if associations are available in objects. All types of associations are present in the extracted data and each CSV column represents one type of association. CSV cells indicate AVAILABLE status if the corresponding object contains association type defined in the CSV column.

  • #Association:<type> and #TargetAttribute:<name> - Used together to first set the name of the association and then the name of the attribute whose value should be shown in the CSV column.

  • Note: An association may point to many referenced objects and in this case values of attributes of all these objects are added to the cell in the CSV column, each separated by a comma.

  • #Unit - Used only with one of the three keywords: #Attribute: <name>, #Attributes or #TargetAttribute:<name>. If attribute contains unit, it is added to the value after putting a single space.

  • #Tracking: <tracking attributes names> - This keyword can be used with the list of Tracking attributes' names separated by a blank character (for example, space, tabulation or a new line), where each blank character represents data in a specific column. All recorded changes (Tracking entries) of extracted data are listed in a chronological order in the CSV cell (see Example 2 of CSV Document).

  • #Tracking: <tracking attributes names> #Filter: <tracking attribute name> : <RegEx expression> - The #Filter: keyword includes only those changes in the CSV report where selected tracking attribute name matches the RegEx.

• column = "<keywords, names...>": Separates long and complex descriptions of columns and defines each description in an XML element (see Example 2). The separate "column" element adds more clarity when the column contains a #Filter with a regular expression.

  Note: Columns and column can be used interchangeably or both at the same time.

The following list details about the settings applicable for the CSV Reporting configuration:

• suffix = "<file name suffix>: (Optional) Defines a suffix to add to the CSV file name. It is used only in Transform configuration. It is ignored when outputFilePath is set.

• outputFolder = "<output folder path>": (Optional) Stores the CSV files in the location indicated by this setting. Otherwise, files are passed to the output folder set in Load settings. It is used only in Transform configuration. It is ignored when outputFilePath is set.

• addHeaderRow = "<true/false>"(in Generate Headers in CSV checkbox): (Optional) Sets to "True" to write the names of the attributes or associations to the first row of the CSV file. This setting is enabled by default.

• append = "<true/valse>": (Optional) Adds current CSV data to file set by ouputFilePath. When option addHeaderRow is enabled, then file to append must contain columns with the same names in the header row. If addHeaderRow is disabled, then number of the columns in the file must be the same as in the data to append.

  In append mode, user cannot provide generic types of columns like #Attributes or #Associations because they may deliver different list of attributes and associations between files. In append mode, list of columns for all processed files is constant.

• outputFilePath = "<output file path>": This attribute must be set when append mode is enabled. It can be provided absolute or path relative to Transform configuration location. When outputFilePath is set then attributes: outputFolder and suffix are ignored. This setting can be also used with disabled append mode.

Tracking attributes names which can be used with keyword #Tracking:

List of the EventType types which can appear in CSV in the EventType column:

• ObjectModified, ObjectAdded and ObjectRemoved

• AssociationAdded, AssociationChanged, AssociationReplaced, AssociationRemoved, AssociationMoved and AssociationsCleared

• AttributeAdded, AttributeChanged, AttributeReplaced, AttributeRemoved, AttributeMoved and AttributesCleared

Example 1 of CSV document viewed in Excel

Example 2 of CSV document with history of changes viewed in Excel

Note: If you want to open CSV reporting in a spreadsheet form, you must extend rows and columns of cells to see complete content.

Object Parameters Available for the CSV Report

The data available to select as columns in the CSV Report are as follows:

• ObjectID

• ClassID

• ClassName

• #Attributes (selects all attributes)

• #Associations (selects all associations)

• #InternalID#

• #SourceID#

• #ObjectType#

• any attribute and property name defined for the entities, defined in property sets.

When a relationship is used, the property names from each relationship get concatenated, for example, HasAssociations:Materials:Name.

Note: Objects containing an attribute "#EXCLUDE_FROM_CSV#" will not be included in the output .csv file. For more information, see Attribute Mapping.

Example 3: CSV Reporting the Base Mapping patterns used for defining ObjectID’s

<extension name="CSVReporting">

<csvReport suffix="_AfterTransformation" addHeaderRow="true" columns="ObjectId, Label">

<column value="#Tracking: AttributeName

BMExpression

BMChanges

#Filter: AttributeName: ObjectID " />

</csvReport>

</extension>

Note: "BMExpression" and "BMChanges" contain the patterns used in the conditions and definition of the ObjectID. These tracking attributes are available to the CSV report when the BaseMapping annotation level is set to "ObjectIdTracking", see Appendix C: Tracking Changes in Data.