Metadata pinning

Since each command in a pipeline operates independently, a mechanism is required to pass information from one command to subsequent commands. This mechanism is to pin metadata to a record when it is output from a command.

For example, when reading a file you want to pin the file name and row in the file where a record originated from.

Example:

<read template="geicl/InfoInterface/Data Sources/{{ $interface }}">

<file

folder="{{ $[input-path] }}"

recurse="true"

report-progress="true"

when-file-errors="Warn">

<excel

name="Capture_Excel"

header-row="{{ $[header-row].evaluate() }}"

first-data-row="{{ $[first-data-row].evaluate() }}"

include-sheet="{{ $[include-sheet].evaluate() }}"

where="{{ $[content-type] equals 'excel' or $[file.extension] in ['xls', 'xlsx'] }}"/>

<pin>

<metadata name="#file-name" value="{{ $[file.name] }}"/>

<metadata name="#file-index" value="{{ $index }}"/>

</pin>

</file>

</read>

Without this metadata pinning, there is no way for a subsequent command to know where this record came from. The following command shows this information is used to indicate a modification of data (to prevent any problems in AIM) and recording exactly where that information came from:

Example:

<log name="System_Policy">

<rule

name="ID Required"

level="Warning"

where="{{ $[this.id].Trim.isUnset }}"

message="Record {{ $[this.#file-index] }} of '{{ $[this.#file-name] }}' skipped as ID '{{ $[this.id] }}' is invalid"/>

<rule

name="No ID Surrounding Whitepace"

level="Warning"

where="{{ $[this.id].Length not equals $[this.id].Trim.Length }}"

message="Record {{ $[this.#file-index] }} of '{{ $[this.#file-name] }}' converted ID '{{ $[this.id] }}' to '{{ $[this.id].Trim }}'"/>

<rule

name="No {} in ID"

level="Warning"

where="{{ $[this.id] contains any ['{','}'] }}"

message="Record {{ $[this.#file-index] }} of '{{ $[this.#file-name] }}' converted ID '{{ $[this.id].Trim }}' to '{{ $[this.id].Trim.Replace('{','(').Replace('}',')') }}'" />

</log>