Metadata Pinning

Since each command in a pipeline operates independently, we need a mechanism to pass information from one command to subsequent commands. This mechanism is to pin metadata to a record when it is output from a command.

For example, when reading a file you want to pin the file name and row in the file where a record originated from.

Example:

<read template="geicl/InfoInterface/Data Sources/{{ $interface }}">

<file folder="{{ $[input-path] }}" recurse="true" report-progress="true" when-file-errors="Warn">

<excel name="Capture_Excel"

header-row="{{ $[header-row].evaluate() }}"

first-data-row="{{ $[first-data-row].evaluate() }}" include-sheet="{{ $[include-sheet].evaluate() }}"

where="{{ $[content-type] equals 'excel' or $[file.extension] in ['xls', 'xlsx'] }}" />

<pin>

<metadata name="#file-name" value="{{ $[file.name] }}" />

<metadata name="#file-index" value="{{ $index }}" />

</pin>

</file>

</read>

Without this metadata pinning, there will be no way for a subsequent command to know where this record came from. The following command shows this information is used to indicate a modification of data (to prevent any problems in AIM) and recording exactly where that information came from:

Example:

<log name="System_Policy">

<rule

name="ID Required" level="Warning"

where="{{ $[this.id].Trim.isUnset }}"

message="Record {{ $[this.#file-index] }} of '{{ $[this.#file-name] }}' skipped as ID '{{ $[this.id] }}' is invalid" />

<rule

name="No ID Surrounding Whitepace" level="Warning"

where="{{ $[this.id].Length not equals $[this.id].Trim.Length }}" message="Record {{ $[this.#file-index] }} of '{{ $[this.#file-name] }}'

converted ID '{{ $[this.id] }}' to '{{ $[this.id].Trim }}'" />

<rule

name="No {} in ID" level="Warning"

where="{{ $[this.id] contains any ['{','}'] }}"

message="Record {{ $[this.#file-index] }} of '{{ $[this.#file-name] }}'

converted ID '{{ $[this.id].Trim }}' to

'{{ $[this.id].Trim.Replace('{','(').Replace('}',')') }}'" />

</log>