Data Expressions
- Last UpdatedAug 04, 2026
- 25 minute read
This section describes in detail about the Data Expressions used in the Registers Gateway.
Language Syntax
This chapter discusses about language syntax such as attributes, variables, text, numbers and so on.
Comments
A high priority to improve clarity of intent – both single and multi-line comments should be supported.
Here:
# end of line comment
<# inline comment #>
The intent here is purely for human understanding, no other uses identified.
Notation without angled < > brackets is also supported, so comments may easily be written inside XML files.
/* inline comment */
Text
Text is represented within an expression as ' ... ', for example, 'Hello Tag'.
If a quote is required in text it may be double single quote escaped, for example, 'Peter''s Tag'.
Note: In the standard expression evaluation mode for the Registers Gateway, the default is to expect text, so This is some text {{ ', this is some text in an expression,' }} and this is some more text would result in "This is some text and, this is some text in an expression, and this is some more text".
Numbers
An integer may be written simply, for example, 0, 95.
If a number has a decimal point it is treated as a double rather than an integer.
The result of an operation between an integer and a double is a double, as is the result of an operation between a double and a double. The result of an operation between an integer and an integer is an integer.
Variables
Variables are denoted by a prefix of $ with [ ... ] used for variables with non-standard variable characters such as a space inside. Standard variable names start with A-z and may continue with A-z, 0-9, - or _. They are case insensitive so $this = $THIS.
For example: $this, $[my variable]
If a variable requires [] inside, a ] may be double escaped, for example, $[my [variable]]]
Variables may be inherited and overridden in the following priority:
Config
Command line
Config folder
Project folder
Environment settings
Application folder
Environment settings support for connection strings facilitates use of container technology such as Docker. This also supports multiple environment deployment without the need to edit configuration files.
If a variable does not exist, the compiler is expected to error before execution.
It may be useful to have a test to see if a variable exists or not, however, this is not currently supported. Instead a default 'no' value must be set, for example, in a config folder settings override file, and this may be overridden with another value to have the same effect. A test may then be written, for example, a WHERE filter may test "$var not equals 'no'".
Temporary Metadata Variables
Temporary variables are explicitly stating they do not want to be stored anywhere. These have specific use in never being stored within Waypoints. This allows metadata to be set on objects (and hence passed down the pipeline with the object) which is used to make decisions within the pipeline but does not get stored anywhere.
For example, we may wish to read an existing waypoint and a new delta from a file and work out what has changed. We can add $[this.#changed] = 'Y' to all objects from the read file (pinning this in the read file command) then sort and merge the two data streams. Any records which do not have $[this.#changed] equals 'Y' are unchanged, so we can filter these out with a where condition and thus submit a small amount of changes to the waypoint (saving potentially a lot of work within the waypoint update). If we were to use a non-temporary metadata for this, the next time we added a delta, we would retrieve the record with $[this.changed] metadata from the waypoint, this would then remember it was changed from some earlier time and incorrectly report a new change.
Secure Variables
You can add configuration storing sensitive information such as database connection strings. However, registers Gateway expressions support dynamic runtime evaluation, such as a configuration received from ISM. This could allow extraction of a variable into an evaluated expression. You can use the same value in a normal expression though, to apply it in the solution configuration logic.
To protect a variable and not allow it to be used in a dynamic evaluation, the first
character of the variable will be !. For example, $[!my-secure-database-connection-string]
Any attempt to evaluate this within an expression using a compiler function will result in an error (from Registers Gateway 2.3+).
Attributes
The attribute concept here reflects a variable from a data source, which could be a field entry from a database or a cell in an Excel or CSV file, or an object-centric attribute such as within gateway processing commands or an object-centric source or target such as EIWM.
Attributes are denoted by a prefix of @ with [ ... ] used for attributes with non-standard variable characters such as a space inside. Standard attribute names start with A-z and may continue with A-z, 0-9, - or _. They are case insensitive so @tag_number = @TAG_NUMBER
If an attribute requires [] inside, a ] may be double escaped, for example, $[my [attribute]]]
The DP Gateway has one concept of variables for both variables and attributes as described here, which is sometimes flexible and useful, but can also lead to difficulty in understanding configuration and make it difficult to predict and understand a result.
Since this language allows more sophisticated expressions, an if then else statement or function may be used as an alternative, which should make intent clearer for readers.
Regular Expressions
A regular expression is not put in text quotes, instead it uses / ... / with optional modifiers after the second /.
/ ... /modifiers
For example,
/^[A-z]$/
/^[a-z]$/i
Regular expressions are a necessary evil as they are very flexible, however it can be difficult to ensure a regular expression does what is desired – especially in more complex scenarios. Additionally, regular expression processing may be notably less performant than simple string operations such as starts, contains and ends. This performance impact is much more noticeable when processing tens or hundreds of thousands of records.
In the initial implementation, regular expression evaluation is always case insensitive, as this seems the most desirable check.
To support effective usage, regexes are a language level concept. The intention is to be able to support checking and highlighting as well as ensuring that in say a replace function, if $3 or $[group] is specified, this matches something in the regex.
The compiler will fail during setup/ compilation phase, before execution, if an invalid regular expression is detected. A smart UI will also be able to highlight and show errors.
Alternatives
Regular expressions are first class citizens in the expression language, though ideally other language features and functions should reduce the need for these since they can in many cases be much less readable than you want, and when applied to many records (millions), regular expression performance is significantly slower.
For example, the language supports expressing start, end, and contains logic which will be more performant as well as more readable than the equivalent regular expression.
Conjunctions
Table 11.1 lists the Conjunctions operator in detail.
Table 11.1: Conjunctions
|
Conjunction |
Description |
|
L AND R |
Are L and R true |
|
L OR R |
Is L or R true (if L true don't evaluate R) |
Conjunctions follow standard precedence rules (AND prioritises over OR, but ( ... ) takes precedence over a conjunction.
For example: A AND B OR C AND D is equivalent to (A AND B) OR (C AND D).
Binary Operators
Logical operators return a true/false Boolean result. The following binary operators (listed in Table 11.2) are required:
Table 11.2: Binary Operators
|
Binary Operator |
Description |
|
L EQUALS R |
Does L equal R |
|
L STARTS R |
Does L start with R |
|
L ENDS R |
Does L end with R |
|
L CONTAINS R |
Does L contain R |
|
L LIKE R |
* or % for 0-n of any char ? for 1 of any char |
|
L MATCHES R |
R must be a Regex for example, @[EnumerationID] matches /[a-z][A- Z]/i |
Note: In general, LIKE and MATCHES will perform worse than other functions over a large set of data, tho ugh the main reason for inclusion is clarity and simplicity. If you do not use regular expressions, then there is no need to worry about the escape of any character.
Array Operators
Array operators allow binary logical operators to be performed against values in an array. Table 11.3 lists the Array operators in detail.
Table 11.3: Array Operators
|
Array Operator |
Description |
|
L BINARY_OPERATOR ANY [R1, ..., RN] |
Does L result in true for any of R1 to RN with the binary operator, for example, @id.slice('|',-1) contains any ['J90', '900', '02A'] |
|
L BINARY_OPERATOR ALL [R1, ..., RN] |
Does L result in true for all of R1 to RN with the binary operator, for example, @id.slice('|',-1) contains all ['J90', '900', '02A'] |
|
L IN [R1, ..., RN] |
IN is a shorthand for EQUALS ANY @id.slice('|',-1) in ['J9002-A', 'P-101', 'V-101'] |
These two statements are equivalent:
'XYZ' contains all ['J', 'O', 'N']
((XYZ contains 'J') and (XYZ contains 'O') and (XYZ contains 'N'))
These operators are primarily available to increase readability of statements and also to include a familiar SQL concept (IN statement).
Not Operator
The NOT operator must be followed by a binary logical operator and will negate that, returning a Boolean true/false result. If the binary operator is followed by an array operator (any / all) this will instead negate that. The need here is to clearly be able to negate some condition. When used with a binary operator, NOT will negate the result. Table 11.4 lists the NOT operators in detail.
Table 11.4: NOT Operators
|
NOT Operator Description |
|
|
L NOT EQUALS R |
Does L not equal R |
Note: The ISFALSE function is used to negate in other contexts.
A NOT alias function for ISFALSE was tested, but this may lead to some statements where the intent is not clear. For example, the Bar.Not.Foo operator is equivalent to FOO(NOT(Bar)) whereas we might be expecting NOT(FOO(BAR)).
Hence, the second option below is available but not the first (see Table 11.5):
Table 11.5: NOT Operators
|
Operator Name |
Description |
|
NOT(Foo(Bar)) |
Is Foo(Bar) false? |
|
Bar.Foo.IsFalse |
Is Foo(Bar) false? |
Functions
Functions may be written in any of the following forms (listed in Table 11.6) and are case insensitive:
Table 11.6: Functions
|
Function |
Description |
|
FREE FUNCTION |
Bar(), bar(Foo) |
|
FREE PROPERTY |
Bar |
|
DOT FUNCTION |
Foo.Bar(), Foo.Bar(Fum) |
|
DOT PROPERTY |
Foo.Bar |
Notes:
If a function 'Bar' has ZERO parameters, it may be written Bar or Bar().
If a function 'Bar' has ONE parameter, it may be written Foo.Bar or Foo.Bar() or Bar(Foo).
If a function 'Bar' has MULTIPLE parameters, it may be written Foo.Bar(Gum,...) or Bar(Foo, Gum,...).
This could feasibly lead to some confusing configuration, but this usage is exploratory as it is not clear which syntax (if any) is most suitable. This allows an SQL style writing and aliases of functions.
Functions may be assigned aliases, with the intention of promoting clarity and allowing for potentially familiar language aliases such as SQL. Similar functionality may be seen in PowerShell, and PML. There are no current means to specify custom aliases, but this would not be difficult to implement.
Conditions
Table 11.7 lists the Conditions operator in detail.
Table 11.7: Conditions
|
Conditions |
Description |
|
IF TEST THEN RESULT |
If TEST evaluates to true then give RESULT otherwise, give an empty string for example, if @[EnumerationID] matches /[a-z][A-Z]/i then @EnumerationID |
|
IF TEST THEN RESULT1 ELSE RESULT2 |
If TEST evaluates to true then give RESULT1, otherwise, give RESULT2 for example, if @[EnumerationID] matches /[a-z][A-Z]/i then @EnumerationID else @EnumerationID.toTitleCase |
Conditions may be chained and nested to give more complex evaluation.
The following example is translated from some fairly complex PML to determine a tag class, with an aim to be more readable since the focus here on data transformation (this is only a part of the logic):
Example:
if @type equals 'VALV' and @name contains '-VG-' then { 'GATE VALVE' }
else if @type equals 'VALV' and @name contains '-VB-' and @dtxrtext contains 'BALL VALVE FB'
then { 'BALL VALVE FULL BORE' }
else if @type equals 'VALV' and @name contains '-VB-' then { 'BALL VALVE REDUCE BORE' }
else if @type equals 'VALV' and @name contains '-VGL-' then { 'GLOBE VALVE' } else if @type equals 'VALV' and @name contains '-VC-' then { 'CHECK VALVE' } else if @type equals 'VALV' and @name contains '-VN-' then { 'NEEDLE VALVE' }
else if @type equals 'VALV' and @name contains '-VBF-' then { 'BUTTERFLY VALVE' } else if @type equals 'VALV' and @name contains '-VP-' then { 'PLUG VALVE' }
else if @type equals 'BEND' and @dtxrtext contains 'BEND' then { '5D BEND' } else if @type equals 'OLET' and @dtxrtext in
['BIOLOGICAL PROBE','CORROSION COUPON','CORROSION PROBE','PIG
DETECTOR','SCALE COUPON'] then { @dtxrtext }
else if @type equals 'PIPE' and @zonename starts '/ELEC-' and @name contains all ['-LV', '-LVT']
and not starts 'BEETLEJUICE-' then { 'LIGHT VOLTAGE TRAY' }
else if @type equals 'PIPE' and @zonename starts '/ELEC-' and @name ends '-HV' then
{ 'HIGH VOLTAGE TRAY' }
else if @type equals 'PIPE' and @zonename starts '/ELEC-' and @name ends '-MVT' then { 'MEDIUM VOLTAGE TRAY' }
else if @type equals 'PIPE' and @zonename starts '/ELEC-' and @name ends '-LTGT' then { 'LIGHTING TRAY' }
else if @type equals 'PIPE' and @zonename starts '/ELEC-' and @name ends '-PAT' then
{ 'PAGA TRAY' }
else if @type equals 'PIPE' and @zonename starts '/ELEC-' and @name ends '-COM' then { 'COMMON TRAY' }
else if @type equals 'PIPE' and @zonename starts '/INST-' then { 'INSTRUMENTATION CABLE TRAY' }
This could also be written in nested form (which the author does not think is as readable).
Note that then may not be directly followed by if, in this case scopes {...} must be used (reason - https://en.wikipedia.org/wiki/Dangling_else):
if @type equals 'VALV' then {
if @name contains '-VG-' then { 'GATE VALVE' } else if @name contains '-VB-' then {
if @dtxrtext contains 'BALL VALVE FB' then { 'BALL VALVE FULL BORE' } else { 'BALL VALVE REDUCE BORE' }
}
else if @name contains '-VGL-' then { 'GLOBE VALVE' } else if @name contains '-VC-' then { 'CHECK VALVE' } else if @name contains '-VN-' then { 'NEEDLE VALVE' }
else if @name contains '-VBF-' then { 'BUTTERFLY VALVE' }
else if @name contains '-VP-' then { 'PLUG VALVE' }
}
else if @type equals 'BEND' then {
if @dtxrtext contains 'BEND' then { '5D BEND' } else 'UNKNOWN'
}
else if @type equals 'OLET' {
and @dtxrtext in ['BIOLOGICAL PROBE','CORROSION COUPON','CORROSION PROBE','PIG DETECTOR','SCALE COUPON']
then { @dtxrtext }
}
else if @type equals 'PIPE' then {
if @zonename starts '/ELEC-' then {
if @name ends '-LV' or '-LVT' and not starts 'BEETLEJUICE-' then { 'LIGHT VOLTAGE TRAY' }
if @name ends '-HV' then { 'HIGH VOLTAGE TRAY' }
if @name ends '-MVT' then { 'MEDIUM VOLTAGE TRAY' } if @name ends '-LTGT' then { 'LIGHTING TRAY' }
if @name ends '-PAT' then { 'PAGA TRAY' }
if @name ends '-COM' then { 'COMMON TRAY' }
}
else if @zonename starts '/INST-' then { 'INSTRUMENTATION CABLE TRAY' }
}
Standard Functions
This section lists standard functions made available in the registers gateway. Function names are case insensitive.
General
Table 11.8 lists the General functions in detail.
Table 11.8: General
|
Function / Aliases |
Returns |
Description |
From |
|
COALESCE FALLBACK |
Depends |
Returns the first value that is not null or empty COALESCE('', @attributethatisnthere, 'hi') => 'hi' |
|
|
CONCAT CONCATENATE |
String |
Concatenates a list into a string Concat('hello', ' ', 'world').ToTitleCase => 'Hello World' 'a'.concatenate('b', 'c') => 'abc' CONCAT('a', 'b', 'c') => 'abc' |
|
|
JOIN CONCAT_WS |
String |
Concatenates a list into a string with a separator '|'.join('IPE','','J-9002A') => 'IPE|J-9002A' |
|
|
MAP TRANSLATE |
String |
Maps a value to another value with optional default $[file.name].left(2).map('01', 'WKMA', '02', 'BGF' , '') $[file.name].left(2).map('01', 'WKMA', '02', 'BGF' ) Without the last optional default (that is, an even number of arguments), a failure to match any item will return the original text. |
|
|
INLIST |
Bool |
Returns true if a value is found within the supplied arguments. If an argument is itself a list, this is searched. |
|
|
COMBINE COLLECT |
String[] |
Collects values from all supplied inputs into a single String[] result. |
|
|
GETREGIVERSI ON |
String |
Returns the Registers Gateway version |
2.6.9 |
|
REQUIREREGI VERSION |
bool |
Returns true if the Registers Gateway version is at least the required version RequireVersion(2.6) => 2.6.9 is good |
2.6.9 |
Yes/ No
Table 11.9 lists the Yes/No functions in detail.
Table 11.9: Yes/No
|
Function / Aliases |
Returns |
Description |
|
ISTRUE |
Boolean |
String evaluates to false if it matches the text 'true' (case insensitive) 'true'.IsTrue |
|
ISFALSE |
Boolean |
String evaluates to false if null, empty or the text 'false' (case insensitive) 'false'.IsFalse Note: No NOT alias is present here leading to confusion. There is only the special NOT logical operator as above. |
|
ISSET |
Boolean |
Indicates if the evaluated expression does not return an empty string (if you have character you are set apparently) @AttributeID.IsSet |
|
ISUNSET ISEMPTY |
Boolean |
Indicates if the evaluated expression returns an empty string or not. Returns the opposite of IsSet. @AttributeID.IsUnset |
Date and Time
Table 11.10 lists the Date and Time function in detail.
Table 11.10: Date and Time
|
Function / Aliases |
Returns |
Description |
|
DATE GETDATE |
DateTime |
Return the date without time info Now.Date => '2018-05-03 00:00:00' |
|
DATESTAMP GETDATESTAMP |
String |
Convert a datetime to standard text date format Now.DateStamp => '20180503' |
|
DAYOFYEAR |
Number |
Gives the day of year of date Now.DayofYear |
|
DAY DAYOFMONTH GETDAY |
Number |
Return the day of month Now.Day => 31 |
|
DAYOFWEEK |
DayOfWeek |
Return the day of week Now.DayofWeek => 'Monday' |
|
MONTH GETMONTH |
Number |
Returns the month part of a date Now.Month => 5 |
|
TIMESTAMP GETTIMESTAMP |
String |
Returns standard text timestamp for a datetime Now.Timestamp => '20180503-132015' |
|
NOW |
Datetime |
Returns the date and time now Now |
|
UTC TOUTC |
Datetime |
Converts a date and time to UTC Now.ToUTC |
|
YEAR GETYEAR |
Number |
Returns the year part of a date Now.Year => 2018 |
|
ISOTIMESTAMP TOISODATE |
String |
Converts a datetime to ISO 8601 format |
|
FORMATDATE |
String |
Converts a datetime to a .NET format |
|
ISDATE |
Bool |
Text
This section describes about general, special characters, and match and replace functions.
General
Table 11.11 lists the General functions in detail.
Table 11.11: General
|
Function / Aliases |
Returns |
Description |
|
CONTAINSANY |
Boolean |
Indicates if text contains any of the provided characters 'c:\temp\my\stuff'.containsany(':ms') => true |
|
COUNTCHAR |
Number |
Count occurences of some text 'c:\temp\my\stuff'.countchar('\') => 3 |
|
COUNTSLICES |
Number |
Count number of slices using separator 'IPE|J-9002A'.countslices('\') => 2 |
|
HASHCODE |
Number |
Returns the (.NET) Hash code, which may not be useful |
|
HEX TOHEX |
String |
Returns the Hex value of the specified text |
|
INDEXOFANY |
Number |
Indicates 0-based position of any of provided characters or -1 if not found 'c:\temp\my\stuff'.indexOfAny(':ms') => 1 |
|
LASTWORD |
String |
Ensures some text is suffixed with a final space then word 'hello'.lastword('world') => 'hello world' |
|
LEFT |
String |
Returns the left N characters (or less if there are not enough) 'IPE'.left(2) => 'IP' 'I'.left(2) => 'I' |
|
LENGTH LEN |
Number |
The length of a string 'text'.Length 'text'.Length() Length('Text') LEN('Text') |
|
MD5HASH MD5 |
String |
Returns the MD5 hash of a string MD5('text') '|'.CONCAT(@site, @facility, @tag).md5 |
|
PADLEFT LPAD |
String |
Left Pad a string to a certain length if shorter than that length 'Hi'.PadLeft(' |