Data discovery query examples
- Last UpdatedOct 10, 2025
- 7 minute read
A discovery against the data source of a Structured Data Files adapter allows you to specify the optional query parameter. The query discovers the contents of the data source and narrows the scope of the discovery. You can add the discovered items to the data selection.
Structured Data Files query string
The string of the query parameter may contain none, any, or all string items in the following form:
INCLUDEDATATYPE=<DATA_TYPE_1>,<DATA_TYPE_2>;EXCLUDEDATATYPE=<DATA_TYPE_1>,<DATA_TYPE_2>;INCLUDEFIELD=<VALUE_COL_1>,<VALUE_COL_2>{UOM=<UOM_FIELD_1>},<VALUE_COL_3>{UOM=<UOM_FIELD_2>,INDEXFIELD=<INDEX_FIELD_2>};EXCLUDEFIELD=<FIELD_1>,<FIELD_2>;INDEXFIELD=<INDEX_FIELD_1>;STREAMIDFIELD=<FIELD_1>,<FIELD_2>;;
|
String item |
Required |
Description |
|---|---|---|
|
INCLUDEDATATYPE |
Optional |
Supported data types to include during discovery. Discovery looks for the specified data types in the files found in the discovery directory in the data source configuration. Note: This filter is applicable only to the valueField parameter in Data selection. Mutual exclusion applied1 |
|
EXCLUDEDATATYPE |
Optional |
Supported data types to exclude during discovery. Discovery ignores the specified data types in the files found in the discovery directory in the data source configuration. Note: This filter is applicable only to the valueField parameter in Data selection. Mutual exclusion applied1 |
|
INCLUDEFIELD |
Optional |
Name of the fields to include during discovery. You can use this string item to define a column field. Discovery looks for the specified fields in the files found in the discovery directory in the data source configuration. Extended properties UOM and INDEXFIELD can be used in braces following the column name. For example, INCLUDEFIELD=Voltage{UOM=V}. Unit of Measurement (UOM) can be defined as a reference point for measuring and comparing data. For example, temperature or voltage. Note: Only single values are permitted for extended properties. If you use more than one value, the string will not work. Multiple value columns can be defined by using commas to separate them. For example, INCLUDEFIELD=Voltage,Current{UOM=A}. Default index column mapping can be overridden by defining INDEXFIELD as an extended property. For example, INCLUDEFIELDS=Voltage,Current{U OM=A},Vrms[kV]{UOM=kV, INDEXFIELD=Local_timestamp2}. Mutual exclusion applied1 |
|
EXCLUDEFIELD |
Optional |
Name of the fields to exclude during discovery. Discovery ignores the specified fields in the files found in the discovery directory in the data source configuration. Mutual exclusion applied1 |
|
INDEXFIELD |
Optional |
Name2 of the field or fields to be used for indexing. |
|
STREAMIDFIELD3 |
Optional |
Name2 of the field or fields to be used as a placeholder for the streamID. Multiple fields can be separated by a comma. For example, STREAMIDFIELDS=SiteID,SiteCode,TagName |
|
COLUMNNAMES |
Optional |
Name2 of the streamId suffix that is used when processing a header less file. Note: This setting is applicable to CSV files only. Default: Stream1, Stream2 |
|
STREAMNAMESEPARATOR |
Optional |
Defines the character to separate streamId field. Supported characters: dash ( - ), period ( . ), colon ( :), underscore ( _ ), and blank value. Default: period ( . ) |
1 INCLUDEDATATYPE and INCLUDEFIELD cannot be used with EXCLUDEDATATYPE or EXCLUDEFIELD and vice versa. See the Query Rules section for valid and invalid combinations.
2 Index value when processing a headerless file
3 "defaultStreamIdPattern": "{FriendlyName}.{ValueField}" results in - StreamIdPrefix.FriendlyName.StreamIdField.ValueColumnName as final stream name
Query rules
The following rules apply for specifying the query string:
-
The query is made up of key=value pairs.
-
Pairs are separated with a semicolon (;).
-
Keys and values are separated with an equals (=).
-
Multiple values per key are supported. These values are separated with a comma (,).
-
Special characters are replaced by the text parser. For more information, see Text parser.
Note: The data source might contain tens of thousands of metrics. Ensure that the query only returns data for the selection items you are interested in.
-
Valid combinations for string items include INCLUDEFIELD and INCLUDEDATATYPE along with EXCLUDEFIELD and EXCLUDEDATATYPE. Defining both INCLUDEFIELD and INCLUDEDATATYPE in same query results in union, that is logical OR operation, of columns covered under both conditions.
-
Invalid combination for string items include
-
INCLUDEFIELD and EXCLUDEFIELD
-
INCLUDEFIELD and EXCLUDEDATATYPE
-
INCLUDEDATATYPE and EXCLUDEFIELD
-
INCLUDEDATATYPE and EXCLUDEDATATYPE
-
-
Unrecognized string items are not processed. A common cause of unrecognized string items is a misspelling. We recommend checking the logs when troubleshooting unprocessed strings.
Single Index, Single Stream, Single Value
|
Timestamp |
Site ID |
Voltage |
|---|---|---|
|
03/15/2024 02:00 |
510 |
23.14 |
|
03/15/2024 02:05 |
510 |
18.54 |
|
03/15/2024 02:10 |
515 |
20.54 |
Discovery query
[
{
"id": "8ff855f1-a636-490a-bb31-207410a6efff",
"query": “INDEXFIELD=Timestamp;STREAMIDFIELD=Site ID”
}
]
The subsequent data selection configuration for this file structure would be:
[
{
"valueField": "Voltage",
"dataFields": null,
"indexField": "Timestamp",
"indexFormat": null,
"dataType": "float64",
"selected": true,
"selectionCritera":"Site ID=510",
“uom”: null,
"streamId": "510.Voltage",
"dataFilterId": null
},
{
"valueField": "Voltage",
"dataFields": null,
"indexField": "Timestamp",
"indexFormat": null,
"dataType": "float64",
"selected": true,
“selectionCritera”:”Site ID=515”,
“uom”: null,
"streamId": "515.Voltage",
"dataFilterId": null
},
]
Single Index, Single Stream, Single Value, Single UOM
|
Timestamp |
Site ID |
Voltage |
Engineering Unit |
|---|---|---|---|
|
03/15/2024 02:00 |
510 |
23.14 |
kV |
|
03/15/2024 02:05 |
510 |
18.54 |
kV |
|
03/15/2024 02:10 |
515 |
20.54 |
kV |
Discovery query
[
{
"id": "8ff855f1-a636-490a-bb31-207410a6efff",
"query": “INDEXFIELD=Timestamp;STREAMIDFIELD=Site ID;INCLUDEFIELD=Voltage{UOM=Engineering Unit}”
}
]
The subsequent data selection configuration for this file structure would be:
[
{
"valueField": "Voltage",
"dataFields": null,
"indexField": "Timestamp",
"indexFormat": null,
"dataType": "float64",
"selected": true,
"selectionCritera":"Site ID=510",
"streamId": "510.Voltage",
“uom”: ”kV”,
"dataFilterId": null
},
{
"valueField": "Voltage",
"dataFields": null,
"indexField": "Timestamp",
"indexFormat": null,
"dataType": "float64",
"selected": true,
"selectionCritera":"Site ID=515",
"streamId": "515.Voltage",
“uom”:”I”,
"dataFilterId": null
},
]
Multiple stream fields
|
Timestamp |
Value 1 |
Value 2 |
Value 3 |
|---|---|---|---|
|
03/15/2024 02:00 |
1.11 |
2.22 |
3.33 |
|
03/15/2024 02:05 |
1.12 |
2.23 |
3.34 |
|
03/15/2024 02:10 |
1.13 |
2.24 |
3.35 |
Discovery query
[
{
"id": "8ff855f1-a636-490a-bb31-207410a6efff",
"query": “INDEXFIELD= Timestamp; INCLUDEFIELD=Value1,Value2”
}
]
The subsequent data selection configuration for this file structure would be:
[
{
"valueField": " Value1",
"dataFields": null,
"indexField": "Timestamp",
"indexFormat": null,
"dataType": "float64",
"selected": true,
"selectionCritera": null,
"streamId": " Value1",
“uom”: null,
"dataFilterId": null
},
{
"valueField": " Value2",
"dataFields": null,
"indexField": "Timestamp",
"indexFormat": null,
"dataType": "float64",
"selected": true, ,
"selectionCritera": null,
"streamId": " Value2",
“uom”: null,
"dataFilterId": null
},
]
Value columns indexed to different columns
|
Angle |
Timestamp1 |
Site ID |
Voltage |
Engineering Unit 1 |
Timestamp2 |
Power |
Engineering Unit 2 |
|---|---|---|---|---|---|---|---|
|
45 |
03/15/2024 02:00 |
510 |
19.31 |
kV |
03/15/2024 02:00 |
39.01581 |
kW |
|
90 |
03/15/2024 02:05 |
510 |
17.7 |
kV |
03/15/2024 02:05 |
39.01581 |
kW |
|
135 |
03/15/2024 02:10 |
515 |
19.4 |
kV |
03/15/2024 02:10 |
39.09051 |
kW |
Discovery query
[
{
"id": "8ff855f1-a636-490a-bb31-207410a6efff",
"query": "STREAMIDFIELD=Site ID;INDEXFIELD= Timestamp1;INCLUDEFIELD=
Voltage{UOM=Engineering Unit1}, Power { UOM=Engineering Unit2,INDEXFIELD=Timestamp2};”
}
]
Note: INCLUDEFIELD defines comma separated column names indexed with the column defined in the INDEXFIELD column unless overridden in the extended property, as it is in the above table.
The subsequent data selection configuration for this file structure would be:
[
{
"valueField": "Angle",
"dataFields": null,
"indexField": "Timestamp1",
"indexFormat": null,
"dataType": "float64",
"selected": true, ,
"selectionCritera":"Site ID=510",
"streamId": "510.Angle",
“uom”: null,
"dataFilterId": null
},
{
"valueField": "Angle",
"dataFields": null,
"indexField": "Timestamp1",
"indexFormat": null,
"dataType": "float64",
"selected": true,
"selectionCritera":"Site ID=515",
“uom”: null,
"streamId": "515.Angle",
"dataFilterId": null
},
{
"valueField": "Voltage",
"dataFields": null,
"indexField": "Timestamp1",
"indexFormat": null,
"dataType": "float64",
"selected": true,
"selectionCritera":"Site ID=510",
“uom”: ”kV”,
"streamId": "510.Voltage",
"dataFilterId": null
},
{
"valueField": "Voltage",
"dataFields": null,
"indexField": "Timestamp1",
"indexFormat": null,
"dataType": "float64",
"selected": true, ,
"selectionCritera":"Site ID=515",
“uom”: ”kV”,
"streamId": "515.Voltage",
"dataFilterId": null
},
{
"valueField": "Power",
"dataFields": null,
"indexField": "Timestamp2",
"indexFormat": null,
"dataType": "float64",
"selected": true,
"selectionCritera":"Site ID=510",
“uom”: ”kW”,
"streamId": "510.Power",
"dataFilterId": null
},
{
"valueField": "Power",
"dataFields": null,
"indexField": "Timestamp2",
"indexFormat": null,
"dataType": "float64",
"selected": true,
"selectionCritera":"Site ID=515",
“uom”: ”kW”,
"streamId": "515.Power"
"dataFilterId": null
}
]
Header less file data selection
|
Timestamp |
Site ID |
Voltage |
Engineering Unit |
Power |
Engineering Unit |
|---|---|---|---|---|---|
|
03/15/2024 02:00 |
510 |
23.14 |
kV |
48.2456 |
kW |
|
03/15/2024 02:05 |
510 |
18.54 |
kV |
48.8756 |
kW |
|
03/15/2024 02:10 |
515 |
20.54 |
kV |
48.3692 |
kW |
Discovery query
[
{
"id": "8ff855f1-a636-490a-bb31-207410a6efff",
"query": “INDEXFIELD=1;STREAMIDFIELD=2;INCLUDEFIELD=3{UOM=4},5{UOM=6};COLUMNNAMES={Voltage=3},{Current=4},{Power=5},{kilovolt=6}”
}
]
Note: COLUMNNAMES defines string literal that get suffixed to resulting streamId. String literals mapped to the value column in the order in which they appear.
The subsequent data selection configuration for this file structure would be:
[
{
"valueField": "3",
"dataFields": null,
"indexField": "1",
"indexFormat": null,
"dataType": "float64",
"selected": true,
"selectionCritera":"2=510",
“uom”:”kV”,
"streamId": "510.Voltage",
"dataFilterId": null
},
{
"valueField": "3",
"dataFields": null,
"indexField": "1",
"indexFormat": null,
"dataType": "float64",
"selected": true,
"selectionCritera":"2=515",
“uom”:”kV”,
"streamId": "515.Voltage",
"dataFilterId": null
},
{
"valueField": "5",
"dataFields": null,
"indexField": "1",
"indexFormat": null,
"dataType": "float64",
"selected": true,
"selectionCritera":"2=510",
“uom”:”kW”,
"streamId": "510.Power",
"dataFilterId": null
},
{
"valueField": "5",
"dataFields": null,
"indexField": "1",
"indexFormat": null,
"dataType": "float64",
"selected": true,
"selectionCritera":"2=515",
“uom”:”kW”,
"streamId": "515.Power",
"dataFilterId": null
}
]
StreamId fields
|
Timestamp |
Location |
Region |
Site ID |
Site Name |
Power |
Engineering Unit |
|---|---|---|---|---|---|---|
|
03/15/2024 02:00 |
NorthAmerica |
East |
510 |
Windy City |
48.2456 |
kW |
|
03/15/2024 02:05 |
NorthAmerica |
West |
510 |
Windy City |
48.8756 |
kW |
|
03/15/2024 02:10 |
APAC |
Southwest |
515 |
Shoreline |
48.3692 |
kW |
Discovery query
[
{
"id": "8ff855f1-a636-490a-bb31-207410a6efff",
"query": “INDEXFIELD=Timestamp;STREAMIDFIELD=Location,Region,Site ID;INCLUDEFIELD=Power{UOM=Engineering Unit};EXCLUDEFIELD=Site Name”
}
]
The subsequent data selection configuration for this file structure would be:
[
{
"valueField": " Voltage",
"dataFields": null,
"indexField": "Timestamp",
"indexFormat": null,
"dataType": "float64",
"selected": true,
“selectionCriteria”:“Location=NorthAmerica;Region=East;Site ID=510,
"streamId": " NorthAmerica.East.510.Voltage”,
“uom”: kW,
"dataFilterId": null
},
{
"valueField": " Voltage",
"dataFields": null,
"indexField": "Timestamp",
"indexFormat": null,
"dataType": "float64",
"selected": true,
“selectionCriteria”:“Location=NorthAmerica;Region=West;Site ID=510”,
"streamId": " NorthAmerica.West.510.Voltage”,
“uom”: kW,
"dataFilterId": null
},
{
"valueField": " Voltage",
"dataFields": null,
"indexField": "Timestamp",
"indexFormat": null,
"dataType": "float64",
"selected": true,
“selectionCriteria”:“Location=APAC;Region=Southwest;Site ID=515”,
"streamId": " APAC.Southwest.515.Voltage”,
“uom”: kW,
"dataFilterId": null
},
{
"valueField": "Power",
"dataFields": null,
"indexField": "Timestamp",
"indexFormat": null,
"dataType": "float64",
"selected": true,
“selectionCriteria”:“Location=NorthAmerica;Region=East;Site ID=510”,
"streamId": "NorthAmerica.East.510.Power”,
“uom”: null,
"dataFilterId": null
},
{
"valueField": "Power",
"dataFields": null,
"indexField": "Timestamp",
"indexFormat": null,
"dataType": "float64",
"selected": true,
“selectionCriteria”:“Location=NorthAmerica;Region= East;Site ID=510”,
"streamId": "NorthAmerica.East.510.Power”,
“uom”: null,
"dataFilterId": null
},
{
"valueField": "Power",
"dataFields": null,
"indexField": "Timestamp",
"indexFormat": null,
"dataType": "float64",
"selected": true,
“selectionCriteria”:“Location= APAC;Region= Southwest;Site ID=515”,
"streamId": "APAC.Southwest.515.Power”,
“uom”: null,
"dataFilterId": null
}
]
Complex data types Coordinates {X, Y, Z} and Geolocation {Latitude, Longitude}
|
Timestamp |
Value1 |
x_col |
y_col |
z_col |
Lat |
Long |
|---|---|---|---|---|---|---|
|
03/15/2024 02:00 |
46.32 |
10 |
10 |
20 |
12587.123 |
4565.126 |
Note: Complex data types are not auto discovered.
The subsequent data selection configuration for this file structure would be:
[
{
"valueField": " Value1",
"dataFields": null,
"indexField": "Timestamp",
"indexFormat": null,
"dataType": "float64",
"selected": true,
"streamId": " StreamName.SimpleValue",
“uom”: null,
"dataFilterId": null
},
{
"valueField": null,
"dataFields": {
"X": "x_col",
"Y": "y_col",
"Z": "z_col"
},
"indexField": " Timestamp",
"indexFormat": null,
"dataType": "COORDINATES",
"selected": true,
"streamId": "Streamname.Coordinates",
“uom”: null,
"dataFilterId": null
},
{
"valueField": null,
"dataFields": {
"Latitude": "lat",
"Longitude": "long"
},
{
"indexField": "Timestamp",
"indexFormat": null,
"dataType": "GEOLOCATION",
"selected": true,
"streamId": "StreamName.GeoLocation",
“uom”: null,
"dataFilterId": null
}
]
Doubling UOM as StreamIdField
|
Timestamp |
Asset |
Value |
Engineering Unit |
|---|---|---|---|
|
8/20/2024 0:00:00 |
Pump-101 |
100.1 |
degF |
|
8/20/2024 1:07:03 |
Pump-101 |
32.4 |
PSIA |
|
8/20/2024 1:16:23 |
Pump-101 |
33.1 |
PSIA |
|
8/20/2024 1:21:12 |
Pump-101 |
2.8 |
GPM |
|
8/20/2024 1:39:41 |
Pump-101 |
102.8 |
degF |
|
8/20/2024 2:09:52 |
Pump-101 |
112.5 |
degF |
|
8/20/2024 2:22:35 |
Pump-101 |
3.6 |
GPM |
|
8/20/2024 2:53:59 |
Pump-101 |
35.9 |
PSIA |
|
8/20/2024 3:14:27 |
Pump-101 |
3.3 |
GPM |
Discovery query
[
{
"id": "8ff855f1-a636-490a-bb31-207410a6efff",
"query": “INDEXFIELD=Timestamp;STREAMIDFIELD=Asset,Engineering Unit;INCLUDEFIELD=Asset{UOM=Engineering Unit}””
}
]
The subsequent data selection configuration for this file structure would be:
[
{
"valueField": "Value",
"dataFields": null,
"indexField": "Timestamp",
"indexFormat": null,
"dataType": "float64",
"selected": true,
"selectionCritera":"Asset=Pump-101,Engineering Unit=degF",
"streamId": "Pump-101.degF",
“uom”: ”degF”,
"dataFilterId": null
},
{
"valueField": "Value",
"dataFields": null,
"indexField": "Timestamp",
"indexFormat": null,
"dataType": "float64",
"selected": true,
"selectionCritera":"Asset=Pump-101,Engineering Unit=PSIA",
"streamId": "Pump-101.PSIA",
“uom”: ”PSIA”,
"dataFilterId": null
},
{
"valueField": "Value",
"dataFields": null,
"indexField": "Timestamp",
"indexFormat": null,
"dataType": "float64",
"selected": true,
"selectionCritera":"Asset=Pump-101,Engineering Unit=GPM",
"streamId": "Pump-101.GPM",
“uom”: ”GPM”,
"dataFilterId": null
}
]