Use array modules split data
- Last UpdatedJun 29, 2026
- 2 minute read
Arrays are common as inputs and outputs, but not an optimal internal format when applying streaming processing.
Examples:
-
To process individual array elements in a streaming pipeline, the array needs to be split into individual messages.
-
If needing to write data into columns in a database, map an object with key/value pairs against database columns.
There are several modules that assist with converting arrays into different formats. The following table lists the most common Array modules.
|
Name |
Description |
Input |
Output |
|---|---|---|---|
|
Array Split |
Splits an array into individual messages. |
Array |
Messages |
|
Array Join |
Combine a stream of messages into an array, by time or message count (opposite of Array Split). |
Messages |
Array |
|
Array to Object |
Convert an array into an object with key/value pairs. Each item in the array must have a property holding the key value and another property with the value for that key. |
Array |
Message |
|
Object to Array |
Convert an object with key/value pairs into an array. Each element in the array will have a property containing the name of the key and another with the value belonging to that key (opposite of Array To Object). |
Message |
Array |
|
Array Filter |
Filter out some elements from an array by matching against one or several conditions. |
Array |
Array |
|
Array GroupBy |
Split an array into sub-arrays by grouping elements based on the value of a property. |
Array |
Arrays |
|
Array Property Pick |
Selects some of the properties available in each array element. A list of properties to pick must be provided. |
Array |
Array |
|
Array Property Omit |
Remove some of the properties available in each array element. A list of properties to remove must be provided. |
Array |
Array |
|
Array Sort By List |
Sort array elements by comparing the values on a specified property against an ordered list of values provided. This is typically used together with Array Property Get to create lists of values where the position in the array is used to identify a source. |
Array |
Array |
|
Array Property Get |
Select the values from a selected property. The output is an array with only values. This is useful when feeding into ML models where the position in the array is used to identify a source, rather than key/value mappings. |
Array |
Array of Values |
|
Join |
This is a variant of the Array Join module that ensures that data for a specified number of sources are always available in each output. If no data has been received for a specific source within the given time period an output will be generated based on the strategy selected. This module is more complex to set up but is useful in cases where you must ensure that each output has a value for each source - for example, when feeding an ML model. |
Messages |
Array |