Get data ready for processing

Data seldom arrives in a usable format that is ready for use. Data may arrive from multiple sources in varying formats, and there may also be non-essential data or data arriving at a higher frequency than needed.

Transform and filter data using modules

There are a large number of modules to help transform and filter this data. The following CONNECT flow modules can help with preparing data for use:

• Property Mapper

• Array Split

• ReportByException

• JSON/XML/CSV

Property Mapper Module

The Property Mapper module is the Swiss Army knife for changing the structure and naming in messages. It assists by removing unwanted parts, adding new properties or moving properties to a new hierarchy level or a assigning a new name.

Array Split

The Array Split module iterates over the elements in an array by splitting them into individual message. The array can be restored later with the Array Join module.

ReportByException

The ReportByException module removes duplicate values.

JSON/XML/CSV

The JSON/XML/CSV module converts structured data to the internal format used in flows.

Flow Studio offers more modules to help with transforming and filtering data in the Module Library under the Transformation category.

Data validation

Data validation ensures that data is correct before running your flow. As a first step, verify that your input modules are running correctly. Each module adds a success property to the output message. By adding a message filter, you can verify that a property is true for the next module in a flow. The message filter helps to avoid propagating data from an external request that failed. You can also combine this filter with other checks by adding more conditions.

JSON Schema Validation module

For more advanced validation, use the JSON Schema Validation module to verify both the structure and the values in messages. This module is located in Flow Studio's Module Library under the Process & Analyze > Conditions & Filtering category.