Work with Datasets

The Dataset management UI provides a centralized interface for managing sample datasets within the Data Pipeline configuration.

Administrators can use this feature to upload representative data samples that facilitate the mapping and configuration process. You can understand data structure and validate field mappings.

The Datasets area shows sample dataset details and configuration actions in a structured view. Datasource groups are expandable and collapsible so that you can focus on one datasource at a time.

Each dataset row shows the following:

• dataset name

• number of fields

• linked dataset information (where applicable)

• created date

• updated date

• available actions

Sample datasets help administrators with the following:

• Understand the structure of source data during configuration.

• Test and validate field mappings.

• Preview data formats and column definitions.

• Configure linked datasets based on actual data relationships.

The sample dataset management page provides the following operations:

• Add Sample Datasets: Upload one or more file-based sample datasets (for configuration reference) or create linked datasets based on existing samples.

• Edit Sample Dataset: Update file-based sample dataset content or change a linked dataset source.

• View Sample Dataset: View columns and preview sample data structure.

• Delete Sample Dataset: Remove a sample dataset after confirmation.

• Filter Datasets: Search sample datasets by datasource name or dataset name.

• Refresh: Reload the sample dataset list from the server.

  

  

Add datasets

1. Select Add Datasets.

   A dialog opens where you can upload up to five datasets at once.

2. For each dataset select an existing datasource or choose to add a new datasource.

   When adding a new datasource, the datasource name must meet the following criteria:

   • Start with a letter or underscore.

   • The name must not start with xml.

   • Include only supported alphanumeric characters and underscores.

   • The name not exceed 50 characters.

3. Specify the dataset name.

   Dataset names follow the same validation rules as datasource names and must be unique within the selected datasource.

   The dialog allows you to choose whether the dataset is file-based or linked:

   • File-based dataset: Upload a supported file.

   • Linked dataset: Reference an existing dataset structure without uploading a new file.

4. For file-based datasets, select supported file formats which include .xlsx, .csv, .tsv, .txt, and .json.

   The file size must be 1 MB or less. Depending on the selected file type, additional configuration fields appear.

   

File-specific configuration

• For Excel files, select the header row and data index.

• For CSV files, select the header row, data index, and delimiter. The delimiter options include comma, semicolon, tab, and pipeline.

• For TSV files, the delimiter is automatically set to tab.

• For TXT files, select the same delimiter options as CSV.

• For JSON files, provide a record selector path for the array of records.

Add multiple datasets

5. Select the plus (+) button to add additional dataset rows .

6. Select the minus (-) button to remove unwanted rows.

   You can include File-based and linked datasets in the same upload batch, with a maximum of five datasets per upload.

7. Select Upload, after all required fields are completed.

   On successful upload, the dialog closes and the dataset list is updated. If validation errors occur, the errors are shown.

   

Edit dataset

8. Select the edit icon on the dataset row to edit a dataset.

   This opens the edit dialog for the selected dataset.

   In edit mode, the datasource name and dataset name are read-only and cannot be modifed. For file-based datasets, you can select a new file to replace the existing file. For linked datasets, you can change the selected source dataset.

9. Select a supported file to edit a file-based dataset.

   Changes to linked datasets do not require a file upload.

   The action button in the edit dialog displays Save for linked dataset changes and Upload for file-based dataset changes.

   

View dataset

10. Select the view icon next to a dataset row to view the dataset details.

    This opens the view dialog for the selected dataset.

    • The dialog header displays the datasource name and dataset name. If the dataset is linked, the dialog also displays the source dataset information.

    • The view dialog provides a column list and a preview section. The column list includes search capability so you can filter columns by name. Virtual scrolling is used to load columns in batches for better performance. The total number of columns is displayed.

    • The preview section can be expanded or collapsed.

    • Displays up to 100 rows of actual data from the dataset. A loading state is shown while preview data is being retrieved.

    • If preview data fails to load, an error message is displayed and you can retry the operation.

    

Delete dataset

11. Select the delete icon next to the dataset row.

    A confirmation dialog appears to confirm the deletion.

12. Select Confirm to delete the dataset.

• A source dataset cannot be deleted while other datasets are linked to it. A pop-up dialog appears stating that the source dataset cannot be deleted until the linked datasets are removed.

• When deleting the last dataset in a datasource, the confirmation dialog indicates that both the dataset and the empty datasource are removed.

  

  

Filter datasets

13. Select the filter icon.

    In the filter pane you can search datasets by dataset name and datasource name.

14. Specify filter criteria.

    You can now apply the filters and the dataset list updates to show matching results.

15. Clear individual filters or clear all of them at once.

    If no datasets match the current filter criteria, a no-results message is shown.

    

Linked datasets

Use linked datasets to create a dataset that references the structure of an existing dataset without uploading another file. This is useful when multiple datasets share the same schema or when duplicate file uploads must be avoided.

16. Select an existing datasource that already contains at least one dataset with a file.

    Linked datasets cannot be created in a new datasource because there is no source dataset available.

17. After selecting the existing datasource, specify the dataset name, turn on the Linked toggle, and select a source dataset from the dropdown list.

    • Only datasets with uploaded files are shown as available source datasets.

    • Linked datasets do not require file upload. They inherit the column structure from the selected source dataset.

    • In the main dataset view, linked datasets are identified with a Link from indication in the dataset information.

    

Supported file formats

The dataset upload supports the following file formats:

• .xlsx: Microsoft Excel spreadsheet

• .csv: Comma-separated values

• .tsv: Tab-separated values

• .txt: Plain text file

• .json: JSON format

All uploaded files are validated by file extension, MIME type, and file size. The file size must be 1 MB or less.

Error handling

• Error messages are displayed in the upload section or relevant dialog with clear information about the issue. When detailed errors are available, you can hover over the error message to see additional details.

• Common validation errors include these:

  • missing datasource

  • missing dataset name

  • unsupported characters

  • duplicate datasource and dataset name combination

  • unsupported file format

  • file size exceeding 1 MB

  • missing file selection

  • missing source dataset for linked datasets

  • exceeding the maximum of five datasets per upload

• Success messages are shown after successful dataset creation or update. Failed operations provide user-friendly messages and can be retried where applicable.

Best practices

The best practices to work with datasets:

• Special characters and spaces must be avoided.

• Related datasets must be grouped under the same datasource.

• Linked datasets must be used when multiple datasets share the same structure.

• Files must be optimized to remain within the 1 MB limit.