Delimited Text

You can extract the Delimited text-specific data from the Gateway by defining specific settings.

Note: The Gateway supports the following Unicode subset ranges:

• Basic Latin (00–7F)

• Latin-1 Supplement (80–FF)

• Latin Extended-A (00–7F)

• Latin Extended-B (80–FF)

To extract the specific settings for Delimited Text data:

1. Click Input Source drop-down list.

2. You can choose either FileSystem or S3 Bucket from the list.

   

For FileSystem:

• Input Path: Browse and select the user input file in the correct format, that is, it must be of the delimited text type, .csv or .txt file.

• Data Source: Provides the Data Source folder path. It is a non-editable field given just to indicate the content.

• File Name: Provides file name. It is a non-editable field to indicate the content.

For S3 Bucket:

• S3: See Accessing an AWS S3 Bucket for more information.

• Download: Click this button to validate the AWS Credentials and to download the specified file from S3 bucket to Input Location of the Project folder. File System Details will be filled with the downloaded location and file details.

Source Structure: Define the following elements of the Source data structure:

• Delimiter: Select the delimiter from the drop-down box. It can be Comma or Tab or any Other delimiter. Select the delimiter from the drop-down box. It can be Comma or Tab or any Other delimiter.

• Has Header: Select Has Header option for the columns having a header.

  Note: If the columns do not have a header then do not select the Has header option. In this case, default names will be assigned to the attributes (columns) of F1, F2, F3, F4 and so on. These can be changed using Transform mapping configuration.

• Text Mode: Select Text Mode option to treat all columns as strings. This is selected by default. Clearing this option allows the extraction method to ‘guess’ what the data types are in each column, based on the first 25 entries. For example, if a column had integers in the first 25 rows, then all entries in that column would be extracted as integers. This may have inconsistent or unexpected extraction results if subsequent rows had text characters.

• Column Filter: Defines the column names and provides the filter to select particular column data for processing. Click Select to select all the columns.

• Query Generator: Type the query to fetch the data in the Query Generator box. Click this to generate the query based on your selection criteria.

  • Generate Query: Click this box to generate the query to fetch the data. Query is based on your selection criteria.

  • Edit Query: Click this box to edit the query.

• Incremental Scan:

  • Required: If you select this option, the Gateway keeps track of the last scan date by storing it in a configuration file of the extractor in the same location where Project.xml exists. When the Gateway is used for the first time, then a text file is created. Next time onwards, it reads the text file and passes this date to the Gateway.

  • DateTime Column: This option allows you to build the queries for a specific date-time.

  • Last Scanned On: This option allows you to store the last scan date-time of the most recent scan. The particular last scan data is appended to the existing query and accordingly you can generate the query. By default, this field is automatically filled with the latest scan.

• Discard Settings: Click this button if you do not require the settings.

• Save Settings: Click this to save the Extract Delimited Text settings options.

  Notes::

• The Gateway can identify some attributes as key attributes, as these attributes will be exported into EIWM file even without mapping in configuration. For example, ObjectID, ObjectName, Revision and so on. For more information about these key reserved attributes, see Appendix E: Reserved Attributes Used in the Gateway.

• If a CSV file has column names that look the same except for differences in upper/lower case (for example: Tag, TAG, and taG), the CSV extraction assigns the same attribute name to all of them, usually the last one read, so this case "taG". The others will be loaded to the EIWM as separate attributes, but they are all named "taG", hence when imported into AIM they will overwrite each other and only the last imported value will be kept. By default, the base mapping logic treats Attribute Names as case insensitive and if we give the value as [Tag] or [taG], it always picks the first occurrence, for example, "Tag". Mapping cannot access the value of the second or third attributes, for example, "TAG" nor "taG".