Features of the Ingestion API

This section describes the features of the Ingestion API.

Multi-Tenancy

The Ingestion API is a multi-tenant application. Each API endpoint can enable uploads for multiple assets belonging to multiple customers. In CONNECT terminology, the tenancy is maintained at Connect-Account-Name/Connect-Folder-Name level.

Example:

Consider the following scenario:

We have two customers CUST1 and CUST2 with their CONNECT accounts set up.

• CUST1 has three folders (or assets) named FOLDER1, FOLDER2, FOLDER3.

• CUST2 has two folders (or assets) named FOLDER1, FOLDER2.

  These two customers can use the same set of API endpoints for uploading files. The final tenancy mapping is maintained as follows:

  Tenant 1:	CUST1/FOLDER1
  Tenant 2:	CUST1/FOLDER2
  Tenant 3:	CUST1/FOLDER3
  Tenant 4:	CUST2/FOLDER1
  Tenant 5:	CUST2/FOLDER2

Each tenant is authenticated and authorized independently and is able to use the API for uploading files.

Transaction

Every file upload using Ingestion API runs as a part of a transaction. So a transaction can have one or more files that is orchestrated together. Each init API call opens a new transaction and each complete API call closes this transaction. After a transaction is marked completed, only then any file in that transaction is processed.

Authentication

The Ingestion API supports CONNECT Authentication. The API support Bearer token mechanism for authenticating any user. The token can be a User Access Token (UAT) or an Account Access Token (AAT). In case, the customer wants to use an AAT, it must be generated beforehand by the CONNECT Administrator.

Once the CONNECT account is setup and respective UAT or AAT is generated, the user can get authenticated by passing the token as a Bearer token in the Authorization header of the request. More information about which endpoint should be used for authentication is available here.

Authorization

A CONNECT authenticated Ingestion API user is authorized to upload files to any asset it can access in CONNECT. Any attempt to access any other asset is not entertained by the API and result in an error.

In case an UAT is used, the user's access is restricted to all the assets the user has access to within the current CONNECT account.

In case an AAT is used, the user can access any asset in the current CONNECT account.

File Upload

The Ingestion API utilizes the concept of AWS S3 pre-signed URLs to enable upload of files of all sizes to the orchestration input bucket. Each file format requires a specific set of request parameters to make sure orchestration understands what needs to be done with this file and how it must be processed.

The request parameters are described in the section Ingestion API Request Parameters.

The Ingestion API supports all file formats for upload and returns specific messages to notify success or failure.

Batch Upload

The Ingestion API also supports batch upload of files in case the user doesn't want to make too many API calls. Each file in a batch upload requires its own set of request parameters.

Multipart Upload

For files larger than 5GB, AWS S3 doesn't support a direct upload. Instead these files must be uploaded using Multipart uploads. The file must be first divided into smaller parts and these parts must be uploaded one by one and then combined again by AWS S3. This process is fully supported by the Ingestion API's multipart upload endpoints (See here).

Cancel an Upload

Any transaction can be cancelled in case the user has accidentally uploaded the wrong files/wrong request parameters.