MultiPartUpload API for Large Files

Any file larger than 5GB must be divided into parts smaller than 5MB before it can be uploaded using the ingestion API. The process and responses for the multipart API requests is quite complex, so it is recommended to use a utility.

The Ingestion API performs a multipart upload in five basic steps:

1. Initiate a transaction: While initiating a transaction, the user token is authenticated and authorized. After it is successfully authorized, the API returns a TransactionId that is the unique identifier for this upload.

2. Initiate a multipart upload: After TransactionId is received, the user can now use this TransactionId to upload large files. A multipart upload request is initiated for the large file. A multipart uploadId is returned which is used in the next steps.

3. Upload part(s): When the uploadId is available, a redirectURL must be requested for each part of the file. Each part is then uploaded to the respective redirectURL. When uploading a single part, in addition to the uploadId, a part number is also specified. The part number is any number between 1 and 10,000. It uniquely identifies a part and its position in the file being uploaded. When a part is successfully uploaded, the API returns an eTag header in its response.

4. Complete multipart upload: When all the parts of a file are successfully uploaded, the multipart upload must be completed. This requires the part number and the eTag header from the upload step and the uploadId from the initiate multipart step.

5. Complete the transaction: After all the files are successfully uploaded into their respective S3 URLs, the transaction can now be completed or aborted.

   To upload files above the size 5 GB, you must use the Multipart Upload API.

   This MultiPartUpload API internally has three calls in sequence in addition to the init and complete Load API request:

   

To perform these steps, the Ingestion API provides the following five endpoints:

• /api/load/init - Initiate a new transaction

• /api/load/multipart/init - Initiate a new multipart upload

• /api/load/multipart/url - Get S3 presigned URLs to upload parts

• /api/load/multipart/complete - Complete the multipart upload

• /api/load/complete - Complete the transaction

Initiate a new multipart upload process for a file

/api/load/multipart/init – Initiate a new multipart upload process for a file.

This endpoint enables you to initiate a new multipart upload process for each specific part of the larger file you want to upload.

HEADERS

You must pass the following two headers:

• Authorization

  This is a valid CONNECT Bearer JWT token.

• TransactionId

  This is generated as part of the init API request.

REQUEST BODY

Example Value

{

"target": "string",

"source": "string",

"publishOriginal": true,

"publishRendition": true,

"publishReference": true,

"documentIdentifier": "string",

"context": "string",

"additionalMetadata": {

"additionalProp1": "string",

"additionalProp2": "string",

"additionalProp3": "string"

},

"fileName": "string",

"registerIdentifier": "string"

}

RESPONSES

Get the redirect URL for single or multiple parts of the file

/api/load/multipart/url – Get the redirect URL for single or multiple parts of the file

This endpoint returns the redirect URL for each part of the larger file. A file larger than 5 GB must be broken into different smaller parts greater than 5 MB other than the last part. You need a new redirect URL for uploading each part of the file.

HEADERS

You must pass the following two headers:

• Authorization

  This is a valid CONNECT Bearer JWT token.

• Transaction ID

  This is generated as part of the init API request.

REQUEST BODY

Example Value

{

"multiPartUploadId": "string",

"partNumber": 0,

"fileName": "string",

"registerIdentifier": "string"

}

RESPONSES

Complete the multipart upload process for a file

/api/load/multipart/complete – Complete the mutipart upload process for a file

This endpoint returns the confirmation of completion of multipart upload for a large file.

This request needs to be recursively triggered until the requestId parameter in the response is null. As multipart takes time based on the number and size of the parts uploaded, this multipart/complete API is designed in such a way that you can make a recursive call to identify the status of multipart upload.

Note: When you have a requestId other than null, the status is InProgress. By the time you get the null value for the requestId in CompleteMultiPartResponse, the Status is either "Completed" or "Error".

HEADERS

You must pass the following two headers:

• Authorization

  This is a valid CONNECT Bearer JWT token.

• TransactionId

  This is generated as part of the init API request.

REQUEST BODY

Example Value

{

"multiPartUploadId": "string",

"multiPartUploads": [

{

"eTag": "string",

"partNumber": 0

}

],

"requestId": "string",

"fileName": "string",

"registerIdentifier": "string"

}

RESPONSES