Connect to a SharePoint with Graph API DMS

The SharePoint sample is a json split into six main sections. Some configuration parts of the sections are already hardcoded in the template so the sample will contain only some of the configurations, as follows:

• Authentication

• DmsConfiguration

• MetadataAIMContext

• AIMIngestion

• HttpSettings

• PipelineRunnerSettings

Authentication

In order to be able to authenticate using Oauth2 we must identify the:

• Client ID

• Client secret

• Tenant ID - this value is received from generating the credentials

SharePoint with Graph API uses the grant_type “client_credentials”

Here are the steps needed to obtain the clientId and clientSecret for SharePoint.

How to configure:

• AuthorizeApi: URL where you can request the access token. https://login.microsoftonline.com/s5m4v.onmicrosoft.com/oauth2/v2.0/token

• BasicLogin

  • username: The application ID that's assigned to your app. Will be present in Azure Portal where the app is registered.

  • password: The client secret that is generated for the app in the app registration portal.

• BodyRequestValues

  • grant_type: client_credentials

  • client_id: The application ID that's assigned to your app. It will be present in the Azure Portal where the app is registered.

  • client_secret: The client secret that is generated for the app in the app registration portal.

DmsConfiguration

The DmsConfiguration is a section where we configure DMS-related settings such as formats, endpoints, query parameters, and so on.

• DMSBaseUrl: Is the base URL of the endpoints that will be used. It is the common part of some of the URLs that will be consumed. For example, for SharePoint with Graph API it is https://s5m4v.sharepoint.com/sites/GraphApiSandbox containing the tenant and the site box.

• DeltaEndpoint: It represents the endpoint that will return an expected list of folders and a list of documents from those folders that should be pushed to the Ingestion API. Usually, this endpoint returns a delta instead of all of the documents. A delta represents all of the documents that are new or were modified between a certain date and the actual moment (document revision). Usually that certain date will be transmitted to the endpoint as a query parameter. For SharePoint with Graph API you can access https://graph.microsoft.com/v1.0/search/query

• RootFolder: Represents the root folder in the SharePoint folder structure from where the documents will be pulled. A delta will get documents from this folder and sub-folders in the folder tree.

• DriveName: Represents the document library name from the SharePoint site.

• DeltaRequestBodyParameters: Represents parameters to be past to the query folders and files in a paginated way in the area where the SharePoint box was set. In this case:

  • the “region” value is the region where the SharePoint box is set, for example: EUR for Europe, APC for Asia Pacific.

  • the “size” value represent the page size.

• Filtering - optional. This subsection will contain a list of FilteringItems and ValidationExpression.

  Note:

  • Supported filters are the following:

    • $.Resource.Name|$.Resource.webUrl|$.Resource.parentReference.driveId|$.Resource.parentReference.id

    • These are not configurable.

  • Since $.Resource.Name|$.Resource.webUrl are relative to file details, then only one of them should be used.

    The FilteringItem - mandatory if Filtering is present, and will contain:

• Identifier: a unique identifier of the filteringItem, that will be used in the ValidationExpression. Mandatory if Filtering is present.

• Mapping: the actual property present in the incoming metadata for all DMSs. Mandatory if Filtering is present.

• Value: the actual value against which the filtering is done, can contain a list of values separated by comma. Mandatory if Filtering is present.

  The ValidationExpression - mandatory if Filtering is present, and will contain:

  • filtering condition based on the identifiers already created and the logical operators in use OR and AND.

    Filtering can be configured by adding within the DmsConfiguration { … } section. The following example contains sample values which must be replaced with actual values before using.

    {

    "Filtering": {

    "Filteringitems": [

    {

    "identifier": "sampleIdentifier",

    "mapping": "sampleMapping",

    "value": "sampleValue"

    },

    {

    "identifier": "testIdentifier",

    "mapping": "testMapping",

    "value": "testValue"

    }

    ],

    "ValidationExpression": "{sampleIdentifier} or {testIdentifier}"

    }

    }

• IsMetadataToStore: boolean. A flag that allows us to save the response from MetadataDownloadLink as a regi file or not.

• ConvertMetadataToJson: fixed boolean value, set to true. A flag where true ensures that the metadata will be subject to conversion to json. This is needed so that the metadata is able to be parsed in the Register Gateway.

• DateTimeFormat: is a string representation of the date format supported by the DMS when calling the DeltaEndpoint. Different DMSs may use different formats so we need to convert our date to the DMS format before starting any request that contains a date.

MetadataAIMContext

The MetadataAIMContext is a section where we configure what we send to the Ingestion API, such as paths, groups, naming formats, extensions, etc.

• context: a value sent to the Ingestion API and used to group files for clients with multiple tenants.

• ClassLibraryRegisterPath

  • DocumentMetadataRegister: a path for the document revisions.

  • ScrapedFileRegister: a path for scraped files.

  • CustomScrapedFileRegisterPatterns: a list of paths for custom name patterns. Every path consists of a rank, a pattern and a path. Check the attached sample file for examples. Use an asterisk character to match any string (e.g. *-test-*.txt will match any file that has the “-test-” string preceded and succeded by any string, e.g. sharepoint-test-regi.txt)

• FileTimeStampFormat: a timestamp format that will be used for delta file names.

• FileNamePrefix: a file name prefix for regi files, used for identification.

• FileNameSuffixForDelta

  • Suffix: a delta filename suffix.

  • Extension: a delta extension. Represents the DeltaEndpoint response type extension to be saved to. If the endpoint response is XML, then the extension will be XML. Same for any other response type.

AIMIngestion

The AIMIngestion section is about Ingestion API configuration.

• BaseUrl: represents the base URL for the Ingestion API.

• AssetId: represents the ID of the asset for which the upload is done.

• ProcessingUnit: represents the processing units with the attached allowed extension type.

  Note: The Document Connector supports service to service integration with the Data Pipeline. If your configuration already has an Ingestion Token, it will be ignored with no impact to the processing. There is no need to remove the key from the config.

HttpSettings

• The HttpSettings section configures timeouts and retries. Most of them are hardcoded in the template, only the DMS timeout should be configured by the user.

  • DmsTimeout: a timeout used for DMS HTTP calls (in minutes - minimum 7, maximum 30).

PipelineRunnerSettings

The PipelineRunnerSettings is a section that will configure the flow of the program. Most of the configurations are already hardcoded in the template.

• DownloadFiles: boolean. A switch for toggling on and off the files transfer. If set to FALSE, only delta files and document links will be transferred.