Connect to a SharePointACS DMS

The SharePoint sample is a json split into six sections. Some configuration parts of the sections are already hardcoded in the template so the sample will contain only some of the configurations, as follows:

• Authentication

• DmsConfiguration

• MetadataAIMContext

• AIMIngestion

• HttpSettings

• PipelineRunnerSettings

Authentication

Note: Microsoft will be retiring from the use of Azure ACS (Access Control Services) for SharePoint Online authentication. Microsoft recommends Microsoft 365 customers will be better served by modern auth offered via Microsoft Entra ID. Azure ACS will stop working for new tenants as of November 1st, 2024 and it will stop working for existing tenants and will be fully retired as of April 2nd, 2026.

SharePoint Online uses the Large List Resource Throttling feature. By default, the list view threshold is configured at 5,000 items. Due to this in the RootFolder queried containing more than 5,000 items, the API will not function. Please consider using in this case the SharePoint with Graph API plugin.

Overview of large lists and libraries - Microsoft Support

In order to be able to authenticate using Oauth2 we must identify the:

• Client ID

• Client secret

• Tenant ID - this value is received from generating the credentials

The SharePoint API uses the grant_type that is, client_credentials. The OAuth 2.0 client credentials grant flow permits a web service (confidential client) to use its own credentials, instead of impersonating a user, to authenticate when calling another web service.

Refer to the How to Integrate SharePoint in GenericDocumentConnector topic to obtain the clientId and clientSecret for SharePoint.

This means that for SharePoint we need to configure the Oauth2 grant type as follows:

• AuthorizeApi: URL that where will request the access token. https://accounts.accesscontrol.windows.net/<tenantId>/tokens/OAuth/2

• BasicLogin

  • username: the Client ID@tenant ID provided by SharePoint

  • password: the Client Secret provided by SharePoint

• BodyRequestValues

  • grant_type: the used Oauth2 grant type. "client_credentials"

  • client_id: the Client ID@tenant ID provided by SharePoint

  • client_secret: the Client Secret provided by SharePoint

  • resource: 00000003-0000-0ff1-ce00-000000000000/aveva-my.sharepoint.com@<tenantId>

DmsConfiguration

• DmsConfiguration is a section where we configure DMS-related settings such as formats, endpoints, query parameters, etc.

• DMSBaseUrl: represents the base URL of the endpoints that will be used. It is the common part of all the URLs that will be consumed. For example, for SharePoint https://aveva-my.sharepoint.com/personal/firstname_lastname_aveva_com

• DeltaEndpoint: For SharePoint this must be left empty.

• DeltaQueryParameters: Represents parameters to be past to the query files in a paginated way In this case:

  • $top: value represents the page size. Example: 400

  • Refer to the Sample SharePointACS Configuration File for more information.

• RootFolder: represents the root folder in the SharePoint folder structure. A delta can get documents from this folder and folders below in the folder tree. For example, for SharePoint /personal/firstname_lastname_aveva_com/Documents/SharePointTest

• DeltaMapping

  • id: Mapping for the document ID in the delta response. In this case: “{UniqueId}”. The mapping must be added surrounded by brackets. See the Sample SharePointACS Configuration File for an example.

  • DocumentDownloadLink: URL used to download the document. This is formed of DMSBaseUrl and the api call: <DMSBaseUrl>/_api/web/GetFileByServerRelativeUrl('{ServerRelativeUrl}')/$value. Only the DMSBaseUrl must be replaced. See the Sample SharePointACS Configuration File for an example.

• Filtering

  • FilteringItem

    • Identifier: a unique identifier of the filteringItem; an identifier that will be used in the ValidationExpression.

    • Mapping: the actual property present in the incoming metadata for all DMSs.

    • Value: the actual value against which the filtering is done; it can contain a list of values separated by commas.

  • ValidationExpression: a filtering condition based on the identifiers already created and the logical operators in use OR and AND.

    Note: The supported filters are:
    Name|ServerRelativeUrl
    These are not configurable.

    Filtering can be configured by adding the following section within the DmsConfiguration section. The example is shown with sample values which must be replaced with actual values before adding to your configuration.

    "Filtering": {

    "Filteringitems": [

    {

    "identifier": "sampleIdentifier",

    "mapping": ['Name'],

    "value": "sampleValue"

    },

    {

    "identifier": "testIdentifier",

    "mapping": ['ServerRelativeUrl'],

    "value": "testValue"

    }

    ],

    "ValidationExpression": "{sampleIdentifier} or {testIdentifier}"

    }

• IsMetadataToStore: boolean. A flag that allows us to save the response from MetadataDownloadLink as regi file or not.

• DateTimeFormat: is a string representation of the date format supported by the DMS when calling the DeltaEndpoint API. SharePoint uses the following format: yyyy-MM-ddTHH:mm:ssZ

MetadataAIMContext

MetadataAIMContext is a section where we configure what we send to the ingestion, such as paths, groups, naming formats, extensions, etc.

• context: a value sent to the ingestion and used to group files for clients with multiple tenants.

• ClassLibraryRegisterPath

  • DocumentMetadataRegister: a path for the document revisions.

  • ScrapedFileRegister: a path for scraped files.

  • CustomScrapedFileRegisterPatterns: a list of paths for custom name patterns. Every path consists of a rank, a pattern and a path. See the Sample SharePointACS Configuration File for an example. Use an asterisk character to match any string (e.g. *-test-*.txt will match any file that has the “-test-” string preceded and succeeded by any string - sharepoint-test-regi.txt)

• FileTimeStampFormat: a timestamp format that will be used for delta file names.

• FileNamePrefix: a file name prefix for regi files, used for identification.

• FileNameSuffixForDelta

  • Suffix: a delta filename suffix.

  • Extension: a delta extension. This must be put json as displayed in the sample.

AIMIngestion

AIMIngestion section is about ingestion configuration.

• BaseUrl: represents the base URL for the Ingestion API.

• AssetId: represents the ID of the asset for which the upload is done.

• ProcessingUnit: represents the processing units with the attached allowed extension type.

HttpSettings

HttpSettings section configures timeouts and retries. Most of them are hardcoded in the template, only the DMS timeout must be configured by the user.

• DmsTimeout: a timeout used for DMS HTTP calls (in minutes - minimum 7, maximum 30).

PipelineRunnerSettings

PipelineRunnerSettings is a section that will configure the flow of the program. Most of the configurations are already hardcoded in the template.

• DownloadFiles: boolean, a switch for toggling on and off the files transfer. If set to FALSE, only delta files will be transferred. If set to TRUE, the document files will be also transferred.

How to Integrate SharePoint in GenericDocumentConnector

The scope of this topic is to provide the necessary steps for configuring the Generic Document Connector with SharePoint integration.

Introduction

SharePoint is a browser-based collaboration and document management platform from Microsoft. SharePoint is structured into a collection of websites – site collections. Every site collection contains sites, and, in turn, each site can contain multiple subsites.

Out of the box, documents have a limited set of properties (metadata) such as: name, file type, path, modified date, modified by, etc. This set of properties can be further enhanced by customization, either manually (by creating new columns and content types) or by enabling some features within SharePoint.

Authentication

Authentication represents the process of obtaining access within the system using either a username and a password (this is called basic authentication) or using more complex steps like OAuth2 which is more suitable in a production environment.

In order to be able to authenticate using OAuth2 we must identify the:

• Client ID

• Client secret

• Tenant ID - this value is received from generating the credentials

The SharePpoint API uses the grant_type client_credentials. The OAuth 2.0 client credentials permit a web service (confidential client) to use its own credentials, instead of impersonating a user, to authenticate when calling another web service.

Obtaining client credentials

1. Register Add-In in SharePoint (URL: https://<sitename>.SharePoint.com/_layouts/15/appregnew.aspx)

   

2. Click Create. Now we have the Client ID and Client Secret.

3. Grant permissions to register add-in:

   In the AppId insert the ClientId value and select Lookup.

   Important: DisableCustomAppAuthentication must be set to false (it’s true by default) before assigning permissions to the add-in.

   URL: https://<sitename>.Sharepoint.com/_layouts/15/appinv.aspx

   

   <AppPermissionRequests AllowAppOnlyPolicy="true">

   <AppPermissionRequest Scope="http://sharepoint/content/sitecollection/web" Right="Read" />

   </AppPermissionRequests>

4. Select Create and Trust It.

   

5. Obtain the Tenant ID.

   When you subscribe to Office 365, an unique identifier known as Tenant ID is assigned. This Tenant ID is a GUID that links all services in your Office 365 environment. To get the Tenant ID go to https://<sitename>.Sharepoint.com/_layouts/15/appprincipals.aspx

   The Tenant ID is the part after the @.

   

6. Get the access token.

   We now have all the information necessary to obtain the access token, we can do this by calling https://accounts.accesscontrol.windows.net/<TenantID>/tokens/OAuth/2.

   

   (Dummy credentials from a tutorial)

7. Use the token retrieved to call the SharePoint REST API.

8. For the AutheticationSection we should now use the following:

   "AuthenticationMethod": "Oauth2",

   "Oauth2AuthenticationDetails": {

   "AuthorizeApi": "https://accounts.accesscontrol.windows.net/<tenantId>/tokens/OAuth/2",

   "BasicLogin": {

   "username": "clientId@tenantId",

   "password": "clientSecret"

   },

   "BodyRequestValues": {

   "grant_type": "client_credentials",

   "client_id": "clientId@teanantId",

   "client_secret": "clientSecret",

   "resource": "00000003-0000-0ff1-ce00-000000000000/aveva-my.sharepoint.com@<tenantId>"

   }

   }

   For the “resource”, the value is always the constant 00000003-0000-0ff1-ce00-000000000000 in an add-in that is accessing SharePoint, this is the principal ID for accessing SharePoint.

Getting a Delta from SharePoint

The application has the possibility of downloading the requested metadata in one single call when retrieving the delta. The delta represents any information changes made to the data between two points in time.

To get the delta, we will need the DMSBaseUrl and the RootFolder as follows:

DMSBaseUrl can be something like https://aveva-my.sharepoint.com/personal/firstname_lastname_aveva_com

RootFolder can be something like /personal/firstname_lastname_aveva_com/Documents/SharePointTest

RootFolder represents the folder from where we will get the delta. The delta will contain every file from the RootFolder and below.

By default, the delta response will be in the format atom-xml. Currently we only accept the media type:

• text/csv

• application/csv

• application/xml

• application/json

We will need to change the delta response from SharePoint to be a json, for this the OptionalHeaderRequestValues value from DmsConfiguration will need to be changed to accept json.

The implementation can be changed in the application to also accept atom-xml media type.

"OptionalHeaderRequestValues": {

"Accept": "application/json"

}

Delta and Metadata mapping

In the DmsConfiguration section make the following changes:

"DeltaMapping": {

"id": "{UniqueId}",

"DocumentDownloadLink": "https://sitename.sharepoint.com/pathToDeltaPage/_api/web/GetFileByServerRelativeUrl('ServerRelativeUrl')/$value

},

DeltaMapping will extract the document ID and DocumentDownloadLink. From the delta response we will receive the ‘ServerRelativeUrl’ value and this is automatically added to the DocumentDownloadLink.