Import Data

Configuring AIM to import data and creating an Import Package can be done in two ways, using either:

• The Deployment Manager - Step 4: Configure Features, or

• The Database Manager tool, see Database Manager Tool, and the Import Controller tool, see Import Controller Tool.

Both methods use the same mechanism and both can be used to start an update, but they differ slightly in the way they collect the necessary information.

After the AIM Workhub database has been created and structured, the information in an Import Package (or just Package) is used by the Import Server to:

1. Scan the Staging Area for new or changed files. See the section The Staging Area.

2. Convert files (if necessary) to formats appropriate for AIM.

3. Create descriptive XML files for each document.

4. Store the converted/created files in the Temporary Area.

5. Copy other documents to the File Store.

6. Parse the XML files to create Objects and Associations in the AIM Workhub database.

   Note: As an Import Package is XML data stored in the tool in which it was created. It can only be seen in that tool.

The Staging Area

A Staging Area is a folder where documents and sub-folders are organized for importing into the AIM Workhub. It can be located anywhere as long as it is visible on the network to the Import Server. A database can contain any number of Import Packages but there should always be a one-to-one correspondence between an Import Package and a Staging Area.

Files in the Staging Area are usually of two different types:

• Data or Document files, for example:

  • a PDMS plot file

  • an MS Excel datasheet

  • a PDF document

• Metadata files

  • [file].vnet – that contains the metadata for an individual file.

  • [folder].vnet – that contains the metadata for all the files in a folder.

For more detail on the correct format for files in the Staging Area, refer to the appropriate AVEVA Gateway or third-party documentation.

Default Permissions

The Import Service virtual account has by default Modify, Read & Execute, List Folder Contents, Read permissions while installing. The Import Service virtual account is by default added to the <Installed Location>\Import and the <Installed Location>\SampleData folders. Add the Import Service virtual account to the Staging Area folder, if it is configured in different location.

File Store and WCF Configuration

The File Store uses Windows Communication Foundation (WCF) streaming to transfer files. The default settings are suitable for many installations but can be configured if required. Typically this is done to change the maximum size of a file that can be used or to change the time out on a slow network.

The file store uses the standard WCF configuration which is publicly documented.

A summary of the most useful configuration parameters is given here:

• maxReceivedMessageSize - the maximum size in bytes for the message. This should be set large enough for the largest file.

• maxBufferSize - this is the maximum size in bytes for buffering the file. Reducing this will reduce the total memory required by the File Store. Increasing it may increase the performance.

• receiveTimeout - with a slow network or very large file the timeout value may need to be increased so that the file transfer completes.

These parameters should be set in both of the following configuration files:

• AVEVA.NET.FileStore.DomainHost.exe.config (for the File Store itself).

• Web.config (for the Workhub website).

If you experience issues using the File Store, enable logging by uncommenting the system.diagnostics and service.model sections of the AVEVA.NET.FileStore.DomainHost.exe.config file.

Linking the File Store to SQL Server

By linking the File Store to SQL Server, all the files are backed up when the SQL database is backed up.

To configure the SQL Server File Store:

7. Enable FileStream at the instance level:

   1. Open SQL Server Configuration Manager from the path: C:\ProgramData\Microsoft\Windows\Start Menu\Programs\Microsoft SQL Server<VersionNumber>\Configuration Tools

   Alternatively, you can search for SQLServerManager<VersionNumber>.msc in the Windows Start menu.

   Note: Ensure that you are using the Configuration Manager.

   

   2. Select the instance of SQL Server on which you want to enable FileStream. Right-click and select Properties.

      

   3. In the SQL Server Properties window, select the FILESTREAM tab, and then select Enable FILESTREAM for Transact-SQL access.

   4. Select Enable FILESTREAM for file I/O access, and then, type the Windows share name.

   5. Select Allow remote clients access to FILESTREAM data.

   6. Select Apply, and then select OK.

8. Open the SQL Server Management Studio and enable the FileStream feature by using one of the following methods:

   7. Select New Query to display the Query Editor.

   8. In the Query Editor, enter the following Transact-SQL code:

      EXEC sp_configure filestream_access_level, 2

      RECONFIGURE

      OR

   9. Right-click on the instance of the Server and select Properties.

   10. On the Advanced tab, select FileStream Access Level and set it to Full access enabled.

   11. Select FileStream Share Name and enter the name of the Windows share (here MSSQLSERVER).

   12. Select OK.

       

9. Create the following path if it does not exist: C:\ProgramData\AVEVA\AVEVA Asset Information Management\SQLFileStore

10. Create the FileStream enabled Database, restart the SQL Server service, and then execute the following query:

    Note: While creating a new database (for example, CygnusSQLFileStore), it is recommended NOT to use the main Cygnus database for storing files.

    CREATE DATABASE CygnusSQLFileStore

    ON PRIMARY (NAME = N'ANetSQLFS', FILENAME = N'C:\ProgramData\AVEVA\AVEVA NET\SQLFileStore\ANetSQLFS.mdf'),

    FILEGROUP FileStoreFG CONTAINS FILESTREAM (NAME = ANetFS, FILENAME= 'C:\ProgramData\AVEVA\AVEVA NET\SQLFileStore\SQLFileStore')

    LOG ON (NAME = N'ANetSQLFSlog', FILENAME = N'C:\ProgramData\AVEVA\AVEVA NET\SQLFileStore\ANetSQLFSlog.ldf')

    WITH FILESTREAM (NON_TRANSACTED_ACCESS = FULL, DIRECTORY_NAME = N'FileStore')

11. Create FileTables.

    USE CygnusSQLFileStore

    CREATE TABLE Sample_Project AS FileTable

    WITH (

    FileTable_Directory = 'SampleProject',

    FileTable_Collate_Filename = database_default

    );

    GO

    Note: The path to the created FileTable is: \\{{SERVERNAME}}\mssqlserver\FileStore\SampleProject

12. Ensure that the SQL Server File Table has been configured correctly by adding or deleting files, as explained in the following steps:

    13. In Object Explorer, navigate to Databases > CygnusSQLFileStore > Tables > FileTables.

    14. Select dbo. Sample_Project table, right-click and then select Select Top 1000 rows.

    15. Ensure that the Sample_Project table is empty.

    16. Select dbo. Sample_Project table, right-click and then select Explore FileTable Directory, that is,

        \\{{SERVERNAME}}\mssqlserver\FileStore\SampleProject

    17. Create a new .txt file (for example, test.txt).

    18. In SQL Server Management Studio > Select Top 1000 rows of Sample_project, ensure that there is a row (or file).

        Note: Files can be added or deleted by SQL Query or Explore Window.

13. Open the AVEVA.NET.FileStore.DomainHost.exe.config file located in {{AVEVA Asset Information Management Installation Directory }} \AVEVA Asset Information Management\FileStore.

    Note: The AIM Installation Directory path is usually C:/Program Files/AVEVA.

14. Replace the root element for SQL fileVault in fileVaults section:

    <fileVaults>

    <fileVault vaultType="NTFS" name="FileSystem" assembly="" root="C:\ProgramData\AVEVA\AVEVA NET\FileStore" locator=""/>

    <fileVault vaultType="SQL" name="SQLFileSystem" assembly="" root="\\{{SERVERNAME}}\mssqlserver\FileStore\SampleProject" locator=""/>

    </fileVaults>

    vaultType: It must be either NTFS or SQL.

    name: It can be any string.

    root: The path to the NTFS or SQL file store depends on the vaultType chosen.

    Locator & assembly: Leave these fields blank.

Loading Users into Workhub

To load a set of users into Workhub:

15. Open the Bootstrap folder. The default location of Bootstrap files is: [installation folder]\Import\Server\Bootstrap

16. Create the following XML file:

    BootstrapSampleUsers.xml

17. To create the BootstrapSampleUsers.xml file:

    19. Add the <Object> tag to each user.

    20. Add one <ID> tag that defines the username to each <Object> tag

        Note: You can assign multiple roles to each user adding the <Association> tag for each role. For example, roles such as the Default User, Administrator.

    21. Copy the following script in the EIWM format:

        <?xml version="1.0" encoding="UTF-8" ?>

        <?vnet xslt="NULL" ?>

        <vl:VNETList xmlns:vl="http://www.aveva.com/VNET/List" xmlns="http://www.aveva.com/VNET/eiwm">

          <Template>

          <ID>BootStrapSampleUsers.xml::template</ID>

          <Object>

          <ID>[domain]\[username]</ID>

          <Association type="fulfills">

          <Object>

          <ID>[Role1]</ID>

          <ClassID>ROLE</ClassID>

          </Object>

          </Association>

          <Association type="fulfills">

          <Object>

          <ID>[Role2]</ID>

          <ClassID>ROLE</ClassID>

          </Object>

          </Association>

          </Object>

        <!--more users can be added using the <Object> tag as shown above-->

          </Template>

        </vl:VNETList>

Watch Folders

Watch Folders enable the AIM Import Server to watch particular physical folders for the arrival of a trigger to start an unattended Import Run from that folder.

When the AIM Import Server sees a trigger.start file arrive in a folder or subfolder that is being watched, its name changes along with its process flow, as explained in the diagram below:

The arrival of a trigger.start file causes the folder to be processed as if an Update command had been issued.

If you want to perform the equivalent of a Force Update command, then use a file called trigger_force.start instead of a trigger.start file.

Note: If the creation of a trigger.start or trigger_force.start file does not automatically start an import, then the system will check every five minutes to see whether one of these files has been created. If one has been created, then the appropriate import will start.

Grouping RVM/ZGL Files into One 3DVis OnPrem Model

The Import Server provides a mechanism for grouping 3D model files into a larger model. This enables the AIM Dashboard to display the overall model and it is achieved by Multiple Model Composition (.mmc) files.

The following are the points to note on grouping:

• The groupings can be done either from a combination of files and folders outside the Staging Area or from within the Staging Area.

• If these files and/or folders are located outside the Staging Area, they must still be accessible to the Import Service.

• An AIM <PORTAL OR W&D> Pointer file with a .mmc suffix can be placed in any convenient folder in the Staging Area.

The format of the .mmc file is as follows:

Sample 1:

<Object>
<Type>rvm document group</Type>
<Location>./3DModel</Location>
<File>./Area 15.rvm</File>
</Object>

Sample 2:

<Object>
<Type>zgl document group</Type>
<Location>D:/Files/3DModel</Location>
<File>./Equipment.mmc</File>
</Object>

Sample 3:

<Object>
<Type>rvm document group</Type>
<Location>D:/Files/3DModel</Location>

<Location>./3DModel</Location>
<File>./Equipment.mmc</File>

<File>./Area 15.rvm</File>
</Object>

Note: There can be multiple Location and/or File elements.

A metadata file with a .vnet suffix is also required to supply the Class and the name of this document. This can be a specific file or a Folder.vnet file applicable to all the files in this folder.

<Object>
<ID>Complete SRP Plant </ID>
<ClassID>3D MODEL<</ClassID>
<Context>
<ID>VPD</ID>
</Context>
</Object>

Note: When the Import Server encounters an AIM Workhub Pointer File (.mmc), it creates 3DVis OnPrem files containing ‘includes’ for each of the individual model files.

Important:

• The Import Server does not process the individual files referenced in the .mmc file. Instead, it converts the referenced model files into 3DVis OnPrem files and places them in the 3DVis OnPrem Drop folder, inside a directory named with the .mmc file.

• During .mmc file processing, the Import Server does not establish tag associations for the underlying models or for main .mmc file.

• The mechanism described overcomes a limit on the size of 3D model that could be translated to 3DVis OnPrem format.

• This mechanism supports the creation of large models, allowing a combined maximum size of up to 4GB for all RVM model files, and up to 5GB for ZGL model files referenced in the .mmc file although Import Controller can take a long time to convert them into 3DVis OnPrem files (and this is a process that can continue long after other update activity has completed).

Managing Item Revision Order

By default, OPE creates an item's new revision at the end of the revision (rev) stack, when a latest revision is loaded.

For example, If we have an existing revision stack 01->02->03->04->05, and a new revision 03 is loaded, it will be placed in the revision stack as 01->02->03->04->05->03.

And if you want to load the revision 03 as per the numerical ordering (between 02 and 03), then you must remove all traces of 02 and 04 from the system, and then load 03 before loading 04 and 05.

You can control this default behaviour of OPE, using the succession association between two items' revision. For this, you must use either of the following associations with a type of:

• Is succeeded by

• Is a successor of

For example, if you want to load a new revision 03 before 04 in an existing revision stack 01->02->04->05, you can use either of the following XML:

Example.1

<Object>

  <ID>ISO1001</ID>

  <Revision>02</Revision>

  <Association type="is succeeded by">

  <Object>

  <ID>ISO1001</ID>

  <Revision>03</Revision>

  </Object>

  </Association>

  </Object>

Example.2

<Object>

  <ID>ISO1001</ID>

  <Revision>04</Revision>

  <Association type="is a successor to">

  <Object>

  <ID>ISO1001</ID>

  <Revision>03</Revision>

  </Object>

  </Association>

  </Object>

Note: The object that is moved in the stack is highlighted in Yellow.

3D Materials Name Translation

When a review model file (.rvm file) is created, all the 3D objects within it have various materials assigned to them, depending on how the model file is created. The names of these materials are of the form Materialnn (for example, Material01). In most cases, such names are not meaningful to the users.

Modifying the Materials Names in the Materials Dialog

You can provide meaningful alternative names for the materials, which display in the Materials dialog area. The definitions for the name translation to be done must be located in an XML configuration file. The following is an example of the content of such an XML file:

<?xml version="1.0" encoding="UTF-8"?>
<Materials>
<NameTranslation>
<MaterialName original="Material01" new="Structures"/>
<MaterialName original="Material02" new="Steelwork"/>
<MaterialName original="Material03" new="Equipment"/>
</NameTranslation>
</Materials>

The file specifies that Material01 (in the original .rvm file) will be displayed as Structures in the Materials dialog table (and it is similar for the other material names mentioned). If a material occurs in the 3D model that is not specified to be translated by the XML file, then the original Materialnn name is displayed.

After you have an XML file in the above format with the required name translation defined, you must then import the file.

Place an entry in the relevant .vnet files in a Staging Area. If you want to apply the XML file (located at, for example, C:\MaterialsMapping.xml) to a single Review model file (like srp.rvm), then you should alter the corresponding srp.vnet file and insert the following piece of code:

<MaterialMappingFile>C:\MaterialsMapping.xml</MaterialMappingFile>

Following the usual rules for use of .vnet files, if you want the same XML file to be used for all .rvm files in a particular folder, you can put the above piece of code into the folder.vnet file for that folder (instead of adding the piece of code in the srp.vnet file).

Advanced Usage

You have the following advanced options for the material names displayed in the Materials dialog area. These are also specified in the XML file.

If you want to limit the names in the Materials dialog in the Dashboard to only those defined in the XML file, then set the outputMappedNamesOnly attribute of the NameTranslation element in the file to true:

<NameTranslation outputMappedNamesOnly="true">

When the above option is enabled, you can specify that any material names omitted due to the above option being in force are displayed using the specified default name:

<NameTranslation outputMappedNamesOnly="true" defaultName="Unmapped Material">

Disable Writing to the Archive Tables

You can disable writing to the Archive tables during the import. This improves the system performance by reducing both the size of the database and the write operations to the database.

Note: Disabling writing to the Archive tables disables the auditing during the import.

To enable or disable archiving while importing, modify the value of the parameter, avevanet:import:disablearchiving, using the following details:

• File name (default location): C:\Program Files\AVEVA\AVEVA Asset Information Management\Import\Server\AVEVA.NET.Import.Server.exe.Config

• Parameter values:

  • True – Disables archiving when importing

  • False – Enables archiving when importing (default value)

    Note: You cannot set this through the Deployment Manager.

Merging of Unclassified Objects

During the Import, you can refine the merging of unknown objects using the Allow Merging Of Unknown Objects option.

This option enables you to merge tags and its aliases even when they are unknown.

When this option is selected: If two unknown objects have the same alias, then they will be merged. Both the objects will be identified by the alias. The Import Controller shows the following warning message if the objects are classified:

"{Object 2} could not be identified by {Alias} - identifier already exists for another object."

When this option is not selected (default): If two unknown objects have the same alias, it shows the following warning message:

"{Object 2} could not be identified by {Alias} - identifier already exists for another object.

Enable 'Allow Merging Of Objects' or 'Allow Merging Of Unknown Objects' option accordingly in order to add this alias by merging those two objects."

Note: The EIWM template containing actual objects must be processed first before its aliases.

Import Controller Logs

By default, import logs will be available in the folder: <installed location>\Import\Server.

To change the location of the import logs folder, give permission to the folder for the NT Service\AVEVANETImportServer service.