Data Loading Performance

There are many aspects that can influence the performance of the overall import process, from sending data to the cloud, through to data availability in the Dashboard. Each of these steps could take more or less time depending on different criteria. The following is a list of the main processes and what can affect their performance.

• Upload of the data to the cloud (via GDP, Work Tasks or other mechanisms):

  • End user internet connection speed.

  • Amount of data to be uploaded.

• Gateway processing:

  • Amount of data.

  • Complexity of the files.

  • Size of the files.

  • Data sent via previous transactions that are still not completed. In this case, transactions will be placed in a queue and will have to wait for the previous transactions to be completed.

• Data Import:

  • Amount of data.

  • Size of the files.

  • Files from other transactions still being processed. Although the loading process can import documents in parallel, this is not the case, for example, with 1D data (XML files).

For more details about specific parameters that could influence performance, depending on the type of data being processed, please see below.

3D Processing

Here we are looking at the expected average performance while loading 3D models, using ZGL or RVM files, of different sizes. The times provided are for the end-to-end process, including all the steps mentioned in the previous section.

For 3D files, the main factors that can influence performance are:

• Size of the file as illustrated above.

• Complexity of the model: the more geometries you have, the longer the conversion will take. For example, due to the internal complexities of the models, a 9 GB RVM file could take 17 hours to process, whereas a 14 GB RVM file could be quicker and take 13 hours to process. The reason could be that the 9 GB RVM file also contains mechanical 3D data that was imported into the main model. To improve performance, it is recommended to look at simplifying the 3D model (for example, removing detailed mechanical data or using model simplification from AVEVA E3D).

2D Processing

Here we are looking at the expected average performance while loading 2D documents using a DWG file. The times provided are for the end-to-end process.

For 2D files, the main factors that can influence performance are:

• Size and number of files as illustrated above.

• Complexity of files.

• Number of patterns defined in the files.

• Number of tags to extract when scraping data (above results performed using the default set of patterns provided by the gateway).

Document Processing (using GDP)

Here we are looking at the expected average performance while loading documents using PDF/DOC files. The times provided are for the end-to-end process.

These results are a guideline only. The processing times for documents will vary depending on:

• Size and number of files to process as illustrated above.

• Number of patterns defined in the files.

• Number of tags to extract, in case of data scraping (Tests have been performed using the default set of patterns provided by the gateway)

1D Processing

Class Library (clib) File

Loading this file should be considered with care and ideally tested before uploading to a production environment. Any change could have a big impact on the system.

Indeed, even a very small change in the class library file means that all 1D objects, and related information, need to be reprocessed and reloaded. So, the more data you have in the system the more impact changes may have.

Processing a clib file on a new environment with no data, may not take a lot of time, but making a change on a clib file where data already exists may take much longer. Aspects that need to be considered when loading a clib are:

• How many objects/attributes/associations exist in the system.

• How complex the class library file is.

Register File

Here we are looking at the expected performance while loading a register file. The times provided are the end-to-end process.

The main factors that can influence performance are:

• Number of records

• Number of attributes

To reduce the time taken to process a register also consider the following:

• Optimize your register files by avoiding duplicated records.

• Empty data is still processed by our gateways. To avoid this you can incorporate a filter that rejects empty data. This will significantly speed up the process.

• Publishing data to the reporting database for reporting purposes takes extra time. So you may not want to publish the data into the database every time you upload a new register, but instead to do so on a less frequent basis depending on your usage of the reports. This can be configured via the clib file.

Parallel Loading

To optimize performance you may also want to consider the frequency of uploads. The following graphic illustrates the impact of the frequency of uploads for an identical register containing 100,000 records.

In the first example, we upload the same register every 10 minutes. For the simplicity of the exercise, we are considering 1 hour for register gateway processing and 1 hour for data import. (The upload time is not considered, as this is very small compared to the other 2 steps.)

If the register gateway is already processing previously received data, then no new register files can be processed.

So in the first example, files are being queued until the gateway finishes its current process. So once the first register gets processed, all the registers accumulated during the hour can be sent for processing together.

For the import, the output file generated by the register gateway can only be processed in sequence by the import tool, hence files are also being queued. With the first example we can see that it is taking longer for the system to digest all the data if the frequency of update is too short.

Finally, if you are processing a clib file in the middle of this process, there may be a delay to get your data visible in the system, especially if the clib has a big impact on the data (see above), as more files will need to get processed in sequence by the loading tool.

For more information on the best practices of using the Registers Gateway with AIM-A, see Advanced Configuration and Best Practice.