EditAttachPrintable
r27 - 2019-10-02 - 12:39:41 - PaulEllisYou are here: TWiki >  Deployment Web > DeploymentAdminstering > ConfiguringAndTuningDNG

Tips for Configuring and tuning DOORS Next Generationupdated.png

Authors: PaulEllis, ErikCraig, BenjaminSilverman, SkyeBischoff
Build basis: DOORS Next Generation(DNG) 5.x, DOORS Next Generation 6.x.

The purpose of this page is to offer additional guidance on specific performance tuning and configuration settings with DOORS Next Generation. The level of explanation here and the general applicability is less than the standard performance reports and official sizing and tuning guides referenced below.

The main guides to reference are:

The Build Basis for this document is for DNG 5.x and 6.x, but where new capabilities are introduced, the specific release will be mentioned and override this.
As of CLM 6.0.6 ifix003 there are significant performance advantages when using this release. Notably, 6.0.6 ifix003 improves the case of opening a module within a changeset for the first time. Caches are now populated from the parent so that the changeset will not have to pay the penalty.

In terms of stability, there is tighter control on SPARQL queries, which if left to run, cause server instability. This is handled via a new Rogue Query Monitor which will automatically kill any queries that are not configured to run over a specific period of time.

In terms of scalability, CLM 6.0.6 ifix003 also completes the use case for Cross-component links across servers, allowing multiple RM repositories to connect together.

This document has also been updated to give more context on the memory-mapped I/O (MMIO) issue experienced by customers using VMWare ESX and Windows 64-bits servers.

Tuning the Java Virtual Machine

Wikipedia quote "The Java virtual machine heap is the area of memory used by the JVM for dynamic memory allocation". IBM advises that the initial heap for DNG is set to be one third of the overall JVM.

However, there are several use cases where altering the JVM heap sizes either temporarily, or due to growing pains is appropriate. The following advice is listed to address the considerations available to administrators since DNG 6.0.2.

Memory Mapped I/O in DNG 6.0.2+

Jena Memory-mapped_file is a new optional memory mode which allows for a different way of managing the DNG indices. Prior to DNG 6.0.2 direct I/O mode was the default mode on Linux and Windows. DNG 6.0.2, 6.0.3 and 6.0.4 default to direct mode for Windows. Since DNG 6.0.5, mapped mode is the default for Windows servers as well as Linux servers.
The two modes differ in how they access the JVM nursery, resulting in different performance for the overall system in memory consumption and garbage collection.

Here is a brief comparison between the two modes:

Jena behavior using Direct I/O Mode:

  • As a query processes, the indices are copied to the JVM nursery causing large chunky disk accesses.
  • These are short-lived and is part of the reason why we recommend the 1/3rd nursery space for DNG.
  • This then results in more Garbage Collection work.
  • Direct mode requires a very large Java heap and considerable additional OS memory to load the indices in the System File Cache.

Jena behavior using Memory Mapped I/O Mode:

  • Jena is able to directly access the address of the blocks on disk
  • In the best case this is cached by filesystem/operating system
  • It's just a memory access outside of the JVM heap no copying - no Garbage Collection.
  • We recommend reducing to 1/4 nursery space for DNG.
  • Mapped mode requires a moderate-large Java heap and less additional OS memory to load the indices in the System File Cache.
  • Some Windows configurations on Virtual Machines will have slower Jena writes due to a virtualized IO issue with memory mapped I/O.

Memory Mapped I/O on Windows 64 bits systems


Note: A suspected defect Memory Mapped I/O on Windows Causes Periodic 20-40 second hangs while writing to the index disk files in Windows 2012 was reported during our testing. This issue has led to an investigation work item 114883 to understand why there is a write slow down in virtualized Windows environments and whether this might also impact other operating systems if also virtualized.
If you are running with memory-mapped I/O and are experiencing hangs similar to this please contact IBM Support.
If you are experiencing issues on a Windows VM or are intending to implement this on a Windows VM, then this runs the risk of write stalls. As stated in investigation work item 114883 this has only been observed when using VMWare ESX and we would recommend the knowledge base article A virtual machine can freeze under load when you take quiesced snapshots or use custom quiescing scripts (5962168) and contacting VMWare to resolve this problem

IBM originally reproduced this using the following in August 2017: Windows 2012 using ESXi 6 with XIV fiber channel LUNs (GEN3 XIV with 4TB disks 7500 RPM)

We turned on this feature on Windows in CLM 6.0.5 and have had very few reports of this issue. However, if you are using 64 bits Windows then this setting is not a default, as there are more common issues for consideration before doing so.
There are two settings in the Advanced Properties (rm/admin) that need to be changed from direct or infer, to mapped. If you do need to do this, note that the scalability referenced in the Sizing Guides will not be achievable.

Jena TDB Default File Access Mode
Jena TDB File Access Mode

The same properties can be set by adjusting teamserver.properties in /server/conf/rm to include the following:

com.ibm.team.jfs.jena.tdb.file-mode-default=mapped
com.ibm.team.jfs.jena.tdb.file-mode=mapped

Note: If done this way, the server must be restarted or done with the server offline for the changes to take effect

Considerations when using Mapped I/O on Windows

One of the main considerations of using memory mapped I/O is that the possibility of contention in the 32 bits space within Windows increases, as detailed in Native Out of Memory Exceptions (Note: Linux does not have this issue).
It is therefore essential to switch -Xcompressedrefs to -Xnocompressedrefs.

We recommend in the Native Out of Memory Exceptions article the

      "IBM Java team recommends that when you use -Xnocompressedrefs you should be 
      doubling current Java Virtual Machine(JVM) heap allocation."

This recommendation does not apply in this context. The usage of the nursery heap will actually go down when using memory-mapped I/O, with a larger demand on native memory. Therefore, it is recommended to use the existing JVM heap sizes initially and monitor your way to your optimum setting.

Lifecycle Query Engine(LQE) and Link Index Provider (LDX)

DOORS Next Generation is not the only implementation with underlying Jena indices. Lifecycle Query Engine(LQE) and Link Index Provider (LDX) also use this technology, therefore it is important to understand this is specific to VMWare + Windows + Memory Mapped I/O. Initial indexing is slower than expected when LQE/LDX is running on a virtualized Microsoft Windows server details how to switch to using direct mode for the purposes of running the reindex. For optimal use during run time, it is still advised to use mapped mode.

Large heap sizes, greater than 24GB

Similar to used mapped I/O, as described above, you would need to switch from compressedrefs to nocompressedrefs when a heap size of more than 24GB is required. This is due to greater heap space usage that could be prevented.


Disk speed's impact on performance

Large imports or deletes of data are predominately write heavy. This is due to having to index the data that is being imported at the same time as importing it into the system. You will see large I/O usage when you import in both the database's area (data files and temp spaces), as well as the indices directory of DNG. If you are using a browser on the server for the import, then this is not advisable as the browser will also add to I/O contention and of course have its own CPU and RAM consumption.


The discussion on large imports is discussed in further detail within the DOORS to DNG Migration Guide. The impact on even memory mapped I/O, described in the previous sections, is when the data is not actually in memory on some level. In these instances, slower disk = slower queries (and indexing time). This is an important point, as we have seen customers who have increased the RAM and vCPUs available, but overlook their Achilles' heel of disk speed.
Mapped I/O would be fast, even on say a 5400rpm disk. However, only if it was all able to be in memory (and all had been accessed at least once), hence populating the caches. This would particularly impact non functional testing and evaluations, if you do not run multiple runs of tests and do not allow the server to fully initialize.
There is a more comprehensive discussion of RAM, CPU and disk speed within the Proof of Concept Sizing in CLM 6.x. The advice there is not specific to DNG and is part of a different discussion. It is worth noting though that there are additional disk speed considerations for large imports of data into DNG, both at the application and database levels.

Tuning repotools commands for upgrade and reindex

The majority of this article is based around ensuring that a typically running system is optimally tuned for performance. This typically means that you need to optimize the read/write speed of your indices and balance this against your RAM and heap requirements.
When you need to reindex, or upgrade, then there are not the same concerns of other running programs and you can utilize more of your available resources. The latest Interactive Upgrade Guide quotes that you can use 8GB RAM for your Xmx value within the repotools-rm file (.bat on Windows, .sh on Linux). We recommend that you can expand this to half RAM for these operations, so as to minimize the duration of the upgrade or reindex operations.

What else?

Upgrade Must Read pertinent information is the best place to check out the known issues outside of this document. At the time of writing, there is particular database advice for Oracle and SQL Server on top of the general advice of Appendix E: Application Server Tuning in the 6.0.6 ifix 3 "Performance test results for Rational DOORS Next Generation 6.0.6 ifix3 with Components".

Here are some known issues:

Related topics: Sizing And Tuning Guide DNG 602, Configuring and tuning the Java Virtual Machine (JVM)

External links:

Additional contributors: IanGreen, RosaNaranjo, KnutRadloff, MattHolt

Edit | Attach | Printable | Raw View | Backlinks: Web, All Webs | History: r42 | r29 < r28 < r27 < r26 | More topic actions...
 
This site is powered by the TWiki collaboration platformCopyright © by IBM and non-IBM contributing authors. All material on this collaboration platform is the property of the contributing authors.
Contributions are governed by our Terms of Use. Please read the following disclaimer.