Attempting to use Collaborative Lifecycle Management on Windows operating system leads to NativeOutOfMemoryException (NOOME). new.png

Authors: NatarajanThirumeni , PaulEllis
Build basis: Engineering Lifecycle Management 5.x, 6.x

This article was originally scoped to explain a situation observed when using Rational Team Concert (RTC) on Microsoft Windows 2008 R2 Operating System which can result in a NativeOutOfMemoryException (NOOME).
This article has been updated due to the same observation being made on Microsoft Windows operating systems in general, also including 2008 and 2012, during evaluation use of Engineering Lifecycle Management (ELM) 6.x.

Typically, we have observed NOOMEs when multiple Engineering Lifecycle Management applications were collocated on the same Windows server. This was especially prevalent when the Lifecycle Query Engine (LQE) had been installed alongside all of the other ELM applications required in a global configuration environment. This issue is one of many discussed in the Proof of Concept Sizing page, where we discuss implications of architectural decisions and what constitutes a supported configuration.

There have now been native out of memory exceptions reported, but with a new use case, as documented below.

As a Jazz Admin you have two solutions

There are a few options to resolve this problem, but only the first is considered a complete solution. Please ensure that you test whichever solution is preferred before implementing in your production environment. There is no problem with -Xnocompressedrefs from a technical point of view. If you wish to use "-Xcompressedrefs" please adopt solution number #2 as the IBM Java team concluded that there is unlikely to be any code fixes to address this use case.

Solution 1

The recommended workaround is to use -Xnocompressedrefs so as not to be restricted by the 4GB memory space that is ultimately causing this out of native memory use case.

1. Switch -xcompressedrefs to -xnocompressedrefs. The IBM Java team recommend that when you use -xnocompressedrefs you should be doubling current Java Virtual Machine(JVM) heap allocation. The reasons is that when you switch to -Xnocompressedrefs option, it disables compressed references, therefore we need to provide enough JVM memory. Please note, only 64-bit JVMs recognize these options. You can read about this setting here: IBM Java Knowledge Center's compressed references page.

JVM heap allocation sample commands

From:

-Xmx8g

-Xms8g

-Xmn2g

-Xcompressedrefs

To:

-Xmx16g <<=== double JVM maximum

-Xms16g <<=== double JVM minimum

-Xmn4g <<=== double JVM nursery

-Xnocompressedrefs <<==== removes compressedrefs to nocompressedrefs

These changes require a an application server restart.

Please note: You need to ensure there is enough physical RAM available on the system to allow 16 GB to be used as the JVM memory. For example to use 16 GB of JVM memory, you need at least 32 GB of RAM available on the machine.
Also note that the above example is for RTC/Rational Quality Manager(RQM) where 25% of the heap allocation is recommended for the nursery setting (Xmn). Rational DOORS Next Generation(RDNG) advocate setting the nursery to 1/3 of the overall heap space.

Solution 2

2. After lengthy discussions with Java team, there is an undocumented option in the JVM which allows memory under 4GB to be reserved for later use. It is likely that a javacore would show that 500MB of low memory is being used when an Native Out of Memory Exceptopion occurs, we suggest that we start with a value of double that (1GB) to see if that fixes the problem. If NOOMEs do occur with this setting, then the value can be revised upwards.

The procedure to implement this would be to add -Xgc:suballocatorInitialSize=1g to your initial JVM settings. If you're selecting this option, it is assumed that you have insufficient RAM available to implement solution 1. It is not expected that you would implement these two solutions simultaneously, as we would need to revert back to the Xcompressedrefs settings as part of this implementation.

Please see a sample command over here.

From:

-Xmx8g

-Xms8g

-Xmn2g

-Xcompressedrefs

To:

-Xmx8g

-Xms8g

-Xmn2g

-Xcompressedrefs

-Xgc:suballocatorInitialSize=1g <<=== JVM which allows memory under 4GB increased to 1GB

These changes require a server restart.

Other Solutions

If you're still reading then it is likely that you are restricted by the available RAM on your system. Using -Xgc:preferredHeapBase with -Xcompressedrefs is a very useful article from the Java team offering more guidance of their options. I will reiterate therefore that solution 1 is our recommendation, but in an interim period where resources are due to be increased (RAM) then you could consider (if you were to look into other java options) the following:
a) -Xmcrs, - forces the JRE to set aside a portion of the 0-4GB address space for monitors, threads and class memory allocation.

However, invariably this just delays the NOOME issue. A server that is prone to NOOME would eventually hit the same issue with this setting whereas, with the nocompressedrefs option, the server would never hit the NOOME issue.
The other question is *iHow would you determine this number when CLM systems can range from all-in-one to single app systems?*i
If you guess low, you hit the NOOME, if you guess too high -- you take away memory that could be utilized. It is difficult to determine a default value when our different customers have such a wide range of hardware: 2-32 vcpus, or 4- xxxx GB of RAM...

Troubleshooting section (verbose logs)

Once Xnocompressedrefs is used you should monitor JVM activity for a week or two. Also note that this information is merely an example. It is important to note that any sizing recommendation really is dependent upon monitoring to confirm whether there is sufficient allocations within your environment to succeed. It maybe that 100% more heap isn't required and the overall sizing is smaller. However monitoring your way to good health and allowing the data to dictate is the only sensible way to proceed.
You should do that by collecting verbose:gc where the application server is running. I've copied a sample verbose report from a real time application server usage, as you can see during normal business operation, there are couple of instances 10 to 15 seconds of pause. Mostly done here as pauses in the 10-15 seconds range once or twice a day are probably within acceptable range.

Longest Garbage Collections 15,619 ms (Mon Feb 9 09:46:48 2015)

11,820 ms (Mon Feb 9 10:55:45 2015)

10,820 ms (Tues Feb 10 17:07:14 2015)

10,502 ms (Wed Feb 11 11:08:43 2015)

9,001 ms ( Feb 13 09:46:29 2015)

Need more helps? Open PMR

Should you have any issues or concerns, open a PMR then we can help you.

IBM Service Request links:

IBM Service Request: https://www.ibm.com/support/servicerequest/

IBM Service Request user information: https://www-946.ibm.com/sr/help/

IBM Service Request Helpdesk: https://www-946.ibm.com/sr/help/sr_helpdesk.html

Related topics: Deployment web home, Deployment web home

External links:
http://www-01.ibm.com/support/knowledgecenter/SSYKE2_6.0.0/com.ibm.java.doc.60_26/vm626/J9/VM/xcompressedrefs.html

Additional contributors: KeithWells, MariusLuca

 
This site is powered by the TWiki collaboration platformCopyright © by IBM and non-IBM contributing authors. All material on this collaboration platform is the property of the contributing authors.
Contributions are governed by our Terms of Use. Please read the following disclaimer.
Dashboards and work items are no longer publicly available, so some links may be invalid. We now provide similar information through other means. Learn more here.