Attempting To use RTC (Rational Team Concert) On Windows 2008 R2 Operating System Leads into NativeOutOfMemoryException (NOOME).

This article was originally scoped to explain a situation observed when using Rational Team Concert (RTC) on Microsoft Windows 2008 R2 Operating System Leads into NativeOutOfMemoryException (NOOME).
This article has been updated due to the same observation being made on Microsoft Windows operating systems in general, also including 2008 and 2012, during evaluation use of Collaborative Lifecycle Management (CLM) 6.x.
The most common situation where we have observed this situation is when the Lifecycle Query Engine (LQE) has been installed on a Microsoft Windows server alongside all of the other applications required for CLM to work in a global configuration environment. This issue is one of many discussed in the Proof of Concept Sizing page.

As a Jazz Admin you have two solutions to this problem.

There are a few options to resolve this problem, but only the first is a solution. Let us discuss one by one. Whatever is convenient to be implemented please test before going into production.

Solution 1

The recommended workaround is to use -Xnocompreessedrefs so as not to be restricted by the 4GB memory space that is ultimately causing this out of native memory use case.

1. Switch -compressedrefs to -xnocompressedrefs. The IBM Java team recommend that when you use -xnocompressedrefs you should be doubling current JVM heap allocation. The reasons is that when you switch to -Xnocompressedrefs option, it disables compressed references, therefore we need to provide enough JVM memory. Please note, only 64-bit JVMs recognize these options. You can read about this settings here: IBM Java Knowledge Center's compressed references page.

JVM heap allocation sample commands

From:

-Xmx8g

-Xms8g

-Xmn2g

-Xcompressedrefs

To:

-Xmx16g <<=== double JVM maximum

-Xms16g <<=== double JVM minimum

-Xmn4g <<=== double JVM nursery

-Xnocompressedrefs <<==== removes compressedrefs to nocompressedrefs

These changes require a an application server restart.

Please note You need to ensure there is enough physical RAM available on the system to allow 16 GB to be used as JVM memory. For example to use 16 GB of JVM memory, you need at least 32 GB of RAM available on the machine.

Solution 2

2. Let's go over option number 2. After lots of discussion with Java team, there is an undocumented option in the JVM which allows memory under 4GB to be reserved for later use. As a javacore shows that 500MB of low memory is being used when an NOOM (Native Out of Memory) occurs, we suggest that we start with a value of double that (1GB) to see if that fixes the problem. If NOOMs do occur with this setting, then the value can be revised upwards.

The procedure is revert all memory allocations (such as -Xmx) to half their current value. Remove -Xnocompressedrefs and replace with -Xcompressedrefs and then to add -Xgc:suballocatorInitialSize=1g

Please see a sample command over here.

From:

-Xmx8g

-Xms8g

-Xmn2g

-Xcompressedrefs

To:

-Xmx8g

-Xms8g

-Xmn2g

-Xcompressedrefs

-Xgc:suballocatorInitialSize=1g <<=== JVM which allows memory under 4GB increased to 1GB

These changes require a server restart.

Please note, from a technical point there is no problem with -Xnocompressedrefs. If wish to use "-Xcompressedrefs" please adapt solution number #2 as Java team concluded this would be way go as there won't any fixes from the code side.

Other Solutions

If you're still reading then it is likely that you are restricted by the available RAM on your system. I will reiterate therefore that solution 1 is our recommendation, but in an interim period where resources are due to be increased (RAM) then you could consider (if you were to look into other java options) the following:
a) -Xmcrs, - forces the JRE to set aside a portion of the 0-4GB address space for monitors, threads and class memory allocation.

However, invariably this just delays the NOOME issue. A server that is prone to NOOME would eventually hit the same issue with this setting whereas, with the nocompressedrefs option, the server would never hit the NOOME issue.
The other question is *iHow would you determine this number when CLM systems can range from all-in-one to single app systems?*i
If you guess low, you hit the NOOME, if you guess too high -- you take away memory that could be utilized. It is difficult to determine a default value when our different customers have such a wide range of hardware: 2-32 vcpus, or 4- xxxx GB of RAM...

Troubleshooting section (verbose logs)

Once Xnocompressedrefs is used you might want to monitor JVM activity for a week or two. Also note that this information is merely an example. It is important to note that any sizing recommendation really is dependent upon monitoring to confirm whether there is sufficient allocations within your environment to succeed. It maybe that 100% more heap isn't required and the overall sizing is smaller. However monitoring your way to good health and allowing the data to dictate is the only sensible way to proceed.
You should do that by collecting verbose:gc where the application server is running. I've copied a sample verbose report from a real time application server usage, as you can see during normal business operation, there are couple of instances 10 to 15 seconds of pause. Mostly done here as pauses in the 10-15 seconds range once or twice a day are probably within acceptable range.

Longest Garbage Collections 15,619 ms (Mon Feb 9 09:46:48 2015)

11,820 ms (Mon Feb 9 10:55:45 2015)

10,820 ms (Tues Feb 10 17:07:14 2015)

10,502 ms (Wed Feb 11 11:08:43 2015)

9,001 ms ( Feb 13 09:46:29 2015)

Need more helps? Open PMR

Should you have any issues or concerns, open a PMR then we can help you.

IBM Service Request links:

IBM Service Request: https://www.ibm.com/support/servicerequest/

IBM Service Request user information: https://www-946.ibm.com/sr/help/

IBM Service Request Helpdesk: https://www-946.ibm.com/sr/help/sr_helpdesk.html

Related topics: Deployment web home, Deployment web home

External links:

• IBM

Additional contributors: KeithWells, TWikiUser