Out of memory error - RTCz 2.0.0.2 IFIX01 + WAS 7

Hi, we're running RTC for z 2.0.0.2 IFIX01 on a WAS 7.

We have problems assigning the correct memory heap size. Our server is currently just a test server (32-bit Windows 2003) so I guess that our maximum memory for the JVM is 1500MB.

Our max heap setting is now 1024MB so that leaves 500 MB for the native heap. We are now getting the error below after running some time (10 days). A reboot temporarily fixes the problem.

If we increase the heap size (to say 1200 MB or so) then we start running out of native heap, and get some other errors.

Anybody have comments on this?

Thanks:

2011-01-13:

RTC has been running for: 10 days, 2 hours, 35 minutes
and is using memory:
Memory Usage
Maximum Memory Allocation 1024 MB
Current Memory Allocation 964 MB
Free Memory (of Current) 1 %

*********
Symptoms:

Accessing "work items" in web gives the result:

Error!

The server is not responding or a page navigation occurred before an operation completed. Check that your server connection is working and that the server is operational, then try the action again or refresh the page.


**********
When logging into WAS admin, then left navigation menu is displaying:

Error Page Exception
SRVE0260E: The server cannot use the error page specified for your application to handle the Original Exception printed below.

Original Exception:
Error Message: java.lang.OutOfMemoryError
Error Code: 500
Target Servlet: null
Error Stack:
java.lang.OutOfMemoryError
at java.io.FileInputStream.readBytes(Native Method)
at java.io.FileInputStream.read(FileInputStream.java:213)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:268)
at java.io.BufferedInputStream.read(BufferedInputStream.java:329)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:230)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:270)
at java.io.BufferedInputStream.read(BufferedInputStream.java:329)
at sun.nio.cs.StreamDecoder$CharsetSD.readBytes(StreamDecoder.java:452)
at sun.nio.cs.StreamDecoder$CharsetSD.implRead(StreamDecoder.java:494)
at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:222)
at java.io.InputStreamReader.read(InputStreamReader.java:177)
at java.io.Reader.read(Reader.java:135)
at java.util.Properties.load(Properties.java:314)
at java.util.Properties.load(Properties.java:280)
at org.apache.struts.util.PropertyMessageResources.loadLocale(Unknown Source)
at org.apache.struts.util.PropertyMessageResources.getMessage(Unknown Source)
at org.apache.struts.util.RequestUtils.message(Unknown Source)
at org.apache.struts.util.RequestUtils.message(Unknown Source)
at com.ibm.isclite.common.util.TitleUtil.getTitle(TitleUtil.java:248)
at com.ibm.isclite.runtime.navigation.util.NavigationUtil.getTitle(NavigationUtil.java:84)
at com.ibm.isclite.datastore.runtime.NavigationNode.getStringTitle(NavigationNode.java:187)
at com.ibm.isclite.service.datastore.navigation.NavigationServiceImpl.processNode(NavigationServiceImpl.java:188)
at com.ibm.isclite.service.datastore.navigation.NavigationServiceImpl.processNode(NavigationServiceImpl.java:397)
at com.ibm.isclite.service.datastore.navigation.NavigationServiceImpl.buildTree(NavigationServiceImpl.java:483)
at _ibmjsp.secure.isclite.tiles._iscnavigatorLayout._jspService(_iscnavigatorLayout.java:672)
at com.ibm.ws.jsp.runtime.HttpJspBase.service(HttpJspBase.java:98)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:831)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1583)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1523)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:104)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain._doFilter(WebAppFilterChain.java:77)
at com.ibm.ws.webcontainer.filter.WebAppFilterManager.doFilter(WebAppFilterManager.java:885)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:865)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:475)
at com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:175)
at com.ibm.wsspi.webcontainer.servlet.GenericServletWrapper.handleRequest(GenericServletWrapper.java:121)
at com.ibm.ws.jsp.webcontainerext.AbstractJSPExtensionServletWrapper.handleRequest(AbstractJSPExtensionServletWrapper.java:239)
at com.ibm.ws.jsp.webcontainerext.AbstractJSPExtensionProcessor.handleRequest(AbstractJSPExtensionProcessor.java:340)
at com.ibm.ws.webcontainer.webapp.WebAppRequestDispatcher.include(WebAppRequestDispatcher.java:673)
at org.apache.jasper.runtime.JspRuntimeLibrary.include(JspRuntimeLibrary.java:1045)
at org.apache.jasper.runtime.PageContextImpl.include(PageContextImpl.java:527)
at org.apache.struts.tiles.TilesUtilImpl.doInclude(Unknown Source)
at org.apache.struts.tiles.TilesUtil.doInclude(Unknown Source)
at org.apache.struts.taglib.tiles.InsertTag.doInclude(Unknown Source)
at org.apache.struts.taglib.tiles.InsertTag$InsertHandler.doEndTag(Unknown Source)
at org.apache.struts.taglib.tiles.InsertTag.doEndTag(Unknown Source)
at _ibmjsp.secure.isclite.tiles._master._jspx_meth_tiles_insert_2(_master.java:313)
at _ibmjsp.secure.isclite.tiles._master._jspService(_master.java:201)
at com.ibm.ws.jsp.runtime.HttpJspBase.service(HttpJspBase.java:98)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:831)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1583)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1523)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:104)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain._doFilter(WebAppFilterChain.java:77)
at com.ibm.ws.webcontainer.filter.WebAppFilterManager.doFilter(WebAppFilterManager.java:885)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:865)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:475)
at com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:175)
at com.ibm.wsspi.webcontainer.servlet.GenericServletWrapper.handleRequest(GenericServletWrapper.java:121)
at com.ibm.ws.jsp.webcontainerext.AbstractJSPExtensionServletWrapper.handleRequest(AbstractJSPExtensionServletWrapper.java:239)
at com.ibm.ws.webcontainer.webapp.WebAppRequestDispatcher.forward(WebAppRequestDispatcher.java:341)
at org.apache.struts.action.RequestProcessor.doForward(Unknown Source)
at org.apache.struts.tiles.TilesRequestProcessor.doForward(Unknown Source)
at org.apache.struts.tiles.TilesRequestProcessor.processTilesDefinition(Unknown Source)
at org.apache.struts.tiles.TilesRequestProcessor.processForwardConfig(Unknown Source)
at com.ibm.isclite.container.controller.InformationController.processForwardConfig(InformationController.java:217)
at org.apache.struts.action.RequestProcessor.process(Unknown Source)
at org.apache.struts.action.ActionServlet.process(Unknown Source)
at org.apache.struts.action.ActionServlet.doGet(Unknown Source)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:718)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:831)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1583)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1523)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:131)
at com.ibm.ws.console.core.servlet.WSCUrlFilter.setUpCommandAssistence(WSCUrlFilter.java:927)
at com.ibm.ws.console.core.servlet.WSCUrlFilter.continueStoringTaskState(WSCUrlFilter.java:494)
at com.ibm.ws.console.core.servlet.WSCUrlFilter.doFilter(WSCUrlFilter.java:315)
at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:188)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:116)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain._doFilter(WebAppFilterChain.java:77)
at com.ibm.ws.webcontainer.filter.WebAppFilterManager.doFilter(WebAppFilterManager.java:885)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:865)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:475)
at com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:175)
at com.ibm.ws.webcontainer.webapp.WebApp.handleRequest(WebApp.java:3799)
at com.ibm.ws.webcontainer.webapp.WebGroup.handleRequest(WebGroup.java:276)
at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:930)
at com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1583)
at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:182)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:455)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewInformation(HttpInboundLink.java:384)
at com.ibm.ws.http.channel.inbound.impl.HttpICLReadCallback.complete(HttpICLReadCallback.java:83)
at com.ibm.ws.ssl.channel.impl.SSLReadServiceContext$SSLReadCompletedCallback.complete(SSLReadServiceContext.java:1772)
at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:165)
at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)
at com.ibm.io.async.AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161)
at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:138)
at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:204)
at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:775)
at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:905)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1550)



Error Page Exception:
Error Message: javax.servlet.jsp.JspException: Missing message for key "errorpage.title"
Error Code: 0
Target Servlet: null
Error Stack:
javax.servlet.jsp.JspException: Missing message for key "errorpage.title"
at org.apache.struts.taglib.bean.MessageTag.doStartTag(Unknown Source)
at _ibmjsp._error._jspx_meth_bean_message_0(_error.java:315)
at _ibmjsp._error._jspx_meth_html_html_0(_error.java:662)
at _ibmjsp._error._jspService(_error.java:235)
at com.ibm.ws.jsp.runtime.HttpJspBase.service(HttpJspBase.java:98)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:831)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1583)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1523)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:104)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain._doFilter(WebAppFilterChain.java:77)
at com.ibm.ws.webcontainer.filter.WebAppFilterManager.doFilter(WebAppFilterManager.java:885)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:865)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:475)
at com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:175)
at com.ibm.wsspi.webcontainer.servlet.GenericServletWrapper.handleRequest(GenericServletWrapper.java:121)
at com.ibm.ws.jsp.webcontainerext.AbstractJSPExtensionServletWrapper.handleRequest(AbstractJSPExtensionServletWrapper.java:239)
at com.ibm.ws.jsp.webcontainerext.AbstractJSPExtensionProcessor.handleRequest(AbstractJSPExtensionProcessor.java:340)
at com.ibm.ws.webcontainer.webapp.WebAppRequestDispatcher.include(WebAppRequestDispatcher.java:673)
at com.ibm.ws.webcontainer.webapp.WebApp.sendError(WebApp.java:3244)
at com.ibm.ws.webcontainer.webapp.WebAppRequestDispatcher.include(WebAppRequestDispatcher.java:681)
at org.apache.jasper.runtime.JspRuntimeLibrary.include(JspRuntimeLibrary.java:1045)
at org.apache.jasper.runtime.PageContextImpl.include(PageContextImpl.java:527)
at org.apache.struts.tiles.TilesUtilImpl.doInclude(Unknown Source)
at org.apache.struts.tiles.TilesUtil.doInclude(Unknown Source)
at org.apache.struts.taglib.tiles.InsertTag.doInclude(Unknown Source)
at org.apache.struts.taglib.tiles.InsertTag$InsertHandler.doEndTag(Unknown Source)
at org.apache.struts.taglib.tiles.InsertTag.doEndTag(Unknown Source)
at _ibmjsp.secure.isclite.tiles._master._jspx_meth_tiles_insert_2(_master.java:313)
at _ibmjsp.secure.isclite.tiles._master._jspService(_master.java:201)
at com.ibm.ws.jsp.runtime.HttpJspBase.service(HttpJspBase.java:98)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:831)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1583)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1523)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:104)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain._doFilter(WebAppFilterChain.java:77)
at com.ibm.ws.webcontainer.filter.WebAppFilterManager.doFilter(WebAppFilterManager.java:885)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:865)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:475)
at com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:175)
at com.ibm.wsspi.webcontainer.servlet.GenericServletWrapper.handleRequest(GenericServletWrapper.java:121)
at com.ibm.ws.jsp.webcontainerext.AbstractJSPExtensionServletWrapper.handleRequest(AbstractJSPExtensionServletWrapper.java:239)
at com.ibm.ws.webcontainer.webapp.WebAppRequestDispatcher.forward(WebAppRequestDispatcher.java:341)
at org.apache.struts.action.RequestProcessor.doForward(Unknown Source)
at org.apache.struts.tiles.TilesRequestProcessor.doForward(Unknown Source)
at org.apache.struts.tiles.TilesRequestProcessor.processTilesDefinition(Unknown Source)
at org.apache.struts.tiles.TilesRequestProcessor.processForwardConfig(Unknown Source)
at com.ibm.isclite.container.controller.InformationController.processForwardConfig(InformationController.java:217)
at org.apache.struts.action.RequestProcessor.process(Unknown Source)
at org.apache.struts.action.ActionServlet.process(Unknown Source)
at org.apache.struts.action.ActionServlet.doGet(Unknown Source)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:718)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:831)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1583)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1523)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:131)
at com.ibm.ws.console.core.servlet.WSCUrlFilter.setUpCommandAssistence(WSCUrlFilter.java:927)
at com.ibm.ws.console.core.servlet.WSCUrlFilter.continueStoringTaskState(WSCUrlFilter.java:494)
at com.ibm.ws.console.core.servlet.WSCUrlFilter.doFilter(WSCUrlFilter.java:315)
at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:188)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:116)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain._doFilter(WebAppFilterChain.java:77)
at com.ibm.ws.webcontainer.filter.WebAppFilterManager.doFilter(WebAppFilterManager.java:885)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:865)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:475)
at com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:175)
at com.ibm.ws.webcontainer.webapp.WebApp.handleRequest(WebApp.java:3799)
at com.ibm.ws.webcontainer.webapp.WebGroup.handleRequest(WebGroup.java:276)
at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:930)
at com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1583)
at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:182)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:455)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewInformation(HttpInboundLink.java:384)
at com.ibm.ws.http.channel.inbound.impl.HttpICLReadCallback.complete(HttpICLReadCallback.java:83)
at com.ibm.ws.ssl.channel.impl.SSLReadServiceContext$SSLReadCompletedCallback.complete(SSLReadServiceContext.java:1772)
at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:165)
at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)
at com.ibm.io.async.AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161)
at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:138)
at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:204)
at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:775)
at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:905)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1550)

What version of WAS and JDK are you on?

I am seeing similar issues on a 32bit Windows 2003 machine with WAS 7.0.0.13 with the matching SDK, it looks like 7.0.0.9 is the current supported version so I was going to try that first. In the meantime I reverted to Tomcat until I can continue troubleshooting, since it was crashing every few hours for me with only a few users on the system.

-Sean