Problems with running tomcat jazz server on Linux Red Hat
Hi all,
we are facing some problems with jazz server which are specific for linux environment. When installing clm solution we followed special instructions for Linux installation (as mentioned here - http://publib.boulder.ibm.com/infocenter/clmhelp/v3r0m1/topic/com.ibm.jazz.install.doc/topics/c_special_considerations_linux.html).
Environmet:
OS - Red Hat Enterprise Linux 5 Update 4 x86-64
DB - DB2 Enterprise Server 9.7 64-bit
Application server - Tomcat 5.5.33
CLM - RTC, RQM, RRC 3.0.1.2
We set in /etc/security/limits.conf these 2 lines:
* hard nofile 500000
* soft nofile 500000
We set fs.file-max to 500000 as well.
So we run the server as root (no other accounts except db2 accounts are there). When I type in the terminal "cat /proc/sys/fs/file-max" or "ulimit -n" I got in both cases value 500000.
But the server falls down usually once per week with an error "too many open files". We don't know how to figure it out, I have read many articles and still don't know what is the cause.
There are some examples from logs:
TOMCAT LOGS:
!ENTRY org.eclipse.core.resources 4 0 2012-04-19 07:32:52.358
!MESSAGE
!STACK 0
java.util.zip.ZipException: Too many open files /opt/IBM/JazzTeamServer/server/tomcat/work/Catalina/localhost/jazz/eclipse/configuration/org.eclipse.osgi/bundles/281/1/bundlefile
at java.util.zip.ZipFile.open(Native Method)
at java.util.zip.ZipFile.<init>(ZipFile.java:241)
...
SEVERE: Endpoint ] ignored exception: java.net.SocketException: Too many open files
java.net.SocketException: Too many open files
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:457)
at java.net.ServerSocket.implAccept(ServerSocket.java:473)
...
JTS LOGS:
2012-04-19 08:28:04,715 WARN m.team.repository.service.internal.LogFeedAppender - CRJAZ0994W Failed to write log event to disk.
java.io.IOException: Too many open files
at java.io.File.checkAndCreate(File.java:1380)
at java.io.File.createTempFile(File.java:1468)
...
java.io.FileNotFoundException: /opt/IBM/JazzTeamServer/server/tomcat/work/Catalina/localhost/jts/eclipse/plugins/org.dojotoolkit.dojo_1.4.3.v20110310_0124/resources/resources/blank.gif (Too many open files)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.<init>(FileInputStream.java:135)
at org.eclipse.osgi.framework.util.SecureAction.getFileInputStream(SecureAction.java:124)
Thanks for any help!
Matous
we are facing some problems with jazz server which are specific for linux environment. When installing clm solution we followed special instructions for Linux installation (as mentioned here - http://publib.boulder.ibm.com/infocenter/clmhelp/v3r0m1/topic/com.ibm.jazz.install.doc/topics/c_special_considerations_linux.html).
Environmet:
OS - Red Hat Enterprise Linux 5 Update 4 x86-64
DB - DB2 Enterprise Server 9.7 64-bit
Application server - Tomcat 5.5.33
CLM - RTC, RQM, RRC 3.0.1.2
We set in /etc/security/limits.conf these 2 lines:
* hard nofile 500000
* soft nofile 500000
We set fs.file-max to 500000 as well.
So we run the server as root (no other accounts except db2 accounts are there). When I type in the terminal "cat /proc/sys/fs/file-max" or "ulimit -n" I got in both cases value 500000.
But the server falls down usually once per week with an error "too many open files". We don't know how to figure it out, I have read many articles and still don't know what is the cause.
There are some examples from logs:
!ENTRY org.eclipse.core.resources 4 0 2012-04-19 07:32:52.358
!MESSAGE
!STACK 0
java.util.zip.ZipException: Too many open files /opt/IBM/JazzTeamServer/server/tomcat/work/Catalina/localhost/jazz/eclipse/configuration/org.eclipse.osgi/bundles/281/1/bundlefile
at java.util.zip.ZipFile.open(Native Method)
at java.util.zip.ZipFile.<init>(ZipFile.java:241)
...
SEVERE: Endpoint ] ignored exception: java.net.SocketException: Too many open files
java.net.SocketException: Too many open files
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:457)
at java.net.ServerSocket.implAccept(ServerSocket.java:473)
...
2012-04-19 08:28:04,715 WARN m.team.repository.service.internal.LogFeedAppender - CRJAZ0994W Failed to write log event to disk.
java.io.IOException: Too many open files
at java.io.File.checkAndCreate(File.java:1380)
at java.io.File.createTempFile(File.java:1468)
...
java.io.FileNotFoundException: /opt/IBM/JazzTeamServer/server/tomcat/work/Catalina/localhost/jts/eclipse/plugins/org.dojotoolkit.dojo_1.4.3.v20110310_0124/resources/resources/blank.gif (Too many open files)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.<init>(FileInputStream.java:135)
at org.eclipse.osgi.framework.util.SecureAction.getFileInputStream(SecureAction.java:124)
Thanks for any help!
Matous
Accepted answer
2 other answers
Matous,
Thanks for sharing your learnings,
We are also hitting this , can you elaborate on where you added this? Not exactly sure it matters, but why not do what worked for you???
to the $CATALINA_HOME/bin/startup.sh,
or the $JAZZ_TEAM_SERVER_HOME/server/server.startup
or even perhaps the /etc/init.d/rtc ???
Hey Dave,
sorry for my late reply.
You can put it to startup.sh as well as /etc/init.d/rtc and it should work (im not sure about startup.sh but guess it should also work). If I remember right, the problem in our case was caused by the fact, that changes in fs.file-max and /etc/security/limits.conf didn't affect our personalized starting script in init.d (if we checked limits of running processes, the value for rtc was still 1024). So we had to add "ulimit -n 65535" directly to our starting script (I guess scripts in init.d are run under default settings even if your system settings got changed).
sorry for my late reply.
You can put it to startup.sh as well as /etc/init.d/rtc and it should work (im not sure about startup.sh but guess it should also work). If I remember right, the problem in our case was caused by the fact, that changes in fs.file-max and /etc/security/limits.conf didn't affect our personalized starting script in init.d (if we checked limits of running processes, the value for rtc was still 1024). So we had to add "ulimit -n 65535" directly to our starting script (I guess scripts in init.d are run under default settings even if your system settings got changed).