CLM 6.0.1 How to successfully start Tomcat 7 as a Windows Service
I recently upgraded a staging server to CLM 6.0.1. I am running under Windows Server 2012 and using the Tomcat web server. After the upgrade, I am able to start Tomcat and the CLM applications from the command line (using server.startup.bat). However, after following the instructions to start Tomcat as a Windows service, I am experiencing errors.
When I try to start the service from Windows Services, I receive the following error: "Windows could not start the Apache Tomcat 7.0 tomcat7 on Local Computer. For more information, review the System Event Log. If this is a non-Microsoft service, contact the service vendor, and refer to service-specific error code 4." When I look in the Windows Event Viewer (Event Viewier-->Windows Logs-->System), I can see the following error: "The Apache Tomcat 7.0 tomcat7 service terminated with the following service-specific error: The System cannot open the file" When I look in the Tomcat logs (tomcat7-stderr.2016-01-06.log), I see: "2016-01-06 09:12:56 Commons Daemon procrun stderr initialized JVMJ9VM015W Initialization error for library j9gc27(2): Failed to instantiate heap; 4G requested" I have set the Java options using tomcat7w.exe as outlined in the documentation: -Dcatalina.home=C:\Program Files\IBM\JazzTeamServer_601\server\tomcat -Dcatalina.base=C:\Program Files\IBM\JazzTeamServer_601\server\tomcat -Djava.endorsed.dirs=C:\Program Files\IBM\JazzTeamServer_601\server\tomcat\endorsed -Djava.io.tmpdir=C:\Program Files\IBM\JazzTeamServer_601\server\tomcat\temp -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.util.logging.config.file=C:\Program Files\IBM\JazzTeamServer_601\server\tomcat\conf\logging.properties -DJAZZ_HOME=file:///C:/PROGRA~1/IBM/JazzTeamServer_601/server/conf -Djava.awt.headless=true -Dorg.eclipse.emf.ecore.plugin.EcorePlugin.doNotLoadResourcesPlugin=true -Dcom.ibm.team.repository.tempDir=C:\Program Files\IBM\JazzTeamServer_601\server\tomcat\temp -Djazz.connector.sslEnabledProtocols="TLSv1,TLSv1.1,TLSv1.2" -Djazz.connector.algorithm=IbmX509 -Dlog4j.configuration=file:///C:/PROGRA~1/IBM/JazzTeamServer_601/server/conf/startup_log4j.properties -Xgcpolicy:gencon -Xcompressedrefs -Xgc:preferredHeapBase=0x100000000 -XX:MaxDirectMemorySize=1G -Xmx4G -Xms4G -Xmn1g -DORACLE_JDBC_DRIVER_FILE=C:\Program Files\IBM\JazzTeamServer_601\server\Oracle\ojdb6.jar I've cleared "Initial memory pool, Maximum memory pool, and Thread stack size". The Java Virtual machine is pointing to jvm.dll in JAZZ_HOME\server\jre\bin\j9vm and the classpath is set appropriately. When I go to JAZZ_Home\server\jre\bin and run java -version, I get: C:\PROGRA~1\IBM\JazzTeamServer_601\server\jre\bin>java -version java version "1.7.0" Java(TM) SE Runtime Environment (build pwa6470_27sr3fp10-20150708_01(SR3 FP10)) IBM J9 VM (build 2.7, JRE 1.7.0 Windows Server 2012 amd64-64 Compressed Referenc es 20150630_255653 (JIT enabled, AOT enabled) J9VM - R27_Java727_SR3_20150630_2236_B255653 JIT - tr.r13.java_20150623_94888.01 GC - R27_Java727_SR3_20150630_2236_B255653_CMPRSS J9CL - 20150630_255653) JCL - 20150628_01 based on Oracle jdk7u85-b15 so I am indeed running a 64 bit version of Java. My host has 10 G memory and the average memory in use is 1.3 GB. I really need to get Tomcat to start as a Windows Service. I have successfully configured Tomcat as a Windows service for CLM 4.0.6, 5.0.1, and 5.0.2. The only difference with 6.0.1 is that Tomcat doesn't come bundled. I had to install it separately but I installed the 64-bit version 7.0.59 and copied over the keystore etc. from my 5.0.2 when I ran the upgrade scripts. I have put in a PMR with IBM but there is still no resolution. Has anyone out there upgraded to CLM 6.0.1 and successfully configured Tomcat to run as a Windows Service? If so, perhaps you can send me your Java Options. There has to be some subtlety with the options since the error message indicates a problem allocating the Java Heap. Thank you |
7 answers
This is not intended as another answer, but rather a summary/clarification on all the answers provided in this post. I hope this can help others to pick the right option.
Based on Becky's answer, to successfully configured Tomcat 7 as a Windows service on Windows Server 2012 for CLM, we need to address three issues - JVM heap base, SSL and APR, and SSL protocols. 1. For the JVM heap base issue, I believe the best option is to remove the -Xgc:preferredHeapBase JVM option completely. I believe the whole reason that IBM suggested this option in the past is to avoid NOOM in the 0-4GB region. Nowadays, the heap size is usually set to greater than 4GB and the heap will no longer be allocated in the 0-4GB region, so this JVM option can be left out. In fact, this option may bring negative effect when not used carefully, such as in this case. http://www-01.ibm.com/support/docview.wss?uid=swg21660890 http://www-01.ibm.com/support/docview.wss?uid=swg1IV37797 https://jazz.net/forum/questions/146018/unable-to-start-jazz-405-as-a-service-on-windows-server-2012 Windows Server 2012 seems to introduce some new memory management that we may not be familiar with, so some suggested settings cannot be brought over without further investigation. As an example, when I remove this option and start Tomcat 7 as Windows service on a Windows Server 2012 machine with only 8GB memory, the heap base is 0x6FFFF0000 (30GB, not a typo) - I really can't explain what I see. 2. SSL and APR. The tomcat settings as suggested by IBM, in particular, the use of a keystore, do not work with APR. We have seen the "APR not found" message in the Tomcat log files too many times, and no one cares about it, not realizing that this is actually the "requirement" for the SSL connection to work. As we have found out, APR requires the parameter SSLCertificateFile. So if you want to use APR and can generate your own SSL Certificate, it should work. But with the provided IBM keystore, APR has to be disabled. You can point Tomcat to an incorrect location so that it cannot find the APR library (as Becky and Kot did), or remove the APR library (tcnative-1.dll) completely (as Hara san did). I believe the latter is the better option. https://tomcat.apache.org/tomcat-7.0-doc/apr.html 3. SSL protocols. We can use either "sslProtocols" or "sslEnabledProtocols" in Tomcat configuration. The switch from "sslProtocols" to "sslEnabledProtocols" is not because of Tomcat 7, but rather we need to disable SSLv3. If you upgrade CLM from an older version (which still allows SSLv3), you may get caught on this one. But if you simply copy the JVM options from the server.startup script, you should notice the difference. http://www-01.ibm.com/support/docview.wss?uid=swg21687762 |
To resolve the issue I performed below steps
1) Set the Java Options to below values -Dcatalina.home=C:\IBM\JazzTeamServer6\server\tomcat -Dcatalina.base=C:\IBM\JazzTeamServer6\server\tomcat -Djava.endorsed.dirs=C:\IBM\JazzTeamServer6\server\tomcat\endorsed -Djava.io.tmpdir=C:\IBM\JazzTeamServer6\server\tomcat\temp -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.util.logging.config.file=C:\IBM\JazzTeamServer6\server\tomcat\conf\logging.properties -DJAZZ_HOME=file:///C:/IBM/JazzTeamServer6/server/conf -Djava.awt.headless=true -Dorg.eclipse.emf.ecore.plugin.EcorePlugin.doNotLoadResourcesPlugin=true -Dcom.ibm.team.repository.tempDir=C:\IBM\JazzTeamServer6\server\tomcat\temp -Djazz.connector.sslEnabledProtocols="TLSv1.0,TLSv1.1,TLSv1.2" -Djazz.connector.algorithm=IbmX509 -Dlog4j.configuration=file:///C:/IBM/JazzTeamServer6/server/conf/startup_log4j.properties -Xgcpolicy:gencon -Xcompressedrefs -XX:MaxDirectMemorySize=1G -Xmx4G -Xms4G 2) Modify tomcat server.xml (JazzTeamServer6\server\tomcat\conf\server.xml) a) Comment below line in server.xml <Listener SSLEngine="on" className="org.apache.catalina.core.AprLifecycleListener"/> b) Under section "Define a SSL HTTP/1.1 Connector on port 443" modify sslEnabledProtocols and set it to sslEnabledProtocols="TLSv1,TLSv1.1,TLSv1.2" The whole section will look like below: <Connector SSLEnabled="true" URIEncoding="UTF-8" acceptCount="100" algorithm="${jazz.connector.algorithm}" ciphers="SSL_RSA_WITH_RC4_128_MD5, SSL_RSA_WITH_RC4_128_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_CBC_SHA, SSL_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA" clientAuth="false" connectionTimeout="20000" disableUploadTimeout="true" enableLookups="false" keystoreFile="ibm-team-ssl.keystore" keystorePass="ibm-team" maxHttpHeaderSize="8192" maxThreads="150" minSpareThreads="25" port="443" protocol="HTTP/1.1" scheme="https" secure="true" sslEnabledProtocols="TLSv1,TLSv1.1,TLSv1.2"/> |
|
I was successfully able to fix the problem and start Tomcat as a Windows service. The fix involved changing some of the Java Options when configuring the service using tomcat7w.exe.
Specifically, I needed to: 1) Remove the option: -Xgc:preferredHeapBase=0x100000000 2) Add the option: -Djava.library.path=C:\PROGRA~1\IBM\JazzTeamServer_601\server\tomcat\lib 3) Modify the option: -Djazz.connector.sslEnabledProtocols * Change FROM: -Djazz.connector.sslEnabledProtocols=TLSv1.2 * Change TO: -Djazz.connector.sslEnabledProtocols="TLSv1,TLSv1.1,TLSv1.2" With these 3 changes, the service successfully started and I was able to navigate to the CLM applications from a web browser. Here are all of the Java Options: -Dcatalina.home=C:\PROGRA~1\IBM\JazzTeamServer_601\server\tomcat -Dcatalina.base=C:\PROGRA~1\IBM\JazzTeamServer_601\server\tomcat -Djava.io.tmpdir=C:\PROGRA~1\IBM\JazzTeamServer_601\server\tomcat\temp -Djava.library.path=C:\PROGRA~1\IBM\JazzTeamServer_601\server\tomcat\lib -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.util.logging.config.file=C:\PROGRA~1\IBM\JazzTeamServer_601\server\tomcat\conf\logging.properties -DJAZZ_HOME=file:///C:/PROGRA~1/IBM/JazzTeamServer_601/server/conf -Djava.awt.headless=true -Dorg.eclipse.emf.ecore.plugin.EcorePlugin.doNotLoadResourcesPlugin=true -Dcom.ibm.team.repository.tempDir=C:\PROGRA~1\IBM\JazzTeamServer_601\server\tomcat\temp -Djazz.connector.sslEnabledProtocols="TLSv1,TLSv1.1,TLSv1.2" -Djazz.connector.algorithm=IbmX509 -Dlog4j.configuration=file:///C:/PROGRA~1/IBM/JazzTeamServer_601/server/conf/startup_log4j.properties -Xgcpolicy:gencon -Xcompressedrefs -XX:MaxDirectMemorySize=1G -Xmx4G -Xms4G -Xmn1g -DORACLE_JDBC_DRIVER_FILE=C:\PROGRA~1\IBM\JazzTeamServer_601\oracle\ojdbc6.jar |
Tomcat version used is 'apache-tomcat-7.0.67-windows-x64.zip'. CLM starts up correctly and is accessible when using server.startup.bat.
When trying to configure it to run as Windows Service,, I see the same error mentioned in this post in tomcat7-stderr.log. I got passed the error by removing the following line from the java options. -Xgc:preferredHeapBase=0x100000000 This allows the Tomcat Service to startup; however, when I access the CLM application, it keeps loading... Found the following error in catalina.out: Jan 12, 2016 4:17:17 PM org.apache.catalina.core.AprLifecycleListener lifecycleEvent INFO: Loaded APR based Apache Tomcat Native library 1.1.33 using APR version 1.5.1. Jan 12, 2016 4:17:17 PM org.apache.catalina.core.AprLifecycleListener lifecycleEvent INFO: APR capabilities: IPv6 [true], sendfile [true], accept filters [false], random [true]. Jan 12, 2016 4:17:18 PM org.apache.catalina.core.AprLifecycleListener initializeSSL INFO: OpenSSL successfully initialized (OpenSSL 1.0.1m 19 Mar 2015) Jan 12, 2016 4:17:18 PM org.apache.coyote.AbstractProtocol init INFO: Initializing ProtocolHandler ["http-apr-9080"] Jan 12, 2016 4:17:18 PM org.apache.coyote.AbstractProtocol init INFO: Initializing ProtocolHandler ["http-apr-9443"] Jan 12, 2016 4:17:18 PM org.apache.coyote.AbstractProtocol init SEVERE: Failed to initialize end point associated with ProtocolHandler ["http-apr-9443"] java.lang.Exception: Connector attribute SSLCertificateFile must be defined when using SSL with APR ... Somehow, it can find the library for APR. This isn't the case when using server.startup.bat. Jun 22, 2015 9:58:12 AM org.apache.catalina.core.AprLifecycleListener lifecycleEvent INFO: The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: <some paths> Jun 22, 2015 9:58:13 AM org.apache.coyote.AbstractProtocol init INFO: Initializing ProtocolHandler ["http-bio-9080"] Jun 22, 2015 9:58:13 AM org.apache.coyote.AbstractProtocol init INFO: Initializing ProtocolHandler ["http-bio-9443"] I could not find where java.library.path is defined in the server.startup.bat, but I assume it takes the value from %PATH% system variable. In order to mimic the old behavior, I added the following java option in tomcat7w: -Djava.library.path=<some path> In my case, I use the path that does not exist ==> -Djava.library.path=/usr/tomcat/shared/lib Start the server and confirm no SEVERE error in tomcat logs. I am able to access CLM after the above change. Comments I tried above steps on Windows server but got below error in localhost.log file and was not able to access CLM
The handshake_failure is likely caused by the following option:
|
Hi.
Thank you for you patience with my English. Although I also had same problem I have resolved below two steps. 1) Change preferredHeapBase to 0x200000000. I referred this page
https://jazz.net/wiki/bin/view/Main/TomcatAsAWindowsServiceScenario
-Xgc:preferredHeapBase=0x100000000 to -Xgc:preferredHeapBase=0x200000000. 2)delete tcnative-1.dll in JazzTeamServer601\server\tomcat\bin tcnative-1.dll is Apache Tomcat Native Library if you downloaded tomcat archive from tomcat site there is tcnative-1.dll in tomcat\bin. There isn't tcnative-1.dll In tomcat\bin CLM6.0 or CLM5.0.2 After above two step you can run tomcat as windows service. Comments I had the very same problem in event viewer I also found:
Acutally I also had to add Shradha Srivastav fix as the server started just fine but I could not access any webpages I think it had to do with the sslEnabledprotocols not being found when running as a service
Thank you, this solved our same problem.
Cheers.
|
Becky's resolution worked for me, the CLM 6.0.2 installation instructions didn't contain the path to the Java library: -Djava.library.path=D:\ibm\JazzTeamServer\server\tomcat\lib |
Your answer
Dashboards and work items are no longer publicly available, so some links may be invalid. We now provide similar information through other means. Learn more here.