It's all about the answers!

Ask a question

CLM 6.0.1 How to successfully start Tomcat 7 as a Windows Service


Becky McDermott (921140) | asked Jan 06 '16, 11:23 a.m.
I recently upgraded a staging server to CLM 6.0.1.  I am running under Windows Server 2012 and using the Tomcat web server.  After the upgrade, I am able to start Tomcat and the CLM applications from the command line (using server.startup.bat).  However, after following the instructions to start Tomcat as a Windows service, I am experiencing errors.

When I try to start the service from Windows Services, I receive the following error:
"Windows could not start the Apache Tomcat 7.0 tomcat7 on Local Computer.  For more information, review the System Event Log.  If this is a non-Microsoft service, contact the service vendor, and refer to service-specific error code 4."

When I look in the Windows Event Viewer (Event Viewier-->Windows Logs-->System), I can see the following error:
"The Apache Tomcat 7.0 tomcat7 service terminated with the following service-specific error:
The System cannot open the file"

When I look in the Tomcat logs (tomcat7-stderr.2016-01-06.log), I see:
"2016-01-06 09:12:56 Commons Daemon procrun stderr initialized
JVMJ9VM015W Initialization error for library j9gc27(2): Failed to instantiate heap; 4G requested"

I have set the Java options using tomcat7w.exe as outlined in the documentation:
-Dcatalina.home=C:\Program Files\IBM\JazzTeamServer_601\server\tomcat
-Dcatalina.base=C:\Program Files\IBM\JazzTeamServer_601\server\tomcat
-Djava.endorsed.dirs=C:\Program Files\IBM\JazzTeamServer_601\server\tomcat\endorsed
-Djava.io.tmpdir=C:\Program Files\IBM\JazzTeamServer_601\server\tomcat\temp
-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
-Djava.util.logging.config.file=C:\Program Files\IBM\JazzTeamServer_601\server\tomcat\conf\logging.properties
-DJAZZ_HOME=file:///C:/PROGRA~1/IBM/JazzTeamServer_601/server/conf
-Djava.awt.headless=true
-Dorg.eclipse.emf.ecore.plugin.EcorePlugin.doNotLoadResourcesPlugin=true
-Dcom.ibm.team.repository.tempDir=C:\Program Files\IBM\JazzTeamServer_601\server\tomcat\temp
-Djazz.connector.sslEnabledProtocols="TLSv1,TLSv1.1,TLSv1.2"
-Djazz.connector.algorithm=IbmX509
-Dlog4j.configuration=file:///C:/PROGRA~1/IBM/JazzTeamServer_601/server/conf/startup_log4j.properties
-Xgcpolicy:gencon
-Xcompressedrefs
-Xgc:preferredHeapBase=0x100000000
-XX:MaxDirectMemorySize=1G
-Xmx4G
-Xms4G
-Xmn1g
-DORACLE_JDBC_DRIVER_FILE=C:\Program Files\IBM\JazzTeamServer_601\server\Oracle\ojdb6.jar

I've cleared "Initial memory pool, Maximum memory pool, and Thread stack size".  The Java Virtual machine is pointing to jvm.dll in JAZZ_HOME\server\jre\bin\j9vm and the classpath is set appropriately.

When I go to JAZZ_Home\server\jre\bin and run java -version, I get:
C:\PROGRA~1\IBM\JazzTeamServer_601\server\jre\bin>java -version
java version "1.7.0"
Java(TM) SE Runtime Environment (build pwa6470_27sr3fp10-20150708_01(SR3 FP10))
IBM J9 VM (build 2.7, JRE 1.7.0 Windows Server 2012 amd64-64 Compressed Referenc
es 20150630_255653 (JIT enabled, AOT enabled)
J9VM - R27_Java727_SR3_20150630_2236_B255653
JIT  - tr.r13.java_20150623_94888.01
GC   - R27_Java727_SR3_20150630_2236_B255653_CMPRSS
J9CL - 20150630_255653)
JCL - 20150628_01 based on Oracle jdk7u85-b15

so I am indeed running a 64 bit version of Java.

My host has 10 G memory and the average memory in use is 1.3 GB.

I really need to get Tomcat to start as a Windows Service.  I have successfully configured Tomcat as a Windows service for CLM 4.0.6, 5.0.1, and 5.0.2.  The only difference with 6.0.1 is that Tomcat doesn't come bundled.  I had to install it separately but I installed the 64-bit version 7.0.59 and copied over the keystore etc. from my 5.0.2 when I ran the upgrade scripts.

I have put in a PMR with IBM but there is still no resolution. 

Has anyone out there upgraded to CLM 6.0.1 and successfully configured Tomcat to run as a Windows Service?  If so, perhaps you can send me your Java Options.  There has to be some subtlety with the options since the error message indicates a problem allocating the Java Heap.

Thank you

7 answers



permanent link
Donald Nong (14.5k614) | answered Jan 14 '16, 11:04 p.m.
This is not intended as another answer, but rather a summary/clarification on all the answers provided in this post. I hope this can help others to pick the right option.

Based on Becky's answer, to successfully configured Tomcat 7 as a Windows service on Windows Server 2012 for CLM, we need to address three issues - JVM heap base, SSL and APR, and SSL protocols.

1. For the JVM heap base issue, I believe the best option is to remove the -Xgc:preferredHeapBase JVM option completely. I believe the whole reason that IBM suggested this option in the past is to avoid NOOM in the 0-4GB region. Nowadays, the heap size is usually set to greater than 4GB and the heap will no longer be allocated in the 0-4GB region, so this JVM option can be left out. In fact, this option may bring negative effect when not used carefully, such as in this case.
http://www-01.ibm.com/support/docview.wss?uid=swg21660890
http://www-01.ibm.com/support/docview.wss?uid=swg1IV37797
https://jazz.net/forum/questions/146018/unable-to-start-jazz-405-as-a-service-on-windows-server-2012

Windows Server 2012 seems to introduce some new memory management that we may not be familiar with, so some suggested settings cannot be brought over without further investigation. As an example, when I remove this option and start Tomcat 7 as Windows service on a Windows Server 2012 machine with only 8GB memory, the heap base is 0x6FFFF0000 (30GB, not a typo) - I really can't explain what I see.

2. SSL and APR. The tomcat settings as suggested by IBM, in particular, the use of a keystore, do not work with APR. We have seen the "APR not found" message in the Tomcat log files too many times, and no one cares about it, not realizing that this is actually the "requirement" for the SSL connection to work. As we have found out, APR requires the parameter SSLCertificateFile. So if you want to use APR and can generate your own SSL Certificate, it should work. But with the provided IBM keystore, APR has to be disabled. You can point Tomcat to an incorrect location so that it cannot find the APR library (as Becky and Kot did), or remove the APR library (tcnative-1.dll) completely (as Hara san did). I believe the latter is the better option.
https://tomcat.apache.org/tomcat-7.0-doc/apr.html

3. SSL protocols. We can use either "sslProtocols" or "sslEnabledProtocols" in Tomcat configuration. The switch from "sslProtocols" to "sslEnabledProtocols" is not because of Tomcat 7, but rather we need to disable SSLv3. If you upgrade CLM from an older version (which still allows SSLv3), you may get caught on this one. But if you simply copy the JVM options from the server.startup script, you should notice the difference.
http://www-01.ibm.com/support/docview.wss?uid=swg21687762

permanent link
Shradha Srivastav (348413) | answered Jan 13 '16, 11:52 p.m.
edited Jan 13 '16, 11:58 p.m.
To resolve the issue I performed below steps

1) Set the Java Options to below values

-Dcatalina.home=C:\IBM\JazzTeamServer6\server\tomcat
-Dcatalina.base=C:\IBM\JazzTeamServer6\server\tomcat
-Djava.endorsed.dirs=C:\IBM\JazzTeamServer6\server\tomcat\endorsed
-Djava.io.tmpdir=C:\IBM\JazzTeamServer6\server\tomcat\temp
-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
-Djava.util.logging.config.file=C:\IBM\JazzTeamServer6\server\tomcat\conf\logging.properties
-DJAZZ_HOME=file:///C:/IBM/JazzTeamServer6/server/conf
-Djava.awt.headless=true
-Dorg.eclipse.emf.ecore.plugin.EcorePlugin.doNotLoadResourcesPlugin=true
-Dcom.ibm.team.repository.tempDir=C:\IBM\JazzTeamServer6\server\tomcat\temp
-Djazz.connector.sslEnabledProtocols="TLSv1.0,TLSv1.1,TLSv1.2"
-Djazz.connector.algorithm=IbmX509
-Dlog4j.configuration=file:///C:/IBM/JazzTeamServer6/server/conf/startup_log4j.properties
-Xgcpolicy:gencon
-Xcompressedrefs
-XX:MaxDirectMemorySize=1G
-Xmx4G
-Xms4G

2) Modify tomcat server.xml (JazzTeamServer6\server\tomcat\conf\server.xml)

a) Comment below line in server.xml
<Listener SSLEngine="on" className="org.apache.catalina.core.AprLifecycleListener"/>

b) Under section "Define a SSL HTTP/1.1 Connector on port 443" modify sslEnabledProtocols and set it to
sslEnabledProtocols="TLSv1,TLSv1.1,TLSv1.2"

The whole section will look like below:

<Connector SSLEnabled="true" URIEncoding="UTF-8" acceptCount="100" algorithm="${jazz.connector.algorithm}" ciphers="SSL_RSA_WITH_RC4_128_MD5, SSL_RSA_WITH_RC4_128_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_CBC_SHA, SSL_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA" clientAuth="false" connectionTimeout="20000" disableUploadTimeout="true" enableLookups="false" keystoreFile="ibm-team-ssl.keystore" keystorePass="ibm-team" maxHttpHeaderSize="8192" maxThreads="150" minSpareThreads="25" port="443" protocol="HTTP/1.1" scheme="https" secure="true" sslEnabledProtocols="TLSv1,TLSv1.1,TLSv1.2"/>

permanent link
Becky McDermott (921140) | answered Jan 14 '16, 10:12 a.m.
I was successfully able to fix the problem and start Tomcat as a Windows service.  The fix involved changing some of the Java Options when configuring the service using tomcat7w.exe.

Specifically, I needed to:

1) Remove the option:  -Xgc:preferredHeapBase=0x100000000
2) Add the option:  -Djava.library.path=C:\PROGRA~1\IBM\JazzTeamServer_601\server\tomcat\lib
3) Modify the option:  -Djazz.connector.sslEnabledProtocols
      * Change FROM:  -Djazz.connector.sslEnabledProtocols=TLSv1.2
       * Change TO:  -Djazz.connector.sslEnabledProtocols="TLSv1,TLSv1.1,TLSv1.2"

With these 3 changes, the service successfully started and I was able to navigate to the CLM applications from a web browser.  Here are all of the Java Options:

-Dcatalina.home=C:\PROGRA~1\IBM\JazzTeamServer_601\server\tomcat
-Dcatalina.base=C:\PROGRA~1\IBM\JazzTeamServer_601\server\tomcat
-Djava.io.tmpdir=C:\PROGRA~1\IBM\JazzTeamServer_601\server\tomcat\temp
-Djava.library.path=C:\PROGRA~1\IBM\JazzTeamServer_601\server\tomcat\lib
-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
-Djava.util.logging.config.file=C:\PROGRA~1\IBM\JazzTeamServer_601\server\tomcat\conf\logging.properties
-DJAZZ_HOME=file:///C:/PROGRA~1/IBM/JazzTeamServer_601/server/conf
-Djava.awt.headless=true
-Dorg.eclipse.emf.ecore.plugin.EcorePlugin.doNotLoadResourcesPlugin=true
-Dcom.ibm.team.repository.tempDir=C:\PROGRA~1\IBM\JazzTeamServer_601\server\tomcat\temp
-Djazz.connector.sslEnabledProtocols="TLSv1,TLSv1.1,TLSv1.2"
-Djazz.connector.algorithm=IbmX509
-Dlog4j.configuration=file:///C:/PROGRA~1/IBM/JazzTeamServer_601/server/conf/startup_log4j.properties
-Xgcpolicy:gencon
-Xcompressedrefs
-XX:MaxDirectMemorySize=1G
-Xmx4G
-Xms4G
-Xmn1g
-DORACLE_JDBC_DRIVER_FILE=C:\PROGRA~1\IBM\JazzTeamServer_601\oracle\ojdbc6.jar


permanent link
Kot T. (1.5k11319) | answered Jan 08 '16, 11:32 a.m.
JAZZ DEVELOPER
edited Jan 12 '16, 4:31 p.m.
Tomcat version used is 'apache-tomcat-7.0.67-windows-x64.zip'. CLM starts up correctly and is accessible when using server.startup.bat.

When trying to configure it to run as Windows Service,, I see the same error mentioned in this post in tomcat7-stderr.log. I got passed the error by removing the following line from the java options.
      -Xgc:preferredHeapBase=0x100000000

This allows the Tomcat Service to startup; however, when I access the CLM application, it keeps loading...

Found the following error in catalina.out:
Jan 12, 2016 4:17:17 PM org.apache.catalina.core.AprLifecycleListener lifecycleEvent
INFO: Loaded APR based Apache Tomcat Native library 1.1.33 using APR version 1.5.1.
Jan 12, 2016 4:17:17 PM org.apache.catalina.core.AprLifecycleListener lifecycleEvent
INFO: APR capabilities: IPv6 [true], sendfile [true], accept filters [false], random [true].
Jan 12, 2016 4:17:18 PM org.apache.catalina.core.AprLifecycleListener initializeSSL
INFO: OpenSSL successfully initialized (OpenSSL 1.0.1m 19 Mar 2015)
Jan 12, 2016 4:17:18 PM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["http-apr-9080"]
Jan 12, 2016 4:17:18 PM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["http-apr-9443"]
Jan 12, 2016 4:17:18 PM org.apache.coyote.AbstractProtocol init
SEVERE: Failed to initialize end point associated with ProtocolHandler ["http-apr-9443"]
java.lang.Exception: Connector attribute SSLCertificateFile must be defined when using SSL with APR
...


Somehow, it can find the library for APR. This isn't the case when using server.startup.bat.

Jun 22, 2015 9:58:12 AM org.apache.catalina.core.AprLifecycleListener lifecycleEvent
INFO: The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: <some paths>

Jun 22, 2015 9:58:13 AM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["http-bio-9080"]
Jun 22, 2015 9:58:13 AM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["http-bio-9443"]



I could not find where java.library.path is defined in the server.startup.bat, but I assume it takes the value from %PATH% system variable.

In order to mimic the old behavior, I added the following java option in tomcat7w:
         -Djava.library.path=<some path>

In my case, I use the path that does not exist ==> -Djava.library.path=/usr/tomcat/shared/lib

Start the server and confirm no SEVERE error in tomcat logs. I am able to access CLM after the above change.

Comments
Shradha Srivastav commented Jan 13 '16, 11:32 p.m.

I tried above steps on Windows server but got below error in localhost.log file and was not able to access CLM

SEVERE: Servlet.service() for servlet [equinoxbridgeservlet] in context with path [/jts] threw exception [Failed to determine authentication method.  The URL from the request is: https://hostname:443/jts] with root cause
javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
    at com.ibm.jsse2.j.a(j.java:9)
    at com.ibm.jsse2.j.a(j.java:6)


Kot T. commented Jan 14 '16, 11:05 a.m.
JAZZ DEVELOPER

The handshake_failure is likely caused by the following option:
          -Djazz.connector.sslEnabledProtocols="TLSv1.2"

It seems some extra steps are needed for Tomcat to support TLSv1.2:
https://jazz.net/help-dev/clm/index.jsp?re=1&topic=/com.ibm.jazz.install.doc/topics/t_enable_tls1.2_tomcat.html&scope=null


permanent link
Tadahiro Hara (927) | answered Jan 14 '16, 3:05 a.m.
edited Jan 14 '16, 3:23 a.m.
Hi.
Thank you for you patience with my English.

Although I also had same problem I have resolved below two steps.

1) Change preferredHeapBase to 0x200000000.

I referred this page

-Xgc:preferredHeapBase=0x100000000 to
-Xgc:preferredHeapBase=0x200000000.

2)delete tcnative-1.dll in JazzTeamServer601\server\tomcat\bin
tcnative-1.dll is Apache Tomcat Native Library
if you downloaded tomcat archive from tomcat site there is tcnative-1.dll in tomcat\bin.

There isn't tcnative-1.dll In tomcat\bin CLM6.0 or CLM5.0.2

After above two step you can run tomcat as windows service.


Comments
Kim Soederhamn commented Jan 14 '16, 8:19 a.m.

I had the very same problem in event viewer I also found:
"The Apache Tomcat 7.0 tomcat7 service terminated with the following service-specific error:
The System cannot open the file"
Tadahiros solution worked like a charm


Kim Soederhamn commented Jan 14 '16, 9:31 a.m.

Acutally I also had to add Shradha Srivastav fix as the server started just fine but I could not access any webpages I think it had to do with the sslEnabledprotocols not being found when running as a service


SEC Servizi commented Apr 10 '19, 11:42 a.m.

Thank you, this solved our same problem.

Cheers.


permanent link
Davyd Norris (20221014) | answered Jan 06 '16, 8:20 p.m.
I had a similar problem with an earlier version of CLM and I found that the heap error message is actually a bit of a red herring. What it's actually telling you is it can't allocate the heap because it can't start the JVM, and the reason it can't start the JVM is that it can't find one of the files (the first error message)

In the end I found that one of my paths in either the tomcat properties or the teamserver.properties was incorrect.

My suspicion in your case is that the space in 'Program Files' is giving you grief in all your catalina.* type properties. Try putting some quotes in, or using the short path name. I am pretty sure it will be a path issues somewhere


Comments
Ralph Schoon commented Jan 07 '16, 2:24 a.m.
FORUM ADMINISTRATOR / FORUM MODERATOR / JAZZ DEVELOPER

In addition, I would recommend to never, ever, ever install the server in the Program Files folder. The folder is protected. Even if logged in as administrator, if you run server.startup.bat the first time you can run into the situation that the server does not start, because it can not create folders. You have to actively open a cmd with "as administrator" and run the startup from there.

So install into some other folder e.g. \clm\6.0.1. This also support upgrade better.


permanent link
John Hannon (1111) | answered Dec 29 '16, 6:11 p.m.

Becky's resolution worked for me, the CLM 6.0.2 installation instructions didn't contain the path to the Java library:

-Djava.library.path=D:\ibm\JazzTeamServer\server\tomcat\lib

Your answer


Register or to post your answer.


Dashboards and work items are no longer publicly available, so some links may be invalid. We now provide similar information through other means. Learn more here.