Can these logs tell me whether my Tomcat is crashing or being shut down?
Hello,
I am troubleshooting some catalina logs to determine whether my CLM server is crashing or whether it is being brought down intentionally by an admin or some rogue process. Here are 4 different log extracts where the server restarts. Can somebody help me figure out what's going on?
Extract 1
Extract 2Feb 10, 2014 10:21:01 AM org.apache.catalina.startup.Catalina stopServerSEVERE: Catalina.stop:java.net.ConnectException: Connection refused: connectat java.net.PlainSocketImpl.socketConnect(Native Method)at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:393)at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:254)at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:241)at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:389)at java.net.Socket.connect(Socket.java:556)at java.net.Socket.connect(Socket.java:504)at java.net.Socket.<init>(Socket.java:412)at java.net.Socket.<init>(Socket.java:226)at org.apache.catalina.startup.Catalina.stopServer(Catalina.java:395)at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:79)at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)at java.lang.reflect.Method.invoke(Method.java:618)at org.apache.catalina.startup.Bootstrap.stopServer(Bootstrap.java:344)at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:435)Feb 10, 2014 10:21:08 AM org.apache.catalina.core.AprLifecycleListener initINFO: The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path:<removed>Feb 10, 2014 10:21:09 AM org.apache.coyote.http11.Http11BaseProtocol initINFO: Initializing Coyote HTTP/1.1 on http-9080Feb 10, 2014 10:21:10 AM org.apache.coyote.http11.Http11BaseProtocol initINFO: Initializing Coyote HTTP/1.1 on http-9443Feb 10, 2014 10:21:10 AM org.apache.catalina.startup.Catalina loadINFO: Initialization processed in 2564 msFeb 10, 2014 10:21:10 AM org.apache.catalina.core.StandardService startINFO: Starting service CatalinaFeb 10, 2014 10:21:10 AM org.apache.catalina.core.StandardEngine startINFO: Starting Servlet Engine: Apache Tomcat/5.5.33Feb 10, 2014 10:21:10 AM org.apache.catalina.core.StandardHost startINFO: XML validation disabledFeb 10, 2014 10:22:09 AM org.apache.catalina.startup.HostConfig deployWARINFO: Deploying web application archive clmhelp.warFeb 10, 2014 10:22:16 AM org.apache.coyote.http11.Http11BaseProtocol startINFO: Starting Coyote HTTP/1.1 on http-9080Feb 10, 2014 10:22:16 AM org.apache.coyote.http11.Http11BaseProtocol startINFO: Starting Coyote HTTP/1.1 on http-9443Feb 10, 2014 10:22:16 AM org.apache.catalina.storeconfig.StoreLoader loadINFO: Find registry server-registry.xml at classpath resourceFeb 10, 2014 10:22:16 AM org.apache.catalina.startup.Catalina startINFO: Server startup in 66349 ms
Extract 3Feb 11, 2014 7:08:11 PM org.apache.coyote.http11.Http11BaseProtocol pauseINFO: Pausing Coyote HTTP/1.1 on http-9080Feb 11, 2014 7:17:59 PM org.apache.catalina.startup.Catalina stopServerSEVERE: Catalina.stop:java.net.ConnectException: Connection refused: connectat java.net.PlainSocketImpl.socketConnect(Native Method)at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:393)at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:254)at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:241)at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:389)at java.net.Socket.connect(Socket.java:556)at java.net.Socket.connect(Socket.java:504)at java.net.Socket.<init>(Socket.java:412)at java.net.Socket.<init>(Socket.java:226)at org.apache.catalina.startup.Catalina.stopServer(Catalina.java:395)at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:79)at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)at java.lang.reflect.Method.invoke(Method.java:618)at org.apache.catalina.startup.Bootstrap.stopServer(Bootstrap.java:344)at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:435)Feb 11, 2014 7:18:08 PM org.apache.catalina.core.AprLifecycleListener initINFO: The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path:<removed>Feb 11, 2014 7:18:08 PM org.apache.coyote.http11.Http11BaseProtocol initINFO: Initializing Coyote HTTP/1.1 on http-9080Feb 11, 2014 7:18:10 PM org.apache.coyote.http11.Http11BaseProtocol initINFO: Initializing Coyote HTTP/1.1 on http-9443Feb 11, 2014 7:18:10 PM org.apache.catalina.startup.Catalina loadINFO: Initialization processed in 2314 msFeb 11, 2014 7:18:10 PM org.apache.catalina.core.StandardService startINFO: Starting service CatalinaFeb 11, 2014 7:18:10 PM org.apache.catalina.core.StandardEngine startINFO: Starting Servlet Engine: Apache Tomcat/5.5.33Feb 11, 2014 7:18:10 PM org.apache.catalina.core.StandardHost startINFO: XML validation disabledFeb 11, 2014 7:19:23 PM org.apache.catalina.startup.HostConfig deployWARINFO: Deploying web application archive clmhelp.warFeb 11, 2014 7:19:31 PM org.apache.coyote.http11.Http11BaseProtocol startINFO: Starting Coyote HTTP/1.1 on http-9080Feb 11, 2014 7:19:31 PM org.apache.coyote.http11.Http11BaseProtocol startINFO: Starting Coyote HTTP/1.1 on http-9443Feb 11, 2014 7:19:31 PM org.apache.catalina.storeconfig.StoreLoader loadINFO: Find registry server-registry.xml at classpath resourceFeb 11, 2014 7:19:31 PM org.apache.catalina.startup.Catalina startINFO: Server startup in 81296 ms
Feb 14, 2014 2:07:08 PM org.apache.coyote.http11.Http11BaseProtocol pauseINFO: Pausing Coyote HTTP/1.1 on http-9080Feb 14, 2014 2:07:08 PM org.apache.coyote.http11.Http11BaseProtocol pauseINFO: Pausing Coyote HTTP/1.1 on http-9443Feb 14, 2014 2:07:09 PM org.apache.catalina.core.StandardService stopINFO: Stopping service CatalinaFeb 14, 2014 2:12:21 PM org.apache.coyote.http11.Http11BaseProtocol destroyINFO: Stopping Coyote HTTP/1.1 on http-9080Feb 14, 2014 2:12:21 PM org.apache.coyote.http11.Http11BaseProtocol destroyINFO: Stopping Coyote HTTP/1.1 on http-9443Feb 14, 2014 2:13:12 PM org.apache.catalina.startup.Catalina stopServerSEVERE: Catalina.stop:java.net.ConnectException: Connection refused: connectat java.net.PlainSocketImpl.socketConnect(Native Method)at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:393)at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:254)at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:241)at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:389)at java.net.Socket.connect(Socket.java:556)at java.net.Socket.connect(Socket.java:504)at java.net.Socket.<init>(Socket.java:412)at java.net.Socket.<init>(Socket.java:226)at org.apache.catalina.startup.Catalina.stopServer(Catalina.java:395)at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:79)at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)at java.lang.reflect.Method.invoke(Method.java:618)at org.apache.catalina.startup.Bootstrap.stopServer(Bootstrap.java:344)at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:435)Feb 14, 2014 4:32:29 PM org.apache.catalina.core.AprLifecycleListener initINFO: The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path:<removed>Feb 14, 2014 4:32:29 PM org.apache.coyote.http11.Http11BaseProtocol initINFO: Initializing Coyote HTTP/1.1 on http-9080Feb 14, 2014 4:32:30 PM org.apache.coyote.http11.Http11BaseProtocol initINFO: Initializing Coyote HTTP/1.1 on http-9443Feb 14, 2014 4:32:30 PM org.apache.catalina.startup.Catalina loadINFO: Initialization processed in 1626 msFeb 14, 2014 4:32:30 PM org.apache.catalina.core.StandardService startINFO: Starting service CatalinaFeb 14, 2014 4:32:30 PM org.apache.catalina.core.StandardEngine startINFO: Starting Servlet Engine: Apache Tomcat/5.5.33Feb 14, 2014 4:32:30 PM org.apache.catalina.core.StandardHost startINFO: XML validation disabledFeb 14, 2014 4:32:56 PM org.apache.catalina.startup.HostConfig deployWARINFO: Deploying web application archive clmhelp.warFeb 14, 2014 4:33:01 PM org.apache.coyote.http11.Http11BaseProtocol startINFO: Starting Coyote HTTP/1.1 on http-9080Feb 14, 2014 4:33:01 PM org.apache.coyote.http11.Http11BaseProtocol startINFO: Starting Coyote HTTP/1.1 on http-9443Feb 14, 2014 4:33:01 PM org.apache.catalina.storeconfig.StoreLoader loadINFO: Find registry server-registry.xml at classpath resourceFeb 14, 2014 4:33:01 PM org.apache.catalina.startup.Catalina startINFO: Server startup in 30344 msFeb 14, 2014 4:33:22 PM org.apache.coyote.http11.Http11BaseProtocol pauseINFO: Pausing Coyote HTTP/1.1 on http-9080Feb 14, 2014 4:33:22 PM org.apache.coyote.http11.Http11BaseProtocol pauseINFO: Pausing Coyote HTTP/1.1 on http-9443Feb 14, 2014 4:33:23 PM org.apache.catalina.core.StandardService stopINFO: Stopping service CatalinaFeb 14, 2014 4:33:26 PM org.apache.coyote.http11.Http11BaseProtocol destroyINFO: Stopping Coyote HTTP/1.1 on http-9080Feb 14, 2014 4:33:26 PM org.apache.coyote.http11.Http11BaseProtocol destroyINFO: Stopping Coyote HTTP/1.1 on http-9443Feb 14, 2014 4:33:35 PM org.apache.catalina.core.AprLifecycleListener initINFO: The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path:<removed>Feb 14, 2014 4:33:35 PM org.apache.coyote.http11.Http11BaseProtocol initINFO: Initializing Coyote HTTP/1.1 on http-9080Feb 14, 2014 4:33:36 PM org.apache.coyote.http11.Http11BaseProtocol initINFO: Initializing Coyote HTTP/1.1 on http-9443Feb 14, 2014 4:33:36 PM org.apache.catalina.startup.Catalina loadINFO: Initialization processed in 1579 msFeb 14, 2014 4:33:36 PM org.apache.catalina.core.StandardService startINFO: Starting service CatalinaFeb 14, 2014 4:33:36 PM org.apache.catalina.core.StandardEngine startINFO: Starting Servlet Engine: Apache Tomcat/5.5.33Feb 14, 2014 4:33:36 PM org.apache.catalina.core.StandardHost startINFO: XML validation disabledFeb 14, 2014 4:34:02 PM org.apache.catalina.startup.HostConfig deployWARINFO: Deploying web application archive clmhelp.warFeb 14, 2014 4:34:06 PM org.apache.coyote.http11.Http11BaseProtocol startINFO: Starting Coyote HTTP/1.1 on http-9080Feb 14, 2014 4:34:06 PM org.apache.coyote.http11.Http11BaseProtocol startINFO: Starting Coyote HTTP/1.1 on http-9443Feb 14, 2014 4:34:06 PM org.apache.catalina.storeconfig.StoreLoader loadINFO: Find registry server-registry.xml at classpath resourceFeb 14, 2014 4:34:06 PM org.apache.catalina.startup.Catalina startINFO: Server startup in 29774 ms
Extract 4
As you can see, there is sometimes a pause a few minutes before the server is stopped and restarted. The fact that the server starts almost immediately leads me to believe that there is some sort of restart being scheduled.Feb 18, 2014 10:28:51 PM org.apache.coyote.http11.Http11BaseProtocol pauseINFO: Pausing Coyote HTTP/1.1 on http-9080Feb 18, 2014 11:38:32 PM org.apache.catalina.startup.Catalina stopServerSEVERE: Catalina.stop:java.net.ConnectException: Connection refused: connectat java.net.PlainSocketImpl.socketConnect(Native Method)at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:393)at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:254)at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:241)at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:389)at java.net.Socket.connect(Socket.java:556)at java.net.Socket.connect(Socket.java:504)at java.net.Socket.<init>(Socket.java:412)at java.net.Socket.<init>(Socket.java:226)at org.apache.catalina.startup.Catalina.stopServer(Catalina.java:395)at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:79)at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)at java.lang.reflect.Method.invoke(Method.java:618)at org.apache.catalina.startup.Bootstrap.stopServer(Bootstrap.java:344)at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:435)Feb 18, 2014 11:38:40 PM org.apache.catalina.core.AprLifecycleListener initINFO: The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path:<removed>Feb 18, 2014 11:38:40 PM org.apache.coyote.http11.Http11BaseProtocol initINFO: Initializing Coyote HTTP/1.1 on http-9080Feb 18, 2014 11:38:42 PM org.apache.coyote.http11.Http11BaseProtocol initINFO: Initializing Coyote HTTP/1.1 on http-9443Feb 18, 2014 11:38:42 PM org.apache.catalina.startup.Catalina loadINFO: Initialization processed in 2455 msFeb 18, 2014 11:38:42 PM org.apache.catalina.core.StandardService startINFO: Starting service CatalinaFeb 18, 2014 11:38:42 PM org.apache.catalina.core.StandardEngine startINFO: Starting Servlet Engine: Apache Tomcat/5.5.33Feb 18, 2014 11:38:42 PM org.apache.catalina.core.StandardHost startINFO: XML validation disabledFeb 18, 2014 11:40:20 PM org.apache.catalina.startup.HostConfig deployWARINFO: Deploying web application archive clmhelp.warFeb 18, 2014 11:40:29 PM org.apache.coyote.http11.Http11BaseProtocol startINFO: Starting Coyote HTTP/1.1 on http-9080Feb 18, 2014 11:40:29 PM org.apache.coyote.http11.Http11BaseProtocol startINFO: Starting Coyote HTTP/1.1 on http-9443Feb 18, 2014 11:40:29 PM org.apache.catalina.storeconfig.StoreLoader loadINFO: Find registry server-registry.xml at classpath resourceFeb 18, 2014 11:40:30 PM org.apache.catalina.startup.Catalina startINFO: Server startup in 107714 ms
I don't expect a definitive solution, but maybe somebody has had a similart occurance and point to something that might be causing the problem (scheduled tasks or similar)
Thanks!
Andrew
One answer
Andrew, if I request a shutdown, my catalina logs in Tomcat/logs show:
INFO: A valid shutdown command was received via the shutdown port. Stopping the Server instance.
Comments
Hello Ralph,
Thanks for your input. When you request a shutdown, are you doing it via windows service or the Jazz Team Server batch files?
Regards,
Andrew
As I usually run demo/development/test systems I start them with the batch files.
If you run a service, the service logs should tell you when the service was shut down.