Who shut down Build Forge??
Hi guys,
Build Forge was down yesterday morning. When I checked the catalna log, I saw it lost communication with its database at one point. This has happened before (we have network issues). However, I also see the following in the log (that I'd never seen before), about an hour after the communication problem occurred:
Aug 19, 2013 9:26:12 AM org.apache.catalina.core.StandardServer await
INFO: A valid shutdown command was received via the shutdown port. Stopping the Server instance.
Aug 19, 2013 9:26:12 AM org.apache.coyote.AbstractProtocol pause
INFO: Pausing ProtocolHandler ["http-bio-8080"]
Everybody swears they did not stop the engine. Is it possible Build Forge sent this message to itself and shut itself down? Does it do that if it realizes its database connection is down?
Thanks,
Jozef
Accepted answer
Comments
Thanks Jason. That server has not been rebooted in weeks, so that can't be it. Now that I know it wasn't BF I'll be looking elsewhere for the reason.
Jozef
Jason,
It actually happened again. Build Forge lost communication with its database and 40 minutes later Apache shut itself down.
Aug 22, 2013 9:23:51 AM com.buildforge.services.server.event.RTCEventReaper run
SEVERE: Exception during RTCEvent reaping
Throwable occurred: com.buildforge.services.common.db.DBException: {SQLException during retrieval of connection from pool}
...
Aug 22, 2013 10:03:37 AM org.apache.catalina.core.StandardServer await
INFO: A valid shutdown command was received via the shutdown port. Stopping the Server instance.
I had to manually restart the engine to get Build Forge operational again.
The previous time there was also 40 minutes between BF losing network connection and turning itself off. I went through the Windows Security event logs and there is no record of anyone logging in around those times (each logon/logoff is recorded in that log).
I would really like to understand this behavior.
Thanks,
Jozef
Build Forge is a database centric product, and any absence of the database connection will prevent Build Forge from functioning properly.
If the database connection is lose, the expected behavior is the following.
- Apache/PHP loses the database connection, and will re-attempt to connect periodically until successful.
- Tomcat rbf-services loses the database connection, and will re-attempt to connect periodically until successful.
- Build Forge engine process will immediately shutdown and die.
So, we could look into why Tomcat was shutdown 40 minutes after the database connection was lost, but it would be moot to the fact that the Build Forge engine was already gone and a full restart of Build Forge will already be required.
1 vote
One other answer
Comments
Two things can happen when the database loses connectivity.