CLMHelp crashing Websphere
When I attempt to visit https://127.0.0.1:9443/clmhelp/updater/updatewar.jsp in my attempt to enable local help (and disable the internet help), the page hangs.
I view my SystemOut.log, and there is lines such as
WSVR0605W: Thread "WebContainer : 12" has been active for 692205 milliseconds and may be hung. There is/are 5 thread(s) in total in the server that may be hung.
And then my server crashes shortly after.
Has anyone else seen this?
I view my SystemOut.log, and there is lines such as
WSVR0605W: Thread "WebContainer : 12" has been active for 692205 milliseconds and may be hung. There is/are 5 thread(s) in total in the server that may be hung.
And then my server crashes shortly after.
Has anyone else seen this?
3 answers
The help system goes to an IBM web site to grab the content - so it may be that the system cannot get to the external site. Is it possible that access might be blocked to external sites?
Comments
This is a good thought, but the network calls to IBM wouldn't start until after the page is pretty much loaded. You would see a loading image next to the 'Site List', and then it would go away after the IBM server can't be contacted.
You can use a network sniffer to check if any network requests are made.
I suppose if that would be the case, eventually there would be a network timeout, and everything would go back to normal.
Hello Chris,
If there were network issues, I would expect to see sometype of timeout in the WASlogs.
SInce we are seeing hung messages, WAS may not have enough threads to manage the work load. I would suggest the following:
1. WAS by default sets the webcontainor thread pool to min=max of 20. I would increase this value to 100
2. Enable PMI to monitor the thread utilization in WAS
3. Enable the Hung thread detection policy in WebSphere to generate cores when a hung thread is detected. This should give some insight on what is going on.
If there were network issues, I would expect to see sometype of timeout in the WASlogs.
SInce we are seeing hung messages, WAS may not have enough threads to manage the work load. I would suggest the following:
1. WAS by default sets the webcontainor thread pool to min=max of 20. I would increase this value to 100
2. Enable PMI to monitor the thread utilization in WAS
3. Enable the Hung thread detection policy in WebSphere to generate cores when a hung thread is detected. This should give some insight on what is going on.
Hi Chris,
Just to eliminate any type of resource contention, I would:
1. Increase the thread pool to min=max=500. NOTE: Seting it this high is to remove any dout that there is an issue with the thread pool being too low.
2. Increase the java heap to min=max=4gig
3. Also to eliminate this is an issue with connectivity, ensure that there is a stable connection to the internet.
Attempt to reproduce the issue. If it persists
1. Capture WAS logs, jts/ccm logs and java cores
2. View the PMI data to determine if the thread pool was exhausted
3.Open a service request against WebSphere. This way we can have the WAS java team review the WAS logs and java cores. If needed they will pass the PMR over to the CLM support team.
4. Also reference this forum post in the PMR.
Just to eliminate any type of resource contention, I would:
1. Increase the thread pool to min=max=500. NOTE: Seting it this high is to remove any dout that there is an issue with the thread pool being too low.
2. Increase the java heap to min=max=4gig
3. Also to eliminate this is an issue with connectivity, ensure that there is a stable connection to the internet.
Attempt to reproduce the issue. If it persists
1. Capture WAS logs, jts/ccm logs and java cores
2. View the PMI data to determine if the thread pool was exhausted
3.Open a service request against WebSphere. This way we can have the WAS java team review the WAS logs and java cores. If needed they will pass the PMR over to the CLM support team.
4. Also reference this forum post in the PMR.
Comments
Chris Austin
JAZZ DEVELOPER Jun 20 '13, 2:58 p.m.Hi Chris, that is odd - I haven't seen anything like that before. Assuming the about page loads, can you tell me what version of the help system are you using (can be found at https://127.0.0.1:9443/clmhelp/about/about.jsp)
Chris Ryan
Jun 20 '13, 3:11 p.m.Thanks for the response, Chris.
The help version I'm using is as follows:
Eclipse Version: 3.7.0v20110613
Version: M4.2012
WAR Type: UA Delivery WAR
...
...
Chris Ryan
Jun 20 '13, 3:11 p.m.And yes, the page loaded. :)
Krzysztof Kaźmierczyk
Jun 21 '13, 4:18 a.m.hi Chris,
What do you mean by crashing? Is the server process running and the server is not accessible? Or there is no server process running?
Chris Ryan
Jun 21 '13, 9:16 a.m.The server process eventually becomes unusable, cannot be shut down using the stopServer script, and must be killed manually. i.e. kill -9 <process ID>
Chris Austin
JAZZ DEVELOPER Jun 21 '13, 10:53 a.m.Ouch. So updatewar.jsp is causing the webserver process to hang. The only thing I can think of is maybe some javascript function is caught in an endless loop. Its possible you could see where the error happens using firebug or some similar tool, but that would be a pain. We might be able to get you a newer version of the WAR to try (M2.2013 is the latest at the moment) - let me look into that.
1 vote
Chris Ryan
Jun 28 '13, 8:39 a.m.Chris, have you had any luck looking into it?
I just attempted to stop the clmhelp application, and that action produced 10 hung threads. I've had no choice but to kill my production WAS process, restart, and then uninstall the CLMHelp application as it's ununsable. Unfortunately.
Any suggestions would be fantastic.