CLMHelp crashing Websphere

Chris Ryan (157●3●24●28) | asked Jun 20 '13, 1:39 p.m.
retagged Jun 21 '13, 1:27 p.m. by Amy Laird (165●1●4)

When I attempt to visit https://127.0.0.1:9443/clmhelp/updater/updatewar.jsp in my attempt to enable local help (and disable the internet help), the page hangs.

I view my SystemOut.log, and there is lines such as

WSVR0605W: Thread "WebContainer : 12" has been active for 692205 milliseconds and may be hung. There is/are 5 thread(s) in total in the server that may be hung.

And then my server crashes shortly after.

Has anyone else seen this?

Comments

Chris Austin commented Jun 20 '13, 2:58 p.m.

JAZZ DEVELOPER

Hi Chris, that is odd - I haven't seen anything like that before. Assuming the about page loads, can you tell me what version of the help system are you using (can be found at https://127.0.0.1:9443/clmhelp/about/about.jsp)

Chris Ryan commented Jun 20 '13, 3:11 p.m.

Thanks for the response, Chris.

The help version I'm using is as follows:

Eclipse Version:    3.7.0v20110613
Version:                M4.2012
WAR Type:           UA Delivery WAR
...
...

Chris Ryan commented Jun 20 '13, 3:11 p.m.

And yes, the page loaded. :)

Krzysztof Kaźmierczyk commented Jun 21 '13, 4:18 a.m.

hi Chris,
What do you mean by crashing? Is the server process running and the server is not accessible? Or there is no server process running?

Chris Ryan commented Jun 21 '13, 9:16 a.m.

The server process eventually becomes unusable, cannot be shut down using the stopServer script, and must be killed manually. i.e. kill -9 <process ID>

Chris Austin commented Jun 21 '13, 10:53 a.m.

JAZZ DEVELOPER

Ouch. So updatewar.jsp is causing the webserver process to hang. The only thing I can think of is maybe some javascript function is caught in an endless loop. Its possible you could see where the error happens using firebug or some similar tool, but that would be a pain. We might be able to get you a newer version of the WAR to try (M2.2013 is the latest at the moment) - let me look into that.

Chris Ryan commented Jun 28 '13, 8:39 a.m.

Chris, have you had any luck looking into it?

I just attempted to stop the clmhelp application, and that action produced 10 hung threads. I've had no choice but to kill my production WAS process, restart, and then uninstall the CLMHelp application as it's ununsable. Unfortunately.

Any suggestions would be fantastic.

showing 5 of 7 show 2 more comments

3 answers

Most liked answers ↑|Newest answers|Oldest answers

permanent link

Anthony Kesterton (7.5k●7●179●136) | answered Jun 21 '13, 10:11 a.m.
JAZZ DEVELOPER

The help system goes to an IBM web site to grab the content - so it may be that the system cannot get to the external site. Is it possible that access might be blocked to external sites?

Comments

Chris Austin commented Jun 21 '13, 10:55 a.m.

JAZZ DEVELOPER

This is a good thought, but the network calls to IBM wouldn't start until after the page is pretty much loaded. You would see a loading image next to the 'Site List', and then it would go away after the IBM server can't be contacted.

Piotr Aniola commented Jun 21 '13, 11:06 a.m.

You can use a network sniffer to check if any network requests are made.
I suppose if that would be the case, eventually there would be a network timeout, and everything would go back to normal.

permanent link

Abraham Sweiss (2.4k●13●31) | answered Jun 21 '13, 11:51 a.m.

Hello Chris,

If there were network issues, I would expect to see sometype of timeout in the WASlogs.

SInce we are seeing hung messages, WAS may not have enough threads to manage the work load. I would suggest the following:
1. WAS by default sets the webcontainor thread pool to min=max of 20. I would increase this value to 100
2. Enable PMI to monitor the thread utilization in WAS
3. Enable the Hung thread detection policy in WebSphere to generate cores when a hung thread is detected. This should give some insight on what is going on.

Comments

Chris Ryan commented Jun 28 '13, 8:40 a.m.

Hey Abraham,

I've done all of the following. My thread pool was already at 50.

PMI is already enabled.

I've also configured the HungThread policy to dump Java when threads hang.

permanent link

Abraham Sweiss (2.4k●13●31) | answered Jun 28 '13, 10:46 a.m.

Hi Chris,

Just to eliminate any type of resource contention, I would:
1. Increase the thread pool to min=max=500. NOTE: Seting it this high is to remove any dout that there is an issue with the thread pool being too low.
2. Increase the java heap to min=max=4gig
3. Also to eliminate this is an issue with connectivity, ensure that there is a stable connection to the internet.

Attempt to reproduce the issue. If it persists
1. Capture WAS logs, jts/ccm logs and java cores
2. View the PMI data to determine if the thread pool was exhausted
3.Open a service request against WebSphere. This way we can have the WAS java team review the WAS logs and java cores. If needed they will pass the PMR over to the CLM support team.
4. Also reference this forum post in the PMR.

It's all about the answers!

Related questions

CLMHelp crashing Websphere

3 answers

Your answer