Performance Problems in 601
CLM 601 WAS Liberty -> JRS 601 (separate machine)
16 GB RAM (8GB for CLM) - 16GB RAM (8GB allocated for JRS)
I can understand during 502/tomcat after a reboot, the first to log in took a long time. With WASLiberty601, the login appears quick on start up, and it takes a while to log in, OK. However, the problem I am having is EVERY DAY this week, when I log in in the morning, it's taking 5-10mins. Sometimes after a long period of idle, it takes time. I logged in, (5-10mins) and it worked fine after that. I went to a couple of meetings and lunch. I came back and I logged in at 2:18pm, it took till 2:29pm to finally log me in to my personal dashboard. Others tried, and they were left waiting as well.
What can I do to problem solve this?
I had the server team and WAS team look at the server. 4GB of the 8GB for Java was in use, Memory of the 16GB machine at 53%. Virtually no CPU usage. 7.xGB available.
Any ideas on what logs or something to look for?
2 answers
Sterling
Is the login time same if you browse to either server? are you using Liberty's own user registry or connected to LDAP? Any different if you use the browser on the server? What is the network connection between the two servers - no unexpected high latency/ping time? You aren't using file shares between them, are you?
Liberty defaults to lazy loading of apps which means there can be a delay the first time you browse to an app (and also when an app is activated by e.g. rich hover). This would account for why the login appears quickly the first time but the app takes longer to appear. However that delay shou;ldn't appear for subsequent logins to the same app while the server is still running, because Liberty will have loaded it.
If you have free memory available on the machines then I would allocate it to jvm - you might be able to allocate 12G in a 16G machine. There are some guidelines (IIRC in the deployment wiki) that the jvm memory allocations for Xmx and Xms should be set to the same, e.g. 12G, and the nursery Xmn should be a third of that, i.e for 12G the Xmn should be 4G.
HTH
Ian
Is the login time same if you browse to either server? are you using Liberty's own user registry or connected to LDAP? Any different if you use the browser on the server? What is the network connection between the two servers - no unexpected high latency/ping time? You aren't using file shares between them, are you?
Liberty defaults to lazy loading of apps which means there can be a delay the first time you browse to an app (and also when an app is activated by e.g. rich hover). This would account for why the login appears quickly the first time but the app takes longer to appear. However that delay shou;ldn't appear for subsequent logins to the same app while the server is still running, because Liberty will have loaded it.
If you have free memory available on the machines then I would allocate it to jvm - you might be able to allocate 12G in a 16G machine. There are some guidelines (IIRC in the deployment wiki) that the jvm memory allocations for Xmx and Xms should be set to the same, e.g. 12G, and the nursery Xmn should be a third of that, i.e for 12G the Xmn should be 4G.
HTH
Ian
Comments
Hello,
based on:
I went from 10GB, to 8GB to leave "resource" for the server. But then again, the server isn't doing anything. (such as right now...haven't logged in since 11am...it' now 5mins) both the JRS server and CLM (all-in-one) are sitting with no CPU crunching, and no memory usage. I am at 8GB, because I thought it was a resource issue when it was at 10GB doing the same thing...
Assuming you have 16G physical RAM memory you should be able to give jvm more memory, e.g. 12G. These are physical machines? What OS? What database are you using?
Virtual windows 2012 with MS SQL Server 2012.
Update:
It just hung for 6 minutes at /jts/admin. When it finally came back, it gave the following message:
The external user registry synchronization is complete. All users have been imported into the Jazz Team Server.
Soooo....is it LDAP (MS Active Directory) that is causing this delay? What are the sync times for LDAP? If I can monitor this, then I might have a clue.
Comments
The settings can be found under "com.ibm.team.repository.service.jts.internal.userregistry.ldap.LDAPNightlySyncTask" in the Advanced Properties page of /jts/admin.
hmmm...
OK, well that tells me that it runs every 24hours, but that actually doesn't explain why some updates throughout the day get added to LDAP and then are "synced/read" in to the JTS within the hour...without the need for an Import Users.
Anywhere else to look?
It looks like this may be a 601 JRS issue. We are going to "un couple" the JRS server for a day or two and see what happens.
Comments
Tiago Moura
Mar 18 '16, 3:06 a.m.Is only the login process taking too long?
Donald Nong
Mar 17 '16, 11:09 p.m.If you have not submitted a PMR with Support, I suggest you do so. If you would like to investigate by yourself, these are the things I will look into (not in order).
1. Network, in particular DNS. If the process is waiting on network/connection, you can expect the same symptom that you have observed. You may need to use Wireshark or tcpdump to capture network packets.
2. Register to https://wait.ibm.com/ and obtain the waitDataCollector utility. Capture multiple (20 - 30) javacore files (of Liberty) during the long log in period. And then analyze the javacore file to find out what those processes are doing.
3. Use a network trace tool (Firebug or Chrome itself) to monitor the network traffic to identify which request takes such a long time.
1 vote