Jazz Forum Welcome to the Jazz Community Forum Connect and collaborate with IBM Engineering experts and users

JTS(4.0) has become very unstable/unresponsive

Questions
Tags
Users
Badges
Jeff Foege
(452●6●69●60) May 21 '14, 3:55 p.m.
 We used to have a problem with RTC not responding to user's request to login. Now it appears as JTS has this problem. The second time this week users weren't able to log into JTS but were able to log into RTC without trouble. The only way I'm able to correct this is to restart the PC, restarting the service doesn't seem to work. I've checked the JTS logs but I'm not seeing anything that stands out. In the admin log we have an entry on 4/21 then the next one is 5/12. I've checked the JTS.log and didn't see any errors there either. I do see a bunch of entries in the catalina log.

May 21, 2014 9:30:41 AM org.apache.catalina.realm.JNDIRealm open
WARNING: Exception performing authentication
Throwable occurred: javax.naming.ServiceUnavailableException: thcg.net:389; socket closed
at com.sun.jndi.ldap.Connection.readReply(Connection.java:423)
at com.sun.jndi.ldap.LdapClient.ldapBind(LdapClient.java:338)
at com.sun.jndi.ldap.LdapClient.authenticate(LdapClient.java:190)
at com.sun.jndi.ldap.LdapCtx.connect(LdapCtx.java:2720)
at com.sun.jndi.ldap.LdapCtx.<init>(LdapCtx.java:296)
at com.sun.jndi.ldap.LdapCtxFactory.getUsingURL(LdapCtxFactory.java:175)
at com.sun.jndi.ldap.LdapCtxFactory.getUsingURLs(LdapCtxFactory.java:193)
(not the whole trace)
May 21, 2014 9:30:42 AM org.apache.catalina.realm.JNDIRealm authenticate
SEVERE: Exception performing authentication
Throwable occurred: javax.naming.CommunicationException: localhost:389 [Root exception is java.net.ConnectException: Connection refused: connect]
at com.sun.jndi.ldap.Connection.<init>(Connection.java:198)
at com.sun.jndi.ldap.LdapClient.<init>(LdapClient.java:116)
at com.sun.jndi.ldap.LdapClient.getInstance(LdapClient.java:1580)
at com.sun.jndi.ldap.LdapCtx.connect(LdapCtx.java:2678)
at com.sun.jndi.ldap.LdapCtx.<init>(LdapCtx.java:296)

I checked the RTC logs and I'm seeing a ton of messages from the com.ibm.team.workitem.service. 

2014-05-21 11:41:25,326 [       http-bio-9443-exec-2358] ERROR com.ibm.team.workitem.service                       - com.ibm.team.workitem.service.internal.oslc.UnacceptableMediaTypeException: Content type '[application/x-oslc-compact+xml, application/x-jazz-compact-rendering]' is not supported.
java.lang.IllegalArgumentException: com.ibm.team.workitem.service.internal.oslc.UnacceptableMediaTypeException: Content type '[application/x-oslc-compact+xml, application/x-jazz-compact-rendering]' is not supported.
at com.ibm.team.workitem.service.internal.oslc.ResourceETagProvider.getETagFor(ResourceETagProvider.java:33)
at sun.reflect.GeneratedMethodAccessor384.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:600)

In fact the workitem.service error gets entered in the log every few seconds. In a span of about 20 minutes I counted over 30.

We have RTC 4.0.0 (with a hot fix applied, number 221037.) Why all of a sudden has JTS become so unstable. I've had to restart the PC twice already this week. Also at times it prevents users from getting work done because its so unstable now. I'd be happy to post more of the trace if need be. I also understand this might be two separate issues.


0 votes

Comments
Karl Weinert
JAZZ DEVELOPER Jun 03 '14, 4:14 p.m.

Hi Jeff,


Can you expand on what you mean by "not responding to user's request to login"
Is there some sort of error, does it hang etc...?

Are the other working applications running under the same instance of Tomcat?
Are all the databases the same server?

Is the LDAP server a single server or is it using some sort of load balancing?

Jeff Foege
Jun 03 '14, 4:43 p.m.

The only errors that I can find are ones I've posted in this thread. We get the login window, input our credentials and hit enter. The login dialog does away for a bit with a "Loading..." text in its place. Then it comes back with red text on the login window saying "Invalid ID/PW"

As far as the LDAP server, I'm not sure what the specs are on that since its in the IT department. We have multiple other apps that don't have a problem with the LDAP server.

Its becoming a real pain since there are times people can't login at all until we restart the service and/or restart  the machine.

3 answers
5,860 views
0 votes

3 answers

Permanent link
Donald Nong
(14.5k●6●14) May 22 '14, 1:30 a.m.
Your JTS server had some problems with connection to the LDAP server. At the time the unresponsiveness occurs, use command "telnet <server> 389" to see if you can connect to the LDAP server manually. Or in a browser on the JTS server, enter the URL ldap://<server> and see if you can search any users.
BTW, did you modify the logs before posting to the forum? It shows that the JTS server was connecting to two LDAP servers, one is "thcg" and the other "localhost".

0 votes

Permanent link
Jeff Foege
(452●6●69●60) Jun 02 '14, 12:22 p.m.

No I didn't modify the logs at all before posting. I'm not sure why it would have localhost. I also tried adding adCompat=true to the tomcat server.xml file based off this thread

https://jazz.net/forum/questions/120073/rtc-v402-tomcat7-ldap-connection-problem

This is now happening every few days. We have contractors in India and are often blocked from logging into RTC/JTS.

0 votes

Comments
Donald Nong
Jun 02 '14, 8:34 p.m.

I don't think the adCompat=true parameter can do anything to your issue, but it does seem like a Tomcat issue than a JTS issue. It is quote hard to imagine that Tomcat will switch the LDAP server form thcg.net to localhost all of a sudden. You may have to analyze network trace to understand why it happens like that.
Another option is to switch to WebSphere Application Server, which is unlikely to have the same problem.
http://pic.dhe.ibm.com/infocenter/clmhelp/v4r0/topic/com.ibm.jazz.install.doc/topics/t_migrate_tomcat_was.html

Permanent link
Jeff Foege
(452●6●69●60) Jun 03 '14, 3:59 p.m.

Unfortunately we won't be able to switch to Websphere. I work in the Medical arena and we would have to do this in a test environment first, validate it, create a report from the validation and submit it to doc control. Once that is done we would be able to switch.

I did notice something in the tomcat logs, I see this error all the time (a lot actually)

Jun 3, 2014 9:29:22 AM org.apache.catalina.realm.JNDIRealm open
WARNING: Exception performing authentication
Throwable occurred: javax.naming.CommunicationException: thcg.net:389

Jun 3, 2014 9:29:23 AM org.apache.catalina.realm.JNDIRealm authenticate
SEVERE: Exception performing authentication
Throwable occurred: javax.naming.CommunicationException: localhost:389

When the error is SEVERE it reports as localhost and not thcg.net. In fact our JTS server is going down/unrepsonsive almost a daily basis now. What happens is someone will try to login and they will get the "Loading..." screen for a few seconds and then it goes back to a login dialog. There are times when restarting Tomcat doesn't help at all until we restart the server. Here is part of the trace when I have the issue of it not restarting.

Jun 3, 2014 9:37:05 AM org.apache.catalina.startup.Catalina start
SEVERE: Catalina.start:
Throwable occurred: org.apache.catalina.LifecycleException: Failed to start component [StandardServer[9005]]
 at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:154)
 at org.apache.catalina.startup.Catalina.start(Catalina.java:646)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:48)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:600)
 at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:322)
 at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:450)
Caused by: org.apache.catalina.LifecycleException: Failed to start component [StandardService[Catalina]]
 at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:154)
 at org.apache.catalina.core.StandardServer.startInternal(StandardServer.java:727)
 at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
 ... 7 more

We certainly have other services/applications running on other machines, some through tomcat and they don't even remotely have the same issue as the JTS does now. The admin.log & jts.log don't really hold any valuable information to determine what is going on.

0 votes

Comments
Donald Nong
Jun 03 '14, 8:12 p.m.

It sounds more and more complicated now. I would suggest you open a ticket with IBM Support and send all the log files for analysis. A network trace may even be required.

Your answer

Register or log in to post your answer.

Dashboards and work items are no longer publicly available, so some links may be invalid. We now provide similar information through other means. Learn more here.

Ask a question
Guidelines
FAQ
Search context
Follow this question

By Email: 

Once you sign in you will be able to subscribe for any updates here.

By RSS:

Answers
Answers and Comments
Question details

Question asked: May 21 '14, 3:55 p.m.

Question was seen: 5,860 times

Last updated: Jun 03 '14, 8:12 p.m.

Related questions
Confirmation Cancel Confirm