[closed] Logging into JTS, RTC, RQM using LDAP authentication takes about 10 mins
Enrique Gaona (134●6●25●26)
| asked Jul 05 '17, 2:19 p.m.
closed Mar 07 '22, 8:52 a.m. by Ralph Schoon (63.3k●3●36●46)
Hi,
When I login to JTS or RTC or RQM, it takes over 10 minutes just to login. The CLM server is using LDAP authenticaiton. Looking at the jts.log, I'm constantly seeing an LDAPUserRegistry and ServiceUnavailableException error in the jts.log. Does anyone know what these errors are and how to resolve them? Thanks.
Output from jts.log:
2017-06-26 11:48:48,031 [http-bio-9443-exec-10760 @@ 11:33 xxxxxx@xx.xxx.com <com.ibm.team.repository.manageUsers/Search@34df35c0-087a-4b85-8243-7f42497b073a> /jts/service/com.ibm.team.repository.service.internal.IExternalUserRegistryRestService/searchRegistry] ERROR ce.jts.internal.userregistry.ldap.LDAPUserRegistry - [ff00c384.2] CRJAZ0745I Error retrieving users from the external user directory.
2017-06-26 13:45:50,596 [http-bio-9443-exec-10997 @@ 13:30 xxxxx@xx.xxx.com <com.ibm.team.repository.editUser@28246360-497f-44dc-8f5f-af7443753aa1> /jts/service/com.ibm.team.repository.service.internal.IAdminRestService/contributorByUUID] ERROR ce.jts.internal.userregistry.ldap.LDAPUserRegistry - [9e5744f] CRJAZ0743I Error retrieving group for user.
javax.naming.ServiceUnavailableException: bluepages.ibm.com:389; socket closed; remaining name 'ou=bluepages,o=ibm.com'
at com.sun.jndi.ldap.Connection.readReply(Connection.java:479)
at com.sun.jndi.ldap.LdapClient.getSearchReply(LdapClient.java:652)
at com.sun.jndi.ldap.LdapClient.search(LdapClient.java:575)
at com.sun.jndi.ldap.LdapCtx.doSearch(LdapCtx.java:1998)
at com.sun.jndi.ldap.LdapCtx.searchAux(LdapCtx.java:1860)
at com.sun.jndi.ldap.LdapCtx.c_search(LdapCtx.java:1785)
at com.sun.jndi.toolkit.ctx.ComponentDirContext.p_search(ComponentDirContext.java:399)
at com.sun.jndi.toolkit.ctx.PartialCompositeDirContext.search(PartialCompositeDirContext.java:369)
at com.sun.jndi.toolkit.ctx.PartialCompositeDirContext.search(PartialCompositeDirContext.java:352)
at javax.naming.directory.InitialDirContext.search(InitialDirContext.java:279)
at com.ibm.team.repository.jndi.internal.DirContextImpl.search(DirContextImpl.java:242)
at com.ibm.team.repository.service.jts.internal.userregistry.ldap.LDAPUserRegistry.fetchUserDN(LDAPUserRegistry.java:634)
at com.ibm.team.repository.service.jts.internal.userregistry.ldap.LDAPUserRegistry.fetchUserDN(LDAPUserRegistry.java:616)
at com.ibm.team.repository.service.jts.internal.userregistry.ldap.LDAPUserRegistry.fetchGroupsForUser(LDAPUserRegistry.java:700)
at com.ibm.team.repository.service.jts.internal.userregistry.ldap.LDAPUserRegistry.fetchGroupsForUser(LDAPUserRegistry.java:607)
at com.ibm.team.repository.service.jts.internal.userregistry.ExternalUserRegistryInternalService.fetchGroupsForUser(ExternalUserRegistryInternalService.java:255)
at sun.reflect.GeneratedMethodAccessor3523.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:56)
at java.lang.reflect.Method.invoke(Method.java:620)
at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord.invoke(ExportProxyServiceRecord.java:361)
at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord.access$0(ExportProxyServiceRecord.java:347)
at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord$ExportedServiceInvocationHandler.invoke(ExportProxyServiceRecord.java:56)
at com.sun.proxy.$Proxy580.fetchGroupsForUser(Unknown Source)
at com.ibm.team.repository.service.jts.internal.userregistry.JtsExternalUserRegistryService.fetchGroupsForUser(JtsExternalUserRegistryService.java:101)
at sun.reflect.GeneratedMethodAccessor3522.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:56)
at java.lang.reflect.Method.invoke(Method.java:620)
at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord.invoke(ExportProxyServiceRecord.java:361)
at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord.access$0(ExportProxyServiceRecord.java:347)
at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord$ExportedServiceInvocationHandler.invoke(ExportProxyServiceRecord.java:56)
at com.sun.proxy.$Proxy581.fetchGroupsForUser(Unknown Source)
at com.ibm.team.repository.service.jts.internal.userregistry.ExternalUserRegistryServiceDelegator.fetchGroupsForUser(ExternalUserRegistryServiceDelegator.java:79)
at sun.reflect.GeneratedMethodAccessor3521.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:56)
at java.lang.reflect.Method.invoke(Method.java:620)
at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord.invoke(ExportProxyServiceRecord.java:361)
at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord.access$0(ExportProxyServiceRecord.java:347)
at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord$ExportedServiceInvocationHandler.invoke(ExportProxyServiceRecord.java:56)
at com.sun.proxy.$Proxy600.fetchGroupsForUser(Unknown Source)
at com.ibm.team.repository.service.internal.AdminRestService.addContributorRoles(AdminRestService.java:671)
at com.ibm.team.repository.service.internal.AdminRestService.getContributorByUUID(AdminRestService.java:645)
at sun.reflect.GeneratedMethodAccessor3977.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:56)
at java.lang.reflect.Method.invoke(Method.java:620)
at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord.invoke(ExportProxyServiceRecord.java:361)
at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord.access$0(ExportProxyServiceRecord.java:347)
at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord$ExportedServiceInvocationHandler.invoke(ExportProxyServiceRecord.java:56)
at com.sun.proxy.$Proxy801.getContributorByUUID(Unknown Source)
at sun.reflect.GeneratedMethodAccessor3976.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:56)
at java.lang.reflect.Method.invoke(Method.java:620)
at com.ibm.team.repository.servlet.AbstractTeamServerServlet.doModelledRestService(AbstractTeamServerServlet.java:573)
at com.ibm.team.repository.servlet.AbstractTeamServerServlet.handleRequest2(AbstractTeamServerServlet.java:2524)
at com.ibm.team.repository.servlet.AbstractTeamServerServlet.handleRequest(AbstractTeamServerServlet.java:2315)
at com.ibm.team.repository.servlet.AbstractTeamServerServlet.service(AbstractTeamServerServlet.java:1794)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
at org.eclipse.equinox.http.registry.internal.ServletManager$ServletWrapper.service(ServletManager.java:180)
at org.eclipse.equinox.http.servlet.internal.ServletRegistration.service(ServletRegistration.java:61)
at org.eclipse.equinox.http.servlet.internal.ProxyServlet.processAlias(ProxyServlet.java:126)
at org.eclipse.equinox.http.servlet.internal.ProxyServlet.service(ProxyServlet.java:76)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
at org.eclipse.equinox.servletbridge.BridgeServlet.service(BridgeServlet.java:120)
at com.ibm.team.repository.server.servletbridge.JazzServlet.service(JazzServlet.java:74)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at com.ibm.team.repository.server.servletbridge.BridgeFilter.processDelegate(BridgeFilter.java:165)
at com.ibm.team.repository.server.servletbridge.BridgeFilter.doFilter(BridgeFilter.java:198)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:613)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
at org.apache.catalina.authenticator.SingleSignOn.invoke(SingleSignOn.java:358)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:421)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1074)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:611)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1157)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:627)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:809)
2017-06-26 13:45:50,596 [http-bio-9443-exec-11000 @@ 13:30 rcerkas@us.ibm.com <com.ibm.team.repository.editUser@b4c89a0b-b17e-41a6-89e9-1d0d0ce00284> /jts/service/com.ibm.team.repository.service.internal.IAdminRestService/contributorByUUID] ERROR ce.jts.internal.userregistry.ldap.LDAPUserRegistry - [9e5744f.2] CRJAZ0743I Error retrieving group for user.
2017-06-26 15:58:00,468 [http-bio-9443-exec-11192 @@ 15:42 rcerkas@us.ibm.com <com.ibm.team.repository.editUser@4be0e120-c2bf-4464-8f50-dadd778dc898> /jts/service/com.ibm.team.repository.service.internal.IAdminRestService/contributorByUUID] ERROR ce.jts.internal.userregistry.ldap.LDAPUserRegistry - [9e5744f.3] CRJAZ0743I Error retrieving group for user.
2017-06-29 11:47:21,297 [http-bio-9443-exec-13655 @@ 11:31 rcerkas@us.ibm.com <com.ibm.team.repository.editUser@75b15595-c6df-4068-87f4-8dfb5b1d487b> /jts/service/com.ibm.team.repository.service.internal.IAdminRestService/contributorByUUID] ERROR ce.jts.internal.userregistry.ldap.LDAPUserRegistry - [9e5744f] CRJAZ0743I Error retrieving group for user.
javax.naming.ServiceUnavailableException: bluepages.ibm.com:389; socket closed; remaining name 'ou=bluepages,o=ibm.com'
at com.sun.jndi.ldap.Connection.readReply(Connection.java:479)
at com.sun.jndi.ldap.LdapClient.getSearchReply(LdapClient.java:652)
at com.sun.jndi.ldap.LdapClient.search(LdapClient.java:575)
at com.sun.jndi.ldap.LdapCtx.doSearch(LdapCtx.java:1998)
at com.sun.jndi.ldap.LdapCtx.searchAux(LdapCtx.java:1860)
|
The question has been closed for the following reason: "Problem is not reproducible or outdated" by rschoon Mar 07 '22, 8:52 a.m.
Accepted answer
4 other answers
Hello,
Comments Please submit a new question rather than try to answer/comment on a 4+ year old question. See the forum etiquette.
Thanks David. I have closed this old question.
Adam, please open a new question with your log data. With the data given, no one will be able to answer anything.
You can also consider to open a case with support. Support can get the logs they need and the data is as safe as possible.
2
Enrique Gaona
commented Mar 07 '22, 9:51 a.m.
I know this thread has been closed but I thought I'd respond to Adam's question.
Adam,
The workaround I came up with, I added a cron job that would login (RTC commandline) to RTC every 5 minutes. Once I had this in place, the issue went away.
Ralph Schoon
commented Mar 07 '22, 10:17 a.m.
| edited Mar 07 '22, 10:18 a.m.
FORUM ADMINISTRATOR / FORUM MODERATOR / JAZZ DEVELOPER
The answer is fine, Enrique, thank you for that. In general there is a pattern that some users really go to town on searching related questions in the forum. Which I like in general, that should hopefully find hints you can follow.
Unfortunately some of the users follow up with answers on very old questions asking for updates and this is where this usually breaks. You usually do not get an answer and it is unlikely that the question is even related.
Plus it is actually also a pattern that spammers spam on old questions, so I always review changed items that have a lot of hits.
I think you have a work around, but that is not satisfactory, especially for major customers. That is why I would not accept any of the answers 8)
Something is configured incorrectly or there is a network or certificate error. This needs to be fixed. Hence support. |
Looking back to a now ancient (circa 2011) Tomcat Jazz configuration, this is the Realm snippet in use then:
|
The server.xml is basically identical to the server.xml from other RTC servers not having this issue. Here are the relevant contents of my server.xml:
|
I don't see this behavior for my dozens of Jazz applications also authenticating to the same LDAP. LDAP really ought to be configured to LDAPS ( port 636 ). One cannot infer your choice of J2EE container from the above (i.e. Tomcat, WAS or WAS Liberty ) so I can't advise further except to verify that you use these settings for lookups
Type: Custom
SSL Enabled: checked Base DN: o=ibm.com Values used for "Advanced LDAP registry" settings under the LDAP configuration
User Filter: (&(preferredIdentity=%v)(objectclass=ePerson))
Things to try on your server: nslookup bluepages.ibm.com host bluepages.ibm.com ping bluepages.ibm.com traceroute bluepages.ibm.com Most should return immediately; if there are delays in the data being returned, you might need to check the DNS server settings on your network stack. You could also try some ldapsearch commands from the machine hosting your front-end. Comments
Enrique Gaona
commented Jul 05 '17, 3:05 p.m.
Kevin,
Thanks for the suggestions. I have other RTC/RQM servers and they don't exhibit the same issue and are all running on Tomcat and configured LDAP the same way. The above commands, including ldapsearch, returns their data without any delay.
Here's a copy of the LDAP in my jts/teamserver.properties file.
com.ibm.team.repository.ldap.membersOfGroup=uniquemember
com.ibm.team.repository.ldap.baseUserDN=ou\=bluepages,o\=ibm.com
com.ibm.team.repository.ldap.userAttributesMapping=userId\=preferredidentity,name\=cn,emailAddress\=mail
com.ibm.team.repository.user.registry.type=LDAP
com.ibm.team.repository.ldap.registryLocation=ldap\://bluepages.ibm.com
com.ibm.team.repository.ldap.baseGroupDN=ou\=memberlist,ou\=ibmgroups,o\=ibm.com
com.ibm.team.repository.ldap.findGroupsForUserQuery=uniquemember\={USER-DN}
com.ibm.team.repository.ldap.findUsersByUserIdQuery=preferredidentity\=?1
Donald Nong
commented Jul 06 '17, 5:00 a.m.
I think you get confused. When authenticating, it is the application server, Tomcat in your case, to talk to the LDAP server. So the configuration in teamserver.properties is not being used at that time. For Tomcat, you should look into the server.xml file.
Enrique Gaona
commented Jul 07 '17, 5:41 p.m.
You're right! I compared the server.xml from another RTC server that does not have the long authentication issue and they're basically the same and posted the contents of the server.xml below. That said, I added JAVA_OPTS="$JAVA_OPTS -Dcom.sun.jndi.ldap.read.timeout=5000" to the server.startup script, which sets the timeout operation for LDAP operations.
|