Jazz Forum Welcome to the Jazz Community Forum Connect and collaborate with IBM Engineering experts and users

Serious LDAP Problems

I installed 3.0.1 RC2 a month ago from scratch and while configuring the LDAP at some point I must have put in my credentials. Things seemed to be fine until May 6 when I changed my LDAP password because we have to change our passwords every three months.

The next day I could no longer login because there were too many failed attempts to login on my account. This went on for about a week until our Identity Services staff, who maintain the LDAP service, investigated are reported that some process on my computer was making thousands of attempts to authenticate with the LDAP server using my account, but the wrong password, and this was happening every night at midnight.

I quickly assumed it was the Jazz Team Server, and I assumed it was the nightly LDAP synchronization because I could see that failing in the jts.log file, so I disabled that. But the next day the problem happened again and the LDAP synchronization was still running. So then I restarted the JTS server - I guess just disabling things in the Admin UI is not good enough. Then the next day the problem happened again, my account was locked out. However, the jts.log showed that the nightly LDAP synchronization was not running, something else was causing the problem.

Finally I checked the catalina.log files and sure enough I could see LDAP requests failing. Looking back over time from the beginning I could see the following in the log:

Apr 15, 2011 3:40:03 PM org.apache.catalina.realm.JNDIRealm authenticate

WARNING: Exception performing authentication
javax.naming.CommunicationException: connection closed [Root exception is java.io.IOException: connection closed]; remaining name 'ou=people,o=Kodak,c=US'
at com.sun.jndi.ldap.LdapCtx.doSearch(LdapCtx.java:2002)
at com.sun.jndi.ldap.LdapCtx.searchAux(LdapCtx.java:1847)
at com.sun.jndi.ldap.LdapCtx.c_search(LdapCtx.java:1772)
at com.sun.jndi.toolkit.ctx.ComponentDirContext.p_search(ComponentDirContext.java:383)
at com.sun.jndi.toolkit.ctx.PartialCompositeDirContext.search(PartialCompositeDirContext.java:353)
at com.sun.jndi.toolkit.ctx.PartialCompositeDirContext.search(PartialCompositeDirContext.java:336)
at javax.naming.directory.InitialDirContext.search(InitialDirContext.java:259)
at org.apache.catalina.realm.JNDIRealm.getUserBySearch(JNDIRealm.java:1069)
at org.apache.catalina.realm.JNDIRealm.getUser(JNDIRealm.java:977)
at org.apache.catalina.realm.JNDIRealm.authenticate(JNDIRealm.java:926)
at org.apache.catalina.realm.JNDIRealm.authenticate(JNDIRealm.java:812)
at org.apache.catalina.authenticator.FormAuthenticator.authenticate(FormAuthenticator.java:259)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:454)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
at org.apache.catalina.authenticator.SingleSignOn.invoke(SingleSignOn.java:420)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:879)
at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665)
at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528)
at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)
at java.lang.Thread.run(Thread.java:811)
Caused by: java.io.IOException: connection closed
at com.sun.jndi.ldap.LdapClient.ensureOpen(LdapClient.java:1590)
at com.sun.jndi.ldap.LdapClient.search(LdapClient.java:536)
at com.sun.jndi.ldap.LdapCtx.doSearch(LdapCtx.java:1985)
... 23 more

I hope people can appreciate that this message is not very helpful in solving the problem - but it is just a warning. Looking through the logs further I can see there is one of these messages every night at midnight, but on May 6 I started seeing

May 6, 2011 12:00:02 AM org.apache.catalina.realm.JNDIRealm authenticate

SEVERE: Exception performing authentication
javax.naming.directory.InvalidAttributeValueException: [LDAP: error code 19 - Exceed password retry limit. Contact system administrator to reset.]; remaining name ''
at com.sun.jndi.ldap.LdapCtx.mapErrorCode(LdapCtx.java:3058)
at com.sun.jndi.ldap.LdapCtx.processReturnCode(LdapCtx.java:2991)
at com.sun.jndi.ldap.LdapCtx.processReturnCode(LdapCtx.java:2792)
at com.sun.jndi.ldap.LdapCtx.connect(LdapCtx.java:2706)
at com.sun.jndi.ldap.LdapCtx.ensureOpen(LdapCtx.java:2605)
at com.sun.jndi.ldap.LdapCtx.ensureOpen(LdapCtx.java:2579)
at com.sun.jndi.ldap.LdapCtx.doSearch(LdapCtx.java:1942)
at com.sun.jndi.ldap.LdapCtx.doSearchOnce(LdapCtx.java:1934)
at com.sun.jndi.ldap.LdapCtx.c_getAttributes(LdapCtx.java:1327)
at com.sun.jndi.toolkit.ctx.ComponentDirContext.p_getAttributes(ComponentDirContext.java:228)
at com.sun.jndi.toolkit.ctx.PartialCompositeDirContext.getAttributes(PartialCompositeDirContext.java:136)
at com.sun.jndi.toolkit.ctx.PartialCompositeDirContext.getAttributes(PartialCompositeDirContext.java:124)
at javax.naming.directory.InitialDirContext.getAttributes(InitialDirContext.java:134)
at org.apache.catalina.realm.JNDIRealm.bindAsUser(JNDIRealm.java:1282)
at org.apache.catalina.realm.JNDIRealm.checkCredentials(JNDIRealm.java:1135)
at org.apache.catalina.realm.JNDIRealm.authenticate(JNDIRealm.java:931)
at org.apache.catalina.realm.JNDIRealm.authenticate(JNDIRealm.java:812)
at org.apache.catalina.authenticator.FormAuthenticator.authenticate(FormAuthenticator.java:259)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:454)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
at org.apache.catalina.authenticator.SingleSignOn.invoke(SingleSignOn.java:420)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:879)
at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665)
at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528)
at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)
at java.lang.Thread.run(Thread.java:811)

May 5 was when I changed my password. And the log is filled with thousands of these messages as what ever software is trying to authenticate against our LDAP service clearly does not know when to give up, but it is using my LDAP account, which forces me to have to get a new password every morning since May 6.

I have gone through the Admin UI for the server and I can find nothing linking my account. I have gone through all the configuration files on the Jazz server and Tomcat and still can find nothing linking my account to this.

Could someone please tell me what this stupid process is, and how I can fix it.

Cheers, Eric

0 votes



One answer

Permanent link
OK, the offending process turned out the be the data warehouse provider configuration. This was not at all obvious because when I went to the advanced settings in the Server Administration the Data Collection User was set to some other account, but not mine.

Finally I had to use Visual Studio to search the server configuration files - because Windows Explorer Search is completely useless. I found my personal ID was set all over the place in the various properies files and their backups.

I hand edited all the files to remove my personal ID - I just hope some other process does not have my personal ID stashed someplace (like the database) and will try to reset it.

Cheers, Eric

0 votes

Your answer

Register or log in to post your answer.

Dashboards and work items are no longer publicly available, so some links may be invalid. We now provide similar information through other means. Learn more here.

Search context
Follow this question

By Email: 

Once you sign in you will be able to subscribe for any updates here.

By RSS:

Answers
Answers and Comments
Question details

Question asked: May 14 '11, 12:10 p.m.

Question was seen: 6,938 times

Last updated: May 14 '11, 12:10 p.m.

Confirmation Cancel Confirm