I installed 3.0.1 RC2 a month ago from scratch and while configuring the LDAP at some point I must have put in my credentials. Things seemed to be fine until May 6 when I changed my LDAP password because we have to change our passwords every three months.
The next day I could no longer login because there were too many failed attempts to login on my account. This went on for about a week until our Identity Services staff, who maintain the LDAP service, investigated are reported that some process on my computer was making thousands of attempts to authenticate with the LDAP server using my account, but the wrong password, and this was happening every night at midnight.
I quickly assumed it was the Jazz Team Server, and I assumed it was the nightly LDAP synchronization because I could see that failing in the jts.log file, so I disabled that. But the next day the problem happened again and the LDAP synchronization was still running. So then I restarted the JTS server - I guess just disabling things in the Admin UI is not good enough. Then the next day the problem happened again, my account was locked out. However, the jts.log showed that the nightly LDAP synchronization was not running, something else was causing the problem.
Finally I checked the catalina.log files and sure enough I could see LDAP requests failing. Looking back over time from the beginning I could see the following in the log:
Apr 15, 2011 3:40:03 PM org.apache.catalina.realm.JNDIRealm authenticate
WARNING: Exception performing authentication
javax.naming.CommunicationException: connection closed [Root exception is java.io.IOException: connection closed]; remaining name 'ou=people,o=Kodak,c=US'
at com.sun.jndi.ldap.LdapCtx.doSearch(LdapCtx.java:2002)
at com.sun.jndi.ldap.LdapCtx.searchAux(LdapCtx.java:1847)
at com.sun.jndi.ldap.LdapCtx.c_search(LdapCtx.java:1772)
at com.sun.jndi.toolkit.ctx.ComponentDirContext.p_search(ComponentDirContext.java:383)
at com.sun.jndi.toolkit.ctx.PartialCompositeDirContext.search(PartialCompositeDirContext.java:353)
at com.sun.jndi.toolkit.ctx.PartialCompositeDirContext.search(PartialCompositeDirContext.java:336)
at javax.naming.directory.InitialDirContext.search(InitialDirContext.java:259)
at org.apache.catalina.realm.JNDIRealm.getUserBySearch(JNDIRealm.java:1069)
at org.apache.catalina.realm.JNDIRealm.getUser(JNDIRealm.java:977)
at org.apache.catalina.realm.JNDIRealm.authenticate(JNDIRealm.java:926)
at org.apache.catalina.realm.JNDIRealm.authenticate(JNDIRealm.java:812)
at org.apache.catalina.authenticator.FormAuthenticator.authenticate(FormAuthenticator.java:259)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:454)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
at org.apache.catalina.authenticator.SingleSignOn.invoke(SingleSignOn.java:420)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:879)
at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665)
at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528)
at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)
at java.lang.Thread.run(Thread.java:811)
Caused by: java.io.IOException: connection closed
at com.sun.jndi.ldap.LdapClient.ensureOpen(LdapClient.java:1590)
at com.sun.jndi.ldap.LdapClient.search(LdapClient.java:536)
at com.sun.jndi.ldap.LdapCtx.doSearch(LdapCtx.java:1985)
... 23 more
I hope people can appreciate that this message is not very helpful in solving the problem - but it is just a warning. Looking through the logs further I can see there is one of these messages every night at midnight, but on May 6 I started seeing
May 6, 2011 12:00:02 AM org.apache.catalina.realm.JNDIRealm authenticate
SEVERE: Exception performing authentication
javax.naming.directory.InvalidAttributeValueException: [LDAP: error code 19 - Exceed password retry limit. Contact system administrator to reset.]; remaining name ''
at com.sun.jndi.ldap.LdapCtx.mapErrorCode(LdapCtx.java:3058)
at com.sun.jndi.ldap.LdapCtx.processReturnCode(LdapCtx.java:2991)
at com.sun.jndi.ldap.LdapCtx.processReturnCode(LdapCtx.java:2792)
at com.sun.jndi.ldap.LdapCtx.connect(LdapCtx.java:2706)
at com.sun.jndi.ldap.LdapCtx.ensureOpen(LdapCtx.java:2605)
at com.sun.jndi.ldap.LdapCtx.ensureOpen(LdapCtx.java:2579)
at com.sun.jndi.ldap.LdapCtx.doSearch(LdapCtx.java:1942)
at com.sun.jndi.ldap.LdapCtx.doSearchOnce(LdapCtx.java:1934)
at com.sun.jndi.ldap.LdapCtx.c_getAttributes(LdapCtx.java:1327)
at com.sun.jndi.toolkit.ctx.ComponentDirContext.p_getAttributes(ComponentDirContext.java:228)
at com.sun.jndi.toolkit.ctx.PartialCompositeDirContext.getAttributes(PartialCompositeDirContext.java:136)
at com.sun.jndi.toolkit.ctx.PartialCompositeDirContext.getAttributes(PartialCompositeDirContext.java:124)
at javax.naming.directory.InitialDirContext.getAttributes(InitialDirContext.java:134)
at org.apache.catalina.realm.JNDIRealm.bindAsUser(JNDIRealm.java:1282)
at org.apache.catalina.realm.JNDIRealm.checkCredentials(JNDIRealm.java:1135)
at org.apache.catalina.realm.JNDIRealm.authenticate(JNDIRealm.java:931)
at org.apache.catalina.realm.JNDIRealm.authenticate(JNDIRealm.java:812)
at org.apache.catalina.authenticator.FormAuthenticator.authenticate(FormAuthenticator.java:259)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:454)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
at org.apache.catalina.authenticator.SingleSignOn.invoke(SingleSignOn.java:420)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:879)
at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665)
at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528)
at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)
at java.lang.Thread.run(Thread.java:811)
May 5 was when I changed my password. And the log is filled with thousands of these messages as what ever software is trying to authenticate against our LDAP service clearly does not know when to give up, but it is using my LDAP account, which forces me to have to get a new password every morning since May 6.
I have gone through the Admin UI for the server and I can find nothing linking my account. I have gone through all the configuration files on the Jazz server and Tomcat and still can find nothing linking my account to this.
Could someone please tell me what this stupid process is, and how I can fix it.
Cheers, Eric
