Requirements data collection job failed
We have a all-in-one clm v.4.0.4 env. on windows 2008 server with oracle 11g back-end db. Authentication is using 8.0.0.3 federated repository
A week ago, jts logged the following error:
2014-05-09 10:46:46,223 [WebContainer : 1] ERROR com.ibm.team.repository - CRJAZ2654E The server was unable to connect to the database because the database connection settings are invalid or because database is unreachable (SQLSTATE: 99999, SQLCODE: 28001). Verify the connection settings in your teamserver.properties file. See the Jazz Team Server log for more information.
java.sql.SQLException: ORA-28001: the password has expired.
We resetted the oracle db user password and restarted the server. After the restart, all functionality returned to normal except that the requirements data collection job is now failing. All other data collection jobs are running OK with no error. It never failed before. All clm application db accounts including the data warehouse db account are ok..status is open. The etl-user userid/password which is used to run the dw jobs is ok and working as expected. The etl_user is a member of the jazzadmin and jazzdwadmin group. The XDC authentication type is set to FORM.
Snippet from the jts-etl-driver.log
2014-05-09 18:06:38,774 [Thread-198] ERROR tional.drivers.jdbc.xml.internal.PageFetcherThread - com.ibm.rational.dataservices.client.auth.AuthenticationException: com.ibm.rational.dataservices.client.auth.AuthenticationException: HTTP Status 500 Internal Server Error
2014-05-09 18:06:38,774 [Thread-198] ERROR tional.drivers.jdbc.xml.internal.PageFetcherThread - com.ibm.rational.etl.common.exception.ETLException: com.ibm.rational.dataservices.client.auth.AuthenticationException: com.ibm.rational.dataservices.client.auth.AuthenticationException: HTTP Status 500 Internal Server Errorcom.ibm.rational.etl.common.exception.ETLException: com.ibm.rational.etl.common.exception.ETLException: com.ibm.rational.dataservices.client.auth.AuthenticationException: com.ibm.rational.dataservices.client.auth.AuthenticationException: HTTP Status 500 Internal Server Error
at com.ibm.rational.drivers.jdbc.xml.internal.PageFetcherThread.exeFetch(PageFetcherThread.java:568)
at com.ibm.rational.drivers.jdbc.xml.internal.PageFetcherThread.run(PageFetcherThread.java:136)
Caused by: com.ibm.rational.etl.common.exception.ETLException: com.ibm.rational.dataservices.client.auth.AuthenticationException: com.ibm.rational.dataservices.client.auth.AuthenticationException: HTTP Status 500 Internal Server Error
at com.ibm.rational.drivers.jdbc.xml.internal.PageFetcherThread.exeFetch(PageFetcherThread.java:519)
.. 1 more
Caused by: com.ibm.rational.dataservices.client.auth.AuthenticationException: com.ibm.rational.dataservices.client.auth.AuthenticationException: HTTP Status 500 Internal Server Error
at com.ibm.rational.dataservices.client.util.URLDataProvider.execute(URLDataProvider.java:196)
at com.ibm.rational.dataservices.client.util.URLDataProvider.openDataStream(URLDataProvider.java:144)
at com.ibm.rational.dataservices.client.util.URLDataProvider.openDataStream(URLDataProvider.java:101)
at com.ibm.rational.drivers.jdbc.xml.internal.PageFetcherThread.exeFetch(PageFetcherThread.java:335)
... 1 more
Caused by: com.ibm.rational.dataservices.client.auth.AuthenticationException: HTTP Status 500 Internal Server Error
at com.ibm.rational.dataservices.client.util.URLDataProvider.execute(URLDataProvider.java:180)
... 4 more snippet from jts-etl-driver.log
Snippet from the jts-etl.log
ERROR ouse.service.rrc.internal.RemoteRRCSnapshotService - java.sql.SQLException: com.ibm.rational.etl.common.exception.ETLException:com.ibm.rational.dataservices.client.auth.AuthenticationException: com.ibm.rational.dataservices.client.auth.AuthenticationException: HTTP Status 500
Internal Server Error
WARN .ibm.team.datawarehouse.service.rrc.util.ETLTracer - Preceding event: 'CALL' from 'RRCRequirementETL::resetInitialCounts()'
2014-05-09 18:06:38,774 [jts: AsynchronousTaskRunner-0 @@ 17:59] WARN .ibm.team.datawarehouse.service.rrc.util.ETLTracer - Preceding event: 'CALL' from
'RRCCustomAttributeETL::initPrepareStatements(?)'
2014-05-09 18:06:38,774 [jts: AsynchronousTaskRunner-0 @@ 17:59] WARN .ibm.team.datawarehouse.service.rrc.util.ETLTracer - Preceding event: 'CALL' from
'RRCCustomAttributeETL::resetInitialCounts'
2014-05-09 18:06:38,774 [jts: AsynchronousTaskRunner-0 @@ 17:59] WARN .ibm.team.datawarehouse.service.rrc.util.ETLTracer - Preceding event: 'CALL' from
'RRCRequirementsLookupsETL::initPrepareStatements(?)'2014-05-09 18:06:38,774 [jts: AsynchronousTaskRunner-0 @@ 17:59] WARN .ibm.team.datawarehouse.service.rrc.util.ETLTracer - Preceding event: 'CALL' from
'RRCRequirementsLookupsETL::resetInitialCounts()'
2014-05-09 18:06:38,774 [jts: AsynchronousTaskRunner-0 @@ 17:59] WARN .ibm.team.datawarehouse.service.rrc.util.ETLTracer - Preceding event: 'CALL' from
'RRCExternalLinkETL::initPrepareStatements(?)'
2014-05-09 18:06:38,774 [jts: AsynchronousTaskRunner-0 @@ 17:59] WARN .ibm.team.datawarehouse.service.rrc.util.ETLTracer - Preceding event: 'CALL' from
'RRCExternalLinkETL::resetInitialCounts()'
2014-05-09 18:06:38,774 [jts: AsynchronousTaskRunner-0 @@ 17:59] WARN .ibm.team.datawarehouse.service.rrc.util.ETLTracer - Preceding event: 'Get page - RRC requirements' from 'RemoteRRCSnapshotService::updateSnapshot(?)' 2014-05-09 18:06:38,774 [jts: AsynchronousTaskRunner-0 @@ 17:59] WARN .ibm.team.datawarehouse.service.rrc.util.ETLTracer - Event subject: 0
2014-05-09 18:06:38,774 [jts: AsynchronousTaskRunner-0 @@ 17:59] WARN .ibm.team.datawarehouse.service.rrc.util.ETLTracer - Preceding event: 'CALL' from
'RemoteRRCSnapshotService::getNextResourcesPage(Connection con, String resourceGroupName, String modifiedSince, int pageSize, boolean linksOnly, boolean[] more'
2014-05-09 18:06:38,774 [jts: AsynchronousTaskRunner-0 @@ 17:59] WARN .ibm.team.datawarehouse.service.rrc.util.ETLTracer - Event subjects:
2014-05-09 18:06:38,774 [jts: AsynchronousTaskRunner-0 @@ 17:59] WARN .ibm.team.datawarehouse.service.rrc.util.ETLTracer -
2014-05-09 18:03:23,940 [WebContainer : 2]
ERROR ibm.rdm.fronting.server.rrs.publish.PublishService - Artifact _h_a5lMoeEeOEBLqIKjkBLg failed to load
2014-05-09 18:03:23,940 [WebContainer : 2] ERROR
ibm.rdm.fronting.server.rrs.publish.PublishService - java.lang.NullPointerException
2014-05-09 18:03:23,972 [WebContainer : 2]
ERROR
ibm.rdm.fronting.server.rrs.publish.PublishService - Server Error: Entry [d787213227e3b2b2]
com.ibm.rdm.fronting.server.exception.RRSException: CRRRS6527E Errors detected. No results available. Please check the log.
Any idea what the issue might be and how to resolve it?
FYI, I have enabled logging for the ETL driver in hopes that something more is revealed.
Thanks,
Pierre
A week ago, jts logged the following error:
2014-05-09 10:46:46,223 [WebContainer : 1] ERROR com.ibm.team.repository - CRJAZ2654E The server was unable to connect to the database because the database connection settings are invalid or because database is unreachable (SQLSTATE: 99999, SQLCODE: 28001). Verify the connection settings in your teamserver.properties file. See the Jazz Team Server log for more information.
java.sql.SQLException: ORA-28001: the password has expired.
We resetted the oracle db user password and restarted the server. After the restart, all functionality returned to normal except that the requirements data collection job is now failing. All other data collection jobs are running OK with no error. It never failed before. All clm application db accounts including the data warehouse db account are ok..status is open. The etl-user userid/password which is used to run the dw jobs is ok and working as expected. The etl_user is a member of the jazzadmin and jazzdwadmin group. The XDC authentication type is set to FORM.
Snippet from the jts-etl-driver.log
2014-05-09 18:06:38,774 [Thread-198] ERROR tional.drivers.jdbc.xml.internal.PageFetcherThread - com.ibm.rational.dataservices.client.auth.AuthenticationException: com.ibm.rational.dataservices.client.auth.AuthenticationException: HTTP Status 500 Internal Server Error
2014-05-09 18:06:38,774 [Thread-198] ERROR tional.drivers.jdbc.xml.internal.PageFetcherThread - com.ibm.rational.etl.common.exception.ETLException: com.ibm.rational.dataservices.client.auth.AuthenticationException: com.ibm.rational.dataservices.client.auth.AuthenticationException: HTTP Status 500 Internal Server Errorcom.ibm.rational.etl.common.exception.ETLException: com.ibm.rational.etl.common.exception.ETLException: com.ibm.rational.dataservices.client.auth.AuthenticationException: com.ibm.rational.dataservices.client.auth.AuthenticationException: HTTP Status 500 Internal Server Error
at com.ibm.rational.drivers.jdbc.xml.internal.PageFetcherThread.exeFetch(PageFetcherThread.java:568)
at com.ibm.rational.drivers.jdbc.xml.internal.PageFetcherThread.run(PageFetcherThread.java:136)
Caused by: com.ibm.rational.etl.common.exception.ETLException: com.ibm.rational.dataservices.client.auth.AuthenticationException: com.ibm.rational.dataservices.client.auth.AuthenticationException: HTTP Status 500 Internal Server Error
at com.ibm.rational.drivers.jdbc.xml.internal.PageFetcherThread.exeFetch(PageFetcherThread.java:519)
.. 1 more
Caused by: com.ibm.rational.dataservices.client.auth.AuthenticationException: com.ibm.rational.dataservices.client.auth.AuthenticationException: HTTP Status 500 Internal Server Error
at com.ibm.rational.dataservices.client.util.URLDataProvider.execute(URLDataProvider.java:196)
at com.ibm.rational.dataservices.client.util.URLDataProvider.openDataStream(URLDataProvider.java:144)
at com.ibm.rational.dataservices.client.util.URLDataProvider.openDataStream(URLDataProvider.java:101)
at com.ibm.rational.drivers.jdbc.xml.internal.PageFetcherThread.exeFetch(PageFetcherThread.java:335)
... 1 more
Caused by: com.ibm.rational.dataservices.client.auth.AuthenticationException: HTTP Status 500 Internal Server Error
at com.ibm.rational.dataservices.client.util.URLDataProvider.execute(URLDataProvider.java:180)
... 4 more snippet from jts-etl-driver.log
Snippet from the jts-etl.log
ERROR ouse.service.rrc.internal.RemoteRRCSnapshotService - java.sql.SQLException: com.ibm.rational.etl.common.exception.ETLException:com.ibm.rational.dataservices.client.auth.AuthenticationException: com.ibm.rational.dataservices.client.auth.AuthenticationException: HTTP Status 500
Internal Server Error
WARN .ibm.team.datawarehouse.service.rrc.util.ETLTracer - Preceding event: 'CALL' from 'RRCRequirementETL::resetInitialCounts()'
2014-05-09 18:06:38,774 [jts: AsynchronousTaskRunner-0 @@ 17:59] WARN .ibm.team.datawarehouse.service.rrc.util.ETLTracer - Preceding event: 'CALL' from
'RRCCustomAttributeETL::initPrepareStatements(?)'
2014-05-09 18:06:38,774 [jts: AsynchronousTaskRunner-0 @@ 17:59] WARN .ibm.team.datawarehouse.service.rrc.util.ETLTracer - Preceding event: 'CALL' from
'RRCCustomAttributeETL::resetInitialCounts'
2014-05-09 18:06:38,774 [jts: AsynchronousTaskRunner-0 @@ 17:59] WARN .ibm.team.datawarehouse.service.rrc.util.ETLTracer - Preceding event: 'CALL' from
'RRCRequirementsLookupsETL::initPrepareStatements(?)'2014-05-09 18:06:38,774 [jts: AsynchronousTaskRunner-0 @@ 17:59] WARN .ibm.team.datawarehouse.service.rrc.util.ETLTracer - Preceding event: 'CALL' from
'RRCRequirementsLookupsETL::resetInitialCounts()'
2014-05-09 18:06:38,774 [jts: AsynchronousTaskRunner-0 @@ 17:59] WARN .ibm.team.datawarehouse.service.rrc.util.ETLTracer - Preceding event: 'CALL' from
'RRCExternalLinkETL::initPrepareStatements(?)'
2014-05-09 18:06:38,774 [jts: AsynchronousTaskRunner-0 @@ 17:59] WARN .ibm.team.datawarehouse.service.rrc.util.ETLTracer - Preceding event: 'CALL' from
'RRCExternalLinkETL::resetInitialCounts()'
2014-05-09 18:06:38,774 [jts: AsynchronousTaskRunner-0 @@ 17:59] WARN .ibm.team.datawarehouse.service.rrc.util.ETLTracer - Preceding event: 'Get page - RRC requirements' from 'RemoteRRCSnapshotService::updateSnapshot(?)' 2014-05-09 18:06:38,774 [jts: AsynchronousTaskRunner-0 @@ 17:59] WARN .ibm.team.datawarehouse.service.rrc.util.ETLTracer - Event subject: 0
2014-05-09 18:06:38,774 [jts: AsynchronousTaskRunner-0 @@ 17:59] WARN .ibm.team.datawarehouse.service.rrc.util.ETLTracer - Preceding event: 'CALL' from
'RemoteRRCSnapshotService::getNextResourcesPage(Connection con, String resourceGroupName, String modifiedSince, int pageSize, boolean linksOnly, boolean[] more'
2014-05-09 18:06:38,774 [jts: AsynchronousTaskRunner-0 @@ 17:59] WARN .ibm.team.datawarehouse.service.rrc.util.ETLTracer - Event subjects:
2014-05-09 18:06:38,774 [jts: AsynchronousTaskRunner-0 @@ 17:59] WARN .ibm.team.datawarehouse.service.rrc.util.ETLTracer -
com.ibm.rational.drivers.jdbc.xml.RDSConnection@27a8f1ab
2014-05-09 18:06:38,774 [jts: AsynchronousTaskRunner-0 @@ 17:59] WARN .ibm.team.datawarehouse.service.rrc.util.ETLTracer - RRC
2014-05-09 18:06:38,774 [jts: AsynchronousTaskRunner-0 @@ 17:59] WARN .ibm.team.datawarehouse.service.rrc.util.ETLTracer - 2014-05-05 00:00:00.0
2014-05-09 18:06:38,774 [jts: AsynchronousTaskRunner-0 @@ 17:59] WARN .ibm.team.datawarehouse.service.rrc.util.ETLTracer - 100
2014-05-09 18:06:38,774 [jts: AsynchronousTaskRunner-0 @@ 17:59] WARN .ibm.team.datawarehouse.service.rrc.util.ETLTracer - [Z@27b727f4
2014-05-09 18:06:38,774 [jts: AsynchronousTaskRunner-0 @@ 17:59] WARN .ibm.team.datawarehouse.service.rrc.util.ETLTracer - Suspect event: 'Querying the XDC table' from 'RemoteRRCSnapshotService::getNextResPage(?,...)'
Snippet from rm.log2014-05-09 18:03:23,940 [WebContainer : 2]
ERROR ibm.rdm.fronting.server.rrs.publish.PublishService - Artifact _h_a5lMoeEeOEBLqIKjkBLg failed to load
2014-05-09 18:03:23,940 [WebContainer : 2] ERROR
ibm.rdm.fronting.server.rrs.publish.PublishService - java.lang.NullPointerException
2014-05-09 18:03:23,972 [WebContainer : 2]
ERROR
ibm.rdm.fronting.server.rrs.publish.PublishService - Server Error: Entry [d787213227e3b2b2]
com.ibm.rdm.fronting.server.exception.RRSException: CRRRS6527E Errors detected. No results available. Please check the log.
Any idea what the issue might be and how to resolve it?
FYI, I have enabled logging for the ETL driver in hopes that something more is revealed.
Thanks,
Pierre
Accepted answer
Hi, Pierre
It seems that you have checked a lot of things already. But the error
==
ETLException:com.ibm.rational.dataservices.client.auth.AuthenticationException: com.ibm.rational.dataservices.client.auth.AuthenticationException: HTTP Status 500
Internal Server Error
==
does point to etl user authentication problem.
There are two things you may want to try:
1) go to
jts/admin > reports > data collection jobs and see if you have set the correct user and password for Requirement job, re-enter the correct user and password(you can test with the user and password by login with it to the admin page and you should be able to login). Specifically password in case something could be wrong
2) Go to jts > user list and check with etl-user's license.
If Data collector license is checked, it is fine but if you use QM/CCM/RM data collector license instead, you need to check 3 of them, see if RM data collector license should be checked.
Thanks
It seems that you have checked a lot of things already. But the error
==
ETLException:com.ibm.rational.dataservices.client.auth.AuthenticationException: com.ibm.rational.dataservices.client.auth.AuthenticationException: HTTP Status 500
Internal Server Error
==
does point to etl user authentication problem.
There are two things you may want to try:
1) go to
jts/admin > reports > data collection jobs and see if you have set the correct user and password for Requirement job, re-enter the correct user and password(you can test with the user and password by login with it to the admin page and you should be able to login). Specifically password in case something could be wrong
2) Go to jts > user list and check with etl-user's license.
If Data collector license is checked, it is fine but if you use QM/CCM/RM data collector license instead, you need to check 3 of them, see if RM data collector license should be checked.
Thanks
Comments
Hello,
I forgot to add
1) I did verify the user and password for the requirements job. Everything is ok there.
2) The etl_user has all 3 licenses -qm,ccm,rm - assigned to him. Everything is ok there.
Now, for some odd reason, after last night server restart, I checked the server this morning and noticed that the requirements data collection job PASSED. There are no more ETL errors in the logs. I find it strange that the job would all of a sudden run normally after a week of failed results. Since we stop/start the server every single night, I would of expected the problem to fix itself when the issue first came up a week ago.
Thanks,