It's all about the answers!

Ask a question
QuestionsTagsUsersBadges
Guidelines | FAQ
Follow this question
Question details

×6,035
×4,321
×3,952

question asked: May 13 '14, 3:20 p.m.
question was seen: 4,691 times
last updated: May 14 '14, 9:50 a.m.

Related questions

Requirements data collection job failed

0
Pierre Friend (19●1●9●15) | asked May 13 '14, 3:20 p.m.
We have a all-in-one clm v.4.0.4 env. on windows 2008 server with oracle 11g back-end db. Authentication is using 8.0.0.3 federated repository

A week ago, jts logged the following error:
2014-05-09 10:46:46,223 [WebContainer : 1] ERROR com.ibm.team.repository - CRJAZ2654E The server was unable to connect to the database because the database connection settings are invalid or because database is unreachable (SQLSTATE: 99999, SQLCODE: 28001). Verify the connection settings in your teamserver.properties file. See the Jazz Team Server log for more information.
java.sql.SQLException: ORA-28001: the password has expired.

We resetted the oracle db user password and restarted the server. After the restart, all functionality returned to normal except that the requirements data collection job is now failing. All other data collection jobs are running OK with no error. It never failed before. All clm application db accounts including the data warehouse db account are ok..status is open. The etl-user userid/password which is used to run the dw jobs is ok and working as expected. The etl_user is a member of the jazzadmin and jazzdwadmin group. The XDC authentication type is set to FORM.

Snippet from the jts-etl-driver.log
2014-05-09 18:06:38,774 [Thread-198] ERROR tional.drivers.jdbc.xml.internal.PageFetcherThread  - com.ibm.rational.dataservices.client.auth.AuthenticationException: com.ibm.rational.dataservices.client.auth.AuthenticationException: HTTP Status 500 Internal Server Error
2014-05-09 18:06:38,774 [Thread-198] ERROR tional.drivers.jdbc.xml.internal.PageFetcherThread  - com.ibm.rational.etl.common.exception.ETLException: com.ibm.rational.dataservices.client.auth.AuthenticationException: com.ibm.rational.dataservices.client.auth.AuthenticationException: HTTP Status 500 Internal Server Errorcom.ibm.rational.etl.common.exception.ETLException: com.ibm.rational.etl.common.exception.ETLException: com.ibm.rational.dataservices.client.auth.AuthenticationException: com.ibm.rational.dataservices.client.auth.AuthenticationException: HTTP Status 500 Internal Server Error
at com.ibm.rational.drivers.jdbc.xml.internal.PageFetcherThread.exeFetch(PageFetcherThread.java:568)
at com.ibm.rational.drivers.jdbc.xml.internal.PageFetcherThread.run(PageFetcherThread.java:136)
Caused by: com.ibm.rational.etl.common.exception.ETLException: com.ibm.rational.dataservices.client.auth.AuthenticationException: com.ibm.rational.dataservices.client.auth.AuthenticationException: HTTP Status 500 Internal Server Error
at com.ibm.rational.drivers.jdbc.xml.internal.PageFetcherThread.exeFetch(PageFetcherThread.java:519)
.. 1 more
Caused by: com.ibm.rational.dataservices.client.auth.AuthenticationException: com.ibm.rational.dataservices.client.auth.AuthenticationException: HTTP Status 500 Internal Server Error
at com.ibm.rational.dataservices.client.util.URLDataProvider.execute(URLDataProvider.java:196)
at com.ibm.rational.dataservices.client.util.URLDataProvider.openDataStream(URLDataProvider.java:144)
at com.ibm.rational.dataservices.client.util.URLDataProvider.openDataStream(URLDataProvider.java:101)
at com.ibm.rational.drivers.jdbc.xml.internal.PageFetcherThread.exeFetch(PageFetcherThread.java:335)
... 1 more
Caused by: com.ibm.rational.dataservices.client.auth.AuthenticationException: HTTP Status 500 Internal Server Error
at com.ibm.rational.dataservices.client.util.URLDataProvider.execute(URLDataProvider.java:180)
... 4 more snippet from jts-etl-driver.log

Snippet from the jts-etl.log
ERROR ouse.service.rrc.internal.RemoteRRCSnapshotService  - java.sql.SQLException: com.ibm.rational.etl.common.exception.ETLException:com.ibm.rational.dataservices.client.auth.AuthenticationException: com.ibm.rational.dataservices.client.auth.AuthenticationException: HTTP Status 500
Internal Server Error
WARN .ibm.team.datawarehouse.service.rrc.util.ETLTracer  - Preceding event: 'CALL' from 'RRCRequirementETL::resetInitialCounts()'
2014-05-09 18:06:38,774 [jts: AsynchronousTaskRunner-0 @@ 17:59]  WARN .ibm.team.datawarehouse.service.rrc.util.ETLTracer  - Preceding event: 'CALL' from
'RRCCustomAttributeETL::initPrepareStatements(?)'
2014-05-09 18:06:38,774 [jts: AsynchronousTaskRunner-0 @@ 17:59]  WARN .ibm.team.datawarehouse.service.rrc.util.ETLTracer  - Preceding event: 'CALL' from
'RRCCustomAttributeETL::resetInitialCounts'
2014-05-09 18:06:38,774 [jts: AsynchronousTaskRunner-0 @@ 17:59]  WARN .ibm.team.datawarehouse.service.rrc.util.ETLTracer  - Preceding event: 'CALL' from
'RRCRequirementsLookupsETL::initPrepareStatements(?)'2014-05-09 18:06:38,774 [jts: AsynchronousTaskRunner-0 @@ 17:59]  WARN .ibm.team.datawarehouse.service.rrc.util.ETLTracer  - Preceding event: 'CALL' from
'RRCRequirementsLookupsETL::resetInitialCounts()'
2014-05-09 18:06:38,774 [jts: AsynchronousTaskRunner-0 @@ 17:59]  WARN .ibm.team.datawarehouse.service.rrc.util.ETLTracer  - Preceding event: 'CALL' from
'RRCExternalLinkETL::initPrepareStatements(?)'
2014-05-09 18:06:38,774 [jts: AsynchronousTaskRunner-0 @@ 17:59]  WARN .ibm.team.datawarehouse.service.rrc.util.ETLTracer  - Preceding event: 'CALL' from
'RRCExternalLinkETL::resetInitialCounts()'
2014-05-09 18:06:38,774 [jts: AsynchronousTaskRunner-0 @@ 17:59]  WARN .ibm.team.datawarehouse.service.rrc.util.ETLTracer  - Preceding event: 'Get page - RRC requirements' from 'RemoteRRCSnapshotService::updateSnapshot(?)' 2014-05-09 18:06:38,774 [jts: AsynchronousTaskRunner-0 @@ 17:59]  WARN .ibm.team.datawarehouse.service.rrc.util.ETLTracer  - Event subject: 0
2014-05-09 18:06:38,774 [jts: AsynchronousTaskRunner-0 @@ 17:59]  WARN .ibm.team.datawarehouse.service.rrc.util.ETLTracer  - Preceding event: 'CALL' from
'RemoteRRCSnapshotService::getNextResourcesPage(Connection con, String resourceGroupName, String modifiedSince, int pageSize, boolean linksOnly, boolean[] more'
2014-05-09 18:06:38,774 [jts: AsynchronousTaskRunner-0 @@ 17:59]  WARN .ibm.team.datawarehouse.service.rrc.util.ETLTracer  - Event subjects:
2014-05-09 18:06:38,774 [jts: AsynchronousTaskRunner-0 @@ 17:59]  WARN .ibm.team.datawarehouse.service.rrc.util.ETLTracer  -   

com.ibm.rational.drivers.jdbc.xml.RDSConnection@27a8f1ab

2014-05-09 18:06:38,774 [jts: AsynchronousTaskRunner-0 @@ 17:59]  WARN .ibm.team.datawarehouse.service.rrc.util.ETLTracer  - RRC

2014-05-09 18:06:38,774 [jts: AsynchronousTaskRunner-0 @@ 17:59]  WARN .ibm.team.datawarehouse.service.rrc.util.ETLTracer  - 2014-05-05 00:00:00.0

2014-05-09 18:06:38,774 [jts: AsynchronousTaskRunner-0 @@ 17:59]  WARN .ibm.team.datawarehouse.service.rrc.util.ETLTracer  - 100

2014-05-09 18:06:38,774 [jts: AsynchronousTaskRunner-0 @@ 17:59]  WARN .ibm.team.datawarehouse.service.rrc.util.ETLTracer  - [Z@27b727f4

2014-05-09 18:06:38,774 [jts: AsynchronousTaskRunner-0 @@ 17:59]  WARN .ibm.team.datawarehouse.service.rrc.util.ETLTracer  - Suspect event: 'Querying the XDC table' from 'RemoteRRCSnapshotService::getNextResPage(?,...)'

Snippet from rm.log
2014-05-09 18:03:23,940 [WebContainer : 2]
ERROR ibm.rdm.fronting.server.rrs.publish.PublishService  - Artifact _h_a5lMoeEeOEBLqIKjkBLg failed to load
2014-05-09 18:03:23,940 [WebContainer : 2] ERROR
ibm.rdm.fronting.server.rrs.publish.PublishService  - java.lang.NullPointerException
2014-05-09 18:03:23,972 [WebContainer : 2]
ERROR
ibm.rdm.fronting.server.rrs.publish.PublishService  - Server Error: Entry [d787213227e3b2b2]
com.ibm.rdm.fronting.server.exception.RRSException: CRRRS6527E  Errors detected. No results available. Please check the log.

Any idea what the issue might be and how to resolve it?

FYI, I have enabled logging for the ETL driver in hopes that something more is revealed.

Thanks,
Pierre

Accepted answer

0
permanent link
Don Yang (7.7k●21●114●139) | answered May 14 '14, 12:15 a.m.
Hi, Pierre

It seems that you have checked a lot of things already. But the error
==
ETLException:com.ibm.rational.dataservices.client.auth.AuthenticationException: com.ibm.rational.dataservices.client.auth.AuthenticationException: HTTP Status 500
Internal Server Error
==
does point to etl user authentication problem.
There are two things you may want to try:

1) go to
jts/admin > reports > data collection jobs and see if you have set the correct user and password for Requirement job, re-enter the correct user and password(you can test with the user and password by login with it to the admin page and you should be able to login). Specifically password in case something could be wrong
2) Go to jts > user list and check with etl-user's license.
If Data collector license is checked, it is fine but if you use QM/CCM/RM data collector license instead, you need to check 3 of them, see if RM data collector license should be checked.

Thanks

Pierre Friend selected this answer as the correct answer
Comments
Pierre Friend commented May 14 '14, 9:50 a.m.

Hello,
I forgot to add
1) I did verify the user and password for the requirements job. Everything is ok there.
2) The etl_user has all 3 licenses -qm,ccm,rm - assigned to him. Everything is ok there.

Now, for some odd reason, after last night server restart, I checked the server this morning and noticed that the requirements data collection job PASSED. There are no more ETL errors in the logs. I find it strange that the job would all of a sudden run normally after a week of failed results. Since we stop/start the server every single night, I would of expected the problem to fix itself when the issue first came up a week ago.
Thanks,

Your answer

Register or to post your answer.


Dashboards and work items are no longer publicly available, so some links may be invalid. We now provide similar information through other means. Learn more here.