It's all about the answers!

Ask a question

Why is the Star Data Collection Job taking longer every day?


Scott Crouch (48532426) | asked Jul 17 '12, 9:01 a.m.
I was wondering if anybody has any ideas why the STAR ETL job is taking longer and longer each day. In the past month it's gone from taking 5 hours, to now over 7 hours.

I can see in the jts-etl.log file there is a significant jump in the log when it's doing something. Is there any way to know why it's taking so long? I pasted the snippet from the log file where it seems to be taking it's time (notice the 7 hour jump in log entries).

2012-07-17 00:39:55,636 [ jts: AsynchronousTaskRunner-4]  INFO ervice.starjobs.internal.StarRemoteSnapshotService  - ***********com.ibm.team.datawarehouse.service.starjobs.internal.factbuild.RequCreatiTplanD
ateMetricsFactETL********************
2012-07-17 00:39:55,636 [ jts: AsynchronousTaskRunner-4]  INFO ervice.starjobs.internal.StarRemoteSnapshotService  - Total Records Selected = 22960
2012-07-17 00:39:55,636 [ jts: AsynchronousTaskRunner-4]  INFO ervice.starjobs.internal.StarRemoteSnapshotService  - Total Records Inserted into DataWarehouse = 22960
2012-07-17 00:39:55,636 [ jts: AsynchronousTaskRunner-4]  INFO ervice.starjobs.internal.StarRemoteSnapshotService  - *************************************************************
2012-07-17 00:39:55,636 [ jts: AsynchronousTaskRunner-4]  INFO ervice.starjobs.internal.StarRemoteSnapshotService  - Java ETL (STAR): Fact build "com.ibm.team.datawarehouse.service.starjobs.internal.factbuil
d.RequCreatiTplanDateMetricsFactETL" passed
2012-07-17 00:39:55,637 [ jts: AsynchronousTaskRunner-4]  INFO ervice.starjobs.internal.StarRemoteSnapshotService  - In com.ibm.team.datawarehouse.service.starjobs.internal.factbuild.RequestTestPlanMetrics
2012-07-17 00:39:55,637 [ jts: AsynchronousTaskRunner-4]  INFO ervice.starjobs.internal.StarRemoteSnapshotService  - SELECT  DISTINCT a."REQUEST_ID", a."REQUEST_CLASS_ID",a."REQUEST_CATEGORY_ID", a."PROJECT_
ID", a."COMPONENT_ID", a."TEAM_ID", a."REQUEST_SEVERITY_ID",  a."REQUEST_STATE_ID", a."REQUEST_STATUS_ID", a."REQUEST_PRIORITY_ID", UPPER(a."PLATFORM") AS "PLATFORM", COALESCE(D."ITERATION_ID",-1) AS "ITERAT
ION_ID",  a."RELEASE_ID", a."FIXEDINRELEASE_ID", a."IS_BLOCKING", a."OWNER_ID", a."REQUEST_TYPE_ID", COALESCE(D."TESTPLAN_ID",-1) AS "TESTPLAN_ID" FROM "RIODS"."REQUEST" a LEFT JOIN "RIODS"."COMPONENT" C ON
C."COMPONENT_ID"=a."COMPONENT_ID" LEFT JOIN (SELECT DISTINCT REQUEST_ID, ITERATION_ID, TESTPLAN_ID FROM(SELECT L."REQUEST_ID", CASE WHEN ER."ITERATION_ID">-1 THEN ER."ITERATION_ID" ELSE COALESCE(I."ITERATION
_ID",-1) END AS "ITERATION_ID", CASE WHEN EWI."TESTPLAN_ID">-1 THEN EWI."TESTPLAN_ID" WHEN TC."TESTPLAN_ID">-1 THEN TC."TESTPLAN_ID" ELSE COALESCE(TL."TESTPLAN_ID",-1) END AS "TESTPLAN_ID" FROM "RIODS"."EXEC
RES_REQUEST_LOOKUP" L LEFT JOIN "RIODS"."EXECUTION_RESULT" ER ON ER."EXECUTION_RESULT_ID"=L."EXECUTION_RESULT_ID" LEFT JOIN "RIODS"."TESTCASE" TC ON TC."TESTCASE_ID"=ER."TESTCASE_ID" LEFT JOIN "RIODS"."TESTP
LAN_TESTCASE_LOOKUP" TL ON TL."TESTCASE_ID"=TC."TESTCASE_ID" LEFT JOIN "RIODS"."EXECRES_EXECWKITEM_LOOKUP" EWIL ON EWIL."EXECUTION_RESULT_ID"=ER."EXECUTION_RESULT_ID" LEFT JOIN "RIODS"."EXECWORKITEM_ITERATIO
N_LOOKUP" I ON I."EXECWORKITEM_ID"=EWIL."EXECWORKITEM_ID" LEFT JOIN "RIODS"."EXECUTION_WORKITEM" EWI ON EWI."EXECWORKITEM_ID"=EWIL."EXECWORKITEM_ID" LEFT JOIN "RIODS"."REQUEST" R ON R."REQUEST_ID"=L."REQUEST
_ID") C ) D ON a."REQUEST_ID"=D."REQUEST_ID" WHERE a."REQUEST_ID"<>-1 AND a."ISSOFTDELETED"=0
2012-07-17 07:45:54,532 [ jts: AsynchronousTaskRunner-4]  INFO ervice.starjobs.internal.StarRemoteSnapshotService  - ***********com.ibm.team.datawarehouse.service.starjobs.internal.factbuild.RequestTestPlanM
etrics********************

Accepted answer


permanent link
Rafik Jaouani (5.0k16) | answered Jul 17 '12, 9:50 a.m.
JAZZ DEVELOPER
Hi Scott, CLM 3.0.1.4 will contain an optimization that should fix the issue. Here is the work item documenting the issue: Warehouse STAR job duration increasing exponentially (185790)
Scott Crouch selected this answer as the correct answer

One other answer



permanent link
ankur sharma (66212228) | answered Jul 17 '12, 2:30 p.m.
JAZZ DEVELOPER

Your answer


Register or to post your answer.


Dashboards and work items are no longer publicly available, so some links may be invalid. We now provide similar information through other means. Learn more here.