Why is the Star Data Collection Job taking longer every day?
I was wondering if anybody has any ideas why the STAR ETL job is taking longer and longer each day. In the past month it's gone from taking 5 hours, to now over 7 hours.
I can see in the jts-etl.log file there is a significant jump in the log when it's doing something. Is there any way to know why it's taking so long? I pasted the snippet from the log file where it seems to be taking it's time (notice the 7 hour jump in log entries). 2012-07-17 00:39:55,636 [ jts: AsynchronousTaskRunner-4] INFO ervice.starjobs.internal.StarRemoteSnapshotService - ***********com.ibm.team.datawarehouse.service.starjobs.internal.factbuild.RequCreatiTplanD ateMetricsFactETL******************** 2012-07-17 00:39:55,636 [ jts: AsynchronousTaskRunner-4] INFO ervice.starjobs.internal.StarRemoteSnapshotService - Total Records Selected = 22960 2012-07-17 00:39:55,636 [ jts: AsynchronousTaskRunner-4] INFO ervice.starjobs.internal.StarRemoteSnapshotService - Total Records Inserted into DataWarehouse = 22960 2012-07-17 00:39:55,636 [ jts: AsynchronousTaskRunner-4] INFO ervice.starjobs.internal.StarRemoteSnapshotService - ************************************************************* 2012-07-17 00:39:55,636 [ jts: AsynchronousTaskRunner-4] INFO ervice.starjobs.internal.StarRemoteSnapshotService - Java ETL (STAR): Fact build "com.ibm.team.datawarehouse.service.starjobs.internal.factbuil d.RequCreatiTplanDateMetricsFactETL" passed 2012-07-17 00:39:55,637 [ jts: AsynchronousTaskRunner-4] INFO ervice.starjobs.internal.StarRemoteSnapshotService - In com.ibm.team.datawarehouse.service.starjobs.internal.factbuild.RequestTestPlanMetrics 2012-07-17 00:39:55,637 [ jts: AsynchronousTaskRunner-4] INFO ervice.starjobs.internal.StarRemoteSnapshotService - SELECT DISTINCT a."REQUEST_ID", a."REQUEST_CLASS_ID",a."REQUEST_CATEGORY_ID", a."PROJECT_ ID", a."COMPONENT_ID", a."TEAM_ID", a."REQUEST_SEVERITY_ID", a."REQUEST_STATE_ID", a."REQUEST_STATUS_ID", a."REQUEST_PRIORITY_ID", UPPER(a."PLATFORM") AS "PLATFORM", COALESCE(D."ITERATION_ID",-1) AS "ITERAT ION_ID", a."RELEASE_ID", a."FIXEDINRELEASE_ID", a."IS_BLOCKING", a."OWNER_ID", a."REQUEST_TYPE_ID", COALESCE(D."TESTPLAN_ID",-1) AS "TESTPLAN_ID" FROM "RIODS"."REQUEST" a LEFT JOIN "RIODS"."COMPONENT" C ON C."COMPONENT_ID"=a."COMPONENT_ID" LEFT JOIN (SELECT DISTINCT REQUEST_ID, ITERATION_ID, TESTPLAN_ID FROM(SELECT L."REQUEST_ID", CASE WHEN ER."ITERATION_ID">-1 THEN ER."ITERATION_ID" ELSE COALESCE(I."ITERATION _ID",-1) END AS "ITERATION_ID", CASE WHEN EWI."TESTPLAN_ID">-1 THEN EWI."TESTPLAN_ID" WHEN TC."TESTPLAN_ID">-1 THEN TC."TESTPLAN_ID" ELSE COALESCE(TL."TESTPLAN_ID",-1) END AS "TESTPLAN_ID" FROM "RIODS"."EXEC RES_REQUEST_LOOKUP" L LEFT JOIN "RIODS"."EXECUTION_RESULT" ER ON ER."EXECUTION_RESULT_ID"=L."EXECUTION_RESULT_ID" LEFT JOIN "RIODS"."TESTCASE" TC ON TC."TESTCASE_ID"=ER."TESTCASE_ID" LEFT JOIN "RIODS"."TESTP LAN_TESTCASE_LOOKUP" TL ON TL."TESTCASE_ID"=TC."TESTCASE_ID" LEFT JOIN "RIODS"."EXECRES_EXECWKITEM_LOOKUP" EWIL ON EWIL."EXECUTION_RESULT_ID"=ER."EXECUTION_RESULT_ID" LEFT JOIN "RIODS"."EXECWORKITEM_ITERATIO N_LOOKUP" I ON I."EXECWORKITEM_ID"=EWIL."EXECWORKITEM_ID" LEFT JOIN "RIODS"."EXECUTION_WORKITEM" EWI ON EWI."EXECWORKITEM_ID"=EWIL."EXECWORKITEM_ID" LEFT JOIN "RIODS"."REQUEST" R ON R."REQUEST_ID"=L."REQUEST _ID") C ) D ON a."REQUEST_ID"=D."REQUEST_ID" WHERE a."REQUEST_ID"<>-1 AND a."ISSOFTDELETED"=0 2012-07-17 07:45:54,532 [ jts: AsynchronousTaskRunner-4] INFO ervice.starjobs.internal.StarRemoteSnapshotService - ***********com.ibm.team.datawarehouse.service.starjobs.internal.factbuild.RequestTestPlanM etrics******************** |
Accepted answer
Hi Scott, CLM 3.0.1.4 will contain an optimization that should fix the issue. Here is the work item documenting the issue: Warehouse STAR job duration increasing exponentially (185790)
Scott Crouch selected this answer as the correct answer
|
One other answer
|
Your answer
Dashboards and work items are no longer publicly available, so some links may be invalid. We now provide similar information through other means. Learn more here.