How to find the test cases which fail most often in RQM
Hi,
To explain my question: In my team we are going to use RQM for automated test which runs each night.
Some test cases are not passing each time. They are unstable and need to be redesigned.
I want to use a report i RQM to identify these test cases. How do I do that?
I have say... 500 test cases with results 356 days a year.
I do not have the time to look in to all results for each test to find the ones that some times fail....
It could be a graph shown the number of fails for each test cases.....most will show 0.
Or just a list .....showing the count of fail or each test cases.....ordered by the highest count.
Does somthing like that exist in RQM or can it be created?
2 answers
Hi Eva,
first I would check whether RQM OOTB reports cover your needs:
the "Execution Status by..." reports could be good candidate to start with.
(e.g. the "Execution Status by Owner using TCER Count (Live)" report may fit. Note that you could also customize it such that it only display failed/inconclusive/etc. executions results).
Since you mention this activity for "tracking the unstable test cases" occurs on a daily basis, I'd suggest you to add the identified report to your dashboard. Every morning, you could drill down to the failing TCERs and associated test cases from there.
If the OOTB reports are not enough, you'd like to envision the use of RRDI.
Regards,
Stéphane
first I would check whether RQM OOTB reports cover your needs:
the "Execution Status by..." reports could be good candidate to start with.
(e.g. the "Execution Status by Owner using TCER Count (Live)" report may fit. Note that you could also customize it such that it only display failed/inconclusive/etc. executions results).
Since you mention this activity for "tracking the unstable test cases" occurs on a daily basis, I'd suggest you to add the identified report to your dashboard. Every morning, you could drill down to the failing TCERs and associated test cases from there.
If the OOTB reports are not enough, you'd like to envision the use of RRDI.
Regards,
Stéphane
Comments
Hi, I have tried to simulate some test results and use the suggested report. It seems to be that the report only pays attention to the latest test result. The intention is to look back on all old test results.
The result I am looking for might on the form:
Test case name x failed 34 times
Test case name y failed 32 times
Test case name z failed 15 times
Test case name p failed 5 times
Test case name g failed 0 times
for a defined period.....
This data might be collect over a period of 14 days to 6 month.
Perhaps this requires a specific way to organizes test cases and use TER's ?
Well. I see some options:
1) try the "Summary > Tester Report using TCER Count (Live)" report
You will get information on Passed/Error/... status splitted by Owner
and from any cell of the tab (defining an owner + a status), you could drill down to the associated "TCER Listing (Live)"
2) try the "Summary > Execution by Test Schedule (Live)"
Note that you'd need to adapt to information it displays as Passed/Error/... are reflecting associated weights (and not the total numbers of times a TCER was run)
If not enough or if requiring more specific reporting , you'd probably want to figure out the capabilities provided of RRDI. Here are some interesting links :
1) the CLM 2011 Reporting workshop
2) the CLM 4.0.1 Reporting workshop
2) this blog post providing a mind map around Reporting for CLM
Regards,
Stéphane
1) try the "Summary > Tester Report using TCER Count (Live)" report
You will get information on Passed/Error/... status splitted by Owner
and from any cell of the tab (defining an owner + a status), you could drill down to the associated "TCER Listing (Live)"
2) try the "Summary > Execution by Test Schedule (Live)"
Note that you'd need to adapt to information it displays as Passed/Error/... are reflecting associated weights (and not the total numbers of times a TCER was run)
If not enough or if requiring more specific reporting , you'd probably want to figure out the capabilities provided of RRDI. Here are some interesting links :
1) the CLM 2011 Reporting workshop
2) the CLM 4.0.1 Reporting workshop
2) this blog post providing a mind map around Reporting for CLM
Regards,
Stéphane