RQM and Test Execution
We've been running with RQM for a while now, but I'm still struggling with getting accurate status information from RQM. I'm hoping someone can help.
When I attempt to run a single test script, the entire test case associated with that test script gets marked, along with all points. For example: - I have Test Case A, with Test Scripts 1, 2, 3, 4, and execution records created for the same. - If I mark Test Script 1 as passed, Test Case A is now marked as passed, with the default being all associated points marked passed. - If I then run Test Script 2 and mark as failed, Test Case A is now marked as failed, with all associated points marked failed (even though points passed in the previous step). - If I then run Test Script 3 and mark as passed, Test Case A is now marked passed, will all associated points marked passed (even though points should still be failed). How do you get RQM to properly keep track of Test Scripts' status and their associated points? The only option I can see is have the tester memorize the point totals prior to executing a test, and then re-adjust the sliders from RQM's forced defaults after execution. - I can get reports and views which show me Test Scripts which have been marked Passed/Failed/Inconclusive. However, I can't seem to find a list of Test Scripts which are remaining. I can also view Test Cases which are 'Not Run', however, this does not help when trying to determine which Test Scripts within a Test Case still remain. |
7 answers
Hi ,
My understanding of the Test Script was that they were step for a Test Case. For you TC you can have many steps (TS) So If I have a TC that have 5 steps and the step 1 to 3 are Pass and step 4 to 5 and fail, my TC will be Fail. I hope It help |
Hi , Sure, but if Scripts 1-3 are passed, then some portion of the points should be marked as What |
Hi , To further explain the way I've approached my Test Plan setup - a Test Case is the category of the test (say, "Test Function X"), where a Test Script is a single test executed against said Function. A single Function may require multiple tests (ie., multiple Test Scripts) to properly validate, and each Test Script will likely have multiple steps to execute each Script. For example:
Is this the same approach others take? If so, how do you handle marking partial completion, when RQM wants to mark all points one way or another, regardless of your actual progress. |
Although the test case is covering a single functional area, our team typically has 1 script per test case. So in your example, we would have 5 different test cases. Additionally we may group all 5 of these test cases into a single Test Suite. Were you get better roll-up of the overall points and status of the scripts that were executed.
RQMs results are actually at the Test Execution Record level. So one thing you may consider is to create multiple Execution Records. This can be done on the Execution Records tab of the test case. You can create one execution record for each of your scripts. That way the execution record table will show the pass/failed/block state of the individual execution records, which in your case would be 1 for each individual script. When viewing Test Case results you can also group by Test Case, so you will see the result of all execution records in one place. Additionally most reports against TERs allowing filtering based on the Test Case you want to report on. |
Although the test case is covering a single functional area, our team typically has 1 script per test case. So in your example, we would have 5 different test cases. Additionally we may group all 5 of these test cases into a single Test Suite. Were you get better roll-up of the overall points and status of the scripts that were executed. Thanks for the input. Unfortunately, we're already grouping functional areas into Test Suites. RQMs results are actually at the Test Execution Record level. So one thing you may consider is to create multiple Execution Records. The downside of this is that our point totals will be screwed up. Since test points are tied to the Test Case (and not Test Script), creating multiple TERs for each Test Script will still produce inaccurate metrics (my current problem with RQM). So, it looks like our only option is to limit Test Cases to contain only one Test Script in the future....however it seems like a waste to have a container (Test Case) for a single container (Test Script). |
Blimey, this really is a strange design of RQM or shows the incapability of the reporting and execution tracking. looking t previous versions of RQM, there used to be a Test Case folder , a Test case and Test Scripts in the hierarchy so someone somewhere has previously realised items needed to be bunched together. I am being told that I need to have a 1:1 relationship between Test Case and Test Script to be able to get accurate progress reporting using a standalone RQM without Rational Common reporting or Cognos . There seems to be a major design flaw if 1:1 is the only way to know what the position of test progress in execution is. I agree with the writer in what is said above.
|
|
You shouldn't need to limit a test case to contain only one script in order to get the information you desire. There are plenty of valid scenarios where you want a test case to contain more than one script.
I assume you are talking about RQM's out-of-box default reports. What you need is 1 test case execution record (TER) per script. You need to be certain, that the default script is saved for the TER. Sometimes I've struggled with getting the default script to save. But you know it's saved, when you go to the view Test Execution Record page and see it selected by default. Once in place, you should not need to select the script when running a test case. Instead select the TER and that should fill in the default script. If you ever have to select the script from the drop down menu when running the Test Case or TER, then you aren't using the default script and it could cause the issues you describe. With that said, the out-of-box execution reports are centered around the TERs, not the Test Case. Some times they may group by Test Case, however, the counts and sums of weights are taken from the TER's last result. If you have your TERs setup correctly, then you shouldn't see the issues you originally mentioned. For example, for a single Test Case, if you run TER 1 for script 1 and it passes, then you run TER 2 for script 2 and it fails. Then you run a report such as Execution status using TER Count you will see both the pass and fail result. Similarly if you run Execution Status using Weight, you will see some points passed and some points failed. I think this issue is a matter of your execution records being setup with the default script. |
Your answer
Dashboards and work items are no longer publicly available, so some links may be invalid. We now provide similar information through other means. Learn more here.