How to best clean results from automated tests
We're starting the process of implementing RQM. We have many hundreds of test scripts that have been written over a long period of time, and one of our concerns is that they don't always return accurate pass/fail results. We get many false failures. We've dealt with this by manually reviewing test log files and deciding what failures are real and which we can ignore. We want to do better with RQM, but cleaning up all our tests will take time.
If we execute these test cases automatically and the results are loaded back into RQM, then any reports will not be accurate because of the original bad data. Therefore, is there a way to efficiently review the test results in RQM and modify the results accordingly? Any suggestions on how to handle this situation are very welcome.
thanks
One answer
Comments
The ability to update multiple results from one screen will be much more efficient then having to go to each result individually. Since we'll likely have several people doing this daily after each nightly regression run, efficiency in doing this for hundreds of test results is essential.
thanks
Michael does this existing bulk change feature satisfy your request? Or are you looking more for presenting more information while viewing the table of results in order to make the decision on whether to change the result to passed? For example would having a column in the table where each cell would contain a link to the generated log file help? And would it be better still that when you hover over a link you get a popup to view the log file content?
I don't have a specific solution in mind. We're currently planning our RQM implementation so I can't really comment yet on how well this helps. We know we're going to face this issue and I was looking for a discussion on how to best deal with the problem. I'm assuming that we aren't the only QA team with this behavior in legacy tests. Having links to the log file is a great idea. Generally, anything that would increase our efficiency reviewing the logs would be greatly appreciated.