Share your top 5 pain points using RQM

1. Continually having to log in every so often.  I'm on RQM all day, and I continually have to log in.  The fact that I keep several tabs open into RQM makes this even more frustrating; I have to re-log in to every tab.

2. Printing is useless.  Sure, I can print a test case, but if I try printing out a list of test cases in a test plan, I get too much information and wasted paper.  Plus, printing doesn't scale - it takes so long to create a big printout.

3. The concept of test scripts per test case.  It took us over a year to realize that we were using RQM wrong - we had multiple scripts per test case, in the assumption that if you ran a TER, the scripts would be strung together.  Not so.  We were actually missing scripts when we ran the TERs.

4. Test scripts are a nice concept, but they are really hard to create/edit.  I'm talking here about the table used for the script steps.  Due to point 3 above, I had to do a bunch of editing, copying steps from one script and pasting them into another.  This was a serious annoyance.  I got into a rhythm, with two browsers open of Tab, Ctrl+C, Atl+Tab, Tab, Ctrl+V.  In general, I love how easy scripts are to run but I absolutely HATE editing them.  I wish there were some way to use a desktop tool, or something. 

5. When I create TERs, I would love to know who ran them last.  Sometimes I want to have the same testers run the same tests; sometimes I want to avoid that.  Unless I actually do a bunch of clicking and navigating, I can't know who ran the last TER for a particular test case.  It takes too long when you're assigning dozens of cases at a time.

We have been using RQM 3.0.1.1 extensively to help us test our product development. In particular we are using the RQMCommandLineAdapter to automate the running of our tests. From this use case we are finding the following problems:

1. So slow to navigate - this seems to be a combination of:
1. lack of responsiveness when there are large numbers of results (e..g the previous results list of a test suite execution record) which can take up to a few minutes to respond
2. the tabs inside the browser window - when you are scrolled down to the bottom of a page of results you can no longer see the tabs to choose another one, and when you switch between tabs you are always taken back to the top again, not where you were viewing. Not a big problem when you are using it occasionally, but when we are using it all day, every day, it becomes very frustrating. I get lost amongst all the tabs. I think RQM 4.0 could solve this problem for us, which is great when we start using it.
   
2. Problems with reliably using execution schedules. Sometimes an execution schedule with fail with an 'Error' state which has no more information to suggest a problem. Trial and error suggests that any change to a test suite will stop an existing execution schedule (that uses the test suites's execution record) from working. The solution is that every time a test suite is modified, any execution schedule has be modified by removing and re-adding the same test suite execution record. Which leads me to my next point
   
3. The slowest part of the whole process: modifying an execution schedule. We have test suites with about 30 tests. Whenever I need to change an execution schedule with one or more test suite execution schedule I have to wait for all the individual test cases to find a machine to run on. It takes 2 or 3 seconds per test case at best. I have spent many minutes several times a day waiting for the execution schedule to refresh itself before I can make a small change. By the way, the ability to re-order test suite execution records in a test schedule is broken although I haven't raised a bug yet.
4. The way that blocking defects are associated with test case results is good, but I would like to automatically remove the blocking defect when the defect is verified. It is a slow process to individually remove all blocking defects (is there any way to do this other than one test case at a time?)
5. No way of seeing the test output until the test has finished. The RQMCommandLineAdapter does not have any way of shown the test output while the test is running. Unlike an RTC build using a jazz build engine, there is no log file that I can tail on the target server. 
6. Am I allowed a number 6? I have just remembered what is probably just a niggle compared to the other problems. When I am in a test suite I would like to add it to a test plan, but I cannot add the test plan from the test suite. Instead I have to go to the list of test plans, open it, then find the test suite to add. 

Sorry this sounds so negative, there are many positives about using RQM but we are finding it harder and harder to use (mainly because of performance and navigation problems) as we add more and more test cases.

Hello Pramod,

1. No screenshot utility when authoring manual tests.
2. Product Structures like a category tree would be more convenient to create in a tree like editor view on testcases for average users of the tool.
3. Test data are bound to a test script which makes it difficult to parametrize the same test with different data. You have 2 test cases A and B and would like to test with the same test script but different data set for A and B. Test data should be bound to TCER and test script should only use variables, in the TCER there would be a mapping from test data columns to test script variables.
4. Test data will not remember which test data rows where already used in tests before.

Best Regards,
René

Hello Pramod,

 

In future will RQM have the customization available for its test artifacts as RTC offers for its work item?

  For large execution teams, this is REALLY a problem:

Here's 3 I can think of off the top of my head

1.  Inability to create dynamic test suites

2. When creating Manual Tests we can't put screenshots in for steps.

3. Inconsistency with RFT adapters

1. Creating and maintaining links to RFT test Scripts via the adapter is a painful exercise. If you could use a standard "Browse" with the adapter, or even paste a string in this would go a long way to improving my workload in maintenance of Test Scripts. The other option available which downloads the scripts locally is not an option in our environment.
   
2. If status of a test script/case/suite is Approved some changes are not available to be implemented until you reopen the the status. Negates the use of approved as we just leave it in an earlier status state so that we can still edit.
3. Unable to debug RFT script runs from RQM unless extensive logging created in the script (console not active, RFT not active - just utilised in background etc.). This has caused us particular pain in debugging errors found when using the Execution Variables.
4. Every time we make an Adapter connection, even from the same PC, we get another entry in the Adapter Hosts list, with the previous version of this same PC adapter being "red" and the new one being "green". Clearing this automatically would help with maintenance.
   
5. If I tell someone what to do in QC they can execute a damn sight quicker than if I get them to execute the same method in RQM. This is due to:

• So many clicks to do anything! Just navigating to a specific test script in a Test Suite requires:
  Open RQM > Select View Suites > filter for suite or search > Select Suite > Select Test Cases on Suite > Select Test script on Suite
  By comparison:
  Open QC > Navigate to TestSet (usually just a flick open of folder then select, depending on the project) > Test Case Instance is visible and ready for editing
• Each click is a delay! Delivery of the RQM screens in our environment does not seem to be anywhere near as quick as presentations I have seen from IBM. Yes, I can see that its the environment, not the application, but users don't care and that reflects badly on the product.
• Views are maintained in QC but not in RQM. If I am working in an area in QC and close the app, I can go back at any time and it will be the same view I left.
  In RQM I would need to follow a number of clicks before I get my "workspace" back

That's it for today... ;)

Overall RAM V4 is quite nicer than V3. Besides what other expertes mentioned, I would like to dream the following  features:

1) Allow dynamic query of test artifacts like what we can do with RTC work items by providing certain values of certain attributes on the fly.

2) Have "RQM client" (eclipse and Visual studio based) in the same way as RTC client to also manage test artifact creation, update, execution and query etc.

3) There can be a "Review" and "Approval" status attributes for test artifacts so that we can easily tell which test artifacts have been reviewed or approved. If not, it would show current status (Pending, In Progress, Not Opened, etc)  Currently we have to open the test artifacts to see the information in the formal review section, unless there is this kind of query I don't know.

1 - editing test scripts is a pain. We find it easier to create test scripts in excel and then import them instead of using the current editor.

2 - auto generation of manual test scripts from description. Needs to be improved to make it easier to separate steps and add expected column in the scripts. Define a nice interface or way to write the description so conversion is better.

3 - provide ability to tie the execution of RQM test cases(?) back to work items. There are references but if a test case is passed, the total time it took to execute (through all the runs) and results should be fed back and update a linked work item (task or defect). Eliminates the need for a user to go into CCM and manual update that information.

Following features can be incorporate in RQM

1) exporting of Test Case in Word document

2) If any requirement change , status of linked test case need to be suspect automatically. Requirement Reconcillation feature is there but it is not automated.

3) Versioning of Test Artifacts as other ALM tools offering this feature in market.

4) customization jazz login page for e.g. to textual change in JTS Login Page, client logo etc.

5) Reporting can be imnproved.