Automate test without compromising coverage of the environment using Rational Quality Manager

Pramod Chandoria
Last updated: April 12, 2012
Build basis: Rational Quality Manager 4.0

Summary

Automated testing is beneficial compared to manual testing in many ways, such as accuracy, saving time, resource utilization, and early regression detection. However, it is equally important that automated tests are executed in correct test environments, according to the test plan. Failing to do that could lead environment-specific defects to leak to the stakeholders. Pramod Chandoria describes how can you make sure that automated tests run on the correct machine, based on the test environment coverage planned, and mitigate the risk of leaking platform-specific issues. He also explains how to configure, manage, and run automated tests in Rational Quality Manager.

Introduction

Software testing is an important part of the software development lifecycle, because it assures developers and owners that the software works as expected. With the increased complexity of today’s software and the software environment, it is becoming difficult to manage testing manually, alone. Organizations are seeking solutions to automate the testing to mitigate the risk of regression defects leaking into the end product.

Fortunately, there are many automation tools available to automate the tests, including IBM Rational Functional Tester and Rational Performance Tester. There are initial problems associated with automating the test and with maintaining the tests as applications change. However, after a well-proven suite of automated tests is created, it proves useful in long run.

After tests has been automated, the next challenge arises while running the automated tests on all supported platforms, such as Microsoft Windows 2008 Server, Android, Firefox 10, IBM WebSphere Application Server, Oracle, IBM DB2, and so forth. Every year, the list of supported platforms changes due to the changes in versions, support for new platforms, and other factors.

As a recent example, Mozilla has released many versions of the Firefox browser since Version 3.0. This means that it is not only important to run the test, but it is equally important to run all tests on all of the supported test environments. This article explains how you can make sure that your automated tests automatically cover the platforms in your test plan, thus ensure maximum test coverage and detection of defects before you release software.

Figure 1: Manual Script

Automated Testing using Rational Quality Manager

IBM Rational Quality Manager is a web-based test management tool. It integrates with most of the leading automation tools, including Rational Functional Tester, Rational Performance Tester, Rational Robot, HP Quick Test Professional, and many more. The Rational Quality Manager open RESTful web API makes it easy to integrate with any other tools.

Essentially Rational Quality Manager provides two types of scripts:

• A native manual script with natural language steps, as shown in Figure 1, above
• A remote script, which represents an external automated script as shown in Figure 2, below

Figure 2: Remote Script

A manual script captures manual steps in natural language statements, as Figure 1 shows, and is useful for manual testing. Remote scripts, on the other hand, represents placeholders for the actual external automation scripts that can be run by specific external automation tools as shown in Figure 2, above. Each remote script has a type that is based on the type of automation tool. Rational Quality Manager cannot run the remote script; it merely coordinates the execution by delegating it to the automation tool that can actually run the remote script.

Automated tests are typically run on lab machines where the application under test (AUT) and the automation tool are configured to run automated tests. To integrate with Rational Quality Manager, each automation tool needs to have a lightweight utility called an execution adapter to bridge the gap between Rational Quality Manager and the automation tool. Each Rational automation tool has Rational Quality Manager-specific adapter to connect the two.

Planning Automated Test Execution

RQM contains testcase and it is implemented by manual or remote script to execute the test. Planning of Testcase execution results into creation of Test Case Execution Records (TCER). TCERs typically provides information which testcase to be executed, when to execute, who will execute it and most importantly for this article, on which test environment to execute it. A single Testcase often have multiple TCERs to cover multiple Test Environment which needs to be covered for the same test environment

Registering an execution adapter

When an automation tool-specific adapter is started, it registers to Rational Quality Manager and thereby makes itself available for execution of automated tests. As a result of this registration, Rational Quality Manager creates two resources: adapter and lab resource (test machine). The adapter represents the registered adapter, and it is linked to the lab resource, which captures machine details where the adapter is running. By default, the adapter captures only the machine name, IP address, domain name, and so forth.

To automatically provision full machine details, Rational Quality Manager can integrate with IBM Tivoli Provisioning Manager, Tivoli Application Dependency Discovery Manager, and open source StAX (Streaming API for XML) or STAF (Software Testing Automation Framework). The Integration section in Rational Quality Manager online help provides more information (see Resources for a link). Alternatively, you can manually edit the lab resource in Rational Quality Manager to provide details about the machine where the adapter is running.

The command-line adapter (CLA) is included with Rational Quality Manager (see the link in Resources for where to get details). CLA can execute any script that can be run from the command line. It could be as simple as ping command to as complex as a path to a .bat file that contains a complex script. Automation tool-specific adapters are often included when you install the automation tool.

Follow these steps to register a command-line adapter to the Rational Quality Manager server:

1. Copy Command line adapter zip from RQM Installation directory to Test machine. It can be found at ..JazzTeamServerserverconfqmadapters RQMCommandLineAdapter.zip
2. Unzip the context of RQMCommandLineAdapter.zip into a folder
3. Make sure JRE 1.5 or above version is installed on Test machine
4. Open Command window and change directory to the location where zip file is extracted
5. Run following command in command window: > start.bat -repository https://<rqmservername>:9443/qm -user <user id> -password <password> -adapter <uniqueAdapter id> -adapterName <My adapter> -projectArea “<Name of the RQM project area>”

Figure 3. Registering Command line Adapter

Upon successful registration an confirmation message will be displayed in Adapter console as shown in Figure 3 above.

Viewing registered execution adapters in the adapter console

In this section, you will learn how to view all execution adapters registered from different machines to Rational Quality Manager server.

1. Open your web browser, and enter the Rational Quality Manager URL, for example: https://server:9443/qm/web/console
2. Provide valid login information to log into the project area
3. Click Execution > Adapter Console to open the console
4. Verify that the adapter that you connected from test machine is listed, as shown in Figure 4, and that the Health column for it shows a green icon. (A red icon means that the adapter is not active for a long time after registration.)

Figure 4. Viewing adapter console

Updating test environment information

After execution adapters are registered, Rational Quality Manager creates a lab resource that corresponds to each machine where the adapter is running. By using this lab resource, Rational Quality Manager determines the test environment where the adapter is running. As mentioned earlier, registering an adapter to Rational Quality Manager provides merely minimal information about the machine, such as the host name and IP address. Such information is not sufficient. You can update test environment information in two ways. One way is to use provisioning tools from Tivoli that will update Rational Quality Manager automatically (Rational Quality Manager and Tivoli).

This article covers only how to update the test environment information for lab resources manually:

1. Follows steps in previous section to open the adapter console.
2. Click the Machine Name link to open the lab resource associated with the adapter.
3. Click the Operating System section. Update it with operating system information.
4. Click the Hardware section and provide any details about the machine,
5. Click the Software section and add all software that you can to update, such as Browser, Database, Application Server, and so on.
6. Click Save to save the test environment information.

Repeat these steps for each adapter that you have registered.

Configuring custom test environment coverage

This section provides step-by-step guidance to make sure that when Rational Quality Manager selects or proposes an execution adapter, the machine where the adapter is running complies with the test environment specified for the test case execution record.

1. Log in to Rational Quality Manager as a Project Administrator.
2. Click Application Administration > Manage Project Properties (see Figure 5).

Figure 5. Manage project properties

3. In the Manage Project Properties view, under the Properties section, click Execution Preference as shown in Figure 6.

Figure 6. Execution preference

4. By default, Rational Quality Manager does not make any attempt to match the test environment when executing automated tests. To change this, select the Custom Search radio button, and then select the check box for the type of test environments (operating system, browsers, and so forth) that you want to make sure are matched for searching for the adapter. If you want to match a specific test environment, choose the Environment radio button instead.
5. Click Save to update execution preferences.

By following these steps, you have configured Rational Quality Manager to run automated tests on the correct machine. This means that if you create an execution record to execute on Windows XP with the Firefox browser, Rational Quality Manager will verify that the chosen machine adheres to the test environment requirement.

Using a test cell for preparing robust test bed

Often, multiple teams work together on single project. Each team will have its own set of execution adapters connected to a Rational Quality Manager project. You do not want your testing to interfere with someone else’s test environment, of course. By using a test cell, you can limit the scope of machines where automated tests will be executed.

To create a test cell:

1. Click Lab Management > Create > Test Cell.
2. Provide a logical name for the test cell.
3. Click the green plus sign icon under the Lab Resource section (see Figure 7).
4. Add all machines where you want to run your tests.
5. Click the Save button to save the test cell.

Figure 7. Test cell

Running an automated test

After test case execution records are generated, you can run individual records manually by clicking the Run button from the Browse Test Case Execution Records view. This is useful when you want to run individual automated tests on an as needed basis.

Often, you will want to create a suite of automated tests and run the full suite to save time. In some situations, you might want to run automated test regularly, according to a schedule. You might also want to run automated tests whenever a new build is created to make sure that there is no regression in the build and to detect problems early, before the whole test team is engaged in testing. In this section, you learn how to run test suites manually for automated test, as well as more advanced options so you can design a workflow of automated tests that can be scheduled to run many times.

Running an automated test by using a test suite

1. Create a test suite by following instruction given in the information center (see Resources). Add all of the automated test cases to it, and make sure that you specify the test environment for each test case.
2. To specify the test environment:
   1. Check rows and then click Change Execution Properties.
   2. Select the test environment.
   3. Click OK, and Save your test suite.
3. Click Run in the test suite editor.
4. By default, Rational Quality Manager will select an adapter for each test. To change the default adapter, click Edit Adapter. Alternatively, let Rational Quality Manager distribute tests at run time by selecting the test cell. This will allow selection of adapters to be done by the software within the specified test cell at run time, based on the availability.
5. Select Run in parallel, in case tests do not depend on each other, to minimize the overall suite execution time.
6. Click OK.

Scheduling an automated test

1. Click Execution > Create Execution Schedule.
2. Provide a name for the test schedule.
3. Select the scheduling options available (see Figure 8):
   1. Schedule for one time at specified time
   2. Schedule daily for given time
   3. Schedule weekly on specified days
   4. Schedule based on build availability
   Note: The first three options are based on the time schedule. The last option is more useful when you want to run tests each time that a new build is created.
   

Figure 8. Types of schedules

4. In the Steps section, click the + (plus sign) button to add a test suite execution record, as shown in Figure 9. Filter to get the record that you want, and add to the Steps section. You can add as many execution records as you need.

Figure 9. Add a schedule step

5. Rational Quality Manager automatically selects available adapters to use for execution. However, to make better use of available adapters at run time, select Choose machines at run time in toolbar. The selected adapter will be set to “Search runtime” as shown in Figure 10.

Figure 10. Search for the adapter at run time

6. Associate a test cell with the execution schedule if you wish to isolate the machines where Rational Quality Manager should search for available adapters and not interfere with another test environment.
7. Click Save.

Now all steps added in the Execution Schedule will be executed in sequence whenever the schedule is triggered.

Tips: To manage a scheduled job, select Next scheduled time. Or to cancel schedules, click Execution > Browse > Scheduled Jobs.

Conclusion

In this article, you learned how to use Rational Quality Manager to run and schedule automated tests for various types of test automation tools, including command-line scripts. The article explained how you can configure Rational Quality Manager so that automated tests run in the test environment according to your test plan and to ensure that all supported platforms have been properly covered. You also learned how to better manage automated testing by using test cells.

For more information

• Running Rational command-line execution tests for more about the command line adapter.
• Referencing automated test scripts created with several other Rational test tools
• Creating a Test Suite using Rational Quality Manager
• Gap from TestManager: RQM is unable to distribute scheduled Execution based on matching Operating system and other attribute of Test Environment (server-relative link to a Jazz project work item)

About the author

Pramod Chandoria works at the Rational Software Lab for IBM, in India. He represents Test Automation in defining the first version of the Automation OSLC specification. He has been leading design and development of Rational Quality Manager since Version 1.0. Test execution and integration with test automation tools are his areas of expertise. Pramod is a graduate of the Indian Institute of Technology, Roorkee, with a degree in computer science, and has nearly nine years of product development experience. He can be contacted at pchandor@in.ibm.com.

---

Copyright © 2012 IBM Corporation