Test execution results

When you execute a test, the execution details are saved in an Execution Result. You can view the results immediately, or you can later access the results through the history of a test case execution record or a test suite execution record.

Overview

The top section of the Execution Result displays the details of the test run:

  • Actual Result
  • Host Name
  • Originator
  • Owner

    The Owner field is set to the last tester who ran the test.

  • Tested By
  • Test Plan
  • Iteration
  • Test Case
  • Test Script
  • Test Data
  • Build
  • Weight
Some of these fields are editable, depending on the type of script that is run. For manual and automated tests, you can edit only the Actual Result field in this section. If you execute a test without a script, all fields are editable until you save the result. After you save the result, only the Actual Result field is editable. If you execute a test without a script, you must first save the result in order to open the associated test case.
Note: You must have permission to edit fields in the Execution Result. For more information about permissions, see Adding and modifying roles and permissions.

Engineering Test Management uses a default set of test execution states. The Actual Result field displays one of the following execution states:

  • Passed (Pass)
  • Incomplete (Incomplete)
  • Inconclusive (Inconclusive)
  • Partially Blocked (Partially blocked)
  • Failed (Fail)
  • Error (Error)
  • Blocked (Blocked)
  • PermFailed (PermFailed)
  • Deferred (Deferred)
  • Skipped (Skipped)
To see standard definitions of test execution states, go to Test execution states.
Tip: An administrator can remove or rename execution states by clicking the Administration (Administration) icon at the upper, right part of the banner, selecting Manage Project Properties, and then clicking Execution States.

The execution result states are only configurable for the overall result of the test case execution. Changes do not apply to the individual test script step result states, test suite result states, or reports that display weight distribution.

Weight Distribution

Weight is a measurement of test results. The total possible weight was set in the test case. For the execution result, the calculated weight is a reflection of how many of the steps are completed successfully. If a test script has 10 steps and 8 of the steps completed successfully, then the weight is calculated as 80 points. You can use the sliders in this section to manually edit weight distributions for execution outcomes. This is useful for adjusting weights to reflect that certain steps are more important to the test effort than other steps.

The following formulas restrict how points can be distributed in a test case result:

  • Pass + Fail + Inconclusive <= Total Weight
  • Pass + Blocked <= Total Weight
  • Attempted >= Pass + Fail + Inconclusive

When running reports to measure the progress of the test effort, the weight value is taken into account when determining status. Weight should not be used to measure time spent as this would cause inaccuracies with reporting. If you intend to measure time rather than results, then the estimate and time spent fields should be used.

Skipped steps and deferred steps are not included in the Attempted weight distribution. Additionally, skipped steps are included in the step count of execution results.

Test Environment

This section lists the software and hardware platforms specified in the test case execution record or test suite execution record. You can link to the test environment directly from the result.

Defects

This section displays the defects that were opened during execution. You can also create new defects (Create new defect), add existing defects (Link to existing defect), or change the blocking status of defects (Change blocking status). The defects associated with the Execution Result are listed in the Related Defects panel on the right side of the page.

Categories

Use this section to categorize the execution record. You can use categories as a tagging or grouping mechanism to organize a test project into logical hierarchies. For more information about categories, see Organizing artifacts by using categories.

Execution Variables

This section includes information about the execution variables that were used during the test execution. The table shows the execution variable name and the value of the variable that was used in the test.

Previous Results

Use this section to view a summary of the results for recent tests. You can click the Execution Result icon (Execution Results) to view the detailed results for a test.

Results Details

For a manual test, this section outlines the steps executed in a test run, along with its result. You can use this information to see at what step a test failed or a defect was reported. You can open the defect by clicking the Defect (Defect) icon.

For other script types, you can click the link in this section to view the report in HTML format.

Tip: If the test script includes test data, the row numbers of the test data that is used in a corresponding step appear under the Row column.

If you run a test without a script, this section displays execution details in a rich-text editor. You can edit these details if you have the appropriate permissions.

You can view information about the testers as they run specific steps of a manual test. When you show the Tested by column, each step shows a list of the testers who have run the test, along with their user ID and a time stamp. You can use this information for audit purposes in a regulatory environment and also use the list of testers to add review and approvals requests. The list of testers involved in the test run is also available in the result table view, and you can filter on it or export results in CSV and PDF format.

Formal Review

This section is used to implement a formal review process that can help your business processes comply with applicable industry standards and regulations. Reviewers and approvers can be assigned to contribute comments to the process.
Note: Only users who have been granted permission can add reviewers and approvers for a test artifact.

State of Test Artifacts

Use this section to view the workflow states of the referenced artifacts when the test was run. For example, you can view the workflow state of the referenced test case, test script, and where applicable, test plan. You might use this information during an audit of the test results.

In the results of test cases and test suites, you can view the start date and time of the test, and the states of the applicable referenced test artifacts. You can also view information about the states of referenced artifacts in the table views, where you can export the data to reports in CSV or PDF format.

You can also display the workflow state of the artifacts referenced from an execution result in the browse lists for test case results and test suite results by adding the related column. When you change column display settings for the browse lists, the following columns are now available: Test Case State When Run, Test Plan State When Run, and Test Script State When Run.


video icon Video

Jazz.net channel
Software Education channel

learn icon Courses

IoT Academy
Skills Gateway

ask icon Community

Jazz.net
Jazz.net forums
Jazz.net library

support icon Support

IBM Support Community
Deployment wiki