Jenkins and Bamboo Integration

qTest Manager allows for out-of-the-box integration with Jenkins and Bamboo. The integration collects executed unit test results while building projects in your CI server and reports them to qTest Manager. Our Plug-in does not execute your unit tests. It will only collect and submit test results (if any) to qTest.

To set up this feature:

  1. Activate the CI Integration feature in your qTest project.

  2. Map the CI execution status with corresponding qTest values.

  3. Configure the Test Run and Test Case organization.

  4. Install the qTest Plug-in on Jenkins or Bamboo and configure it to connect to your qTest project.

Supported Testing Frameworks

We support testing frameworks that are capable of generating JUnit XML format test reports.

  • For Jenkins, if your testing framework does not generate JUnit XML format test reports, you can utilize xUnit Plug-in to generate and publish test results before qTest Plug-in collects and reports them to qTest.

  • For Bamboo, the qTest Bamboo Plug-in does not support Bamboo Specs, and only JUnit XML format test reports are supported.

Supported Build Tools

  • Maven

  • Ant

  • Gradle

Activating the CI Integration

To receive test logs submitted from your CI server and associate them with builds, first you need to enable this feature in each individual qTest project where you want to use it.

To activate the CI Integration, go to the Gear icon and select Automation settings from the dropdown menu. Then, open the CI Tool Integration section and switch the Activation status to On.

Mapping the CI Status

Activating the CI Tool Integration will also activate the Automation Integration feature. Therefore, it is required that execution statuses for your CI tools are mapped to the corresponding qTest values. The CI tools' default execution statuses are:

Jenkins:

  • PASSED

  • FAILED

  • SKIPPED

  • FIXED

  • REGRESSION

Bamboo:

  • SUCCESS

  • FAILED

  • SKIPPED

To map the CI statuses:

  1. Select the Add Mapping button.

  2. Enter the CI tools status and the corresponding qTest value.

  3. Select Save.

Organizing Test Runs

While setting up the CI Tool Integration, you have different options to manage your Test Runs and Test Cases.

  1. In one Test Suite per selected Release: One Test Suite is created under each selected Release on qTest Plug-in's configuration in your CI server. This is the default option.

    • Jenkins: Test Suites are named as [Jenkins Project Name] [mm-dd-yyyy], where [mm-dd-yyyy] is the suite's created date.

    • Bamboo: Test Suites are named as [Bamboo Plan Key] [mm-dd-yyyy], where [mm-dd-yyyy] is the suite's created date.

  2. In one Test Suite for all Releases: Test Runs are created under only one Test Suite regardless of selected release in your CI server's qTest plug-in configuration.

    • Jenkins: Test Suites are named as [Jenkins Project Name].

    • Bamboo: Test Suites are named as [Bamboo Plan Key].

  3. Test Case submitted from your CI server are located under a Module named [Jenkins Project Name] or [Bamboo Plan Key]. You can optionally select a parent module to contain the CI module. If no parent module is selected, the CI module is created in the root module.

Jenkins Integration

The Jenkins integration is available in two configurations:

  • Freestyle: Use qTest Jenkins Plug-in in your post-build actions to submit test results to your qTest Manager project.

  • Pipeline: Generate pipeline script from qTest Jenkins Plug-in and put it in your pipeline code as a step to submits test results to a qTest Manager project.

You can use either type of Jenkins Integration to submit your test results to qTest Manager.

Install the qTest Jenkins Plug-in

After you choose your preferred configuration, use the Jenkins Plugin Manager to search for and install the qTest Plugin.

Once you install the qTest Plugin, add Submit JUnit Test Results to qTest as a post-build action.

If your testing framework does not generate JUnit XML format test reports, you can use the xUnit Plugin to generate and publish test results before the qTest Plugin collects and reports them to qTest. To do this, install the xUnit Plug-in and add Publish xUnit test result report as a post-build action before you add Submit JUnit Test Results to qTest as a post-build action.

Notes

  • If you use xUnit Plug-in to generate and publish test results, turn off the Delete temporary JUnit files option.

  • If you're working in a Maven project, you don't need a prior task to publish test results to Jenkins.

  • If you're working in a Freestyle or Multi-configuration project, you need a prior task to publish test results to Jenkins. If you don't have a prior task, Jenkins doesn't submit test results to qTest and the Submission Status for the build is SKIPPED.

Find API Token and App Secret Key

Whether you're setting up in Freestyle or Pipeline, you'll need the API token and App Secret Key from qTest as a part of your set up. The API token is different for each user, while the App Secret Key is the same across the entire qTest instance.

To find these, select the Download qTest Resources   icon and open the CI Tool Integration section.

From here, copy the API Token and App Secret Key from the Integration with Jenkins section to allow Jenkins to access your qTest Manager instance.

Configure the qTest Jenkins Plugin for Freestyle

Once you have the API token and App Secret Key, go to the Jenkins Plugin Configuration page. From here, you can configure the qTest Jenkins Plugin for Freestyle to both execute tests from Tosca and submit test results to qTest Manager.

Configure the Jenkins Plugin for Freestyle to Execute Tricentis Tosca Tests

To configure Jenkins plugin for Freestyle to execute Tricentis Tosca tests, follow these instructions:

  1. Select the Execute Tricentis Tosca Tests checkbox.

  2. Enter the following information for Tricentis Tosca:

    • Path to Tricentis Tosca CI Executable: The path to the Tosca CI client tool. This is either ToscaCIClient.exe or ToscaCIClient.rar, depending your operating system.

    • Tricentis Tosca Command Line Arguments: Command line arguments to execute Tosca test.

    • Path to Results: The path to the completed test results the Tosca CI client tool generates.

  3. Enter the URL you use to access your qTest instance in the qTest URL field. For example, https://demo.qtestnet.com.

  4. Paste the API Key and App Secret Key. These are the API Token and App Secret Key that you obtained from the qTest Manager Resources page.

  5. Select Retrieve Data to load data from qTest Manager.

  6. Select the project you want to send test results to from the Selected Project dropdown.

Configure the Jenkins Plugin for Freestyle to Submit Test Results to Manager

To submit your test results to qTest Manager, configure the following options:

  1. Select where you want to submit test results:

    • Submit test result to a Release as settings in your qTest Manager: The qTest Plugin submits the test run results to the qTest Manager Release that you select. For more information on how to set up releases in qTest, refer to the Test Plan for Releases and Builds topic.

    • Submit Test Results to an Existing Container: The qTest Jenkins Plugin submits test results to a specific container in qTest Manager, such as a release, test cycle, or test suite.

      If you choose this option, use the Container field to select the container you want to submit test results to. You can also select the available checkbox to create a new test suite for your test runs each day. If a test suite already exists, then the plugin submits test results to the existing test suite instead of creating a new one. For example, you have am existing test suite from a previous Jenkins build in the same day.

      If you leave the checkbox unselected, the plugin saves all test runs in a single test suite.

  2. Optionally, select an Environment to submit as the test run's property.

    Note that this field only affects newly created test runs and doesn't update existing test runs. To have test runs inherit the Environment field, delete your existing test runs and create new ones.

    The following fields are not available if you select the Execute Tricentis Tosca Test checkbox: Parse test results from testing tools, Utilize test results from the CI tool, Each JUnit Test Suite (class) = a qTest Test Case, and Each JUnit Test Case (method) = a qTest Test Case.

  3. The Overwrite existing test steps checkbox is selected by default. This means Jenkins overwrites any existing test case steps in qTest Manager when it sends test results to qTest Manager. Clear this checkbox to retain test step details in the existing test case in qTest Manager.

  4. The Parse test results from testing tools is selected by default. This means that the plugin scans for XML files that contain the test results. Optionally, you can use ANT Style Pattern to specify a directory pattern where these XML files are located. If you don't specify a pattern, the plugin scans the whole project.

  5. Select the Utilize test results from the CI tool if you want the qTest Jenkins Plugin to use existing test results scanned and published to Jenkins in a prior task without scanning again. For example, from the xUnit Plugin.

  6. Select one of the following radio buttons:

    • Each JUnit Test Suite (class) = a qTest Test Case: Create qTest test cases from a JUnit Test Suite (class). This option is selected by default.
    • Each JUnit Test Case (method) = a qTest Test Case: Create qTest test cases from a JUnit Test Case (method).

  7. Select Save to complete the configuration.

Configure the qTest Jenkins Plugin for Pipeline

If your Jenkins project uses the Jenkins Pipeline feature for advanced continuous integration scenarios, you can use the qTest Jenkins Plugin to generate a pipeline script to submit test results to qTest Manager. After you generate the script, you can add it as a step in your pipeline scripts.

Prerequisites

Make sure your Jenkins system already has Jenkins Pipeline setup. This requires the following:

Set up Pipeline Syntax for qTest

In your Jenkins Project , navigate to the Pipeline Syntax page and select Snippet Generator. In the Steps section, select submitJUnitTestResultsToqTest: Submit jUnit test result to qTest from the Sample Step list.

Configure the Jenkins Plug-in for Pipeline to Execute Tricentis Tosca Tests

After you have the API key and App Secret key, return to the Jenkins Pipeline Syntax page.

To configure Jenkins plugin for Pipeline to execute Tricentis Tosca tests, follow these instructions:

  1. Select the Execute Tricentis Tosca Tests checkbox.

  2. Enter the following information for Tricentis Tosca:

    • Path to Tricentis Tosca CI Executable: The path to the Tosca CI client tool. This is either ToscaCIClient.exe or ToscaCIClient.rar, depending your operating system.

    • Tricentis Tosca Command Line Arguments: Command line arguments to execute Tosca test.

    • Path to Results: The path to the completed test results the Tosca CI client tool generates.

  1. Enter the URL you use to access your qTest instance in the qTest URL field. For example, https://demo.qtestnet.com.

  2. Paste the API Key and App Secret Key. These are the API Token and App Secret Key that you obtained from the qTest Manager Resources page.

  3. Select Retrieve Data to load data from qTest Manager.

  4. Select the project you want to send test results to from the Selected Project dropdown.

Configure the Jenkins Plugin for Pipeline to Submit Test Results to Manager

To submit your test results to qTest Manager, configure the following options:

  1. Select where you want to submit test results:

    • Submit test result to a Release as settings in your qTest Manager: The qTest Plugin submits the test run results to the qTest Manager Release that you select. For more information on how to set up releases in qTest, refer to the Test Plan for Releases and Builds topic.

    • Submit Test Results to an Existing Container: The qTest Jenkins Plugin submits test results to a specific container in qTest Manager, such as a release, test cycle, or test suite.

      If you choose this option, use the Container field to select the container you want to submit test results to. You can also select the available checkbox to create a new test suite for your test runs each day. If a test suite already exists, then the plugin submits test results to the existing test suite instead of creating a new one. For example, you have am existing test suite from a previous Jenkins build in the same day.

      If you leave the checkbox unselected, the plugin saves all test runs in a single test suite.

  2. Optionally, select an Environment to submit as the test run's property.

    Note that this field only affects newly created test runs and doesn't update existing test runs. To have test runs inherit the Environment field, delete your existing test runs and create new ones.

    The following fields are not available if you select the Execute Tricentis Tosca Test checkbox: Parse test results from testing tools, Utilize test results from the CI tool, Each JUnit Test Suite (class) = a qTest Test Case, and Each JUnit Test Case (method) = a qTest Test Case.

  3. The Overwrite existing test steps checkbox is selected by default. This means Jenkins overwrites any existing test case steps in qTest Manager when it sends test results to qTest Manager. Clear this checkbox to retain test step details in the existing test case in qTest Manager.

  4. The Parse test results from testing tools is selected by default. This means that the plugin scans for XML files that contain the test results. Optionally, you can use ANT Style Pattern to specify a directory pattern where these XML files are located. If you don't specify a pattern, the plugin scans the whole project.

  5. Select the Utilize test results from the CI tool if you want the qTest Jenkins Plugin to use existing test results scanned and published to Jenkins in a prior task without scanning again. For example, from the xUnit Plugin.

  6. Select one of the following radio buttons:

    • Each JUnit Test Suite (class) = a qTest Test Case: Create qTest test cases from a JUnit Test Suite (class). This option is selected by default.
    • Each JUnit Test Case (method) = a qTest Test Case: Create qTest test cases from a JUnit Test Case (method).

  7. Select Generate Pipeline Script to generate a script that you can copy and add as a step in your pipeline scripts.

    Sample Jenkins file that shows how to use the generated script in the context of other pipeline scripts. The submitJUnitTestResultstoqTest step submits the test results to qTest Manager once the step that executes the test is finished.

Bamboo Integration

Install the qTest Bamboo Plug-in

Install the qTest Plug-in for Bamboo from the Manage Add-ons page in Bamboo.

Configure qTest Bamboo Plug-in

  1. Navigate to qTest Manager and locate the Download qTest Resources   icon to open the page.

  2. Select the CI Tool Integration section to expand.

  3. Copy the API token from the Integration with Bamboo section. This allows Bamboo to access your qTest Manager instance.

    The API token is at a user level, which means it is different between qTest users.

  4. Navigate back to the Bamboo Plug-in Configuration page.

  5. Configure a build plan for your Bamboo server.

  6. Select a build stage under the plan.

  7. Select a build job under the stage.

  8. Add a new task to the selected build.

  9. On the pop-up, search for the qTest Integration task and add it to the stage.

    This task needs to be added to the job in which unit tests are executed.

  10. After the task is added, you will need to configure the task to connect to your qTest Manager instance.

  11. Enter the following qTest Manager information:

    • qTest URL: The URL to access your qTest instance, such as https://demo.qtestnet.com.

    • API Key: This is the Integration with Bamboo token you obtained from the qTest Manager Download Resources page.

  12. Select the Retrieve Projects button to load data from qTest Manager.

  13. Select the qTest project that the build is integrating with and where you want to submit test results.

  14. Select one Release that will be the target Release of submitted Test Runs.

  15. (Optional) The selected Environment will be set as submitted Test Run's property.

Notes

  • The selected Environment only affects the newly created Test Runs. It will not update the existing Test Runs with the new Environment if you have selected a new one. We are working on the fix for that and will release it soon.

  • The Plug-in will scan for the XML files that contain the test results. You can optionally specify a directory pattern where these XML files are located, using ANT Style Pattern. If not specified, the Plug-in will scan the whole project.

CI Integration Reports

CI Tool Integration Report

This is an export file that contains details of tests submitted from your CI server. It includes two views:

  • Build by Build: Details of all tests executed in each CI build

  • Test by Test: Execution results of each test through different CI builds

To download the export file:

  1. On the Test Execution tab, select the Export button.

  2. Select CI Tool Integration Report.

    • Select filters before exporting.

      The export is not filtered by what you select on Test Execution tree. It is only filtered by values selected in this pop-up.

Test Result Submission Report

On your CI tool side, we provide a report allowing you to track the status of test submissions to qTest after each build is completed.

If a failure occurs during test submission, the Failure Log will be included as a text file attachment in the Test Log Details.

  • Jenkins: Select a build project. Select the qTest Plug-in on the left panel.

  • Bamboo: Select a build plan. Open the qTest Plug-in tab.