Test analysis configuration

When running test analysis, it's important to have the right metrics to better understand your test results. With NeoLoad Web, you can embed up to ten metrics into your test results.

You can also use Work with regression detection to automatically compare current test results with static or custom dynamic metrics. For example, this helps you keep track of unwanted changes in the core behavior of your feature.

Specify metrics for your test results

To adjust the metrics displayed in a test view's graph, follow these steps:

  1. In NeoLoad Web, go to Tests.

  2. Inside the test you want to configure, select Test objectives to open the Test Analysis configuration dialog.

  3. Within the Metrics section, add or remove relevant metrics to your test results.

  4. Optionally, Work with regression detection.

The resulting graph can show up to ten metrics, displayed as solid lines. For more information on the available metric values, see Navigate the Values tab.

Work with regression detection

To detect regression within your test results, NeoLoad Web uses Objectives. When you specify an Objective as a metric, NeoLoad Web automatically compares your current test results with the values associated with the Objective. You can use different types of Objectives:

If a value in your test results surpasses the value of your specified Objective, it indicates regression, leading to test failure. This failure can also trigger Webhooks that depend on quality status Failure.

In some scenarios, NeoLoad Web doesn't detect regression:

  • You don't set any Objectives in the Test Analysis configuration.

  • You don't enter a value for the metric in the Test results.

Static Objective Mode

You can use static Objectives to compare one of your metric's actual value with a fixed value you specify.

To specify a static Objective, follow these steps:

  1. In the Test Analysis configuration dialog, check one or more Objective boxes to specify a custom metric value.

  2. In the Objective Mode section, select Static Objective Mode.

Baseline Objective Mode

You can use baseline Objectives to use existing test results as the comparison baseline for your current test. This automatically populates the Objective fields with the values of the chosen baseline.

Note: You can't manually edit baseline Objective fields.

To select existing test results as a baseline, follow these steps:

  1. In the Test Analysis configuration dialog, check one or more Objective boxes to specify a custom metric value.

  2. In the Objective Mode section, select Baseline Objective Mode.

  3. Specify a Reference result for that metric and its Tolerance. This automatically populates the value fields.

Note: The tolerance percentage defines a permissible deviation range for a metric without indicating a regression. It allows minor changes in the test results without marking the test as failed.

Automated Objective Mode

With an automated Objective, NeoLoad calculates the Objective's value at the end of the test result, using the most recent Passed results. Specifically, NeoLoad goes through the last 50 successful test results to calculate the automated Objective's value.

To specify an Objective, which uses a computed target value, follow these steps:

  1. In the Test Analysis configuration dialog, check one or more Objective boxes to specify a custom metric value.

  2. In the Objective Mode section, select Automated Objective Mode.

When you set your Objectives to Automated, there's an option to expand the deviation range. By default, this deviation is set to twice the standard deviation. However, for more flexibility, you can increase it to three or four times the standard deviation.

Note: To obtain an Objective value, you must complete at least 2 tests that include the required metric.