Best practices | TestCases | Error handling
We recommend that you always add recovery steps to your tests. First, they can compensate for problems in your application. This lets Tosca get your test runs back on track, even if your tests encounter problems at runtime. Consequently, you still get conclusive results, without having lost valuable time and resources on stopped or failed tests. Second, you can focus on fixing the things that are actually broken, instead of having to double-check everything.
This topic explains important considerations for the main components of Tosca's recovery feature: Recovery Scenarios, RetryLevels, and CleanUp Scenarios.
Recovery Scenarios
Recovery Scenarios define what Tosca should do if something unexpected happens.
There are some generic error handling settings that tell Tosca to continue anyway, if possible, or to move on to the next TestCase. These settings are a useful option during the discovery phase.
But to create a stable, resilient test set, you need more precise Recovery Scenarios that solve specific problems. This means you need to have a fairly good understanding of your application and the things that can happen. For example, sudden cookie notifications, delays in the user interface, or data miscalculations. You also need to know what to do to resolve these situations. These are the steps you'll put into your Recovery Scenarios.
Let's say you have a modal dialog, Message from your sys admin. This dialog can appear at any time on any screen of your application. Unfortunately, it blocks the actual screen, which causes the current TestStep to fail. In this case, add the following steps to your Recovery Scenario: check whether the modal dialog is open. If it is, select OK to close it and unblock the screen.
RetryLevels
RetryLevels determine when Tosca applies Recovery and where it resumes test execution after a successful Recovery.
Make sure to set an appropriate RetryLevel for your Scenarios, so you don't run into new issues after Recovery. By default, Tosca re-runs the entire TestCase from the start, which may not always be the best option. For example, if a TestCase starts with opening an application and signing in. If the application is already open and the user already signed in when Tosca tries to repeat these steps, you get another error.
Generally, your best bet is to set the RetryLevel to TestStep. In this case, Tosca runs your Recovery Scenario, then retries the TestStep that originally failed and continues from there.
CleanUp Scenarios
CleanUp Scenarios define what Tosca should do when Recovery fails. For example, force a shutdown of the application with the Module TBox Start Program. The goal is to reset the test environment, so Tosca can try again with other tests in that environment.
Whatever you do, don't use CleanUp Scenarios as a cover for vulnerable test design.
Let's say you're testing the transfer of customer data from a file to a database. The database doesn't allow duplicate entries; if a customer already exists, the database triggers a pop-up dialog. That's a routine part of the database process, and it should be part of the normal test flow. For example, with an IF Statement. Don't treat this as a CleanUp Scenario.
Need a hand with
What's next
If you haven't yet, check out our other best practices articles.