Vision AI for mobile automation

Vision AI for mobile automation is an early access feature that is still under development. We can't guarantee full functionality in this or future versions.

The Mobile Engine 3.0 lets you steer screen objects in mobile apps with the power of Vision AI. You can start right away:

Enable Vision AI for mobile automation

To use Vision AI for mobile automation, first you need to enable it via Tosca > Project > Settings > TBox > Engines > Mobile30.

There are two different settings that allow you to connect to the Vision AI service:

  • Tosca Cloud URL: Allows you to connect to the Vision AI service through your Tosca Cloud identity.
    In the Value field enter your Tosca Cloud URL: https://<your tenant name>.my.tricentis.com

  • Vision AI personal access token: Allows you to connect to the Vision AI service through a personal access token.
    In the Value field, enter your personal access token.

If both settings are enabled, the primary access method will be Vision AI personal access token. If none of the above settings are enabled, the Vision AI option will not be displayed in the Mobile Scan window.

Create Vision AI-based Modules

To create a Vision AI-based Module, follow these steps:

  1. Start the Mobile Scan.

  2. In the Mobile Scan dialog, select Connect.

  3. Once connected, select Vision AI. If Mobile Scan doesn't display the Vision AI option, you haven't enabled it yet.

  4. Select Scan. This opens the live view.

  5. Select the controls you want to scan either from the tree or the live view.

  6. When you are ready to end your scan, select Save.

  7. Create a TestStep out of the Module.

Vision AI technical properties

The controls that you scan with Vision AI display specific technical properties:

Property

Description

Area

Specifies the section of the application interface where the control is located.

ControlId

Uniquely identifies the control within the application. This property is selected by default to ensure accurate recognition, improving efficiency and avoiding ambiguity during testing.
IsMandatory

Specifies that the control is required for the test to proceed. If no matching controls are found, the locator is updated to be less specific. This action occurs during the Vision AI automatic healing process and defines how the process adapts.
This is selected by default, which means that Tosca cannot execute the test successfully without interacting with this control.

If your control is a hidden or disappearing element change this property to False.

IsUnique

Specifies that the control must be unique within the application. If multiple matching controls are found, the locator is updated to be more specific. This action occurs during the Vision AI automatic healing process and defines how the process adapts.

This is selected by default to make sure the test interacts with the correct control and avoids any conflicts with duplicate elements.

Set up mobile tests with Vision AI

If you want to use DEX to execute your tests or override the value from settings, you'll need to take a few more steps as the settings are available only in Commander. Once you've created TestSteps from your Vision AI-based Modules, set up your tests:

  1. Add a test configuration parameter (TCP) and name it Automation_Mobile30_VisionAIPersonalToken.

  2. Enter your Vision AI personal access token as the value for the TCP.