Gil Sever is co-founder and CEO of Applitools.
In my previous article, I dug into autonomous testing, why it’s the future of test automation, and what the path to fully autonomous testing looks like.
In this article, let’s analyze the different technologies needed to create an end-to-end autonomous test platform. First, I will provide an overview of the technologies needed for such a platform before describing and analyzing two technologies from this list.
Here are the must-have technologies for a complete, end-to-end autonomous test platform.
• Automatic creation of test cases.
• Validation that the test cases fully cover the functionality of the application we want to test.
• Automatic generation of test code.
• Automatic generation (or collection) of test data.
• Test Execution: Execute all required tests, ensuring that the required coverage has been achieved (and on time).
• Automatic validation of the expected result after each test step and each test scenario.
• Automatic test code maintenance when application changes.
• Automatic decision if the application is ready to be published.
Today, companies are working on different technologies to implement subsets of the above capabilities. At some point, when enough components are available, they can be combined into one complete standalone test tool.
As with any complex project, there should be a gradual path to implementation. In the case of stand-alone tests, automatic validation and automatic test creation (or automatic navigation) are the first two steps and form the core of the end-to-end solution.
There are at least two automatic validation technologies in existence today: the use of test scripts to validate the content of various software objects in the application under test, and automatic visual validation based on the comparison of images of expected result screen with actual screen images being processed.
Although both are automation-based technologies, object-based technology requires writing a lot of code that needs to be maintained. This requires human intervention that is difficult to replace. Visual validation requires only a single line of code for each test step, which can make it an ideal technology for standalone no-code solutions.
When it comes to visual validation technology, AI-based visual validation may be a better solution than heuristic image processing algorithms.
When it comes to automatic navigation in the application we want to test, we see three main approaches: record and playback, script-based, and design-based. These approaches generate code that can navigate an application under test, provide necessary inputs to the application, and validate results at each stage of test execution.
The first two approaches have an initial phase where a quality assurance (QA) professional or automation engineer must determine the navigation paths and validation criteria.
The third approach (design-based) avoids a manual initiation phase by building tests based on usage paths defined in the application design phase. In the long run, this may be the preferred method of automatically accessing navigation patterns for testing. However, this requires full integration of stand-alone tests with application design documents, which is not yet common practice. Additionally, there are always lags between design and implementation, and there are additional insights and usage paths that aren’t obvious at the design stage or aren’t reflected in design documents. design.
Over the next few years, software testing teams will need a manual initiation phase and may need to manually intervene in automated decisions. Hopefully the automatic process can cover most cases and limit the number of manual interventions.
Two additional approaches that can help reduce the manual initiation phase are analyzing user behavior patterns and analyzing user interface (UI) elements.
Analyzing user behavior patterns requires the app to have users who have already been using the app for some time. However, it cannot be used until a new application has been deployed where user behavior can be monitored to allow ML algorithms to build a set of behavior patterns.
Analyze user interface elements can start from scratch, but it is the most complex problem to solve and it relies entirely on the predictive capabilities of AI. Moreover, such an approach is almost guaranteed to have coverage gaps, missing certain use cases and user behavior patterns.
Therefore, until stand-alone testing has matured, automatic navigation should include a phase of manual initiation and intervention to create an initial set of tests, including navigation paths, test data and validation criteria.
When standalone testing becomes a reality, it will be codeless and require minimal human intervention. Let’s look at the pros and cons of record-and-playback and script-based approaches.
recording and playback the automatic navigation approach provides an easy path for manual testers, while the script-based approach This approach provides flexibility and scalability to test automation engineers. A design based on approach allows to import different potential flows in an app from popular design apps like Figma, Adobe XD, Sketch, etc.
A record-and-playback approach or a design-based approach can be a great starting point for users who don’t want to worry about the internal details of test code, and we should consider these approaches, as they can be the simplest ones first. step towards stand-alone testing for potential users.
However, it is important that the code for stand-alone tests is exposed and available to test engineering groups so that human intervention can be made possible when a stand-alone test generator fails to achieve the expected result or produces a result. wrong. Since the transformation from no-code to code can only work in one direction (it is practically impossible to go from code to no-code), it is important that all subsequent processes, such as automatic test maintenance, are based on the application’s code base.
In future articles, I will dig a deeper level into the various subunits of a standalone test platform and the implementation considerations associated with them.
Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs, and technology executives. Am I eligible?