Smart automation – reaping the rewards of testing’s third wave
The benefits of test automation are plentiful and well publicised, and the process isn’t new. Indeed, for many, automating testing is now entering its ‘third wave’.
The first wave, filled with some good old-fashioned vendor tools like WinRunner, Silk Test, and QTP, started the move towards true automation – and set the stage for the innovations which we all now rely on, like Selenium and Appium.
In the second wave of test automation things got more sophisticated, and promised to resolve the most pressing challenges that development teams faced. New testing tools provided visibility on utilisation and upcoming demand, helped build sufficient development and test environments for agile development processes and promised easy integration at early stages of testing.
And today, entering the third wave, the watchword is ‘smart’ automation. Simply put, smart practices make test automation easier. They give us faster release cycles, better tests and smoother collaboration between all practitioners.
Smart test execution allows elasticity in scale, smart labs can self-heal itself when a platform either disconnects or is busy, and smart analysis can shrink the time needed to analyse and debug issues – especially when dealing with big test data – all crucial in ensuring that speed-to-market and quality assurance go hand-in-hand.
Moving toward smart automation
However, even though most companies believe that smart automation is important, at the early stages of this ‘new wave’, few companies can claim true success. It’s true that the authoring part is less complicated, but the matching of such testing to the overall process, and combining it with existing code-based tests, is still a challenge.
The result is that many organisations are still unsure of what to automate and how to do it smartly versus what to leave for code-based testing.
So, how can businesses turn the tide and make smart automation efforts a success?
First and foremost, the complexity involved in automation strategies shouldn’t be underestimated. Automation in the digital era is more difficult than ever before.
Desktop and mobile browsers support increasingly more advanced capabilities. Services like FaceID, location capabilities and virtual reality (VR) / augmented reality (AR) are tough to automate and scale across multiple permutations. In addition, the available time within software iterations is continually shrinking – and so, for many, that means that there isn’t sufficient time to create automation that really works.
Test automation creation isn’t always structured or able to follow best practices, and this leads to flaky and inconsistent test scenarios – something which is exacerbated by teams that don’t have the test automation skills (or time) to lend to the task.
So, there’s a lot to contend with.
Getting it right
It’s the organisations which take a strategic approach, with clear goals, and with the right tools in place, who will ultimately make smart automation a success. So, how do they get there?
Firstly, it’s important to remember that not all test automation is created equal.
At its most basic level, automation stands for the process of taking a set of requirements and creating code to implement them in an automated fashion. Traditional automation refers primarily to static test scenarios, such that whenever a change occurs in the app – it requires code changes, refactoring and other time consuming tasks.
In addition, traditional automation is error-prone since it is created manually, and so subject to human error (wrong object IDs, duplications, wrong prerequisites to the test scenario implemented, lab issues, etc.).
Smart automation, on the other hand, takes traditional automation to the next level by minimising the human risk error, as well as eliminating the need for test maintenance. Since smart automation often relies on record and playback, there is close to zero code that needs to be managed and maintained, and this is clearly a huge benefit to teams. Coders at every skill level can use these tools – the output (the tests themselves) are more reliable, robust, and the flakiness that traditional code-based automation creates is reduced to a minimum.
So, automating everything shouldn’t be the goal of continuous testing in DevOps. Instead, what’s important is automating only what is relevant from a risk mitigation / reduction perspective, and focusing on what provides value and feedback to the developer.
The first step is for an organisation to assess their current state: looking at what tools are available today, what resources and skills are available, and what the teams’ future roadmap looks like. They should then drill down deeper still, looking into the pipeline to realise the percentage of reliable tests, the percentage of successful CI builds – and the covered vs. uncovered functional areas in the application.
This will allow them to build a pipeline of tests that are candidates for smart automation migration – and, equally importantly, decide which ones are not.
In our next step, as well as scaling the number of tests, teams must put fast feedback capabilities into place – to analyse and record the tests effectively on a big scale. It’s a common mistake not to put fast feedback loops in, the outcome being that teams drown in data. There’s no point in automating hundreds of scrips it they’re not effectively analysed.
The final stage is putting together a process on how to maintain these tests and keep them valuable at the same time.
A focus on tools
By following these steps, teams will be well on the way to effective smart automation. But of course, they must also have the right tools in place.
When it comes to technology and tools, there’s a lot of choice in the market, so this means working with trusted partners, evaluating providers and tools against your objectives, and perhaps even looking at hybrid options to fit your needs.
Support for highly complex scenarios especially in the mobile space requires a more mature version of the smart automation tools, many of which are currently available. However, more tools are being released to market all the time.
As testing cycles become shorter, it’s inevitable that teams will forgo testing extraneous aspects of their application. It’s essential that the risks of doing so are profiled adequately, so developers can ensure they’re testing the most important features first. By harnessing AI, systems can identify what should to be tested or prioritised based on prior test history or risks that are ranked by how critical they are, their complexity, and performance.
AI classification learns from correlations in past results to refine application test coverage through recommended test scenarios based on risk and identified root causes to quickly resolve defects. Furthermore, using ML and AI in smart record and playback test creation allows robust object identification and analysis.
Eran Kinsbruner, chief technical evangelist for Perfecto