Maciej Wyrodek
Table of Contents
Testing, especially end-to-end, is currently enjoying a revival. A lot of new testing tools and approaches have emerged over the last few years. Selenium has finally launched version 4, Cypress and Playwright have brought coded test automation to a new level and companies like Bugbug, Mabl, Testim, have revitalized the idea of record and playback testing.
Since a lot is happening, let's take a look at specific tools and compare the strengths and weaknesses of 3 approaches: Custom Made Frameworks, Out of the Box Framework and Record and Playback. We will look at the pros and cons of all the solutions and when they work best.
However, first we need to discuss the reasons for using automation and then we will discuss how to use record and playback testing.
Why we automate tests?
If we look hard enough, all goes back to the one and only goal of companies, as stated by Eliyahu M. Goldratt in The Goal - Making money. Testing by itself often is considered a cost - as it does not bring more money.
But let's not forget that more money can be made not only by selling more - but by increasing the margin - either by making the product worth the price or by reducing the cost.
Testing can help to protect the bottom line, by reducing the cost of mistakes and finding them more easily and faster. This also leads to reducing the cost of customer support, refunds, etc.
But as DevOps and Modern Testing state - testing can synergize with a great experimentation culture, leading to possible delivery of a quality product in a shorter time and cost-savings on the development cycles.
Traits of good automated testing tools
For automation to be useful in the DevOps world, it needs to be:
- Reliable
- Fast
- Easy to write and maintain new test
- Easy to run
Reliability
A reliable testing framework consistently produces accurate and consistent results. It should be able to detect defects reliably and identify regression issues when changes are made to the software.
Improve your testing, fast
The development cycle is fast. If developers need to wait for the results of their work more than a coffee break, they will switch to the next task. Now the results will need to wait which leads to the cost of context switching. And longer lead time of a task.
Ease of writing and maintaining
Automated testing should be a regular part of a software development lifecycle, benefit must justify the cost. If writing a test takes 4 days, it'd better be worth it - especially if developing the tested feature takes less than that.
The same holds true for maintenance: we need to be able to easily understand why a test has failed and whether this is a code issue or a test issue and how to fix it.
Ease of running
This relates to being fast: anybody who needs to use a test to check a code should be able to do it without extra hoops, for example automatically running them locally or via CI/CD.
How to choose the best test automation tool?
- Understand testing requirements: Define project objectives, scope, and the types of tests to automate. Consider application type (web, mobile, desktop), supported platforms, and required test coverage.
- Check technology compatibility: Verify if the tool supports your application's technologies, programming languages, and frameworks and testing needs.
- Evaluate ease of use: Consider your team's expertise. People can benefit from user-friendly interfaces or record-playback features, while developers may prefer script-oriented tools.
- Ensure cross-platform support: If testing is needed on multiple platforms (e.g., Windows, macOS, Linux, Android, iOS), choose a tool that supports cross-platform testing.
- Assess scripting languages and frameworks: Pick a tool that aligns with your team's skills, facilitating maintenance and extensions.
- Examine community and support: Use modern testing tools with strong online resources, documentation, and active user communities for better assistance.
- Consider vendor support and reliability: Choose reputable vendors providing regular updates, bug fixes, and customer support.
- Analyze cost and licensing: Assess total ownership costs, including licensing, training, and maintenance, seeking value for money within budget constraints.
Record playback testing tools
Record and playback test automation also known as capture-replay or scriptless tools, are a type of automated software testing tool that allow employees to create test cases by recording their interactions with the application under test (AUT).
The tool records these interactions as a sequence of user actions (test steps) and system responses, which can be played back later to re-execute the test scenario automatically.
Here's how record and playback testing tools generally work:
- Record The Test: User interacts with the application as if performing manual testing. The tool records each action taken by the tester, such as clicks, keystrokes, form submissions, and other user interactions. It captures the sequence of events and builds a test script based on these interactions.
- Playback: In the playback phase, the recorded test cases script is executed automatically against the application. The tool replicates the actions recorded during the recording phase and verifies the responses from the application against the expected outcomes.
Record and replay testing tools are commonly used for web applications and graphical user interface (GUI) testing. They are particularly beneficial for QA engineers who may not have strong programming skills or for quickly generating test scripts during the early stages of testing.
Which testing approach is for you?
We will look at the 3 most popular approaches, starting from ones requiring most coding and maintenance effort, like "Custom Frameworks", through usage of "Out of the box" Frameworks and finishing with playback automation. We will look at the pros and cons.
As it is my belief, all these approaches have made a name for themselves on the market and suit different project needs.
Pros
- Great customizability
- Ownership
- "On-site support"
Cons
- Huge maintenance cost
- Time consuming
- Requires lots of skill to maintain
When I was starting test automation, this was the way to go. Mainly because Selenium WebDriver was the tool of choice back then. And as its description stated, Selenium automates browsers. Selenium was just a tool that did exactly what it said it delivered.
It didn't provide any functions for running test, or making assertions. At that time, practically every company had to build its own framework.
In most cases, they were just learning how to automate less complex tests. Making lots of mistakes in the process. But some companies managed to make great frameworks and, as a result, decided to share them with others. Some of them are still in use and developed but other have not stood the test of time. Ocaramba,Selino, Serenity,Selendie, Protractor
This approach is still used today. Why?
Pros
The main advantage of this approach is control. It gives a company the greatest level of control in customization. If a specific feature for a project is required, it is relatively easy to find someone who can develop this functionality.
Additionally, it gives a company considerable independence; there is no risk that at one day, the tool support will end or repo will be deleted, leading to massive issues.
What is more, there is someone who knows how the tool works on site and can teach new team members..
Cons
The main drawback of this solution is the cost - maintaining internal test automation framework costs time and money. Many companies underestimate the cost of developing test automation.
Time is of essence here: this automation needs a long time to be developed, from creating a framework to writing a test. And this is the aspect that can be most damning. As discussed before, for automation to bring value, it needs to provide fast feedback.
Creating a custom framework is as big a project as any and it requires a good dose of skills, ideas, and planning. This means a lot of expertise while typically, companies are not willing to spend precious resources on test automation.
Out of the box frameworks: Pros & Cons
Pros
- Shorter time needed to write first test
- Much less need for framework maintenance
- Lots of available online trainings and knowledge
Cons
- Often only limited support from community
- Writing and Test Maintenance takes time
- Huge dependence on framework developers
As mentioned before, the growth of Selenium was accompanied by a lot of frameworks created on its base. Over time, the issues with the web driver became increasingly prominent, leading to the re-emergence of new automation frameworks not based on Selenium: Cypress, Puppeter, Playwright, to name a few most recent ones.
All these tools had one thing in common - they provided the user with out-of-the-box, ready options to write automation. These tools usually provide built-in test runners, an extensive assertions library, boiler plates for starting writing project and tests, and - most importantly- a built-in mechanism to deal with test flakiness and general instability.
Additionally, these frameworks usually provide their own philosophy of writing test automation. As long as what we need to do is with accordance to their approach, everything goes well. Problems start when we need to go beyond what is suggested.
Until recently, the best example has been a test covering multiple domains in Cypress, but as of version 9.6, it is no longer such a grievous case.
In another scenario, we need to rely heavily on framework developers, if they introduce breakthrough changes. Usually, we are left with two options: update and fix the code. Alternatively, don't update and hope that there won't be any issues in the framework that need fixing.
Pros
Compared to the custom frameworks, we can start writing tests almost immediately without the need of spending weeks or months on writing the toolset first.
Similarly, we don't need to wait whenever the operating system or browsers have an update. The tool maintainer will update the framework for us.
And since most tools in this category are open-source ones, there is lots of information easily available online.
Cons
As mentioned before, these tools have their own test philosophy, which is much more restrictive. Quite often support is limited - these tools are usually open-source and support in case of issues can be unavailable where we need to resort to forums and chats where we hope someone will answer a question; sometimes we get the answer in minutes but it may never come just as well.
However, the greatest risk of all is that we are at the mercy of framework developers - these tools become a part of our code base and if we want to keep support, sooner all later we will need to update even if we don't want to.
A case in point is Cypress 10 - it introduced breakthrough changes. As a result, many plugins and customer functionalities won't work without extensive updating.
Record and Playback Testing Tools: Pros & Cons
Pros
- Fast Test Writing Time
- Low Entry Level
- Professional Customer Support
- Easy Scalability
Cons
- Usually cloud-based, requires access to test environments
- Still young, "untested" technology
- Requires a new mindset
- Data management can be quite complicated
- Usually a commercial paid tool
The idea of record and playback is not new, even Selenium has Selenium IDE which, after a few years on developments lately, has won some users' recognition.
Microsoft had Coded UI Test as part of the Visual Studio from 2010 and only in 2019 announced it deprecation
Back then this technology was long before its time, the frameworks were very unstable and the maintenance of this test was problematic.
But in the last 4-5 years, a lot has changed. Solutions like Bugbug.io, Mabl, Testim have taken over what modern technology has to offer and reinvented the concept.
Pros
Most coded options require great investment of time into writing and maintaining test - usually even a trivial change costs around half an hour.
In case of record and playback feature, test are usually meant to be disposed of after they stop working, as it is much easier to re-record a test instead of fixing it.
This is of special importance for startups and projects with a lot of changes in the UI layer. When a team is spread thin, it is hard to find the time to write a test. But usually, you can record a few tests while checking if what has been done even works.
On the other hand out-of-the-box coded frameworks offer usually an easier learning curve. Record And Play tools usually do not require coding knowledge - although in more complicated cases some JavaScript knowledge can prove very helpful.
Since most of the record and play tools are paid solutions, it also means that companies have professional Trainers and Customer Support to help when issues arise. Usually, it means that in the case of young projects, we can even influence their feature roadmap.
These modern record and play solutions come with their own cloud runners, making scalability unbelievingly easy. Typically, paralyzing a test takes a few clicks.
Cons
Unfortunately, Cloud Running is also a huge Con. First and foremost, we need to give this tool access to our test environments. An additional issue related to the Cloud is Term of Service - not all projects and customers can afford cloud services.
Most of these Record & Play tools are relatively young and not battle-tested. For long-term projects, this uncertainty can be very risky.
The biggest issue is Data Management. So far, no Record and Play solution has provided an easy way to manage test data. And this can be a huge blocker. Consequently, we need to change our approach to E2E automated tests, with a desire to cover all the important and easy paths as fast as possible.
Lastly, there is the price - cloud solutions are not cheap and if we need to run a lot of tests, it may be quite expensive. But some tools like BugBug are on the other hand very cheap and offer unlimited cloud runs for a fixed mohtly fee. Arguably, it is still less expensive than maintaining coded automation and its infrastructure.
No clear-cut distinction
Before the final thought, some clarification is required:
As much as we humans like clear categorizing, the real-word borders are much more blurred.
For example: should Playwright be in the Custom frameworks or in the Out-of-the-box category? After all, its strength is the ability to use its own test runners, reporting services and easy building of the new features we need.
Similarly, it has the Record and Play functionally and so did Cypress (at the moment of writing this post, the latest version of Cypress is 10.3 and since 10 Cypress Studio, which allowed test recording, has been deactivated).
There are also tools that I like to refer to as combine-harvesters: basically, they do everything and claim that they can test anything, from mobile via web to desktop and many others -- the examples include TOSCA and Test Complete.
That's why we need to look at the numerous different dimensions of automation - and how much coding is just one of them.
Summary
It is my belief that over time, both no-code development and record and play automation become the dominant players on the market. Yet there is a lot of room for improvement.
All the types of approaches to automation, starting from custom frameworks, or using out-of-the-box coded solutions ending on Record and Play tools, have their uses on the market.
They answer to different needs and solve different problems. So, before selecting any tool, it is always good to ask yourselves: "What are we trying to gain?"
