Paweł Bylina
Are you a tech leader, CEO or CTO in a tech startup? You want to implement automation testing for your SaaS product? We've written this guide especially for you. It's packed with actionable tips on how to approach test automation in an agile way.
We're going to tackle the following questions:
-
What is the business goal of automation testing?
-
How to implement automation incrementally?
-
What resources do you need to develop automated tests?
-
What are typical problems and obstacles and how to overcome them?
The strategic approach to automation testing
Automation testing is a very broad term. There is no single goal of automation. Depending on the level of automation you will get different benefits for your organization.
Figure 1: Strategic matrix of different goals of test automation
See this strategic matrix above. You can achieve all of these goals, but not all at once. You should pick the right order of goals, starting with the ones that are low-effort and high-value.
To achieve each of these goals, you need a different combination of resources and skills. We'll guide you through each of the goals, explaining the approach, time & resources and how to achieve it with BugBug.
Step 1: low cost & high impact goals
Goal:
- Prevent loss of business due to production downtimes
Approach
-
You don't need to hire new people, you can do it yourself
-
Create 5-10 automated tests and run them every hour to monitor your production, for things like login, landing pages, registration form, contact form, basic app navigation
Time & Resources
- 2-4 hours of your own time
BugBug features that you should use
Step 2: low hanging fruit
Goal:
- Improve team collaboration on quality assurance
Approach
-
Give all the scrum teams access to the testing platform
-
Share direct links to failed tests in your tickets or on a chat
-
Configure who gets notifications
Time & Resources
- Inviting people only takes 5 - 10 minutes, but you should spend 2-4 hours on communicating its purpose
BugBug features that you should use
Step 3: incremental development of automated tests
Goals
-
Save time on manual regression testing
-
Release faster
-
Prevent cost of regression bugs
Approach
-
Talk to the whole team about what you want to achieve and what's the incremental plan
-
Choose 1-2 people responsible for automated tests
-
Create dedicated test user accounts that will be used for automation only
-
Run automation on some testing environment (staging)
-
Create separate test suites for staging and prod (run all tests on staging, but only essential smoke tests on prod)
-
Add more tests incrementally, 2-3 test cases per week (after 3 months you should have about 50 test cases automated)
-
Start adding data-test-id attributes in your HTML
Time & Resources
-
You will see the benefits after 4-8 weeks of ongoing work
-
You need to engage 1-2 people on the team. Make sure they have at least 2 hours per week to work on automation.
-
You need at least one technical person on the team who knows how selectors work.
BugBug features that you should use
-
Custom javascript might be needed
Step 4: integrate automation with CI/CD
Goal:
- Improve software development lifecycle, being sure that no code is merged without tests
Approach
-
Integrate automation with your development pipelines
-
Trigger automated tests with every pull request
-
Show test results directly in your repo tool (Bitbucket, GitHub, GitLab, etc.)
-
Ask developers to run automated tests locally on the localhost environment before they commit code
Time & Resources
-
You will see the benefits after 4-8 weeks of ongoing work
-
You need to engage 1-2 people on the team. Make sure they have at least 2 hours per week to work on automation.
-
You need at least one technical person on the team who knows how selectors work.
BugBug features that you should use
Step 5: increase coverage
Goal:
- Get 100% test coverage, be confident that all the features, all the browsers, all the devices work correctly
Approach
-
Pick your battles, choose what is the most essential to automate
-
We recommend you keep adding 1-2 test cases every week and don't try to achieve the impossible 100% coverage
Time & Resources
-
To automate everything you would need an Infinite amount of time 😃
-
To automate 2-3 new test cases per week you still need 2-3 hours weekly plus a bit more experience with tests and javascript.
-
Note that the more test cases you have, the longer the maintenance. The trick is to find the right balance.
BugBug features that you should use
-
Project settings > Browser width (for testing mobile screens)
-
Lots of custom Javascript
Common problems and solutions in implementing automation testing:
Here are some typical problems you may encounter when you want to start automating tests for a SaaS product. We've made a short list with possible solutions that worked for other BugBug clients.
Captcha
You need to disable the captcha only for automation testing. Unfortunately, it's not trivial and you need to read through captcha docs or ask a developer for help.
Date-based features, calendars
Is your app searching train timetables for tomorrow? It's difficult to automate because the date and the expected outcome can change every day.
Solution A: In the automated test, pick a date that's far in the future. Then update the tests regularly, changing this date and the expected outcome.
Solution B: create a dedicated environment for automation with a mocked-up hardcoded date on the backend
Dynamic lists such as search results, product listings
It's hard to automate a test if some data is changing all the time. You need to create independent tests that are always sure what to expect, for example, if you're checking a list of promoted products on your homepage, your test should first go to the admin panel and mark such products as promoted.
You can also have a separate dedicated test automation environment where you have a hardcoded set of data.
Can't use the same user email twice
Create a new user with each test - use the built-it variable "testRunId" in combination with a temporary email catcher always to run tests with a fresh new user. You can also delete these users at the end of the test by recording test steps in an admin panel (if you have one) Read more in how to test user registration.
Selectors are not reliable
The only way to make sure your tests are stable and easy to update is to start using "data-testid" attributes in your HTML structure, dedicated attributes for automation testing selectors. You need to communicate that to the front-end developer and make sure you remember about it during the code review. No worries, adding additional attributes have no risk, it's just additional data for automation with no other function, it will not introduce new bugs.
Multilanguage
Your page can run in different languages and you want to verify all of them. Try to assess the risk of having a bug only in one language without affecting the others. Perhaps it's not likely that such a bug would occur?
If you still need to test different languages, you have two options:
Option A: Record separate tests for each language. Hard to maintain on a bigger scale
Option B: Use variables & profiles to run the same test on different languages
-
Add data-testid to all of your app elements, so that the selectors are independent of the app language and the displayed test
-
Add a way in your app to change the language via a URL, for example, "myapp.com/?lang=fr"
-
Create a variable "language code"
-
Create different profiles with different language codes
-
Replace the language code in the "Go to URL" step with the variable
-
Create separate suites for all the languages, overriding their profile with a chosen language
Tests take long time to finish
Upgrade your BugBug plan to be capable of running cloud tests in parallel
Downloading files
Currently BugBug can't verify if the files have been download. Soon we'll add a new type of assertion for that, talk to our support if you need it ASAP.
Drag & drop interactions
BugBug support for drag&drop is in beta, you can achieve it by combining the "Mouse press", "Hover" and "Release mouse button" actions. Talk to our support for details.
A recorded selector doesn't work
Sometimes it's not possible to record a reliable selector and you need to use your human intelligence and debug HTML a bit.
Use our No-code XPath Selector Builder which makes it easy to craft a good selector manually and can teach you in-depth about best practices in selectors.
Success in test automation depends on the regularity of your progress
Implementing automation testing in a SaaS company is a long-term process. You should do it step by step. The biggest challenge for tech leaders is lack of time to dedicate on QA area. The truth is you don't need a lot to push it forward. Better slow than never. 😉 Block yourself a slot in calendar every week to work on QA automation and keep the momentum going.
