Automation Testing Guide for Startups - Level 2: Automate Hundreds of Regression Tests

In the first episode of our 3-part guide, we covered the basics and described how to get the most business value from simple test automation. Now it’s time to learn some more advanced concepts and increase coverage of your automated tests. Here we go!

What will you achieve on this level?

Once you finished this level you will:

• have 100+ nicely structured tests
• be able to update multiple tests at once when needed
• run tests on different environments
• automate testing of new user registration

TLDR:

• Organize your test steps in groups
• Convert some groups into shared components
• Duplicate tests efficiently with “new test from here”
• Set up a URL variable to quickly swap between environments
• Set up a testing email inbox to automate tests of new user registration
• Use built-in variables to generate random user email aliases

Scaling the test automation

In this episode, we will focus on scaling your test automation to a larger number of test cases so that your regression tests also cover corner cases and niche features. This is “Level 2” so things will get more complicated, but BugBug makes everything easy to achieve.

Example: in the episode 1 of this guide we’ve created just one test that checks if log-in works. Now we want to create several tests that check different outcomes of logging in, for example:

• Log in - success
• Log in - incorrect password
• Log in - account blocked
• Log in - password reset required
• Log in - account doesn’t exist

You can see that we aim to increase the test coverage dramatically. Instead of 5-10 tests, we’re going to have hundreds of them.

Keep tests nice and tidy

If you want to have more than 10 tests, you should start organizing the test steps and keep them in order. Why is it important? Imagine that you’ve created 100 tests, but later you changed something in the UI. All of your tests started to fail. If you haven’t organized test steps, you will need to go to each test individually, see what’s wrong and re-record the failed steps 100 times. Fortunately, BugBug allows you to easily avoid such situations.

Group your test steps

Keep your steps tidy by grouping them. Give a clear name to each group, which will help you to navigate through long test cases. Move your cursor in between the steps to see a plus icon, then click “split group here”.

Convert groups to shared components

Once you start grouping your test steps, soon you will realize that some groups are exactly the same in multiple tests. This makes them a perfect candidate for components. Components are magic groups: if you change it in one test, it will change in all other tests that have it.

Start with creating a component for the first step that goes to your starting URL. Then go to each test and replace the normal group with the component by using “Insert existing component”.

Duplicate the tests rather than create new ones

Since you already have components, you may not want to record each test case from scratch. A lot of your tests will start with the same components (for example: go to homepage, log in, enter a password, etc.). You can quickly create a new test “in the middle” of a different test. When you click “new test from here” you will create a duplicate of the test that keeps all the preceding test steps.

If you use components in a smart way, you will be able to fix hundreds of tests with just one re-recording.

Use data-testid attributes as selectors

Automated tests are most reliable if your selectors are clearly identifying an element. The best way is to clearly annotate such elements in your HTML structure. Ask your frontend developer to add data-testid HTML attributes to elements that you click in your automated tests.

Why is it better? Let’s see an example. If you don’t use a dedicated attribute, your login button selector might look like //SPAN[normalize-space(text())="Log in"]. This selector points to any element with text "Log in". If you change the text in the button, such a selector will stop working and your test would fail.

If you used an attribute, for example [data-testid=”login-submit”] your tests will be more resilient to changes. Even if you change the text in the button, you will not need to edit your automated test. What’s more, the developers will see this attribute in the code so they will be aware that this specific item is a subject of test automation. Once you have such attributes in your HTML, BugBug will automatically create selectors based on “data-testid” attributes during the recording.

Build robust selectors without coding

Sometimes it's not possible to create a reliable selector with recording. Dynamic lists, tables and hover actions are notorious for being difficult to record. You just need to use your human intelligence and use a more complex XPath selector. We created a dedicated tool for that with an in-depth guide - see No-code XPath Builder.

Work with different environments

If you are fully committed to testing automation, chances are you will need to run tests on different environments. Each environment can be accessed with different URLs, for example:

• Production - acme.com
• Staging - stage.acme-test.com
• Local - localhost:3000

You should store these URLs in a variable with 3 profiles. After you set up more than one profile, a new dropdown menu will appear in each test, that lets you run tests with different environments. Learn more in our detailed guide.

Automate tests for new user registration

To reliably automate user registration you need to use a combination of several advanced features

• Third-party inbox that supports plus-addressing (for example Outlook.com)

• Built-in “testRunId” variable for generating random e-mail address alias

Learn more in our separate step-by-step guide on testing sign-up.

Keep it simple

Don’t try to overcomplicate your test steps. Use components and variables only when it will actually save time for you in case of product changes. Remember that re-recording is a really fast way of fixing existing tests - sometimes it’s just faster to re-record steps in 5 tests than debugging a complex net of variables and dependencies. Keep it simple and if you have any problems feel free to talk to our friendly customer support. 😊

Read next: advanced guide with tips on CI/CD integration