Guides to Automation

Software developer currently working as a tech writer and PM.  
[linkedin.com/in/marcin-%C5%82ojewski-a4191518/](https://www.linkedin.com/in/marcin-%C5%82ojewski-a4191518/)

Marcin Łojewski

small_Testing Tools for Web Application.png

thumbnail_Testing Tools for Web Application.png

Testing Tools for Web Application.png

Testing Tools for Web Application - The Ultimate Guide 2023

CEO of BugBug.io

[linkedin.com/in/pbylina/](https://www.linkedin.com/in/pbylina/)

Paweł Bylina

Are you a tech leader, CEO or CTO in a tech startup? You want to implement automation testing for your SaaS product? We've written this guide especially for you. It's packed with actionable tips on how to approach test automation in an agile way.

We're going to tackle the following questions:

-   What is the business goal of automation testing?

-   How to implement automation incrementally?

-   What resources do you need to develop automated tests?

-   What are typical problems and obstacles and how to overcome them?


The strategic approach to automation testing
--------------------------------------------

Automation testing is a very broad term. There is no single goal of automation. Depending on the level of automation you will get different benefits for your organization.

*Figure 1: Strategic matrix of different goals of test automation*
![bugbug-goals-of-automation-testing.png](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/bugbug_goals_of_automation_testing_9308b9cb1b.png)

See this strategic matrix above. You can achieve all of these goals, but not all at once. You should pick the right order of goals, starting with the ones that are low-effort and high-value. 

To achieve each of these goals, you need a different combination of resources and skills. We'll guide you through each of the goals, explaining the approach, time & resources and how to achieve it with BugBug.
->[Automate your tests for free](https://app.bugbug.io/sign-up/){.markdown-cta}<-




Step 1: low cost & high impact goals
------------------------------------

### **Goal:**

-   Prevent loss of business due to production downtimes

### **Approach**

-   You don't need to hire new people, you can do it yourself

-   Create 5-10 automated tests and run them every hour to monitor your production, for things like login, landing pages, registration form, contact form, basic app navigation

### **Time & Resources**

-   2-4 hours of your own time

### **BugBug features that you should use**

-   [Recording tests](https://docs.bugbug.io/creating-tests/recording-the-tests)

-   [Cloud runs](https://docs.bugbug.io/running-tests/in-cloud-on-server)

-   [Schedules](https://docs.bugbug.io/running-tests/schedules)

-   [Notifications](https://docs.bugbug.io/collaboration/notifications)

-   [Re-recording test steps](https://docs.bugbug.io/recording-tests-steps/re-recording-steps)

Step 2: low hanging fruit
-------------------------

### **Goal:**

-   Improve team collaboration on quality assurance

### **Approach**

-   Give all the scrum teams access to the testing platform

-   Share direct links to failed tests in your tickets or on a chat

-   Configure who gets notifications

### **Time & Resources**

-   Inviting people only takes 5 - 10 minutes, but you should spend 2-4 hours on communicating its purpose

### **BugBug features that you should use**

-   [Invite users](https://docs.bugbug.io/collaboration/inviting-team-members)

-   [Project settings > Notifications](https://docs.bugbug.io/collaboration/notifications#choose-when-people-get-email-notifications)

Step 3: incremental development of automated tests
--------------------------------------------------

### **Goals**

-   Save time on manual regression testing

-   Release faster

-   Prevent cost of regression bugs

### **Approach**

-   Talk to the whole team about what you want to achieve and what's the incremental plan

-   Choose 1-2 people responsible for automated tests 

-   Create dedicated test user accounts that will be used for automation only

-   Run automation on some testing environment (staging)

-   Create separate test suites for staging and prod (run all tests on staging, but only essential smoke tests on prod)

-   Add more tests incrementally, 2-3 test cases per week (after 3 months you should have about 50 test cases automated)

-   Start adding [data-test-id attributes](https://docs.bugbug.io/preventing-failed-tests/selectors#use-custom-attributes-for-selectors) in your HTML

### **Time & Resources**

-   You will see the benefits after 4-8 weeks of ongoing work

-   You need to engage 1-2 people on the team. Make sure they have at least 2 hours per week to work on automation.

-   You need at least one technical person on the team who knows how selectors work.
->[Automate your tests for free](https://app.bugbug.io/sign-up/){.markdown-cta}<-


### **BugBug features that you should use**

-   [New test from here](https://docs.bugbug.io/creating-tests/new-test-from-here)

-   [Duplicating tests](https://docs.bugbug.io/creating-tests/duplicating-tests)

-   [Components](https://docs.bugbug.io/editing-tests/wip-make-component)

-   [Variables](https://docs.bugbug.io/editing-tests/variables)

-   [Profiles](https://docs.bugbug.io/editing-tests/variables#profiles) to [test on different environments](https://docs.bugbug.io/editing-tests/variables#work-with-different-development-environments)

-   [Suites](https://docs.bugbug.io/organizing-tests/suites)

-   [Manually editing steps](https://docs.bugbug.io/editing-tests/manually-creating-the-test)

-   [Testing registration](https://docs.bugbug.io/editing-tests/variables#test-user-registration-and-login-using-variables)

-   [Custom javascript](https://docs.bugbug.io/editing-tests/custom-javascript-actions) might be needed

Step 4: integrate automation with CI/CD
---------------------------------------

### **Goal:**

-   Improve software development lifecycle, being sure that no code is merged without tests

### **Approach**

-   Integrate automation with your development pipelines

-   Trigger automated tests with every pull request

-   Show test results directly in your repo tool (Bitbucket, GitHub, GitLab, etc.)

-   Ask developers to run automated tests locally on the localhost environment before they commit code

### **Time & Resources**

-   You will see the benefits after 4-8 weeks of ongoing work

-   You need to engage 1-2 people on the team. Make sure they have at least 2 hours per week to work on automation.

-   You need at least one technical person on the team who knows how selectors work.

### **BugBug features that you should use**

-   [CLI](https://docs.bugbug.io/integrations/api)

-   [Overriding variables](https://docs.bugbug.io/integrations/api)

-   [Running on localhost](https://docs.bugbug.io/running-tests/test-in-your-browser)

Step 5: increase coverage
-------------------------

### **Goal:**

-   Get 100% test coverage, be confident that all the features, all the browsers, all the devices work correctly

### **Approach**

-   Pick your battles, choose what is the most essential to automate

-   We recommend you keep adding 1-2 test cases every week and don't try to achieve the impossible 100% coverage

### **Time & Resources**

-   To automate everything you would need an Infinite amount of time :)

-   To automate 2-3 new test cases per week you still need 2-3 hours weekly plus a bit more experience with tests and javascript.

-   Note that the more test cases you have, the longer the maintenance. The trick is to find the right balance.

### **BugBug features that you should use**

-   Project settings > Browser width (for [testing mobile screens](https://docs.bugbug.io/workflow-tips/mobile-version-testing))

-   Lots of custom Javascript

-   [Javascript variables](https://docs.bugbug.io/editing-tests/variables#javascript-variables)

-   [Javascript assertions](https://docs.bugbug.io/editing-tests/custom-javascript-actions#advanced-javascript-assertions)

-   [Parallel running](https://docs.bugbug.io/running-tests/parallel-mode)

->[Automate your tests for free](https://app.bugbug.io/sign-up/){.markdown-cta}<-

Common problems and solutions in implementing automation testing:
-----------------------------------------------------------------

Here are some typical problems you may encounter when you want to start automating tests for a SaaS product. We've made a short list with possible solutions that worked for other [BugBug clients](https://bugbug.io/case-study/).

### **Captcha**

You need to [disable the captcha only for automation testing](https://docs.bugbug.io/troubleshooting/captcha-in-automation-testing). Unfortunately, it's not trivial and you need to read through captcha docs or ask a developer for help.

### **Date-based features, calendars** 

Is your app searching train timetables for tomorrow? It's difficult to automate because the date and the expected outcome can change every day.

Solution A: In the automated test, pick a date that's far in the future. Then update the tests regularly, changing this date and the expected outcome.

Solution B: create a dedicated environment for automation with a mocked-up hardcoded date on the backend

### **Dynamic lists such as search results, product listings**

It's hard to automate a test if some data is changing all the time. You need to create independent tests that are always sure what to expect, for example, if you're checking a list of promoted products on your homepage, your test should first go to the admin panel and mark such products as promoted.

You can also have a separate dedicated test automation environment where you have a hardcoded set of data.

### **Can't use the same user email twice**

Create a new user with each test - use the [built-it variable "testRunId"](https://docs.bugbug.io/editing-tests/variables#use-built-in-variables-for-dynamic-or-random-values) in combination with a [temporary email catcher](https://www.google.com/search?q=temporary+email&ei=ARVtY_X8IomKrwS_i4k4&ved=0ahUKEwi1gZ2u86P7AhUJxYsKHb9FAgcQ4dUDCA8&uact=5&oq=temporary+email&gs_lcp=Cgxnd3Mtd2l6LXNlcnAQAzIECAAQQzIFCAAQkQIyBQgAEJECMgUIABCABDIFCAAQgAQyBQgAEIAEMgUIABCABDIFCAAQgAQyBQgAEIAEMgUIABCABDoKCAAQRxDWBBCwA0oECE0YAUoECEEYAEoECEYYAFDhAVjhAWC1CmgCcAF4AIABa4gBa5IBAzAuMZgBAKABAcgBCMABAQ&sclient=gws-wiz-serp) always to run tests with a fresh new user. You can also delete these users at the end of the test by recording test steps in an admin panel (if you have one) Read more in how to test user registration.

### **Selectors are not reliable**

The only way to make sure your tests are stable and easy to update is to start using "data-testid" attributes in your HTML structure, [dedicated attributes for automation testing selectors](https://docs.bugbug.io/preventing-failed-tests/selectors#use-custom-attributes-for-selectors). You need to communicate that to the front-end developer and make sure you remember about it during the code review. No worries, adding additional attributes have no risk, it's just additional data for automation with no other function, it will not introduce new bugs.
->[Automate your tests for free](https://app.bugbug.io/sign-up/){.markdown-cta}<-


### **Multilanguage**

Your page can run in different languages and you want to verify all of them. Try to assess the risk of having a bug only in one language without affecting the others. Perhaps it's not likely that such a bug would occur?

If you still need to test different languages, you have two options:

Option A: Record separate tests for each language. Hard to maintain on a bigger scale

Option B: Use variables & profiles to run the same test on different languages

-   Add data-testid to all of your app elements, so that the selectors are independent of the app language and the displayed test 

-   Add a way in your app to change the language via a URL, for example, "myapp.com/?lang=fr"

-   Create a variable "language code"

-   Create different profiles with different language codes

-   Replace the language code in the "Go to URL" step with the variable

-   Create separate suites for all the languages, overriding their profile with a chosen language

**Tests take long time to finish**

[Upgrade your BugBug plan](https://bugbug.io/pricing/) to be capable of [running cloud tests in parallel](https://docs.bugbug.io/running-tests/parallel-mode)

### **Downloading files**

Currently BugBug can't verify if the files have been download. Soon we'll add a new type of assertion for that, [talk to our support](https://bugbug.io/contact/) if you need it ASAP.

### **Drag & drop interactions**

BugBug support for drag&drop is in beta, you can achieve it by combining the "Mouse press", "Hover" and "Release mouse button" [actions](https://docs.bugbug.io/editing-tests/actions). Talk to our support for details.

### **A recorded selector doesn't work**

Sometimes it's not possible to record a reliable selector and you need to use your human intelligence and debug HTML a bit. 

Use our [No-code XPath Selector Builder](https://bugbug.io/xpath-selector-builder/) which makes it easy to craft a good selector manually and can teach you in-depth about [best practices in selectors](https://bugbug.io/xpath-selector-builder/#How-to-create-reliable-XPath-selectors-for-test-automation).

Success in test automation depends on the regularity of your progress
---------------------------------------------------------------------

Implementing automation testing in a SaaS company is a long-term process. You should do it step by step. The biggest challenge for tech leaders is lack of time to dedicate on QA area. The truth is you don't need a lot to push it forward. Better slow than never. [😉](https://emojipedia.org/winking-face/) Block yourself a slot in calendar every week to work on QA automation and keep the momentum going.

medium_bugbug_matrix_illustration_cc881af8ed.png

small_bugbug_matrix_illustration_cc881af8ed.png

thumbnail_bugbug_matrix_illustration_cc881af8ed.png

bugbug_matrix_illustration_cc881af8ed.png

Tech Leader's Guide To Automation Testing

We’ve prepared a short guide with essential recommendations on test automation that we created specifically for startups. We divided the guide into 3 levels:

[★☆☆ Level 1: get the most out of basic automation](https://bugbug.io/blog/software-testing/automation-testing-guide-for-startups-level-1/)

[★★☆ Level 2: automate hundreds of regression tests](https://bugbug.io/blog/software-testing/automation-testing-guide-for-startups-level-2/)

[★★★ Level 3: integrate with CI/CD builds](https://bugbug.io/blog/software-testing/automation-testing-guide-for-startups-level-3/)
  
Also see our guide dedicated to non-technical people: [beginner's tutorial to automation testing with BugBug](https://docs.bugbug.io/in-depth-guides/beginners-tutorial-to-automation-testing)
 
### What will you achieve on Level 1?
After finishing this level you will run 5-10 automated tests every hour and check if your core features are working properly. You will detect production bugs with big impact on users sooner than before.

### TLDR:
-   Start with short and simple tests that check your core features
-   Remember about recording “hover”
-   Create special test user accounts dedicated to automation
-   Schedule the tests to run every hour
-   Incrementally add more tests

### Focus on the most important features

![80-20 rule (1).png](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/80_20_rule_1_c2126d9815.png)

Many companies spend endless hours on a neverending pursuit to automate everything. This leads to very costly automation that requires a whole team to create and maintain. Don’t be like these companies and keep it simple. Agile startups should start with automated testing of the most essential functionality. 

> 👍 **Rule of thumb:**
> Usually, it just takes 5 to 10 automated tests to be sure that most users are able to achieve their goals.

What should you test? Think of your features. What percentage of users are using a particular feature? If the answer is higher than 80%, then this is a core feature. The goal of automating tests of such important features is to make sure that you can detect bugs that have a huge impact on your business as soon as possible (new user registration is for sure a core feature, but it’s a little bit more tricky to automate, which we will cover in the next episode)

->[Automate your tests for free](https://app.bugbug.io/sign-up/){.markdown-cta}<-




### Start with short & atomic tests

When you [record your first tests](https://docs.bugbug.io/creating-tests/recording-the-tests) start with something very simple and atomic. Atomic means that the test is small, independent and checks just one thing.

A good idea is to start with testing the basic navigation of your landing page. Click “record” and create a test that just clicks one item of your navigation link in the menu. Add [assertion](https://docs.bugbug.io/recording-tests-steps/recording-assertions) for some text in a heading. Finish the test and run it. Awesome! You’ve created your first automated test that touches the most essential area of your business!

![Screenshot 2022-07-25 at 13.44.12 (1).png](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/Screenshot_2022_07_25_at_13_44_12_1_0ebde0ddbb.png)

Now create a separate test for each navigation link. Here are screenshots of how BugBug team set up their landing page tests.

![Screenshot 2022-07-25 at 13.44.15 (1).png](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/Screenshot_2022_07_25_at_13_44_15_1_74fb39e9de.png)


### Don’t forget about recording hovers

If your website navigation appears on hover, you need to tell BugBug that you want to record a mouse moving to a specific element. To do this you need to click “Hover” during the recording and then click an element. When running the test, BugBug will move the mouse cursor to this element.

![bugbug record hovers.png](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/bugbug_record_hovers_81639fbd30.png)


### Create special test users

Another idea is to test if login works. Create a separate test that only checks that one thing.

To achieve that you need to prepare a “test user” - a user account that is dedicated only to testing if login works properly

-   Manually register a new user with a new unique email, for example with john.smith+testautomation@email.com (you can use [disposable mail](https://www.google.com/search?q=disposable+mail) or [plus aliasing)](https://support.google.com/a/users/answer/9308648?hl=en)
-   Go to BugBug and record a test that clicks the login button, enters the user login and password and submits the form
-   Add an assertion for some text that confirms that the login was successful
-   Finish your test
    

  

You might be tempted to test other features in the same test, but don’t do this now. We want to be sure that the login test is atomic and independent from other features.

  

Remember that you can create as many test users as needed. For example, if your product offers multiple combinations of preferences, you can create a separate user for each combination.

  

> 👍 **Rule of thumb:**
> Always create new test users if this leads to less test steps

  ->[Automate your tests for free](https://app.bugbug.io/sign-up/){.markdown-cta}<-

### Schedule all tests to run every hour

With BugBug you just need a couple of clicks to start running the tests every hour. Just go to “Schedules”, select the “Hourly” one and add “All tests”.

![hourly schedule (1).png](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/hourly_schedule_1_445d7fe266.png)


  

That’s it, now you will start getting emails every hour with the results of your test automation. Tip: create a filter on your email account so that you avoid inbox clutter or [configure notifications to only get an email on failures.](https://docs.bugbug.io/running-tests/schedules#configure-email-notifications)

![email notifications (1) (1).png](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/email_notifications_1_1_a23f96ba47.png)


  

### Incrementally add more tests

You’ve successfully created test automation that monitors your production! Now keep adding more short tests one by one. Don’t forget to focus on being cost-effective - if you have trouble automating a use case, skip it and go to the next one or find a different way of checking if it works. And remember: you can always [ask BugBug support to help you with any troubles](https://bugbug.io/contact).

Read next: [next level guide on how to easily maintain hundreds of automated tests ](https://bugbug.io/blog/software-testing/automation-testing-guide-for-startups-level-2/)
  
  

medium_illustration_guide_level1_98f13902d2.png

small_illustration_guide_level1_98f13902d2.png

thumbnail_illustration_guide_level1_98f13902d2.png

illustration_guide_level1_98f13902d2.png

Automation Testing Guide for Startups - Level 1: Get The Most out of Basic Automation

In the [first episode](https://bugbug.io/blog/software-testing/automation-testing-guide-for-startups-level-1/) of our 3-part guide, we covered the basics and described how to get the most business value from simple test automation. Now it’s time to learn some more advanced concepts and increase coverage of your automated tests. Here we go!
 
  
### What will you achieve on this level?

Once you finished this level you will:
-   have 100+ nicely structured tests
-   be able to update multiple tests at once when needed
-   run tests on different environments
-   automate testing of new user registration
    

  

### TLDR:

-   Organize your test steps in groups
-   Convert some groups into shared components
-   Duplicate tests efficiently with “new test from here”
-   Set up a URL variable to quickly swap between environments
-   Set up a testing email inbox to automate tests of new user registration
-   Use [built-in variables](https://docs.bugbug.io/editing-tests/variables#use-built-in-variables-for-dynamic-or-random-values) to generate random user email aliases
    
### Scaling the test automation

![illustration-number-of-tests.png](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/illustration_number_of_tests_7ee4d49d68.png)


In this episode, we will focus on scaling your test automation to a larger number of test cases so that your regression tests also cover corner cases and niche features. This is “Level 2” so things will get more complicated, but BugBug makes everything easy to achieve.

Example: in [the episode 1 of this guide](https://bugbug.io/blog/software-testing/automation-testing-guide-for-startups-level-1/) we’ve created just one test that checks if log-in works. Now we want to create several tests that check different outcomes of logging in, for example:

-   Log in - success
-   Log in - incorrect password
-   Log in - account blocked
-   Log in - password reset required
-   Log in - account doesn’t exist
    
You can see that we aim to increase the test coverage dramatically. Instead of 5-10 tests, we’re going to have hundreds of them.
->[Automate your tests for free](https://app.bugbug.io/sign-up/){.markdown-cta}<-

### Keep tests nice and tidy

If you want to have more than 10 tests, you should start organizing the test steps and keep them in order. Why is it important? Imagine that you’ve created 100 tests, but later you changed something in the UI. All of your tests started to fail. If you haven’t organized test steps, you will need to go to each test individually, see what’s wrong and re-record the failed steps 100 times. Fortunately, BugBug allows you to easily avoid such situations.

### Group your test steps

Keep your steps tidy by grouping them. Give a clear name to each group, which will help you to navigate through long test cases. Move your cursor in between the steps to see a plus icon, then click “[split group here](https://docs.bugbug.io/editing-tests/grouping-steps)”.

![split-group-here.png](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/split_group_here_fce12649d5.png)


  

### Convert groups to shared components

Once you start grouping your test steps, soon you will realize that some groups are exactly the same in multiple tests. This makes them a perfect candidate for [components](https://docs.bugbug.io/editing-tests/wip-make-component). Components are magic groups: if you change it in one test, it will change in all other tests that have it.

![component-made (1).png](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/component_made_1_afa56da91d.png)
  

Start with creating a component for the first step that goes to your starting URL. Then go to each test and replace the normal group with the component by using “Insert existing component”.

![insert-existing (1).png](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/insert_existing_1_985f99b0c0.png)

->[Automate your tests for free](https://app.bugbug.io/sign-up/){.markdown-cta}<-
  

### Duplicate the tests rather than create new ones

Since you already have components, you may not want to record each test case from scratch. A lot of your tests will start with the same components (for example: go to homepage, log in, enter a password, etc.). You can quickly create a new test “in the middle” of a different test. When you click “[new test from here](https://docs.bugbug.io/creating-tests/new-test-from-here)” you will create a duplicate of the test that keeps all the preceding test steps.
 
![new-test-from-here.png](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/new_test_from_here_5d79377e02.png)


  

If you use components in a smart way, you will be able to fix hundreds of tests with just one [re-recording](https://docs.bugbug.io/recording-tests-steps/re-recording-steps).

### Use data-testid attributes as selectors

Automated tests are most reliable if your [selectors](https://docs.bugbug.io/preventing-failed-tests/selectors) are clearly identifying an element. The best way is to clearly annotate such elements in your HTML structure. Ask your frontend developer to add `data-testid` HTML attributes to elements that you click in your automated tests. 

Why is it better? Let’s see an example. If you don’t use a dedicated attribute, your login button selector might look like `//SPAN[normalize-space(text())="Log in"]`. This selector points to any element with text "Log in". If you change the text in the button, such a selector will stop working and your test would fail. 

If you used an attribute, for example `[data-testid=”login-submit”]` your tests will be more resilient to changes. Even if you change the text in the button, you will not need to edit your automated test. What’s more, the developers will see this attribute in the code so they will be aware that this specific item is a subject of test automation. Once you have such attributes in your HTML, BugBug will automatically create selectors based on “data-testid” attributes during the recording. 

![data-testid.png](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/data_testid_669fc25baa.png)
->[Automate your tests for free](https://app.bugbug.io/sign-up/){.markdown-cta}<-

### Build robust selectors without coding

Sometimes it's not possible to create a reliable selector with recording. Dynamic lists, tables and hover actions are notorious for being difficult to record. You just need to use your human intelligence and use a more complex XPath selector. We created a dedicated tool for that with an in-depth guide - see [No-code XPath Builder](https://bugbug.io/xpath-selector-builder/).

![no-code-selector-builder.png](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/no_code_selector_builder_b90d46b1c1.png)

### Work with different environments

If you are fully committed to testing automation, chances are you will need to run tests on different environments. Each environment can be accessed with different URLs, for example:

-   Production - acme.com
-   Staging - stage.acme-test.com
-   Local - localhost:3000

You should store these URLs in a variable with 3 [profiles](https://docs.bugbug.io/editing-tests/variables#profiles). After you set up more than one profile, a new dropdown menu will appear in each test, that lets you run tests with different environments. Learn more in our [detailed guide](https://docs.bugbug.io/editing-tests/variables#work-with-different-development-environments).

![profiles (1).png](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/profiles_1_e2d0e47b36.png)

->[Automate your tests for free](https://app.bugbug.io/sign-up/){.markdown-cta}<-
  

### Automate tests for new user registration

To reliably automate user registration you need to use a combination of several advanced features

-   Third-party inbox that supports [plus-addressing](https://www.msoutlook.info/question/886#:~:text=Exchange%20Online%20(Microsoft%20365)%2C,also%20known%20as%20%E2%80%9Csubaddressing.%E2%80%9D) (for example Outlook.com)
    
-   Built-in “testRunId” variable for generating random e-mail address alias
    

  

Learn more in our separate step-by-step [guide on testing sign-up.](https://docs.bugbug.io/editing-tests/variables#test-user-registration-and-login-using-variables)

### Keep it simple

Don’t try to overcomplicate your test steps. Use components and variables only when it will actually save time for you in case of product changes. Remember that re-recording is a really fast way of fixing existing tests - sometimes it’s just faster to re-record steps in 5 tests than debugging a complex net of variables and dependencies. Keep it simple and if you have any problems feel free to [talk to our friendly customer support](https://bugbug.io/contact).  😊

[Read next: advanced guide with tips on CI/CD integration](https://bugbug.io/blog/software-testing/automation-testing-guide-for-startups-level-3/)

medium_illustration_guide_level2_cdfaa2b64b.png

small_illustration_guide_level2_cdfaa2b64b.png

thumbnail_illustration_guide_level2_cdfaa2b64b.png

illustration_guide_level2_cdfaa2b64b.png

Automation Testing Guide for Startups - Level 2: Automate Hundreds of Regression Tests

If you followed our [level 1](https://bugbug.io/blog/software-testing/automation-testing-guide-for-startups-level-1/) and [level 2](https://bugbug.io/blog/software-testing/automation-testing-guide-for-startups-level-2/) of this startup guide, you already know how to create and maintain hundreds of automated tests for your web app. In this episode, we will show you how to introduce automated regression testing into your software development process. Integration with CI/CD is a professional way to achieve high-quality code and prevent bugs before they are released to customers.

  

### What will you achieve on this level?

Once you finished this level you will:
-   collaborate with all team members on test automation
-   run automated regression tests with every build
-   see test results directly in your CI/CD tool

<br/>

--------------------

⚠️ **Technical skills required!**

To achieve this level you need to have software engineering knowledge and experience with development tools. If you’re not a technical person, ask your developers for help.

  --------------

### TLDR:

-   Use [BugBug CLI tool](https://docs.bugbug.io/integrations/api) to integrate with your CI/CD tool
-   Run tests every time an environment is deployed
-   Invite all your team members to BugBug, so that anyone can view test results
-   Copy & paste links to test results and share them with colleagues
    

### Run tests with every deployment

Your goal is to make sure that every time someone makes changes to the code, this new code is automatically checked against bugs before releasing to clients. If you use Github, Gitlab, Bitbucket or similar, you can configure these tools to run a “build” every time someone merges new code to a specific git branch. Our goal is to make sure that every time the build is run it will also trigger an automated regression tests suite. If the suite fails, the build also fails. If the build failed, you should not merge the code to master. This approach lowers the risk of introducing bugs to production.
->[Automate your tests for free](https://app.bugbug.io/sign-up/){.markdown-cta}<-

### Check the ID of test suites

Simply go to “Suites”, and select “Run via CLI”. You will see terminal commands that you can copy and paste into your CI/CD scripts.

  

![pasted image 0 (2) (1).png](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/pasted_image_0_2_1_c03d47d839.png)


![pasted image 0 (3) (1).png](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/pasted_image_0_3_1_f5a9ebf1e6.png)


### Integrate test suites with your existing CI/CD pipeline

You probably already have a build script. It’s very easy to initiate BugBug tests from it. You just need to add a couple of lines of code that will trigger the tests and wait for the results. Learn more in the documentation of [BugBug CLI](https://docs.bugbug.io/integrations/api) and ask your DevOps for help if needed.

![api-example-script-safe.png](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/api_example_script_safe_cbb33eeae4.png)



  

Once you successfully integrate your build scripts with BugBug, you are able to see the tests results directly in your build status.

![pipeline-example-safe.png](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/pipeline_example_safe_88bb40c38a.png)

->[Automate your tests for free](https://app.bugbug.io/sign-up/){.markdown-cta}<-


  

### Share links to test results

You noticed that a test failed and you wonder how can you send the report to your colleague? The easiest way is to simply copy the URL and send it via chat. Each item from the “Runs history” has a unique URL address.

![Screenshot 2022-07-26 at 15.37.47 (1).png](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/Screenshot_2022_07_26_at_15_37_47_1_b21615dcd5.png)


  

To access it the recipient needs to be invited to your [organization](https://docs.bugbug.io/collaboration/organization-settings). BugBug allows you to invite an unlimited number of people to your organization free of charge. Testing automation is more powerful if you make it available to the whole team!


### Don’t forget about "Level 2" recommendations

CI/CD integration doesn’t make sense if your tests are not resilient to changes and easy to maintain. See our previous tips in [the level 2 test automation guide, where we covered topics such as custom attributes, reusable components and variables.](https://bugbug.io/blog/software-testing/automation-testing-guide-for-startups-level-2/) 

medium_illustration_guide3b_1634140130.png

small_illustration_guide3b_1634140130.png

thumbnail_illustration_guide3b_1634140130.png

illustration_guide3b_1634140130.png

Automation Testing Guide for Startups - Level 3:  Integrate With Your CI/CD

Reliable, modern end-to-end test automation tool that improves testers’ and developers’ work.


Software Testing Knowledge

Before you release your product into the market, it is extremely important to properly test whether it’s functioning as intended. To do that, software developers utilize a variety of testing techniques that verify the software’s integrity. To thoroughly check their software from start to finish, developers implement **End-to-End testing**, often simply called **E2E** testing – an elaborate technique that mimics end user experience. 

There are many layers to End to End testing, as it is one of the most complex testing methods. This type of testing involves testing every part of your software that the end-user interacts with in any imaginable way.

With this guide, we want to show you the importance of E2E tests, as well as tell you precisely what they are, including a few specific examples.

**Also see:** if you're new to software automation, take a look at this [beginner's guide to automation with step-by-step screenshots](https://docs.bugbug.io/in-depth-guides/beginners-tutorial-to-automation-testing).

[TABLE_OF_CONTENTS]

## What exactly is End-to-End testing?

**End-to-End testing** is one of the most useful and informative techniques for software testing, providing valuable data about the inner workings of the program from the end-user perspective. With E2E testing, developers can thoroughly verify the behaviour of their software. During the test, you will be trying to replicate key user behaviour scenarios when using your software. The actual list of things to do varies, depending on the type of software you’re testing – but in all cases, it is rather extensive.

When performing a software testing method, like an End-to-End test on a web application, you would start with opening your browser and navigating to the proper URL, and then proceeding to use the software as intended for the end-user.

Similarly, in the case of a regular desktop application or mobile application, you would open the app just like a user would, checking out every bit of functionality that is intended for the users to work with. Keep in mind that during E2E testing you won’t need to check developer-specific aspects of the software, only the end-user perspective if it is important to you.

## End-to-End test goals

During an E2E test, your goal is to verify all of the dependencies and data integrity that directly or indirectly influence the reception of the software by the end-user. This usually includes any external services, as well as databases used by the software and metrics, as well as any other system dependencies. 

E2E tests will ideally be performed in the production environment, but this isn’t always possible. The tests can, of course, be performed in an independent End-to-End testing environment, but you should make sure it is as close to the production environment as possible.

Within the SDLC process, E2E tests are performed in the Testing phase, which is situated after Coding and before Deployment. 
->[Automate your tests for free](https://app.bugbug.io/sign-up/){.markdown-cta}<-


## Why is End-to-End testing important?

Is End-to-End testing necessary? It is true that more and more developers make sure to implement an E2E testing phase before the final deployment of their software. There are many benefits to E2E tests, but the most crucial ones are:

- The ability to verify software performance from the end-user perspective
- Testing verifies not only the software itself but also any interrelated sub systems, increasing test coverage
- Validation of back-end systems
- Immediate end-user-like feedback on test failing

While tests such as unit testing or integration testing are very useful (functional testing), allowing you to verify the working of each component and providing quick results, they don’t give you the whole picture. By not performing E2E tests, you won’t know how the experience of using your software will look like from the end-user perspective.

For example, individual component testing (integration tests) might show promising results and full functionality. However, when combined, something goes wrong and produces a bug. By only checking individual components, you wouldn’t be able to find out about the bug. 

With a proper End-to-End test, you can verify whether everything works correctly together as it should. Even integration testing doesn’t cover all of the system components of your software, and you might never know whether every aspect of your app clicks together.

E2E testing doesn’t necessarily have to discover bugs – it can also give you useful insight into the user scenario of your product. Overall, all types of tests play their respective roles in the whole vertical end-to-end testing process (differently to horizontal end-to-end tests). Just because E2E tests give you a more complex understanding of your software as a whole doesn’t mean that other test types are less valuable. 

Before releasing your app, always make sure to proceed with a thought-through testing strategy. However, while they’re certainly extremely useful, keep in mind that E2E tests occupy the top of the testing pyramid, making them the most expensive and most difficult to perform.

End-to-End testing is also one of the few functional types of tests that can be performed manually, as well as automated - which is done especially often in Continuous Integration. In such a scenario, a chosen set of tests can be run automatically each time a new iteration is uploaded to the testing server.

->[Automate your tests for free](https://app.bugbug.io/sign-up/){.markdown-cta}<-

## Benefits of end-to-end testing

One of the goals of this software testing technique is to simulate the end-user's perspective. This means that this type of testing helps to verify not only the security or performance, but the business aspect as well. It will check whether the software collects and processes data as it was designed to do. End-to-end testing is essential for a comprehensive validation of your product. This testing ensures that all integrated parts of the application work together as expected.

Another purpose of E2E testing is to mitigate risks. E2E testing verifies if the software is ready for deployment. The confidence in stability is a crucial factor which the stakeholders consider while approving the next release. 

There are obviously much more obvious benefits of this form of testing (like bug fixing or testing across different user scenarios), but we described them already in this article.

## End-to-End Testing Framework

Alright, so now you know what End-to-End testing is – but how to exactly implement it? There is a wide variety of checks and individual tests that must be covered by your end-to-end testing plan, some of which include:

- Establishing test requirements in the preparation phase (including software and hardware requirements)
- Creating proper documentation
- Deciding what type of test environment will be used (production environment or independent test environment)
- Preparation of test cases and general E2E design
- Tracking of data input and output
- Test execution and evaluation

When executing E2E testing methods, developers usually follow an established E2E testing framework that guides you throughout the process, enumerating various tasks and responsibilities. The framework is relatively simple and consists of only three elements:

### Establish user functions

The User Functions relates to listing the features of your software, as well as all of its interconnected software systems. To properly measure and test software functionality, you need to first think about what you’re going to test. After you’re finished with listing user functions, remember to record everything relevant to each of them, including both the input and output data. Since E2E testing can be considered holistic, you also need to pay attention to all of the relations between every individual function – how they work together and whether there are any clashes.

### Come up with specific conditions

Now that you have established the user functions you’re going to test, you need to think about the testing conditions. Try to come up with as many quantifiable indicators that can be measured for each of the functions. These can range from response time to consistency of data flow, to simple convenience of use. Remember – you’re trying to imagine yourself as an end-user, so try to keep track of anything that might be an issue for a non-developer.

### Create individual test cases

Finally, before the E2E testing can commence, you need to prepare specific test cases for each of the conditions. At this point, you have prepared a list of user functions to test, as well as data conditions that will be measured during the test. Now, you need to create at least one test case for each of the conditions. By test cases, we mean user scenarios during which specific indicators will be able to be measured.

And by test case preparation status, we mean the precise location of the test instances being developed, as compared to intended test cases.

->[Automate your tests for free](https://app.bugbug.io/sign-up/){.markdown-cta}<-

## End-to-end test cases for your testing process

Congratulations, you’ve almost reached the end! At this point, you should pretty much be a walking fountain of knowledge about End-to-End tests. However, the raw theory is one thing, and implementing those types of tests for system testing in reality is another. To make the process a bit more digestible, consider this End-to-End testing example for a website with shopping functionality:

- Go to the specified URL
- Open a list of products
- Check whether the information on the list correlates with specific product pages
- Try to add the product to the cart and look for visible confirmation
- Navigate to the cart and check whether the product has been successfully added
- Double-check all of the relevant information (price, product info, etc.)
- Proceed with the check-out and verify payment options integration

See how simple it is in the end? Now you can implement E2E testing methodology in your software to ensure better end-user reception, increase your product quality, and build a positive brand image.

This is just a high level example. If you would like to learn how to create a proper test case from start to end, please check out our article on [test case creation](https://bugbug.io/blog/software-testing/how-to-write-automation-test-cases/). You will have a better understanding on how to design test cases for manual and automatic testing. Also check out our other topics in our testing guide. 

## End to End testing tools

Proper tools may play a crucial role in reducing your effort by automating your end-to-end tests. These tools provide a range of features and capabilities to support horizontal testing and vertical testing of the entire system. They allow testers to perform comprehensive testing processes. Choosing a proper tool for testing is typically a long-term investment. If you research and choose a tool that will enable you to execute and record complex user scenarios and allow testing across multiple technologies, your job will be way easier. 

One of the tools that meet such requirements is BugBug. This automated testing tool proved its efficiency for many clients. On the picture below you might see how many steps might an end-to-end scenario actually have.

![n example of a passed test with a proper output statement.](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/88bc1d9e_16f8_4e1a_b33d_15e2a8d17cdf_cdfc84abfc.webp)

## Summary

End-to-end testing is a type of testing that focuses on validating the whole system from start to finish rather then testing individual functionalities. It is often seen as a software testing methodology, but it is not the case. It may be a part of a methodology (i.e. Agile testing) but it is by definition a type of software testing. 

Performing end-to-end testing covers an important task of checking data flows and focuses on testing if any functional gaps occur. This type of testing is necessary to verify if the software is ready for the next release cycle.

## Frequently Asked Questions

1. **What is automated testing?** \
Automated testing in software development is a form of software testing that uses the execution of predefined sets of operations to probe and validate the software system. The fundamental purpose of automation testing in software development is to generate objective feedback about the quality of the product. One such testing tool is Selenium. It can run UI tests. The ideal situation is to run E2E tests with both manual testing and automated testing approaches. If your application has a lot of complicated user interfaces, you'll need more end-to-end testing.

2. **What is software development lifecycle?** \
Software development lifecycle (SDLC) is the process used by developers to create, deploy and maintain software. This process includes several steps that are all closely connected with one another. These steps include requirements gathering, analysis and design, coding and testing.

3. **What is a testing pyramid?** \
In software development projects, we have limited time and resources. This is why it's important to consider our test efforts carefully and find the right balance between different types of testing. Basically, there are 3 layers in the testing pyramid: Unit Tests, Integration Tests, and E2E Tests.

3. **What is a selector?** \
A selector is a small piece of code that describes a specific button in your app. It allows you to clearly define what should be clicked in automated end-to-end tests. You can learn more about selectors using [this interactive XPath learning tool](https://bugbug.io/xpath-selector-builder/?ref=blogarticle)



Happy (automated) testing!


large_testowanie_end_to_end_98e59a7257_deae9fab1e.png

medium_testowanie_end_to_end_98e59a7257_deae9fab1e.png

small_testowanie_end_to_end_98e59a7257_deae9fab1e.png

thumbnail_testowanie_end_to_end_98e59a7257_deae9fab1e.png

testowanie_end_to_end_98e59a7257_deae9fab1e.png

What is End-to-End (E2E) Testing?

Professional Quality Evangelist, experienced in testing web and mobile applications.  
[linkedin.com/in/kinga-witko-b5bba659/](https://linkedin.com/in/kinga-witko-b5bba659/)

Kinga Witko

[TABLE_OF_CONTENTS]

The term “shift left” expresses movement. It can describe the movement of an entire team or individuals.
What does it do in a project or an entire development lifecycle? Putting it simply,  it reflects the position of testing on a timeline. 

**Shift Left means testing earlier in the development process.** It also means repositioning the engagement, development, and testing efforts towards the customer.


Typically, in a traditional (waterfall) model of software making, testing lands at the end of the pipeline or is one of the last phases of the process. Bringing testing at the very last stage usually causes many problems such as:

- delays in the deployment
- a lot of bugs at the end of the development lifecycle
- bugs discovered just before or after the release to production
- lack of collaboration between developers and testers through the entire process.

Poor software quality is one of the most harmful consequences of all of these pains listed above.

At first, the team members do not communicate with each other and they fail to communicate the problems early. 

It must not necessarily be a bug; sometimes there is some issue with the application setup, there is no testing environment, access to certain resources (such as specific tooling or external software providers), the task description is vague, or the team setup is incorrect. 

If communicated and fixed earlier, the entire process can be smoother and cause less frustration in the team. It also encourages people to focus more on the code quality and to deliver earlier and faster.

Most of those issues can be avoided by changing the style of working and shifting testing efforts at the beginning of the process.
->[Automate your tests for free](https://app.bugbug.io/sign-up/){.markdown-cta}<-


## Agile way of working

If you care about the quality of your product, your mindset has probably shifted left already.

This approach is strongly connected with the agile way of thinking and using agile work methods in software projects.


In this setup, testing is an integrated part of the team and accelerates software development at higher quality. As a result, testers can identify bugs sooner, saving time and effort. Notably, the earlier you test, the less expensive bugs are to fix.


[T The Seven Testing Principles (ISTQB)](https://www.istqb.org/downloads/syllabi/foundation-level-syllabus.html) tell that testing everything is not possible unless a system or functionality is very simple. In practice, however,  the testing efforts should be targeted appropriately at the risk analysis, techniques, and bugs prioritization rather than be attempted thorough testing. 


One of the good methods of approaching testing strategy for a specific application can be the guideline from [Agile Testing Quadrants](https://bugbug.io/blog/software-testing/agile-testing-quadrants/). The quadrants are a taxonomy to help teams plan the testing process and testing activities and make sure they have all the resources they need to accomplish it.


## Shift Left approach and DevOps and agile background

With the rise of the DevOps movement, there is an increased focus on delivering automated, continuous improvement. It holds true for development and testing alike.

What is more, Continuous Integration (CI) and Continuous Delivery (CD) are core software development disciplines integrated into the agile software development cycle. The point is to release fully-tested and stable builds early and as often as possible. Development, delivery, and test automation are crucial components of this process. 

## What it means to “Shift Left”

As mentioned at the beginning, Shift Left testing pushes testing to the “left”, to the earlier stages in the pipeline. This improves software quality and reduces time spent on resolving issues later in the pipeline.

Introducing testing early on the “left-side” of the development process, starting with unit tests and complementing with functional and integration tests is fundamental to making CI/CD happen. But that may not be enough.

The team can go even further: to eliminate unclear and contradicting requirements and improve the end-user experience, it is best to involve test engineers in the clarifying and prioritising application requirements. Test engineers can check the available or newly created requirements or user stories for clarity, consistency, testability, and traceability.

This also improves the shift towards client happiness. An early review of the requirements and spotting the flaws marks an early start of the communication with the end-user. 

The Shift Left approach introduces more care and thought exchange concerning the client. The team can be perceived as more proactive and more dedicated to the best quality standards of a project.
->[Automate your tests for free](https://app.bugbug.io/sign-up/){.markdown-cta}<-


## Benefits of Shift Left Testing for Agile and Devops teams

Agile teams benefit most from Shift Left Testing in terms of clearer specifications, smoother implementations, earlier testing, and reduced costs. However, Shift Right Testing has more benefits like the ability to incorporate user and system feedback based on real usage scenarios. 

In opposition to testing early, Shift Right can provide an additional context to the testing scenarios. Shifting right means testing at the end of the process, typically when the software is deployed to production (the post-production environment). 

It is performed by a group of experts or selected end-users. It can utilize testing methods such as API Monitoring, using Feature Toggles, or measuring the performance of a live application. 

This helps to understand the application better and discover some other flaws that could have been omitted earlier.

Nevertheless, when quality comes first:

- developers spend more time on coding because they discuss potentially good solutions - they do not investigate who pushes the code first into production without testing

- there are Unit Tests beside the application code, available and maintainable

- there is an infrastructure that allows testing before pushing to production, usually the entire team is responsible for the process health

- there is room for improvement and mistakes

- developers pair up to catch obvious errors on the fly and learn from one another

- teams estimate tasks together, taking quality and risks into account

- potential risks in the systems are spotted and minimized rather than left alone

And finally - the application is tested - not only by one automated or manual test  but by a set of different functional and non-functional scenarios. 

The tests are regularly reviewed, adjusted, and observed. This approach introduces a new way of working and highlights the value of testing.


## Benefits of the Shift-Left Testing approach

The idea of shift left testing is simple to understand but hard to implement in the development cycle. It is worth the effort though, especially when the team and the entire organization understands the main benefits of this approach, such as:

- Lower costs - as explained above, bugs detected earlier are less expensive to fix.

- Increased efficiency in the software development process.

- Improved product quality and reliability because more bugs are detected at earlier stages.

- Reduced time to market since the QA process covers the entire development cycle, not only its’ last phase.

- A competitive advantage gained by letting development teams move fast, build and defend their credibility by innovating in iterations.
->[Automate your tests for free](https://app.bugbug.io/sign-up/){.markdown-cta}<-


## Get started with “Shift Left” Testing in Agile

How to approach the change and switch to early software testing? Here is the basic list: 


1. **Plan**

Shape development in a way that is friendly to Shift Left testing. Once the requirements have been established, developers and testers should ideate how to shape the coding and software testing as early as possible. 


Depending on the product expectations as well as the limitations of the team, specific strategies, such as involving testers in the project since the planning phase or using the TDD approach, might have to be crafted to accommodate the Shift Left way of working.


2.** Review the requirements**

It is best to verify not only the existing code but also all the requirements necessary for the code to be written. Early verification can spot obvious mistakes.


3. **Use Static Code Analysis**

Static code analysis is the practice of checking the code without executing it. It runs through the basic code structure and ensures that the code is aligned with many standards and guidelines. By following the approach the team can early detect:

- Programming errors

- Violations of commonly followed coding standards

- Syntax anomalies

- Security issues

A static code analysis is best done via automation. The software should scan all untested codes in a project before it moves to the test phase.

4. **Collaborate**

As in any other [agile testing](https://bugbug.io/blog/software-testing/agile-testing-organized-chaos-or-a-planned-way-of-improving-your-software/) technique - early software testing requires a lot of collaboration with the team: Developers, Product Managers, Business Analysts, and other stakeholders responsible for the overall product. 

Everybody needs to understand the value of the Shift Left approach and spot the difference after some time of the new process introduction because as a result, the client benefits from better software quality and receives a lovable product.

## Conclusion for Left Shift Testing in test automation

There are a number of benefits to left shift testing in test automation. One of the primary benefits is that it helps to identify defects earlier in the development cycle. This means that defects can be caught and addressed before they become more complex and costly to fix. In addition, left shift testing can help to reduce the overall time and effort required for testing by identifying and addressing issues earlier in the process.

Another benefit of left shift testing in test automation is that it can help to improve the quality of the software being developed. By identifying and addressing defects earlier in the process, developers can ensure that the software they are creating is of higher quality and meets the requirements of the end-users.
->[Automate your tests for free](https://app.bugbug.io/sign-up/){.markdown-cta}<-


One of the keys to successful left shift testing in test automation is to incorporate testing into the development process **from the very beginning**. This means that testing is not an afterthought or an additional step at the end of the development process, but rather an integral part of the development process itself. Test automation tools and frameworks can be used to automate testing activities and make it easier to incorporate testing into the development process.

It is also important to involve testers in the development process as early as possible. This allows them to provide feedback and identify potential issues before they become more complex and difficult to fix. Testers can also work with developers to ensure that the code being developed is testable, and that the testing process is efficient and effective.

Overall, **left shift testing is an important approach in test automation that can help to improve the quality of software, reduce testing time and effort, and identify defects earlier in the development process**. By incorporating testing into the development process from the very beginning, developers and testers can work together to create software that is of higher quality and meets the needs of the end-users.

Happy (automated) testing!

large_pexels-pixabay-40716.jpg

medium_pexels-pixabay-40716.jpg

small_pexels-pixabay-40716.jpg

thumbnail_pexels-pixabay-40716.jpg

pexels-pixabay-40716.jpg

Shift Left Testing Explained

Testing is an ever-present part of a software developer’s job, performed at every stage of the production process. There is a wide range of test types used for software testing, some performed at various points during production, and some performed after production to confirm that the software functions as intended.

Regression testing is performed on a product that has already been finished and released. How exactly is it performed? What are the objectives and how does the procedure look? If you’re keen to find out, keep on reading as we explain everything you should know about regression testing.

[TABLE_OF_CONTENTS]

## The definition of Regression Testing

**Regression testing** is a software testing method that aims to investigate and locate any possible flaws or bugs in software that has already been fully developed. Regression Testing is performed to find out whether your software has regressed (surprise, surprise) as you continue introducing new updates and bug fixes. Whenever a piece of software is changed in any way, new bugs might emerge and some of the older ones re-emerge.

Most often, regression testing is performed immediately after introducing a new update, but it isn’t the only factor that might call for the test. If any software or hardware-related factor has changed, regression testing should be committed to find out whether everything performs as it has before the change. 

This can include any sort of bug fixes and software changes, but regression might also be caused by hardware configuration changes and electronic component degradation over time. Regression testing isn’t easy – it requires cooperation and diligence out of all of the team members, which makes it perfect for agile software development practices.

->[Automate your tests for free](https://app.bugbug.io/sign-up/){.markdown-cta}<-

## Regression Testing process

As with most testing efforts regression testing is an art rather than science. Let us describe the regression testing process that our development team uses. You may then use and adapt it to your existing test strategy. As regression testing is a type of testing that occurs whenever any change in your software occurs, the smartest way to start is to..

### Identify code changes

This will allow you to asses the number of test cases for regression testing that will need to be created. The testing team will need information on the impact that the current changes may have on the whole application. This is done also during this step.

### Prioritize changes

You finished step one with a list. Now you need to create a queue out of it. By proper prioritizing the changes in your software application you reduce the amount of time you will use on 

### Select for re-run

This is the part where the selection of test cases is performed. After the first test is done, test cases are categorized into reusable and obsolete ones. The reusable ones create a regression test suite. The others are will not be used for regression. 

### Categorize manual and automated regression test cases

Here is where the test automation actually begins. Test cases that occur more often within one process are perfect candidates to for the automation tool to do the work for you. If you want to automate tests in a fast and efficient way, try BugBug as your tool for regression testing. You will soon find out that it is great to reduce regression testing costs.

### Prioritize test cases

Here is where regression testing becomes a tad tricky. One would think, that the most often occurring test cases should have the biggest priority, right? Well the theory on that one is a tad different. The need for regression testing, therefor the priority, is the highest for those cases, that cover all essential features. The lowest on the other hand are stated as "technical complexities". 

### Plan and execute

This one is rather obvious, but we have to put it here. After doing all the analysis, you have to plan when regression testing is done and "push the execute button". 

## Regression Testing techniques

Regression testing can include various functional and non-functional tests in its process to thoroughly investigate the software in search of flaws. There are also many techniques with which developers and QA specialists perform regression testing, including:

- Retest All
- Regression Test Selection
- Test Case Prioritization
- Hybrid approach

### Retest All – an accurate yet expensive solution

**The Retest All ** technique is the most accurate and all-around method of regression testing, but it is also the most expensive one. In Retest All, every test case is checked again on the software to confirm if all of its parts are working as intended. That’s all there is to Retest All – it might be simple in definition, but it is definitely the hardest and most costly technique to actually perform.

### Regression Test Selection – when you realize you might not have enough for a Retest All

**Regression Test Selection** has been designed as a solution to the expensive nature of the Retest All technique. In RTS we don’t run all of the tests in our test suite, but instead we divide all of the tests into three categories – reusable test cases, retestable test cases, and obsolete test cases. In addition, RTS might actually create new test cases that haven’t been in the suite before, but might be relevant for regression tests.

If you’re interested in finding out more about regression testing examples, we recommend taking a look at Understanding Regression Testing Techniques by Gaurav Duggal and Bharti Suri. The authors further divide RTS techniques, naming two most important categories – coverage techniques and minimization techniques, and explain what are safe techniques.

**Coverage techniques** feature specific coverage criteria, often provided by the client who wants their software regression tested. These techniques sift through all test cases, selecting only the ones that are relevant for the covered parts of your software. **Minimization techniques** work in a way similar to coverage techniques, but instead of selecting cases based on set criteria, you select a minimum set of cases that you think will do the job.

->[Automate your tests for free](https://app.bugbug.io/sign-up/){.markdown-cta}<-

### Test Case Prioritization – a rational approach

**Test Case Prioritization** aims to maximize the rate of fault detection, creating the most thorough regression test for the lowest cost. First you prioritize the test cases one by one. Then all of them are performed in order, starting from the most important one and finishing on the least important one. Thanks to that, even if your resources aren’t enough to perform all of the tests in your suite, you will still perform the ones that should detect the most flaws.

**TSP** might be performed in two ways – the first one, called **General Prioritization**, selects test cases that should be effective (on average) on all versions of the software you’re working with. The other way, called Version Specific **Prioritization**, only takes into account a specific version of the software, without considering other existing or potential future versions.

### Hybrid approach – the best of both worlds

When choosing one technique of regression testing is impossible – why not combine the best features of Regression Test Selection and Test Case Prioritization? 

The hybrid approach is still being developed, and every researcher gives a personal spin on the idea. Since there isn’t an exact definition of what a hybrid approach actually is, there are tons of versions available – each one with its own upsides and downsides.

## Regression Testing tools

Regression testing ensures that each version of your software runs smoothly. We have described various regression testing techniques, but our article would not be complete without a few hints of what tool to use for proper test execution. 

Regression testing is used on an incremented basis. Each release of your software means building another layer of an already high pile of multiple regression test cases. A good tool will allow you to perform regression tests without writing a single line of code. 

If you want a specific list of tools, please redirect to our article [HERE](https://bugbug.io/blog/guides/testing-tools-for-web-application/).

->[Automate your tests for free](https://app.bugbug.io/sign-up/){.markdown-cta}<-

## Disadvantages of regression testing

Well now that's a shocker isn't it? We have many articles on testing in the BugBug blog, and never have I ever wrote anything negative about testing. Until now. We found four "buts" on regression testing and we tried to tackle them immediately.

- "but regression testing is time consuming" - yes, but only if you only perform a manual test, this is why selecting a good testing tool is essential
- "but you have to do it after even the smallest patch" - yes, but this is why you build a good regression cycle. You store all the recurring tests and only build a few on top of them for that given patch. It is like CI/CD for your testing process
- "but that process might interfere with the sprint planning" - only if you do regression tests without automation. Think of it from a business perspective - would you have a task that goes cross-sprint or a large bug in the production environment?
- "but it is difficult, to write those test cases" - ah, an oldie but a goodie. There are great no-code regression testing tools. Those might be used not only by QA or other technical personnel but by a PM. Such a person would be even able to automate the test or record and recreate it. 

Regression testing is one of the key ways to have your code thoroughly checked and ready to start earning you money!

## Conclusion – is Regression Testing necessary?

As you have already learned, regression testing plays a key role in keeping the quality of your software as high as possible for as long as possible. Updates and changes to your software might introduce new problems, which is exactly why regression testing is so important – to keep user experience high throughout all iterations of your program.

Not all projects require regression testing, of course. In projects that are already finished and won’t be further updated, regression testing is mostly irrelevant. However, if you are planning to continue development further – never forget to properly regression test your software before releasing an update.


Happy (automated) testing!

large_regression_tests_blog_07c72a81f5.jpg

medium_regression_tests_blog_07c72a81f5.jpg

small_regression_tests_blog_07c72a81f5.jpg

thumbnail_regression_tests_blog_07c72a81f5.jpg

regression_tests_blog_07c72a81f5.jpg

What is Regression Testing? Everything You Need To Know

The Best Software Testing Newsletters in 2023

The Ultimate XPath Cheat Sheet

CI/CD Interview Questions and Answers

Get more tips and product related content. Zero spam.

Don't miss any updates

Automate web app testing easier than ever. Without excessive costs. Faster than coding. Free forever.

End-to-end testing made simple! BugBug is a new, more intuitive tool for test automation of web apps. Get all the benefits of test automation without hiring new people or infrastructure changes.

BugBug.io | Intuitive and reliable test automation for startups

large_opengraph_image.png

medium_opengraph_image.png

small_opengraph_image.png

thumbnail_opengraph_image.png

opengraph_image.png

Everything you need for the entire test automation workflow

Record new tests simply by clicking in your browser. Automatically capture every click or text input. Add assertions directly while recording. Forget about writing selectors by hand; BugBug will do it for you.

Create tests without coding

Enjoy excellent performance when you run tests locally, allowing you to work more efficiently. Run tests in the cloud to regularly monitor your web application and catch bugs earlier.

Run tests in your own browser or in the cloud

Now you don't need to record your test from scratch every time you want to change it! Record new steps right in the middle of your test, just where you need it. Spend less time on manual writing of CSS or XPath Selectors.

Fix tests by re-recording

Run cloud tests every day and monitor if your app works properly. Get comprehensive notifications when a test fails. You will receive notifications with scheduled test results directly to your inbox.

Schedule test automation

Use BugBug API and Command Line Interface to run regression tests along with your CI/CD build pipelines, such as Github, Gitlab, Travis, Jenkins, or Bitbucket. Integrate with Zapier to get notifications on Slack when a test fails.

Integrate with your CI/CD pipelines

Invite your teammates to work together on test automation without time-consuming training and debates. You can involve software developers or product owners in testing within minutes.

Collaborate with your team

Unparalleled simplicity powered by unique features

Powerful features for advanced testing

You can turn test steps into "Components" that are shared across multiple tests. Maintain hundreds of tests without worrying that you will have to spend half a day fixing them. 

Edit multiple tests at once

Work in multiple environments. Create random numbers. Register users with unique email addresses. 

Use variables to handle complex tasks

You have full control over the test steps. You can update selectors, manually add advanced step types or assertions, and even custom JavaScript code.

Customise test steps with JavaScript

Need to show the results to your stakeholders? You can download a PDF report for each suite run in your project.

Generate PDF reports

Record automated tests easily for both desktop and mobile resolutions.

Build tests for mobile resolutions

<div class="cms-section">
      <div class="cms-wrapper">
        <div class="cms-testimonials-row">
          <div class="cms-testimonials-item">
            <div class="cms-testimonials-item__icon"><img src="https://assets.website-files.com/62826448121477c0eb54a773/633456764c0c618517865056_bugbug-quote-icon.png" loading="lazy" alt="" class="cms-testimonials-item__icon-img"></div>
            <div class="cms-testimonials-item__texts">
              <div class="cms-testimonials-item__main-text">“Recording tests in BugBug works surprisingly well. I've tested various competing solutions, but none did as smoothly as BugBug.”</div>
              <div class="cms-testimonials-item__author">— Krzysztof Studniarek, Product Manager, BRAND24</div>
            </div>
          </div>
          <div class="cms-testimonials-item">
            <div class="cms-testimonials-item__icon"><img src="https://assets.website-files.com/62826448121477c0eb54a773/633456764c0c618517865056_bugbug-quote-icon.png" loading="lazy" alt="" class="cms-testimonials-item__icon-img"></div>
            <div class="cms-testimonials-item__texts">
              <div class="cms-testimonials-item__main-text">“In two days, we managed to automate test cases that took us weeks to write up and execute using other software. It is easy to use, yet can still perform advanced testing should the tester want to.”</div>
              <div class="cms-testimonials-item__author">— Panayiotis, Software Tester, ATM Marketing</div>
            </div>
          </div>
          <div class="cms-testimonials-item">
            <div class="cms-testimonials-item__icon"><img src="https://assets.website-files.com/62826448121477c0eb54a773/633456764c0c618517865056_bugbug-quote-icon.png" loading="lazy" alt="" class="cms-testimonials-item__icon-img"></div>
            <div class="cms-testimonials-item__texts">
              <div class="cms-testimonials-item__main-text">"BugBug is super intuitive and lowers the barrier of becoming an end-to-end tester. It reduces development time and significantly lowers the cost of software testing. "</div>
              <div class="cms-testimonials-item__author">— Wojciech, CEO Davinci-Studio.com</div>
            </div>
          </div>
        </div>
        <div class="cms-testimonials-row"></div>
      </div>
    </div>

<style>

.cms-testimonials-row {
  display: -webkit-box;
  display: -webkit-flex;
  display: -ms-flexbox;
  display: flex;
  padding-bottom: 32px;
  grid-column-gap: 32px;
  grid-row-gap: 32px;
}

.cms-testimonials-item {
  display: -webkit-box;
  display: -webkit-flex;
  display: -ms-flexbox;
  display: flex;
  -webkit-box-align: start;
  -webkit-align-items: flex-start;
  -ms-flex-align: start;
  align-items: flex-start;
  -webkit-box-flex: 0;
  -webkit-flex: 0 33.33%;
  -ms-flex: 0 33.33%;
  flex: 0 33.33%;
  grid-column-gap: 16px;
}

.cms-testimonials-item__main-text {
  min-height: 100px;
  padding-bottom: 16px;
  color: #666;
  font-size: 18px;
  line-height: 1.6em;
  font-style: italic;
  letter-spacing: 0em;
}

.cms-testimonials-item__icon {
  width: 64px;
  height: 64px;
  -webkit-box-flex: 0;
  -webkit-flex: 0 0 auto;
  -ms-flex: 0 0 auto;
  flex: 0 0 auto;
}

.cms-testimonials-item__author {
  min-height: 100px;
  color: #666;
  font-size: 16px;
  line-height: 1.6em;
  text-align: right;
  letter-spacing: 0em;
}

@media screen and (max-width: 991px) {
  .cms-testimonials-row {
    -webkit-flex-wrap: wrap;
    -ms-flex-wrap: wrap;
    flex-wrap: wrap;
  }

  .cms-testimonials-item {
    -webkit-flex-basis: 100%;
    -ms-flex-preferred-size: 100%;
    flex-basis: 100%;
  }
}


@media screen and (max-width: 479px) {
  .cms-testimonials-item__main-text {
    min-height: 0px;
    font-size: 18px;
  }

  .cms-testimonials-item__author {
    min-height: 0px;
    font-size: 13px;
  }
}
</style>


Start testing for free! No credit card required.

Features

Customer Success Stories


<div class="cms-section">
      <div class="cms-wrapper">
        <div class="cms-testimonials-row">
          <div class="cms-testimonials-item">
            <div class="cms-testimonials-item__icon"><img src="https://assets.website-files.com/62826448121477c0eb54a773/633456764c0c618517865056_bugbug-quote-icon.png" loading="lazy" alt="" class="cms-testimonials-item__icon-img"></div>
            <div class="cms-testimonials-item__texts">
              <div class="cms-testimonials-item__main-text">“Recording tests in BugBug works surprisingly well. I've tested various competing solutions, but none did as smoothly as BugBug.”</div>
              <div class="cms-testimonials-item__author">— Krzysztof Studniarek, Product Manager, BRAND24</div>
            </div>
          </div>
          <div class="cms-testimonials-item">
            <div class="cms-testimonials-item__icon"><img src="https://assets.website-files.com/62826448121477c0eb54a773/633456764c0c618517865056_bugbug-quote-icon.png" loading="lazy" alt="" class="cms-testimonials-item__icon-img"></div>
            <div class="cms-testimonials-item__texts">
              <div class="cms-testimonials-item__main-text">“In two days we managed to automate test cases that took us weeks to write up and execute using other software. It is easy to use, yet can still perform advanced testing should the tester want to.”</div>
              <div class="cms-testimonials-item__author">— Panayiotis, Software Tester, ATM Marketing</div>
            </div>
          </div>
          <div class="cms-testimonials-item">
            <div class="cms-testimonials-item__icon"><img src="https://assets.website-files.com/62826448121477c0eb54a773/633456764c0c618517865056_bugbug-quote-icon.png" loading="lazy" alt="" class="cms-testimonials-item__icon-img"></div>
            <div class="cms-testimonials-item__texts">
              <div class="cms-testimonials-item__main-text">"Very good and fast support from the BugBug team. Have clear documentation which really helps you to get started. Features like scheduling test cases and getting results are perfect for our needs."</div>
              <div class="cms-testimonials-item__author">— Imalka, QA Engineer, Calcey.com</div>
            </div>
          </div>
        </div>
        <div class="cms-testimonials-row">
          <div class="cms-testimonials-item">
            <div class="cms-testimonials-item__icon"><img src="https://assets.website-files.com/62826448121477c0eb54a773/633456764c0c618517865056_bugbug-quote-icon.png" loading="lazy" alt="" class="cms-testimonials-item__icon-img"></div>
            <div class="cms-testimonials-item__texts">
              <div class="cms-testimonials-item__main-text">"BugBug is super intuitive and lowers the barrier of becoming an end-to-end tester. It reduces development time and significantly lowers the cost of software testing. "</div>
              <div class="cms-testimonials-item__author">— Wojciech, CEO Davinci-Studio.com</div>
            </div>
          </div>
          <div class="cms-testimonials-item">
            <div class="cms-testimonials-item__icon"><img src="https://assets.website-files.com/62826448121477c0eb54a773/633456764c0c618517865056_bugbug-quote-icon.png" loading="lazy" alt="" class="cms-testimonials-item__icon-img"></div>
            <div class="cms-testimonials-item__texts">
              <div class="cms-testimonials-item__main-text">"We always wanted to automate, but didn't have resources. BugBug empowered us to do so. Very simple, quick to use, and looks really cool!"</div>
              <div class="cms-testimonials-item__author">— Kris, CEO, Innential</div>
            </div>
          </div>
          <div class="cms-testimonials-item">
            <div class="cms-testimonials-item__icon"><img src="https://assets.website-files.com/62826448121477c0eb54a773/633456764c0c618517865056_bugbug-quote-icon.png" loading="lazy" alt="" class="cms-testimonials-item__icon-img"></div>
            <div class="cms-testimonials-item__texts">
              <div class="cms-testimonials-item__main-text">"It works great! BugBug saves me hours of monotonous work writing and maintaining tests."</div>
              <div class="cms-testimonials-item__author">— Martin, Python developer</div>
            </div>
          </div>
        </div>
        <div class="cms-testimonials-row">
          <div class="cms-testimonials-item">
            <div class="cms-testimonials-item__icon"><img src="https://assets.website-files.com/62826448121477c0eb54a773/633456764c0c618517865056_bugbug-quote-icon.png" loading="lazy" alt="" class="cms-testimonials-item__icon-img"></div>
            <div class="cms-testimonials-item__texts">
              <div class="cms-testimonials-item__main-text">"BugBug is a great choice for our business because it is easily accessible and intuitive. In the free plan, it offers tools that meet our daily needs."</div>
              <div class="cms-testimonials-item__author">— Mariana Barbosa, Azzo Tecnologia</div>
            </div>
          </div>
          <div class="cms-testimonials-item">
            <div class="cms-testimonials-item__icon"><img src="https://assets.website-files.com/62826448121477c0eb54a773/633456764c0c618517865056_bugbug-quote-icon.png" loading="lazy" alt="" class="cms-testimonials-item__icon-img"></div>
            <div class="cms-testimonials-item__texts">
              <div class="cms-testimonials-item__main-text">“A very interesting software testing tool that allows you to run automated tests without any programming knowledge.”</div>
              <div class="cms-testimonials-item__author">— Michał, QA Engineer</div>
            </div>
          </div>
        </div>
      </div>
   </div>

<style>

.cms-testimonials-row {
  display: -webkit-box;
  display: -webkit-flex;
  display: -ms-flexbox;
  display: flex;
  padding-bottom: 32px;
  grid-column-gap: 32px;
  grid-row-gap: 32px;
}

.cms-testimonials-item {
  display: -webkit-box;
  display: -webkit-flex;
  display: -ms-flexbox;
  display: flex;
  -webkit-box-align: start;
  -webkit-align-items: flex-start;
  -ms-flex-align: start;
  align-items: flex-start;
  -webkit-box-flex: 0;
  -webkit-flex: 0 33.33%;
  -ms-flex: 0 33.33%;
  flex: 0 33.33%;
  grid-column-gap: 16px;
}

.cms-testimonials-item__main-text {
  min-height: 100px;
  padding-bottom: 16px;
  color: #666;
  font-size: 18px;
  line-height: 1.6em;
  font-style: italic;
  letter-spacing: 0em;
}

.cms-testimonials-item__icon {
  width: 64px;
  height: 64px;
  -webkit-box-flex: 0;
  -webkit-flex: 0 0 auto;
  -ms-flex: 0 0 auto;
  flex: 0 0 auto;
}

.cms-testimonials-item__author {
  min-height: 100px;
  color: #666;
  font-size: 16px;
  line-height: 1.6em;
  text-align: right;
  letter-spacing: 0em;
}

@media screen and (max-width: 991px) {
  .cms-testimonials-row {
    -webkit-flex-wrap: wrap;
    -ms-flex-wrap: wrap;
    flex-wrap: wrap;
  }

  .cms-testimonials-item {
    -webkit-flex-basis: 100%;
    -ms-flex-preferred-size: 100%;
    flex-basis: 100%;
  }
}


@media screen and (max-width: 479px) {
  .cms-testimonials-item__main-text {
    min-height: 0px;
    font-size: 18px;
  }

  .cms-testimonials-item__author {
    min-height: 0px;
    font-size: 13px;
  }
}
</style>


Case studies

Create tests in your browser for free. Run the tests in the cloud for 14 days at no charge. No credit card required. 

Effortless cloud testing that fits your budget

Discover the differences of each plan and choose the offer that fits your testing needs

Industry leaders automate tests with BugBug

Frequently Asked Questions

Contact our Support Team.
They’ll give you all the information you need.

Free trial of premium features for 14 days

Automate web app testing easier than ever. Without excessive costs. Faster than coding. 