🚀 NEW: Pop-up Windows Recording - [See how to test seamlessly OAuth integrations](https://docs.bugbug.io/recording-tests-steps/record-pop-up-window-actions)

Software Testing Knowledge

Test Automation Expert and Test Architect. He is a seasoned tester who can’t stay long in one place. He is looking for perfection and a place to challenge his skills. He gathered experience working for different companies with different working models, From small to big corporations, From Products via In-house development to software house. Thanks to that he has a wide perspective on testing quality and delivering value.  
[linkedin.com/in/wyrodek/](https://linkedin.com/in/wyrodek/)

Maciej Wyrodek

[TABLE_OF_CONTENTS]

Testing, especially end-to-end, is currently enjoying a revival. A lot of new testing tools and approaches have emerged over the last few years. Selenium has finally launched version 4, Cypress and Playwright have brought coded test automation to a new level and companies like Bugbug, Mabl, Testim, have revitalized the idea of record and playback testing. 

Since a lot is happening, let's take a look at specific tools and compare the strengths and weaknesses of 3 approaches: **Custom Made Frameworks**, **Out of the Box Framework** and **Record and Playback**. We will look at the pros and cons of all the solutions and when they work best. 

However, first we need to discuss the reasons for using automation and then we will discuss how to use record and playback testing. 

## Why we automate tests?
-----------------------------------

If we look hard enough, all goes back to the one and only goal of companies, as stated by [Eliyahu M. Goldratt in The Goal](https://en.wikipedia.org/wiki/The_Goal_(novel)) - Making money. Testing by itself often is considered a cost - as it does not bring more money. 

But let's not forget that more money can be made not only by selling more - but by increasing the margin - either by making the product worth the price or by reducing the cost. 

Testing can help to protect the bottom line, by reducing the cost of mistakes and finding them more easily and faster. This also leads to reducing the cost of customer support, refunds, etc. 

But as [DevOps](https://itrevolution.com/the-devops-handbook/) and [Modern Testing](https://www.moderntesting.org/) state - testing can synergize with a great experimentation culture, leading to possible delivery of a quality product in a shorter time and cost-savings on the development cycles.


->[Automate your tests for free](https://app.bugbug.io/sign-up/){.markdown-cta}<-

## Traits of good automated testing tools
-------------------------

For automation to be useful in the DevOps world, it needs to be: 

-   Reliable 

-   Fast

-   Easy to write and maintain new test 

-   Easy to run

### Reliability

A reliable testing framework consistently produces accurate and consistent results. It should be able to detect defects reliably and identify regression issues when changes are made to the software.

### Improve your testing, fast

The development cycle is fast. If developers need to wait for the results of their work more than a coffee break, they will switch to the next task. Now the results will need to wait which leads to the cost of context switching. And longer [lead time of a task](https://kanbantool.com/kanban-library/analytics-and-metrics/kanban-definition-of-lead-time-and-cycle-time).

### Ease of writing and maintaining

**Automated testing should be a regular part of a software development lifecycle**, benefit must justify the cost. If writing a test takes 4 days, it'd better be worth it - especially if developing  the tested feature takes less than that. 

The same holds true for maintenance: we need to be able to easily understand why a test has failed and whether this is a code issue or a test issue and how to fix it.

### Ease of running

This relates to being fast: anybody who needs to use a test to check a code should be able to do it without extra hoops, for example automatically running them locally or via CI/CD.


->[Automate your tests for free](https://app.bugbug.io/sign-up/){.markdown-cta}<-

## How to choose the best test automation tool?
------------------------------------------

1. **Understand testing requirements:** Define project objectives, scope, and the types of tests to automate. Consider application type (web, mobile, desktop), supported platforms, and required test coverage.
2. **Check technology compatibility:** Verify if the tool supports your application's technologies, programming languages, and frameworks and testing needs. 
3. **Evaluate ease of use:** Consider your team's expertise. People can benefit from user-friendly interfaces or record-playback features, while developers may prefer script-oriented tools.
4. **Ensure cross-platform support:** If testing is needed on multiple platforms (e.g., Windows, macOS, Linux, Android, iOS), choose a tool that supports cross-platform testing.
5. **Assess scripting languages and frameworks:** Pick a tool that aligns with your team's skills, facilitating maintenance and extensions.
6. **Examine community and support:** Use modern testing tools with strong online resources, documentation, and active user communities for better assistance.
7. **Consider vendor support and reliability:** Choose reputable vendors providing regular updates, bug fixes, and customer support.
8. **Analyze cost and licensing:** Assess total ownership costs, including licensing, training, and maintenance, seeking value for money within budget constraints.

## Record and playback testing tools
------------------------------------------

Record and playback test automation also known as capture-replay or scriptless tools, are a type of automated software testing tool that **allow employees to create test cases by recording their interactions with the application under test (AUT)**. 

The tool records these interactions as a sequence of user actions (test steps) and system responses, which can be played back later to re-execute the test scenario automatically.

Here's how record and playback testing tools generally work:

1. **Record The Test:** User interacts with the application as if performing manual testing. The tool records each action taken by the tester, such as clicks, keystrokes, form submissions, and other user interactions. It captures the sequence of events and builds a test script based on these interactions.
2. **Playback:** In the playback phase, the recorded test cases script is executed automatically against the application. The tool replicates the actions recorded during the recording phase and verifies the responses from the application against the expected outcomes.

Record and replay testing tools are commonly used for web applications and graphical user interface (GUI) testing. They are particularly beneficial for QA engineers who may not have strong programming skills or for quickly generating test scripts during the early stages of testing.

## Which testing approach is for you?
------------------------------------------

We will look at the 3 most popular approaches, starting from ones requiring most coding and maintenance effort, like "Custom Frameworks", through usage of "Out of the box" Frameworks and finishing with playback automation. We will look at the pros and cons. 

As it is my belief, all these approaches have made a name for themselves on the market and suit different project needs.

**Pros**

-   Great customizability

-   Ownership

-   "On-site support"

**Cons**

-   Huge maintenance cost

-   Time consuming

-   Requires lots of skill to maintain

When I was starting test automation, this was the way to go. Mainly because Selenium WebDriver was the tool of choice back then. And as its description stated, [Selenium automates browsers](https://web.archive.org/web/20120208115222/http://seleniumhq.org/). Selenium was just a tool that did exactly what it said it delivered.

It didn't provide any functions for running test, or making assertions. At that time, practically every company had to build its own framework.

In most cases, they were just learning how to automate less complex tests. Making lots of mistakes in the process. But some companies managed to make great frameworks and, as a result, decided to share them with others. Some of them are still in use and developed but other have not stood the test of time. [Ocaramba](https://github.com/ObjectivityLtd/Ocaramba),[Selino](https://github.com/TestStack/TestStack.Seleno), [Serenity](https://serenity-bdd.github.io/theserenitybook/latest/index.html),[Selendie](https://selenide.org/), [Protractor](https://www.protractortest.org/#/)

This approach is still used today. Why?

### Pros

The main advantage of this approach is control. It gives a company the greatest level of control in customization. If a specific feature for a project is required, it is relatively easy to find someone who can develop this functionality.

Additionally, it gives a company considerable independence; there is no risk that at one day, the tool support will end or [repo will be deleted](https://nakedsecurity.sophos.com/2022/01/11/javascript-developer-destroys-own-projects-in-supply-chain-lesson/), leading to massive issues.

What is more, there is someone who knows how the tool works on site and can teach new team members..

### Cons

The main drawback of this solution is the cost - **maintaining internal test automation framework costs time and money**. Many companies underestimate the cost of developing test automation.

**Time is of essence here:** this automation needs a long time to be developed,  from creating a framework to writing a test. And this is the aspect that can be most damning. As discussed before, for automation to bring value, it needs to provide fast feedback.

**Creating a custom framework is as big a project as any and it requires a good dose of skills**, ideas, and planning. This means a lot of expertise while typically, companies are not willing to spend precious resources on test automation.

->[Automate your tests for free](https://app.bugbug.io/sign-up/){.markdown-cta}<-

## Out of the box frameworks: Pros & Cons
--------------------------------------

**Pros**

-   Shorter time needed to write first test

-   Much less need for framework maintenance

-   Lots of available online trainings and knowledge

**Cons**

-   Often only limited support from community

-   Writing and Test Maintenance takes time

-   Huge dependence on framework developers

As mentioned before, the growth of Selenium was accompanied by a lot of frameworks created on its base. Over time, the issues with the web driver became increasingly prominent, leading to the re-emergence of new automation frameworks not based on Selenium: [Cypress](https://www.cypress.io/), [Puppeter](https://github.com/puppeteer/puppeteer), [Playwright](https://playwright.dev/), to name a few most recent ones.

All these tools had one thing in common - they provided the user with **out-of-the-box, ready options to write automation**. These tools usually provide built-in test runners, an extensive assertions library, boiler plates for starting writing project and tests, and - most importantly- a built-in mechanism to deal with test flakiness and general instability.

Additionally, these frameworks usually provide their own philosophy of writing test automation. As long as what we need to do is with accordance to their approach, everything goes well. Problems start when we need to go beyond what is suggested.

Until recently, the best example has been a test covering multiple domains in Cypress, but as of [version 9.6](https://docs.cypress.io/guides/references/changelog#9-6-0), it is no longer such a grievous case.

In another scenario, we need to rely heavily on framework developers, if they introduce breakthrough changes. Usually, we are left with two options: update and fix the code. Alternatively, don't update and hope that there won't be any issues in the framework that need fixing.

### Pros

Compared to the custom frameworks, we can **start writing tests almost immediately** without the need of spending weeks or months on writing the toolset first.

Similarly, we don't need to wait whenever the operating system or browsers have an update. The tool maintainer will update the framework for us. 

And since most tools in this category are open-source ones, there is lots of information easily available online.


### Cons

As mentioned before, these tools have **their own test philosophy, which is much more restrictive**. Quite often support is limited - these tools are usually open-source and support in case of issues can be unavailable where we need to resort to forums and chats where we hope someone will answer a question; sometimes we get the answer in minutes but it may never come just as well.

However, the greatest risk of all is that **we are at the mercy of framework developers** - these tools become a part of our code base and if we want to keep support, sooner all later we will need to update even if we don't want to. 

A case in point is Cypress 10 - [it introduced breakthrough changes](https://docs.cypress.io/guides/references/changelog#10-0-0). As a result, many plugins and customer functionalities won't work without extensive updating.


->[Automate your tests for free](https://app.bugbug.io/sign-up/){.markdown-cta}<-

## Record and Playback Testing Tools: Pros & Cons
------------------------------------------

**Pros**

-   Fast Test Writing Time

-   Low Entry Level

-   Professional Customer Support

-   Easy Scalability

**Cons**

-   Usually cloud-based, requires access to test environments

-   Still young, "untested" technology

-   Requires a new mindset

-   Data management can be quite complicated

-   Usually a commercial paid tool


The idea of **record and playback is not new**, even Selenium has Selenium IDE which, after a few years on developments lately, has won some users' recognition.

Microsoft had Coded UI Test as part of the Visual Studio from 2010 and only in [2019 announced it deprecation](https://devblogs.microsoft.com/devops/changes-to-coded-ui-test-in-visual-studio-2019/)

Back then this technology was long before its time, the frameworks were very unstable and the maintenance of this test was problematic.

But in the last 4-5 years, a lot has changed. Solutions like [Bugbug.io](https://bugbug.io/), [Mabl](https://www.mabl.com/), [Testim](https://www.testim.io/) have taken over what modern technology has to offer and reinvented the concept.

### Pros

Most **coded options require great investment of time into writing and maintaining test** - usually even a trivial change costs around half an hour. 

In case of record and playback feature, test are usually meant to be disposed of after they stop working, as it is **much easier to re-record a test instead of fixing it**. 

This is of special importance for **startups** and projects with a lot of changes in the UI layer. When a team is spread thin, it is hard to find the time to write a test. But usually, you can record a few tests while checking if what has been done even works.

On the other hand out-of-the-box coded frameworks offer usually an **easier learning curve**. Record And Play tools usually **do not require coding knowledge** - although in more complicated cases some JavaScript knowledge can prove very helpful.

Since most of the record and play tools are paid solutions, it also means that companies have professional Trainers and Customer Support to help when issues arise. Usually, it means that in the case of young projects, we can even **influence their feature roadmap**.

These modern record and play solutions come with their own cloud runners, making **scalability unbelievingly easy**. Typically, paralyzing a test takes a few clicks.



### Cons

Unfortunately, **Cloud Running** is also a huge Con. First and foremost, we need to give this tool access to our test environments. An additional issue related to the Cloud is Term of Service - **not all projects and customers can afford cloud services**.

Most of these Record & Play tools are relatively young and not battle-tested. For long-term projects, this uncertainty can be very risky.

The biggest **issue is Data Management.** So far, no Record and Play solution has provided an easy way to manage test data. And this can be a huge blocker. Consequently, we need to change our approach to E2E automated tests, with a desire to cover all the important and easy paths as fast as possible.

Lastly, there is the price - **cloud solutions are not cheap** and if we need to run a lot of tests, it may be quite expensive. But some tools like [BugBug](https://bugbug.io) are on the other hand very cheap and offer unlimited cloud runs for a fixed mohtly fee. Arguably, it is still less expensive than maintaining coded automation and its infrastructure.

->[Automate your tests for free](https://app.bugbug.io/sign-up/){.markdown-cta}<-

## No clear-cut distinction
------------------------

Before the final thought, some clarification is required:

**As much as we humans like clear categorizing, the real-word borders are much more blurred.**

For example: should Playwright be in the Custom frameworks or in the Out-of-the-box category? After all, its strength is the ability to use its own test runners, reporting services and easy building of the new features we need.

Similarly, it has the Record and Play functionally and so did Cypress (at the moment of writing this post, the latest version of Cypress is 10.3 and since 10 Cypress Studio, which allowed test recording, has been deactivated).

There are also tools that I like to refer to as **combine-harvesters**: basically, they do everything and claim that they can test anything, from mobile via web to desktop and many others -- the examples include TOSCA and Test Complete.

That's why we need to look at the numerous different dimensions of automation - and how much coding is just one of them.

## Summary
-------

It is my belief that over time, both no-code development and record and play automation become the dominant players on the market. Yet there is a lot of room for improvement. 

All the types of approaches to automation, starting from custom frameworks, or using out-of-the-box coded solutions ending on Record and Play tools, have their uses on the market.

They answer to different needs and solve different problems. So, before selecting any tool, it is always good to ask yourselves: "What are we trying to gain?"

Record and Play vs Coded Automation Testing

large_image_1_9678043200.jpg

medium_image_1_9678043200.jpg

small_image_1_9678043200.jpg

thumbnail_image_1_9678043200.jpg

image_1_9678043200.jpg

Record and Play vs Coded Automation Testing - How To Choose What Works For Your Team?

[TABLE_OF_CONTENTS]

Testing can be difficult, there are many questions and many issues we can face with every new feature and every new story. Every day, testers need to decide what to test, how to test, what to automate, and what risks to accept. These decisions are hard and require a lot of experience. 

Fortunately, there are techniques, heuristics, and models which can help with planning tests. One of the oldest and still very useful techniques is **Agile Testing Quadrants** (**ATQ**). It is a simple table that helps us select the tools we need to use, depending on the required outcome. In this article, we will look into the history of Agile Testing Quadrants, their uses, and limitations. We will also discuss the attempts at modernizing it.

![Agile-Testing-Quadrants](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/Agile_Testing_Quadrants_4b2d044860.png)




## Agile Testing Quadrants or Agile Testing Matrix?

First, we need to clear one thing: as almost everything in the IT sector, even this simple table might have many names! Chances are that you will come across it as an Agile Matrix, Testing Matrix, Testing Quadrants, or Agile Quadrants. All of them refer to the same thing. The only exception is the Agile Testing Matrix which may refer to both Agile Testing Quadrants and its precursor the Named Agile Testing Matrix.

![Testing matrix](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/test_matrix_00ac277513.jpg)


## The Story of Matrix

When speaking about **Testing in Agile**, there is probably not a single more important person to its development than [Lisa Crispin](https://lisacrispin.com/) and [Janet Gregory](https://janetgregory.ca/). Their book, [Agile Testing](https://www.goodreads.com/book/show/5341009-agile-testing), has helped many testers. [Testing Pyramid ](https://bugbug.io/blog/software-testing/testing-pyramid-a-successful-approach-to-test-automation/) was born while [Mike Cohn](https://twitter.com/mikewcohn) tried to explain the nuances of automation to Lisa’s team (*Experience of Test Automation - chapter 1 Section 1.3.*)  This book, a true agile manifesto,  played a key role in the popularization of Agile Testing Quadrants. The creation of ATQ is often misattributed to this wonderful duo. As Lisa Crispin explained in [her blog](https://lisacrispin.com/2011/11/08/using-the-agile-testing-quadrants/) - they improved on the older concept by [Brian Marick](http://www.exampler.com/old-blog/2003/08/22/#agile-testing-project-2) - aan Agile Testing Matrix created as part of his blog series presenting his vision of testing in Agile. So Agile Testing Quadrants have almost 20- year-long history and they are still one of the best models used for various types of testing.
->[Automate your tests for free](https://app.bugbug.io/sign-up/){.markdown-cta}<-


## What is the ATQ anyway?

Agile Testing Quadrants are a visual tool that allows to differentiate business facing tests and technology facing tests. The quadrants help to change your thinking about testing as a whole. The matrix consists of four basic quadrants and is able to support the team by better planning the testing effort as well as explaining the reasoning behind different types to a non-technical stakeholder.

![Agile Testing Quadrants](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/0cec20b5_414a_4aa3_9fbc_9cdc6e99c32a_31b629100e.png)

Let's briefly go thru the 4 testing quadrants:

**Quadrant 1**

Unit tests and Component tests are those most commonly associated with automated testing. They are performed throughout your applications' lifecycle. The testing activities aim to give feedback to your developers with test cases focused for example on code quality.

**Quadrant 2**

The mix of automated testing and manual testing are focused more on the end client of your app. Functional testing and story testing will aim to check if your software meets the user requirements. Manual and automated tests focus on simulations as well. 

**Quadrant 3 **

The third out of four quadrants holds all your manual tests. Not only the testing that needs to be done by your QA team, but the user alpha and beta testing as well. We mentioned before that allowing a small group of end users to critique the product is usually vital for success. The feedback they provide is amazing!

**Quadrant 4 ** 

Last but not least there are those tests that need all the fancy software, especially if your QA team is not that experienced. Load testing and performance testing are vital. All those technology based tests aim to improve your applications security.


## How to use the Agile Testing Quadrants? Examples for different test types

Agile Testing Quadrants have two axes which split the matrix into 4 quadrants. Both axes pose the same questions, namely **What is the goal of your testing?** The major difference between the two axes is the flavor of the question: the horizontal axis asks what is currently more important: helping the team to deliver faster or ensuring that the product is as good as possible? The vertical axis asks if we are focusing on the aspects of technology, ensuring that everything is working well, or are we looking from the perspective of business and customers, making sure that our program is worth using? Obtaining both answers points to one of the Quadrants with a list of useful examples. This can be a little confusing, so let’s look at some of the examples.

### Example 1: E-commerce Shop

Let’s say we have an e-commerce project that is going nicely. However, there is one enormous risk on the horizon, namely quite a lot of features. Each one is tested separately so nobody is sure how they will work in the real world. The team believes that the standard paths will work. However, nobody is sure what kind of edge test cases will come from the real world.

Looking at our matrix we can see that the problem is definitely business-facing, because we don’t know what the user will do. We also need to make sure that our product will handle it, ergo - Critique Product. 

Looking at it we land in quadrant 3 where we have propositions of the following techniques:
 - Exploratory Testing
 - Scenarios
 - Usability Testing
 - User Acceptance Testing (UAT)
 - Alpha/Beta Testing.

So our goal is to find edge cases. Looking at the list, all the techniques could prove helpful. But some may require too much investment so late in the development (Scenarios). Some are a poor fit for an e-commerce shop (alpha/beta testing, UAT). Which leaves us with Usability Testing and Exploratory testing. ATQ is not an extensive matrix, it contains only examples of techniques rather than a complete list. 

For example, a different technique that could help is [Dogfooding](https://en.wikipedia.org/wiki/Eating_your_own_dog_food) where company employees themselves use a product to satisfy their everyday needs.
->[Automate your tests for free](https://app.bugbug.io/sign-up/){.markdown-cta}<-

### Example 2: Messaging App

A company has created a very successful messaging app, but now they are facing the problem of unexpected success. Their small app, designed for a small community, is rapidly gaining popularity. The team feels that the app is not ready for all this level of attention. The situation is clear - from a business perspective, everything works well enough. The risk is technology, and again the problem lies in the critique of a product. This leads us to Q4 where we have, among others, Performance and Security testing. There is no doubt that we will need both. Lots of users means a load on an application and nothing will kill its momentum like failing servers. Security is also important. Attention also attracts bad actors who will try to exploit the system.

### Example 3: Early Development

The team has a problem with poor code quality. In the case of some features, developers are reluctant to fix a familiar bug because of the high risk of breaking something else. One glance at the ATQ shows that we need something to support the development and its technology which leads us straight to Q1. Both Unit Test and Component Test would improve team productivity and code quality.

The above example may give a wrong impression that Agile Testing Quadrants are used to select one and only one tool. **THIS IS NOT TRUE:** we can use ATQ for each goal of our testing, for each need, for each problem we are trying to address. However, trying to improve too much at once may lead to too much-wasted effort and no result. Concentrate on one subject at a time.
->[Automate your tests for free](https://app.bugbug.io/sign-up/){.markdown-cta}<-

## Evolution

While the concept of AQT has changed little over the years, the techniques in each quadrant have evolved, for example:

![Updated Agile Testing Quadrants](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/updated_atq_06346e83b7.jpg)

In [More Agile Testing](https://www.informit.com/articles/article.aspx?p=2253544), the author presented a new version with Micheal Hüttermann changes, with a circle of Outside-in, Barrier-free, Collaborative in the middle of the quadrants. The author believes that it will stretch beyond any quadrants. The main issue with this update is that it is a catchphrase and provides no information about the tools and techniques that may help.

![Agile Testing Quadrants next iteration](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/Agile_Testing_Quadrants_next_d74569f53c.jpg)

A few years back, [Lisa and Janet](https://agiletester.ca/applying-the-agile-testing-quadrants-to-continuous-delivery-and-devops-culture-part-1-working-towards-continuous-delivery/) also presented a version more in line with DevOps. This version adds a lot more techniques to each of the quadrants but also changes one side, from Supporting Team to Guide Development. While it may seem like a single name change, fortunately, the original name leads to many misunderstandings and the new label is much more descriptive.

![RST Agile Testing Ecosystem 1.1](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/bb_ATQ_1fb44a3b67.png)
->[Automate your tests for free](https://app.bugbug.io/sign-up/){.markdown-cta}<-


Michael Bolton and James Bach have also made the [above version](https://www.slideshare.net/HcmcStc/the-new-agile-testing-quadrants-bringing-skilled-testers-and-developers-together-james-bach) which is compatible with Rapide Software Testing. Most of these updates are only fine-tuning, adding elements that their author thinks are better in this context. So which one to choose? It is the author’s opinion that the original version holds well? It is still relevant, because:

## It is just a model.

“All models are wrong, but some are useful” is an old aphorism attributed to George Box. And it is important to bear it in mind in the case of Test Pyramid, Agile Testing Quadrants, etc. Models are what they are: models which need to help us think about a problem. **They are not the golden rules that should be followed to the letter.**They provide us with a good way of tackling the problem. But it is up to the user to decide if a specific model will work for a specific problem.

## Summary

The **Agile Testing Quadrant** is a model which is nearly 20 years old. Owing to its simplicity, it is still relevant. It is a great tool for teams and testers who have a problem with choosing the right testing method. For junior testers, it is a great place to look into different tools and approaches, and for more advanced testers it will be an excellent aid in building or improving the testing process. 



Happy (automated) testing!

large_beetle_g2d0bd3bb5_1280_e81aa7fc64.jpg

medium_beetle_g2d0bd3bb5_1280_e81aa7fc64.jpg

small_beetle_g2d0bd3bb5_1280_e81aa7fc64.jpg

thumbnail_beetle_g2d0bd3bb5_1280_e81aa7fc64.jpg

beetle_g2d0bd3bb5_1280_e81aa7fc64.jpg

Agile Testing Quadrants

As a movement, DevOps is over 10 years old, and it is one of the most dominating approaches to modern software development. In some companies, however, testing in DevOps has become a critical issue.

[TABLE_OF_CONTENTS]

Lots of old methods and techniques are no longer good enough. DevOps requires changes in testers’ mindsets. Not everybody could adopt to this new era. Today we will try to help. We will look at some practices and techniques that actually work! We will also talk about quality mindset and look at automation. But first, we need to answer a few questions about DevOps and Testing, starting with:

## What do we mean by DevOps?

As with everything, DevOps has become a generic term. Everybody understands it differently:
- for some, it is about tools like CI/CD,
- for others it is a role in a company - usually a cloud and a CI/CD expert.

We will use a definition by [Gene Kim](https://itrevolution.com/faculty/gene-kim/){rel="nofollow"}, author of books like [Phoneix project](https://itrevolution.com/the-phoenix-project/){rel="nofollow"} and a [DevOps handbook](https://itrevolution.com/the-devops-handbook/){rel="nofollow"}, one of the greatest evangelists of DevOps.
In short: DevOps is a culture which helps to create learning organizations which can outperform competition, thanks to fast learning and experimentation enabled by fast delivery of the product.

In [Beyond the Phoenix project](https://itrevolution.com/book/beyond-phoenix-project-audiobook/){rel="nofollow"}, Gene Kim summed it up as 3 ways of DevOps:
- All about Flow,
- All about Feedback
- All about Learning

In short: You need to deliver a product fast to see how it works and learn what to improve, then repeat the cycle.
->[Automate your tests for free](https://app.bugbug.io/sign-up/){.markdown-cta}<-

## What About testing?
Testing is good about collecting feedback and giving information, so it can be helpful with the feedback part.
But what about speed? Management Often sees testing as cost and a bottleneck.
Some even say you can’t have a good quality product if you are going fast. As we learn later, this is not true, but first, let’s discuss another important topic:

## What is quality?
I want you to answer one question. Which product is of better quality:
 - both are doing similar things,
- both have been on the market for some time
- one is a popular product with millions of users and known bugs in hundreds.
- the other one has less than a thousand users but virtually no bugs.

*So which is of better quality?*
At first glance, the other one feels as a better answer since it has fewer bugs.
But commercially it is a failure, it barely has any users! Users chose the other one  - maybe it was first on the market, maybe has had better marketing, maybe a better price.
The fact remains that, despite its all flows, the product is good enough for the users. We call this shippable quality, a product that good enough for users doesn’t need to be perfect as long as it satisfies the users’ needs.
The first thing you need to understand about Testing in DevOps is that **shippable quality today is better than perfect quality tomorrow**!

## Quality at Speed in DevOps
There is also one interesting paradox: the faster and more often you deploy, the better your product quality.
Why? In short:
1. First, you deploy frequently,
2. Each deployment contains less code, instead of months ita takes weeks, instead of weeks it takes days or even a single story.
3. Less code means fewer merging issues, fewer dependencies to check.
4. It also means smaller changes that are easier to test.
5. Resulting in fewer bugs.

It means you can also deploy quick fix if we find a serious issue.
->[Automate your tests for free](https://app.bugbug.io/sign-up/){.markdown-cta}<-

## The Goal of Testing in DevOps
Summing all the concepts, we add all this together. The goal of testing in DevOps is clear: Empower teams to achieve testing as fast as possible.

## Ways of Testing in Devops

### Shift Left
To put it simply, Shift left is about testing as early as possible - and it doesn’t mean testing the finished product as early as possible. It means that, if you can find a partial concept or even an idea that you can verify before everything is done, you should do it at first chance.

A few Examples:
Start with the documentation. If the team has refinements, or there is a meeting where new features/stories are discussed, this is the place to put on the testing hat and do some lateral thinking. Pinpointing the issues here will save time later.

This is the time to ask a few important questions:
- How are we going to test it?
- What can we do to test it as fast as possible?

If the team is too big and the entire team meeting is out of the question, it is a good idea to have a 3 amigos meeting with the developer, the tester and the product owner/analyticto discuss how to develop a story and how to test it. The goal of this discussion should be to decide what needs to be automated and what to check manually.

Another example of Shift left is Dev Testing - instead of having a dedicated testing expert, developers with enough training can very well test their own code. It requires practice and discipline but canal in all, it saves a lot of time on the hand over imagine standard approach:
1.  developers write a code;
2.  the code waits for testing
3.  the testers test and find bug
4.  the bug waits for the developer to have the time to fix it
5.  the developer fixes it
6.  the bug waits for retest.

You get the idea. Now let’s look at a much shorter loop:
1. the developers write a code
2. they test it and fix issues immediately
3. Deployment

[As The Goal](https://www.amazon.com/Goal-Process-Ongoing-Improvement/dp/0884271951){rel="nofollow"} has showed us, a physical product wastes a lot of time just waiting to be worked on by different stations. Our work is not so different.
Of course developers need to be first taught how to test - and there are ways to do it: Dev-Test Pairing, or [Session Based Testing](https://www.satisfice.com/download/session-based-test-management){rel="nofollow"} with debriefings.
There is a lot more to shift left, but in general you can describe it in one phase: test as soon as possible.

### No code automation
Automation is a vast topic which we won’t cover completely here. Test automation should be a part of shift left, meaning: Automate as early as possible and only as much as you need. There are two tools that help in deciding what and how should be automated – a [Test Pyramid](/blog/software-testing/testing-pyramid-a-successful-approach-to-test-automation/) and Testing Quadrants.

However, the basic rule of thumb isthat if something can be done as unit test simply it should be done.

Unfortunately, even using this rule, we still need some end-to-end testing - this type is difficult to write and hard to maintain. Flaky(unstable) test are the bone of the industry.  Even the best companies, like Google, have problems with achieving 0% of flaky tests when creating end-to-end tests using tools like Selenium Web Driver or Cypress.

Usually, writing and stabilizing an end-to-end test takes a long time. Now imagine that you want to deploy often, but practically each new story requires fixing/changing the test automation, which takes a long time and slows you down.

Fortunately, the popularity of no-code leads to the rise of a new generation of Test Automation Tools like [BugBug.io](https://bugbug.io) and Mabl. Both allow to create a stable test in minutes!
Previously, the biggest issue related to this tool was that if a test was failed, your only option was to rerecord it. While fixing these test is much simpler, remember that Testing in DevOps is about speed!

Even if you would have to rerecord the test, it still will be faster than fixing a coded end to end test Written in Selenium Web Driver or Cypress.

### Monitoring and Testing in production
Unit and integration testing is all very nice but your system testing is also important. You need to make sure that everything works nicely together. There is one problem: the test environments. 

Usually they are not really on a par with production - they may have too low resources, run on different OS and many, many other issues. Even in the staging environment, often it is not a good representation of the production environment. Yes, in theory they should but in reality, if you want to test in an environment resembling production, there is no better place than production itself!

Ok, but how do you do that?
There are lots of approaches.
For example, you can do [cannery releases](https://martinfowler.com/bliki/CanaryRelease.htm){rel="nofollow"} of your application to see how it works in the wild.

If you have [a/b testing](https://hbr.org/2017/06/a-refresher-on-ab-testing){rel="nofollow"} - you can use these tools to hide all the new good things behind the a/b test and, after team ensures it is working, you can slowly start to show it to more and more users.
Another approach is [blue/green deployment](https://martinfowler.com/bliki/BlueGreenDeployment.html){rel="nofollow"}.

However, first YOU NEED TO HAVE GOOD MONITORING - with good observability. especially of user activity.
Remember what DevOps is about: “all about flow, all about feedback, and all about learning” - monitoring helps with both feedback and learning!
With a good rollback strategy or feature flags, you can safely deploy a code to production. If something goes wrong, swiftly disable it.

If you want to learn more, Cindy Sridharan runs an amazing blog [post](https://copyconstruct.medium.com/testing-microservices-the-sane-way-9bb31d158c16){rel="nofollow"} [series](https://copyconstruct.medium.com/testing-in-production-the-safe-way-18ca102d0ef1){rel="nofollow"} on [this topic](https://copyconstruct.medium.com/testing-in-production-the-hard-parts-3f06cefaf592){rel="nofollow"}.
->[Automate your tests for free](https://app.bugbug.io/sign-up/){.markdown-cta}<-

### Quality Culture
We have talked about DevOps and we have talked about testing in it; there is one more issue to address as even the best tools and technical solutions won’t save the day if there is no culture promoting quality.  In order to create this culture, we need good principles.
Fortunately, [Modern Testing](https://www.moderntesting.org/){rel="nofollow"} comes to the rescue, trying to address the necessary cultural changes for testing in DevOps.
The basic principles include:

1. Our priority is improving the business.
2. We accelerate the team, and use models like Lean Thinking and the Theory of Constraints to help identify, prioritize and mitigate bottlenecks from the system.
3. We are a force for continuous improvement, helping the team adapt and optimize in order to succeed, rather than providing a safety net to catch failures.
4. We care deeply about the quality culture of our team, and we coach, lead, and nurture the team towards a more mature quality culture.
5. We believe the customer is the only one capable of judging and evaluating the quality of our product
6. We use data extensively to deeply understand customer usage and then close the gaps between product hypotheses and business impact.
7. We expand testing abilities and know-how across the team; understanding that this may reduce (or eliminate) the need for a dedicated testing specialist.

## Summary

Today we started with the same definition of DevOps - culture concentrating on flow, feedback and learning.

We have explained the goal of testing in DevOps - empowering teams to achieve shippable quality as fast as possible.

Then we looked at a different way to help us achieve this goal from shifting left, and no-code test automation, to testing on production, ending on testing culture.

Notably, at first many of the discussed issues were not directly related to testing.

Most of the technical solutions require changes either in the process the deployment strategy or the code structure.

This is another important aspect of Testing in DevOps - testing cannot be left to the end. Just like quality, it needs to be built into both the product and the process. Since the beginning, it cannot be an afterthought.

If there is one thing that can bring your team closer to excellent DevOps testing, start with asking these questions: How can we test it? When is the first opportunity we can test it?

large_bug_g27e425b83_1280_0571add848.jpg

medium_bug_g27e425b83_1280_0571add848.jpg

small_bug_g27e425b83_1280_0571add848.jpg

thumbnail_bug_g27e425b83_1280_0571add848.jpg

bug_g27e425b83_1280_0571add848.jpg

Testing in DevOps

Reliable, modern end-to-end test automation tool that improves testers’ and developers’ work.


Before you release your product into the market, it is extremely important to properly test whether it’s functioning as intended. To do that, software developers utilize a variety of testing techniques that verify the software’s integrity. To thoroughly check their software from start to finish, developers implement **End-to-End testing**, often simply called **E2E** testing – an elaborate technique that mimics end user experience. 

There are many layers to End to End testing, as it is one of the most complex testing methods. This type of testing involves testing every part of your software that the end-user interacts with in any imaginable way.

With this guide, we want to show you the importance of E2E tests, as well as tell you precisely what they are, including a few specific examples.

**Also see:** if you're new to software automation, take a look at this [beginner's guide to automation with step-by-step screenshots](https://docs.bugbug.io/in-depth-guides/beginners-tutorial-to-automation-testing).

[TABLE_OF_CONTENTS]

## What exactly is End-to-End testing?

**End-to-End testing** is one of the most useful and informative techniques for software testing, providing valuable data about the inner workings of the program from the end-user perspective. With E2E testing, developers can thoroughly verify the behaviour of their software. During the test, you will be trying to replicate key user behaviour scenarios when using your software. The actual list of things to do varies, depending on the type of software you’re testing – but in all cases, it is rather extensive.

When performing a software testing method, like an End-to-End test on a web application, you would start with opening your browser and navigating to the proper URL, and then proceeding to use the software as intended for the end-user.

Similarly, in the case of a regular desktop application or mobile application, you would open the app just like a user would, checking out every bit of functionality that is intended for the users to work with. Keep in mind that during E2E testing you won’t need to check developer-specific aspects of the software, only the end-user perspective if it is important to you.

## End-to-End test goals

During an E2E test, your goal is to verify all of the dependencies and data integrity that directly or indirectly influence the reception of the software by the end-user. This usually includes any external services, as well as databases used by the software and metrics, as well as any other system dependencies. 

E2E tests will ideally be performed in the production environment, but this isn’t always possible. The tests can, of course, be performed in an independent End-to-End testing environment, but you should make sure it is as close to the production environment as possible.

Within the SDLC process, E2E tests are performed in the Testing phase, which is situated after Coding and before Deployment. 
->[Automate your tests for free](https://app.bugbug.io/sign-up/){.markdown-cta}<-


## Why is End-to-End testing important?

Is End-to-End testing necessary? It is true that more and more developers make sure to implement an E2E testing phase before the final deployment of their software. There are many benefits to E2E tests, but the most crucial ones are:

- The ability to verify software performance from the end-user perspective
- Testing verifies not only the software itself but also any interrelated sub systems, increasing test coverage
- Validation of back-end systems
- Immediate end-user-like feedback on test failing

While tests such as unit testing or integration testing are very useful (functional testing), allowing you to verify the working of each component and providing quick results, they don’t give you the whole picture. By not performing E2E tests, you won’t know how the experience of using your software will look like from the end-user perspective.

For example, individual component testing (integration tests) might show promising results and full functionality. However, when combined, something goes wrong and produces a bug. By only checking individual components, you wouldn’t be able to find out about the bug. 

With a proper End-to-End test, you can verify whether everything works correctly together as it should. Even integration testing doesn’t cover all of the system components of your software, and you might never know whether every aspect of your app clicks together.

E2E testing doesn’t necessarily have to discover bugs – it can also give you useful insight into the user scenario of your product. Overall, all types of tests play their respective roles in the whole vertical end-to-end testing process (differently to horizontal end-to-end tests). Just because E2E tests give you a more complex understanding of your software as a whole doesn’t mean that other test types are less valuable. 

Before releasing your app, always make sure to proceed with a thought-through testing strategy. However, while they’re certainly extremely useful, keep in mind that E2E tests occupy the top of the testing pyramid, making them the most expensive and most difficult to perform.

End-to-End testing is also one of the few functional types of tests that can be performed manually, as well as automated - which is done especially often in Continuous Integration. In such a scenario, a chosen set of tests can be run automatically each time a new iteration is uploaded to the testing server.

->[Automate your tests for free](https://app.bugbug.io/sign-up/){.markdown-cta}<-

## Benefits of end-to-end testing

One of the goals of this software testing technique is to simulate the end-user's perspective. This means that this type of testing helps to verify not only the security or performance, but the business aspect as well. It will check whether the software collects and processes data as it was designed to do. End-to-end testing is essential for a comprehensive validation of your product. This testing ensures that all integrated parts of the application work together as expected.

Another purpose of E2E testing is to mitigate risks. E2E testing verifies if the software is ready for deployment. The confidence in stability is a crucial factor which the stakeholders consider while approving the next release. 

There are obviously much more obvious benefits of this form of testing (like bug fixing or testing across different user scenarios), but we described them already in this article.

## End-to-End Testing Framework

Alright, so now you know what End-to-End testing is – but how to exactly implement it? There is a wide variety of checks and individual tests that must be covered by your end-to-end testing plan, some of which include:

- Establishing test requirements in the preparation phase (including software and hardware requirements)
- Creating proper documentation
- Deciding what type of test environment will be used (production environment or independent test environment)
- Preparation of test cases and general E2E design
- Tracking of data input and output
- Test execution and evaluation

When executing E2E testing methods, developers usually follow an established E2E testing framework that guides you throughout the process, enumerating various tasks and responsibilities. The framework is relatively simple and consists of only three elements:

### Establish user functions

The User Functions relates to listing the features of your software, as well as all of its interconnected software systems. To properly measure and test software functionality, you need to first think about what you’re going to test. After you’re finished with listing user functions, remember to record everything relevant to each of them, including both the input and output data. Since E2E testing can be considered holistic, you also need to pay attention to all of the relations between every individual function – how they work together and whether there are any clashes.

### Come up with specific conditions

Now that you have established the user functions you’re going to test, you need to think about the testing conditions. Try to come up with as many quantifiable indicators that can be measured for each of the functions. These can range from response time to consistency of data flow, to simple convenience of use. Remember – you’re trying to imagine yourself as an end-user, so try to keep track of anything that might be an issue for a non-developer.

### Create individual test cases

Finally, before the E2E testing can commence, you need to prepare specific test cases for each of the conditions. At this point, you have prepared a list of user functions to test, as well as data conditions that will be measured during the test. Now, you need to create at least one test case for each of the conditions. By test cases, we mean user scenarios during which specific indicators will be able to be measured.

And by test case preparation status, we mean the precise location of the test instances being developed, as compared to intended test cases.

->[Automate your tests for free](https://app.bugbug.io/sign-up/){.markdown-cta}<-

## End-to-end test cases for your testing process

Congratulations, you’ve almost reached the end! At this point, you should pretty much be a walking fountain of knowledge about End-to-End tests. However, the raw theory is one thing, and implementing those types of tests for system testing in reality is another. To make the process a bit more digestible, consider this End-to-End testing example for a website with shopping functionality:

- Go to the specified URL
- Open a list of products
- Check whether the information on the list correlates with specific product pages
- Try to add the product to the cart and look for visible confirmation
- Navigate to the cart and check whether the product has been successfully added
- Double-check all of the relevant information (price, product info, etc.)
- Proceed with the check-out and verify payment options integration

See how simple it is in the end? Now you can implement E2E testing methodology in your software to ensure better end-user reception, increase your product quality, and build a positive brand image.

This is just a high level example. If you would like to learn how to create a proper test case from start to end, please check out our article on [test case creation](https://bugbug.io/blog/software-testing/how-to-write-automation-test-cases/). You will have a better understanding on how to design test cases for manual and automatic testing. Also check out our other topics in our testing guide. 

## End to End testing tools

Proper tools may play a crucial role in reducing your effort by automating your end-to-end tests. These tools provide a range of features and capabilities to support horizontal testing and vertical testing of the entire system. They allow testers to perform comprehensive testing processes. Choosing a proper tool for testing is typically a long-term investment. If you research and choose a tool that will enable you to execute and record complex user scenarios and allow testing across multiple technologies, your job will be way easier. 

One of the tools that meet such requirements is BugBug. This automated testing tool proved its efficiency for many clients. On the picture below you might see how many steps might an end-to-end scenario actually have.

![n example of a passed test with a proper output statement.](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/88bc1d9e_16f8_4e1a_b33d_15e2a8d17cdf_cdfc84abfc.webp)

## Summary

End-to-end testing is a type of testing that focuses on validating the whole system from start to finish rather then testing individual functionalities. It is often seen as a software testing methodology, but it is not the case. It may be a part of a methodology (i.e. Agile testing) but it is by definition a type of software testing. 

Performing end-to-end testing covers an important task of checking data flows and focuses on testing if any functional gaps occur. This type of testing is necessary to verify if the software is ready for the next release cycle.

## Frequently Asked Questions

1. **What is automated testing?** \
Automated testing in software development is a form of software testing that uses the execution of predefined sets of operations to probe and validate the software system. The fundamental purpose of automation testing in software development is to generate objective feedback about the quality of the product. One such testing tool is Selenium. It can run UI tests. The ideal situation is to run E2E tests with both manual testing and automated testing approaches. If your application has a lot of complicated user interfaces, you'll need more end-to-end testing.

2. **What is software development lifecycle?** \
Software development lifecycle (SDLC) is the process used by developers to create, deploy and maintain software. This process includes several steps that are all closely connected with one another. These steps include requirements gathering, analysis and design, coding and testing.

3. **What is a testing pyramid?** \
In software development projects, we have limited time and resources. This is why it's important to consider our test efforts carefully and find the right balance between different types of testing. Basically, there are 3 layers in the testing pyramid: Unit Tests, Integration Tests, and E2E Tests.

3. **What is a selector?** \
A selector is a small piece of code that describes a specific button in your app. It allows you to clearly define what should be clicked in automated end-to-end tests. You can learn more about selectors using [this interactive XPath learning tool](https://bugbug.io/xpath-selector-builder/?ref=blogarticle)



Happy (automated) testing!


large_testowanie_end_to_end_98e59a7257_deae9fab1e.png

medium_testowanie_end_to_end_98e59a7257_deae9fab1e.png

small_testowanie_end_to_end_98e59a7257_deae9fab1e.png

thumbnail_testowanie_end_to_end_98e59a7257_deae9fab1e.png

testowanie_end_to_end_98e59a7257_deae9fab1e.png

What is End-to-End (E2E) Testing?

Professional Quality Evangelist, experienced in testing web and mobile applications.  
[linkedin.com/in/kinga-witko-b5bba659/](https://linkedin.com/in/kinga-witko-b5bba659/)

Kinga Witko

[TABLE_OF_CONTENTS]

The term “shift left” expresses movement. It can describe the movement of an entire team or individuals.
What does it do in a project or an entire development lifecycle? Putting it simply,  it reflects the position of testing on a timeline. 

**Shift Left means testing earlier in the development process.** It also means repositioning the engagement, development, and testing efforts towards the customer.


Typically, in a traditional (waterfall) model of software making, testing lands at the end of the pipeline or is one of the last phases of the process. Bringing testing at the very last stage usually causes many problems such as:

- delays in the deployment
- a lot of bugs at the end of the development lifecycle
- bugs discovered just before or after the release to production
- lack of collaboration between developers and testers through the entire process.

Poor software quality is one of the most harmful consequences of all of these pains listed above.

At first, the team members do not communicate with each other and they fail to communicate the problems early. 

It must not necessarily be a bug; sometimes there is some issue with the application setup, there is no testing environment, access to certain resources (such as specific tooling or external software providers), the task description is vague, or the team setup is incorrect. 

If communicated and fixed earlier, the entire process can be smoother and cause less frustration in the team. It also encourages people to focus more on the code quality and to deliver earlier and faster.

Most of those issues can be avoided by changing the style of working and shifting testing efforts at the beginning of the process.
->[Automate your tests for free](https://app.bugbug.io/sign-up/){.markdown-cta}<-


## Agile way of working

If you care about the quality of your product, your mindset has probably shifted left already.

This approach is strongly connected with the agile way of thinking and using agile work methods in software projects.


In this setup, testing is an integrated part of the team and accelerates software development at higher quality. As a result, testers can identify bugs sooner, saving time and effort. Notably, the earlier you test, the less expensive bugs are to fix.


[T The Seven Testing Principles (ISTQB)](https://www.istqb.org/downloads/syllabi/foundation-level-syllabus.html) tell that testing everything is not possible unless a system or functionality is very simple. In practice, however,  the testing efforts should be targeted appropriately at the risk analysis, techniques, and bugs prioritization rather than be attempted thorough testing. 


One of the good methods of approaching testing strategy for a specific application can be the guideline from [Agile Testing Quadrants](https://bugbug.io/blog/software-testing/agile-testing-quadrants/). The quadrants are a taxonomy to help teams plan the testing process and testing activities and make sure they have all the resources they need to accomplish it.


## Shift Left approach and DevOps and agile background

With the rise of the DevOps movement, there is an increased focus on delivering automated, continuous improvement. It holds true for development and testing alike.

What is more, Continuous Integration (CI) and Continuous Delivery (CD) are core software development disciplines integrated into the agile software development cycle. The point is to release fully-tested and stable builds early and as often as possible. Development, delivery, and test automation are crucial components of this process. 

## What it means to “Shift Left”

As mentioned at the beginning, Shift Left testing pushes testing to the “left”, to the earlier stages in the pipeline. This improves software quality and reduces time spent on resolving issues later in the pipeline.

Introducing testing early on the “left-side” of the development process, starting with unit tests and complementing with functional and integration tests is fundamental to making CI/CD happen. But that may not be enough.

The team can go even further: to eliminate unclear and contradicting requirements and improve the end-user experience, it is best to involve test engineers in the clarifying and prioritising application requirements. Test engineers can check the available or newly created requirements or user stories for clarity, consistency, testability, and traceability.

This also improves the shift towards client happiness. An early review of the requirements and spotting the flaws marks an early start of the communication with the end-user. 

The Shift Left approach introduces more care and thought exchange concerning the client. The team can be perceived as more proactive and more dedicated to the best quality standards of a project.
->[Automate your tests for free](https://app.bugbug.io/sign-up/){.markdown-cta}<-


## Benefits of Shift Left Testing for Agile and Devops teams

Agile teams benefit most from Shift Left Testing in terms of clearer specifications, smoother implementations, earlier testing, and reduced costs. However, Shift Right Testing has more benefits like the ability to incorporate user and system feedback based on real usage scenarios. 

In opposition to testing early, Shift Right can provide an additional context to the testing scenarios. Shifting right means testing at the end of the process, typically when the software is deployed to production (the post-production environment). 

It is performed by a group of experts or selected end-users. It can utilize testing methods such as API Monitoring, using Feature Toggles, or measuring the performance of a live application. 

This helps to understand the application better and discover some other flaws that could have been omitted earlier.

Nevertheless, when quality comes first:

- developers spend more time on coding because they discuss potentially good solutions - they do not investigate who pushes the code first into production without testing

- there are Unit Tests beside the application code, available and maintainable

- there is an infrastructure that allows testing before pushing to production, usually the entire team is responsible for the process health

- there is room for improvement and mistakes

- developers pair up to catch obvious errors on the fly and learn from one another

- teams estimate tasks together, taking quality and risks into account

- potential risks in the systems are spotted and minimized rather than left alone

And finally - the application is tested - not only by one automated or manual test  but by a set of different functional and non-functional scenarios. 

The tests are regularly reviewed, adjusted, and observed. This approach introduces a new way of working and highlights the value of testing.


## Benefits of the Shift-Left Testing approach

The idea of shift left testing is simple to understand but hard to implement in the development cycle. It is worth the effort though, especially when the team and the entire organization understands the main benefits of this approach, such as:

- Lower costs - as explained above, bugs detected earlier are less expensive to fix.

- Increased efficiency in the software development process.

- Improved product quality and reliability because more bugs are detected at earlier stages.

- Reduced time to market since the QA process covers the entire development cycle, not only its’ last phase.

- A competitive advantage gained by letting development teams move fast, build and defend their credibility by innovating in iterations.
->[Automate your tests for free](https://app.bugbug.io/sign-up/){.markdown-cta}<-


## Get started with “Shift Left” Testing in Agile

How to approach the change and switch to early software testing? Here is the basic list: 


1. **Plan**

Shape development in a way that is friendly to Shift Left testing. Once the requirements have been established, developers and testers should ideate how to shape the coding and software testing as early as possible. 


Depending on the product expectations as well as the limitations of the team, specific strategies, such as involving testers in the project since the planning phase or using the TDD approach, might have to be crafted to accommodate the Shift Left way of working.


2.** Review the requirements**

It is best to verify not only the existing code but also all the requirements necessary for the code to be written. Early verification can spot obvious mistakes.


3. **Use Static Code Analysis**

Static code analysis is the practice of checking the code without executing it. It runs through the basic code structure and ensures that the code is aligned with many standards and guidelines. By following the approach the team can early detect:

- Programming errors

- Violations of commonly followed coding standards

- Syntax anomalies

- Security issues

A static code analysis is best done via automation. The software should scan all untested codes in a project before it moves to the test phase.

4. **Collaborate**

As in any other [agile testing](https://bugbug.io/blog/software-testing/agile-testing-organized-chaos-or-a-planned-way-of-improving-your-software/) technique - early software testing requires a lot of collaboration with the team: Developers, Product Managers, Business Analysts, and other stakeholders responsible for the overall product. 

Everybody needs to understand the value of the Shift Left approach and spot the difference after some time of the new process introduction because as a result, the client benefits from better software quality and receives a lovable product.

## Conclusion for Left Shift Testing in test automation

There are a number of benefits to left shift testing in test automation. One of the primary benefits is that it helps to identify defects earlier in the development cycle. This means that defects can be caught and addressed before they become more complex and costly to fix. In addition, left shift testing can help to reduce the overall time and effort required for testing by identifying and addressing issues earlier in the process.

Another benefit of left shift testing in test automation is that it can help to improve the quality of the software being developed. By identifying and addressing defects earlier in the process, developers can ensure that the software they are creating is of higher quality and meets the requirements of the end-users.
->[Automate your tests for free](https://app.bugbug.io/sign-up/){.markdown-cta}<-


One of the keys to successful left shift testing in test automation is to incorporate testing into the development process **from the very beginning**. This means that testing is not an afterthought or an additional step at the end of the development process, but rather an integral part of the development process itself. Test automation tools and frameworks can be used to automate testing activities and make it easier to incorporate testing into the development process.

It is also important to involve testers in the development process as early as possible. This allows them to provide feedback and identify potential issues before they become more complex and difficult to fix. Testers can also work with developers to ensure that the code being developed is testable, and that the testing process is efficient and effective.

Overall, **left shift testing is an important approach in test automation that can help to improve the quality of software, reduce testing time and effort, and identify defects earlier in the development process**. By incorporating testing into the development process from the very beginning, developers and testers can work together to create software that is of higher quality and meets the needs of the end-users.

Happy (automated) testing!

large_pexels-pixabay-40716.jpg

medium_pexels-pixabay-40716.jpg

small_pexels-pixabay-40716.jpg

thumbnail_pexels-pixabay-40716.jpg

pexels-pixabay-40716.jpg

Shift Left Testing Explained

Guides to Test Automation

Software developer currently working as a tech writer and PM.  
[linkedin.com/in/marcin-%C5%82ojewski-a4191518/](https://www.linkedin.com/in/marcin-%C5%82ojewski-a4191518/)

Marcin Łojewski

small_Testing Tools for Web Application.png

thumbnail_Testing Tools for Web Application.png

Testing Tools for Web Application.png

Testing Tools for Web Application - The Ultimate Guide 2023

Testing is an ever-present part of a software developer’s job, performed at every stage of the production process. There is a wide range of test types used for software testing, some performed at various points during production, and some performed after production to confirm that the software functions as intended.

Regression testing is performed on a product that has already been finished and released. How exactly is it performed? What are the objectives and how does the procedure look? If you’re keen to find out, keep on reading as we explain everything you should know about regression testing.

[TABLE_OF_CONTENTS]

## The definition of Regression Testing

**Regression testing** is a software testing method that aims to investigate and locate any possible flaws or bugs in software that has already been fully developed. Regression Testing is performed to find out whether your software has regressed (surprise, surprise) as you continue introducing new updates and bug fixes. Whenever a piece of software is changed in any way, new bugs might emerge and some of the older ones re-emerge.

Most often, regression testing is performed immediately after introducing a new update, but it isn’t the only factor that might call for the test. If any software or hardware-related factor has changed, regression testing should be committed to find out whether everything performs as it has before the change. 

This can include any sort of bug fixes and software changes, but regression might also be caused by hardware configuration changes and electronic component degradation over time. Regression testing isn’t easy – it requires cooperation and diligence out of all of the team members, which makes it perfect for agile software development practices.

->[Automate your tests for free](https://app.bugbug.io/sign-up/){.markdown-cta}<-

## Regression Testing process

As with most testing efforts regression testing is an art rather than science. Let us describe the regression testing process that our development team uses. You may then use and adapt it to your existing test strategy. As regression testing is a type of testing that occurs whenever any change in your software occurs, the smartest way to start is to..

### Identify code changes

This will allow you to asses the number of test cases for regression testing that will need to be created. The testing team will need information on the impact that the current changes may have on the whole application. This is done also during this step.

### Prioritize changes

You finished step one with a list. Now you need to create a queue out of it. By proper prioritizing the changes in your software application you reduce the amount of time you will use on 

### Select for re-run

This is the part where the selection of test cases is performed. After the first test is done, test cases are categorized into reusable and obsolete ones. The reusable ones create a regression test suite. The others are will not be used for regression. 

### Categorize manual and automated regression test cases

Here is where the test automation actually begins. Test cases that occur more often within one process are perfect candidates to for the automation tool to do the work for you. If you want to automate tests in a fast and efficient way, try BugBug as your tool for regression testing. You will soon find out that it is great to reduce regression testing costs.

### Prioritize test cases

Here is where regression testing becomes a tad tricky. One would think, that the most often occurring test cases should have the biggest priority, right? Well the theory on that one is a tad different. The need for regression testing, therefor the priority, is the highest for those cases, that cover all essential features. The lowest on the other hand are stated as "technical complexities". 

### Plan and execute

This one is rather obvious, but we have to put it here. After doing all the analysis, you have to plan when regression testing is done and "push the execute button". 

## Regression Testing techniques

Regression testing can include various functional and non-functional tests in its process to thoroughly investigate the software in search of flaws. There are also many techniques with which developers and QA specialists perform regression testing, including:

- Retest All
- Regression Test Selection
- Test Case Prioritization
- Hybrid approach

### Retest All – an accurate yet expensive solution

**The Retest All ** technique is the most accurate and all-around method of regression testing, but it is also the most expensive one. In Retest All, every test case is checked again on the software to confirm if all of its parts are working as intended. That’s all there is to Retest All – it might be simple in definition, but it is definitely the hardest and most costly technique to actually perform.

### Regression Test Selection – when you realize you might not have enough for a Retest All

**Regression Test Selection** has been designed as a solution to the expensive nature of the Retest All technique. In RTS we don’t run all of the tests in our test suite, but instead we divide all of the tests into three categories – reusable test cases, retestable test cases, and obsolete test cases. In addition, RTS might actually create new test cases that haven’t been in the suite before, but might be relevant for regression tests.

If you’re interested in finding out more about regression testing examples, we recommend taking a look at Understanding Regression Testing Techniques by Gaurav Duggal and Bharti Suri. The authors further divide RTS techniques, naming two most important categories – coverage techniques and minimization techniques, and explain what are safe techniques.

**Coverage techniques** feature specific coverage criteria, often provided by the client who wants their software regression tested. These techniques sift through all test cases, selecting only the ones that are relevant for the covered parts of your software. **Minimization techniques** work in a way similar to coverage techniques, but instead of selecting cases based on set criteria, you select a minimum set of cases that you think will do the job.

->[Automate your tests for free](https://app.bugbug.io/sign-up/){.markdown-cta}<-

### Test Case Prioritization – a rational approach

**Test Case Prioritization** aims to maximize the rate of fault detection, creating the most thorough regression test for the lowest cost. First you prioritize the test cases one by one. Then all of them are performed in order, starting from the most important one and finishing on the least important one. Thanks to that, even if your resources aren’t enough to perform all of the tests in your suite, you will still perform the ones that should detect the most flaws.

**TSP** might be performed in two ways – the first one, called **General Prioritization**, selects test cases that should be effective (on average) on all versions of the software you’re working with. The other way, called Version Specific **Prioritization**, only takes into account a specific version of the software, without considering other existing or potential future versions.

### Hybrid approach – the best of both worlds

When choosing one technique of regression testing is impossible – why not combine the best features of Regression Test Selection and Test Case Prioritization? 

The hybrid approach is still being developed, and every researcher gives a personal spin on the idea. Since there isn’t an exact definition of what a hybrid approach actually is, there are tons of versions available – each one with its own upsides and downsides.

## Regression Testing tools

Regression testing ensures that each version of your software runs smoothly. We have described various regression testing techniques, but our article would not be complete without a few hints of what tool to use for proper test execution. 

Regression testing is used on an incremented basis. Each release of your software means building another layer of an already high pile of multiple regression test cases. A good tool will allow you to perform regression tests without writing a single line of code. 

If you want a specific list of tools, please redirect to our article [HERE](https://bugbug.io/blog/guides/testing-tools-for-web-application/).

->[Automate your tests for free](https://app.bugbug.io/sign-up/){.markdown-cta}<-

## Disadvantages of regression testing

Well now that's a shocker isn't it? We have many articles on testing in the BugBug blog, and never have I ever wrote anything negative about testing. Until now. We found four "buts" on regression testing and we tried to tackle them immediately.

- "but regression testing is time consuming" - yes, but only if you only perform a manual test, this is why selecting a good testing tool is essential
- "but you have to do it after even the smallest patch" - yes, but this is why you build a good regression cycle. You store all the recurring tests and only build a few on top of them for that given patch. It is like CI/CD for your testing process
- "but that process might interfere with the sprint planning" - only if you do regression tests without automation. Think of it from a business perspective - would you have a task that goes cross-sprint or a large bug in the production environment?
- "but it is difficult, to write those test cases" - ah, an oldie but a goodie. There are great no-code regression testing tools. Those might be used not only by QA or other technical personnel but by a PM. Such a person would be even able to automate the test or record and recreate it. 

Regression testing is one of the key ways to have your code thoroughly checked and ready to start earning you money!

## Conclusion – is Regression Testing necessary?

As you have already learned, regression testing plays a key role in keeping the quality of your software as high as possible for as long as possible. Updates and changes to your software might introduce new problems, which is exactly why regression testing is so important – to keep user experience high throughout all iterations of your program.

Not all projects require regression testing, of course. In projects that are already finished and won’t be further updated, regression testing is mostly irrelevant. However, if you are planning to continue development further – never forget to properly regression test your software before releasing an update.


Happy (automated) testing!

large_regression_tests_blog_07c72a81f5.jpg

medium_regression_tests_blog_07c72a81f5.jpg

small_regression_tests_blog_07c72a81f5.jpg

thumbnail_regression_tests_blog_07c72a81f5.jpg

regression_tests_blog_07c72a81f5.jpg

What is Regression Testing? Everything You Need To Know

Product Updates

New Feature: Pop-Up Window Recording

Testing Tools Comparisons

Top Cypress Alternatives in 2023

Uncategorized

No-code Testing Automation: Guide to Codeless Testing and Automation Tools

Get more tips and product related content. Zero spam.

Don't miss any updates

Automate web app testing easier than ever. Without excessive costs. Faster than coding. Free forever.

End-to-end testing made simple! BugBug is a new, more intuitive tool for test automation of web apps. Get all the benefits of test automation without hiring new people or infrastructure changes.

BugBug.io | Intuitive and reliable test automation for startups

large_opengraph_image.png

medium_opengraph_image.png

small_opengraph_image.png

thumbnail_opengraph_image.png

opengraph_image.png

Features

Why Choose Us

Integrations

Book a demo

Customer Success Stories

Case studies

Create tests in your browser for free. Run the tests in the cloud for 14 days at no charge. No credit card required. 

Effortless cloud testing that fits your budget

Discover the differences of each plan and choose the offer that fits your testing needs

Industry leaders automate tests with BugBug

Frequently Asked Questions

Contact our Support Team.
They’ll give you all the information you need.

Free trial of premium features for 14 days

Automate web app testing easier than ever. Without excessive costs. Faster than coding. 

Pricing

Blog about test automation

About us

Contact

Everything you need for the entire test automation workflow

Record new tests simply by clicking in your browser. Automatically capture every click or text input. Add assertions directly while recording. Forget about writing selectors by hand; BugBug will do it for you.

Create tests without coding

Enjoy excellent performance when you run tests locally, allowing you to work more efficiently. Run tests in the cloud to regularly monitor your web application and catch bugs earlier.

Run tests in your own browser or in the cloud

Now you don't need to record your test from scratch every time you want to change it! Record new steps right in the middle of your test, just where you need it. Spend less time on manual writing of CSS or XPath Selectors.

Fix tests by re-recording

Run cloud tests every day and monitor if your app works properly. Get comprehensive notifications when a test fails. You will receive notifications with scheduled test results directly to your inbox.

Schedule test automation

Use BugBug API and Command Line Interface to run regression tests along with your CI/CD build pipelines, such as Github, Gitlab, Travis, Jenkins, or Bitbucket. Integrate with Zapier to get notifications on Slack when a test fails.

Integrate with your CI/CD pipelines

Invite your teammates to work together on test automation without time-consuming training and debates. You can involve software developers or product owners in testing within minutes.

Collaborate with your team

Unparalleled simplicity powered by unique features

Powerful features for advanced testing

You can turn test steps into "Components" that are shared across multiple tests. Maintain hundreds of tests without worrying that you will have to spend half a day fixing them. 

Edit multiple tests at once

Work in multiple environments. Create random numbers. Register users with unique email addresses. 

Use variables to handle complex tasks

You have full control over the test steps. You can update selectors, manually add advanced step types or assertions, and even custom JavaScript code.

Customise test steps with JavaScript

Need to show the results to your stakeholders? You can download a PDF report for each suite run in your project.

Generate PDF reports

Record automated tests easily for both desktop and mobile resolutions.

Build tests for mobile resolutions

See what our clients are saying

Start testing for free! No credit card required.

Test automation is too complicated. BugBug is here to make end-to-end testing ultra simple and effective. Limit your testing workload and focus on delivering great features. Without straining your budget.

Because we want you to catch bugs easier than before

Join hundreds of happy software testers that automated testing process with BugBug

We solve many problems of the test automation world

**TRULY NO-CODE SOLUTION**  
Not a tester? No problem. Our no-code BugBug test recorder will make you feel like a pro QA engineer.

**CLEAN INTERFACE**  
We are fed up with complicated testing tools. That's why we created BugBug. Our software is simple and very intuitive. No matter if you are a QA testing beginner or an experienced engineer.

Built to empower non-technical testers

**UNLIMITED FREE USERS**  
Collaborate with others when creating your test suites. Finally you can invite as many teammates as you want for free.

**UNLIMITED LOCAL TESTS**  
With BugBug you can run unlimited tests in your browser with the Free Forever Plan. Also you may run tests in the cloud without additional infrastructure or DevOps costs. Fun right?

Test automation tool that suits every pocket

**BUILT-IN AND CUSTOM VARIABLES**  
Anytime you need, you can run tests with our built-in variables that we prepared for you. If you need something more specific, you can also create your own custom variables.

**POWERFUL FEATURES**  
Run custom JavaScript, use automatic selectors, create profiles as presets for different variable values, and work with different development environments - BugBug offers advanced features for you to use when you need it.



Advanced features to boost your testing

Integrate with anything via BugBug API. Run cloud tests in your CI/CD pipelines such as Github, Travis, Jenkins, Bitbucket, or GitLab. The setup is easy thanks to BugBug's streamlined Command Line Interface and built-in Zapier support.

Are you fresh in the test automation world? We’ve got your back - read our comprehensive tutorial for beginners and see how easy it is to automate your first test.