Software Testing Knowledge

Test Automation Expert and Test Architect. He is a seasoned tester who can’t stay long in one place. He is looking for perfection and a place to challenge his skills.  He gathered experience working for different companies with different working models, From small to big corporations, From Products via In-house development to software house. Thanks to that he has a wide perspective on testing quality and delivering value.\

[linkedin.com/in/wyrodek/](https://linkedin.com/in/wyrodek/){ref="nofollow"}

large_https://i0.wp.com/wyrodek.pl/wp-content/uploads/2017/01/T1A0194.jpg

medium_https://i0.wp.com/wyrodek.pl/wp-content/uploads/2017/01/T1A0194.jpg

small_https://i0.wp.com/wyrodek.pl/wp-content/uploads/2017/01/T1A0194.jpg

thumbnail_https://i0.wp.com/wyrodek.pl/wp-content/uploads/2017/01/T1A0194.jpg

https://i0.wp.com/wyrodek.pl/wp-content/uploads/2017/01/T1A0194.jpg

Maciej Wyrodek

Test automation, especially end-to-end, is currently enjoying a revival. A lot of new tools and approaches have emerged over the last few years. Selenium has finally launched version 4, Cypress and Playwright have brought coded automation to a new level and tools like [Bugbug.io](https://bugbug.io/), [Mabl](https://www.mabl.com/), [Testim.io](https://www.testim.io/), have revitalized the idea of record and play automation. Since a lot is happening, let's take a look at automation and compare the strengths and weaknesses of 3 approaches to Automation: Custom Made Frameworks, Out of the box framework and Record and play. We will look at the pros and cons of all the solutions and when they work best. However, first we need to discuss the reasons for using test automation.

Why we automate tests
-----------------------------------

If you ask typical testers why they automate tests, they can provide many reasons: from shortening the Regression time, improvement of test coverage, to providing faster feedback. Ultimately, if we look hard enough, all goes back to the one and only goal of companies, as stated by [Eliyahu M. Goldratt in The Goal](https://en.wikipedia.org/wiki/The_Goal_(novel)) - Making money. Testing by itself often is considered a cost - as it does not bring more money. But let's not forget that more money can be made not only by selling more - but by increasing the margin - either by making the product worth the price or by reducing the cost. Testing can help to protect the bottom line, by reducing the cost of mistakes and finding them more easily and faster. This also leads to reducing the cost of customer support, refunds, etc. But as [DevOps](https://itrevolution.com/the-devops-handbook/) and [Modern Testing](https://www.moderntesting.org/) state - testing can synergize with a great experimentation culture, leading to possible delivery of a quality product in a shorter time and cost-savings on the development cycles. But this cannot be achieved without robust test automation. And automation needs to follow a few rules. For that, we need to consider the properties of great automation.

Traits of good automation
-------------------------

For automation to be useful in the DevOps world, it needs to be: 

-   Reliable 

-   Fast

-   Easy to write and maintain new test 

-   Easy to run

### Reliability

If test automation is not reliable, we won`t believe its results. As a result, we will still depend heavily on manual rechecking of everything and that is time-consuming. Or even worse, it can lead to wasting time as now no one will even bother to check the results.

### Fast

The development cycle is fast. If developers need to wait for the results of their work more than a coffee break, they will switch to the next task. Now the results will need to wait which leads to the cost of context switching. And longer [lead time of a task](https://kanbantool.com/kanban-library/analytics-and-metrics/kanban-definition-of-lead-time-and-cycle-time).

### Ease of Writing and Maintaining

If automation should be a regular part of a software development lifecycle, their benefit must justify the cost. If writing a test takes 4 days, it'd better be worth it - especially if developing  the tested feature takes less than that. The same holds true for maintenance: we need to be able to easily understand why a test has failed and whether this is a code issue or a test issue and how to fix it.

### Ease of running

This relates to being fast: anybody who needs to use a test to check a code should be able to do it without extra hoops, for example automatically running them locally or via CI/CD.

How to choose the best automation approach
------------------------------------------

There are a lot of ways of choosing tools for a project. When looking for tools that resolve a problem, we need to choose a tool that contributes the coveted value. The list of questions requiring answers is long, but today we will look at only one aspect, namely automation development and maintenance cost. We will look at the 3 most popular approaches, starting from ones requiring most coding and maintenance effort, like "Custom Frameworks", through usage of "Out of the box" Frameworks and finishing with Record and Play tools. We will look at the pros and cons. As it is my belief, all these approaches have made a name for themselves on the market and suit different project needs.

**Pros**

-   Great Customizability

-   Ownership

-   "On-site support"

**Cons**

-   Huge Maintenance Cost

-   Time Consuming

-   Requires lots of skill to Maintain

When I was starting test automation, this was the way to go. Mainly because Selenium Web Driver was the tool of choice back then. And as its description stated, [Selenium automates browsers](https://web.archive.org/web/20120208115222/http://seleniumhq.org/). Selenium was just a tool that did exactly what it said it delivered.

It didn't provide any functions for running test, or making assertions. 

At that time, practically every company had to build its own framework.

Often it was the role of test automation testers. In most cases, they were just learning how to automate. Making lots of mistakes in the process. But some companies managed to make great frameworks and, as a result, decided to share them with others. Some of them are still in use and developed but other have not stood the test of time. [Ocaramba](https://github.com/ObjectivityLtd/Ocaramba),[Selino](https://github.com/TestStack/TestStack.Seleno), [Serenity](https://serenity-bdd.github.io/theserenitybook/latest/index.html),[Selendie](https://selenide.org/), [Protractor](https://www.protractortest.org/#/)

This approach is still used today. Why?

### Pros

The main advantage of this approach is control. It gives a company the greatest level of control in customization. If a specific feature for a project is required, it is relatively easy to find someone who can develop this functionality.

Additionally, it gives a company considerable independence; there is no risk that at one day, the tool support will end or [repo will be deleted](https://nakedsecurity.sophos.com/2022/01/11/javascript-developer-destroys-own-projects-in-supply-chain-lesson/), leading to massive issues.

What is more, there is someone who knows how the tool works on site and can teach new team members.

### Cons

The main drawback of this solution is the cost - maintaining internal test automation framework costs time and money. Many companies underestimate the cost of developing test automation.

Time is of essence here: this automation needs a long time to be developed,  from creating a framework to writing a test. And this is the aspect that can be most damning. As discussed before, for automation to bring value, it needs to provide fast feedback.

Creating a custom framework is as big a project as any and it requires a good dose of skills, ideas, and planning. This means a lot of expertise while typically, companies are not willing to spend precious resources on test automation.

Out of the box frameworks: Pros & Cons
--------------------------------------

**Pros**

-   Shorter time needed to write first test

-   Much less need for framework maintenance

-   Lots of available online trainings and knowledge

**Cons**

-   Often only limited support from community

-   Writing and Test Maintenance takes time

-   Huge dependence on framework developers
As mentioned before, the growth of Selenium was accompanied by a lot of frameworks created on its base. Over time, the issues with the web driver became increasingly prominent, leading to the re-emergence of new automation frameworks not based on Selenium: [Cypress](https://www.cypress.io/), [Puppeter](https://github.com/puppeteer/puppeteer), [Playwright](https://playwright.dev/), to name a few most recent ones.

All these tools had one thing in common - they provided the user with out-of-the-box, ready options to write automation. No matter if this was Ocarmaba based on Selenium or Cypress: the goal of the solution was the minimalism needed for framework maintenance.

These tools usually provide built-in test runners, an extensive assertions library, boiler plates for starting writing project and tests, and - most importantly- a built-in mechanism to deal with test flakiness and general instability.

Additionally, these frameworks usually provide their own philosophy of writing test automation. As long as what we need to do is with accordance to their approach, everything goes well. Problems start when we need to go beyond what is suggested.

Until recently, the best example has been a test covering multiple domains in Cypress, but as of [version 9.6](https://docs.cypress.io/guides/references/changelog#9-6-0), it is no longer such a grievous case.

In another scenario, we need to rely heavily on framework developers, if they introduce breakthrough changes. Usually, we are left with two options: update and fix the code. Alternatively, don't update and hope that there won't be any issues in the framework that need fixing.

### Pros

Compared to the custom frameworks, we can start writing tests almost immediately without the need of spending weeks or months on writing the toolset first. Similarly, we don't need to wait whenever the operating system or browsers have an update. The tool maintainer will update the framework for us. And since most tools in this category are open-source ones, there is lots of information easily available online.

### Cons

As mentioned before, these tools have their own test philosophy, which is much more restrictive. Quite often support is limited - these tools are usually open-source and support in case of issues can be unavailable where we need to resort to forums and chats where we hope someone will answer a question; sometimes we get the answer in minutes but it may never come just as well.

However, the greatest risk of all is that we are at the mercy of framework developers - these tools become a part of our code base and if we want to keep support, sooner all later we will need to update even if we don't want to. A case in point is Cypress 10 - [it introduced breakthrough changes](https://docs.cypress.io/guides/references/changelog#10-0-0). As a result, many plugins and customer functionalities won't work without extensive updating.

Record and Play Testing Tools: Pros & Cons
------------------------------------------

**Pros**

-   Fast Test Writing Time

-   Low Entry Level

-   Professional Customer Support

-   Easy Scalability

**Cons**

-   Usually cloud-based, requires access to test environments

-   Still young, "untested" technology

-   Requires a new mindset

-   Data management can be quite complicated

-   Usually a commercial paid tool


The idea of record and play automation is not new, even Selenium has Selenium IDE which, after a few years on developments lately, has won some users' recognition.

Microsoft had Coded UI Test as part of the Visual Studio from 2010 and only in [2019 announced it deprecation](https://devblogs.microsoft.com/devops/changes-to-coded-ui-test-in-visual-studio-2019/)

Back then this technology was long before its time, the frameworks were very unstable and the maintenance of this test was problematic.

But in the last 4-5 years, a lot has changed. Solutions like [Bugbug.io](https://bugbug.io/), [Mabl](https://www.mabl.com/), [Testim](https://www.testim.io/) have taken over what modern technology has to offer and reinvented the concept.

### Pros

Most coded automation requires great investment of time into writing and maintaining test - usually even a trivial change costs around half an hour. In case of Automation Records and Play, test are usually meant to be disposed of  after they  stop working, as it is much easier to re-record a test instead of fixing it. This is of special importance for startups and projects with a lot of changes in the UI layer. When a team is spread thin, it is hard to find the time to write a test. But usually, you can record a few tests while checking if what has been done even works.

On the other hand,e out-of-the-box coded frameworks offer usually an easier learning curve. Record And Play tools usually do not require coding knowledge - although in more complicated cases some JavaScript knowledge can prove very helpful.

Since most of the record and play tools are paid solutions, it also means that companies have professional Trainers and Customer Support to help when issues arise. Usually, it means that in the case of young projects, we can even influence their feature roadmap.

These modern record and play solutions come with their own cloud runners, making scalability of test automation unbelievingly easy. Typically, paralyzing a test takes a few clicks.

### Cons

Unfortunately, Cloud Running is also a huge Con. First and foremost, we need to give this tool access to our test environments. 

An additional issue related to the Cloud is Term of Service - not all projects and customers can afford cloud services.

Most of these Record & Play tools are relatively young and not battle-tested. For long-term projects, this uncertainty can be very risky.

In my opinion, the biggest issue is Data Management. So far, no Record and Play solution has provided an easy way to manage test data. And this can be a huge blocker. Consequently, we need to change our approach to E2E test automation, with a desire to cover all the important and easy paths as fast as possible, to make this automation possibly most valuable.

Lastly, there is the price - cloud solutions are not cheap and if we need to run a lot of tests, it may be quite expensive. But some tools like [BugBug](https://bugbug.io) are on the other hand very cheap and offer unlimited cloud runs for a fixed mohtly fee. Arguably, it is still less expensive than maintaining coded automation and its infrastructure.

No clear-cut distinction
------------------------

Before the final thought, some clarification is required:

As much as we humans like clear categorizing, the real-word borders are much blurred.

For example: should Playwright be in the Custom frameworks or in the Out-of-the-box category? After all, its strength is the ability to use its own test runners, reporting services and easy building of the new features we need.

Similarly, it has the Record and Play functionally and so did Cypress (at the moment of writing this post, the latest version of Cypress is 10.3 and since 10 Cypress Studio, which allowed test recording, has been deactivated).

There are also tools that I like to refer to as combine-harvesters: basically, they do everything and claim that they can test anything, from mobile via web to desktop and many others -- the examples include TOSCA and Test Complete.

That's why we need to look at the numerous different dimensions of automation - and how much coding is just one of them.

Summary
-------

It is my belief that over time, both no-code development and record and play automation become the dominant players on the market. Yet there is a lot of room for improvement. All the types of approaches to automation, starting from custom frameworks, or using out-of-the-box coded solutions ending on Record and Play tools, have their uses on the market.

They answer to different needs and solve different problems. So, before selecting any tool, it is always good to ask yourselves: "What are we trying to gain?"

large_image (3).jpg

medium_image (3).jpg

small_image (3).jpg

thumbnail_image (3).jpg

image (3).jpg

Record and Play vs Coded Automation testing - how to choose what works for your team

record-and-play-vs-coded-automation-testing

Testing can be difficult, it is hard to master, there are many questions and many issues we can face with each new feature, each new story. Every day, testers need to decide what to test, how to test, what to automate, and what risks to accept. These decisions are hard and require a lot of experience. Fortunately, there are techniques, heuristics, and models which can help with planning tests. One of the oldest and still very useful techniques is **Agile Testing Quadrants** (**ATQ**). it is a simple table that helps us select the tools we need to use, depending on the required outcome. In this article, we will look into the history of Agile Testing Quadrants, their uses, and limitations. We will also discuss the attempts at modernizing it.

![Agile-Testing-Quadrants](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/Agile_Testing_Quadrants_4b2d044860.png)

[TABLE_OF_CONTENTS]


## Agile Testing Quadrants or Agile Testing Matrix?

First, we need to clear one thing: as with everything in IT, even this simple table needs to have many names! Chances are that you will come across it as an Agile Matrix, Testing Matrix, Testing Quadrants, or Agile quadrants. All of them refer to the same thing. The only exception is the Agile Testing Matrix which may refer to both Agile Testing Quadrants and its precursor the Named Agile Testing Matrix. 

![Testing matrix](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/test_matrix_00ac277513.jpg)


## The Story of Matrix

When speaking about **Testing in Agile**, there is probably not a single more important person to its development than [Lisa Crispin](https://lisacrispin.com/) and [Janet Gregory](https://janetgregory.ca/). Their book, [Agile Testing](https://www.goodreads.com/book/show/5341009-agile-testing), has helped many testers. [Testing Pyramid ](https://bugbug.io/blog/software-testing/testing-pyramid-a-successful-approach-to-test-automation/) was born while [Mike Cohn](https://twitter.com/mikewcohn) tried to explain the nuances of automation to Lisa’s team (*Experience of Test Automation - chapter 1 Section 1.3.*) Agile Testing Book played a key role in the popularization of Agile Testing Quadrants, where the creation of ATQ is often misattributed to this wonderful duo. As Lisa Crispin explained in [her blog](https://lisacrispin.com/2011/11/08/using-the-agile-testing-quadrants/) - they improved on the older concept by [Brian Marick](http://www.exampler.com/old-blog/2003/08/22/#agile-testing-project-2) - an Agile Testing Matrix created as part of his blog series presenting his vision of testing in Agile. So Agile Testing Quadrants have almost 20- year-long history and they are still one of the best models used for testing.

## The Usefulness of Agile Testing Quadrants

Agile Testing Quadrants have two axes which split the matrix into 4 quadrants. Both axes pose the same questions, namely **What is the goal of your testing?** The major difference between the two axes is the flavor of the question: the horizontal axis asks what is currently more important: helping the team to deliver faster or ensuring that the product is as good as possible? the vertical axis asks if we are concentrating on the aspects of technology, ensuring that everything is working well, or are we looking from the perspective of business and customers, making sure that our program is worth using? Obtaining both answers points to one of the Quadrants with a list of useful examples. This can be a little confusing, so let’s look at some of the examples.

### Example 1: E-commerce shop

Let’s say we have e-commerce shop development that is going nicely. However, there is one enormous risk on the horizon, namely quite a lot of features, each tested separately so no one is sure how they will work in the real world. The team believes that the standard paths will work. However, no one is sure what kind of edge cases will come from the real world.

Looking at our matrix we can see that problem is definitely business-facing - we don’t know what the user will do. We also need to make sure that our product will handle it, ergo - Critique Product.
Looking at it we land in Q3  where we have propositions of the following techniques:
 - Exploratory Testing
 - Scenarios
 - Usability Testing
 - User Acceptance Testing (UAT)
 - Alpha/Beta Testing.

So our goal is to find edge cases. Looking at the list, all the techniques could prove helpful. But some may require too much investment so late in the developments (Scenarios). Some are a poor fit for an e-commerce shop (alpha/beta testing, UAT). Which leaves us with Usability Testing and Exploratory testing.
ATQ is not an extensive matrix, it contains only examples of techniques rather than a complete list. For example, a different technique that could help is [Dogfooding](https://en.wikipedia.org/wiki/Eating_your_own_dog_food) where company employees themselves use a product to satisfy their everyday needs.

### Example 2: Messaging App

A company has made a very successful messaging app, but now they are facing the problem of success. Their small app, designed for a small community, is gaining popularity fast. The team feels that the app is not ready for all this attention. The situation is clear - from a business perspective, everything works well enough. The risk is technology, and again the problem lies in the critique of a product. This leads us to Q4 where we have, among others, *Performance* and *Security* testing. There is no doubt that we will need both. Lots of users mean a load on an application and nothing will kill its momentum like failing servers. Security is also important. Attention also attracts bad actors who will try to exploit the system.

### Example 3: Early Development.

The team suffers a problem of poor quality code. In the case of some features, developers are reluctant to fix a familiar bug because of the high risk of breaking something else. One glance at the ATQ shows that we need something to support the development and its technology which leads us straight to Q1. Both Unit Test and Component Test would improve team productivity and code quality.

The above example may give a wrong impression that Agile Testing Quadrants are used to select one and only one tool. **THIS IS NOT TRUE**: we can use ATQ for each goal of our testing, for each need, for each problem we are trying to address. However, trying to improve too much at once may lead to too much-wasted effort and no result. Concentrate on one subject at a time.

## Evolution

While the concept of AQT has changed little over the years, the techniques in each quadrant have evolved, for example: 

![Updated Agile Testing Quadrants](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/updated_atq_06346e83b7.jpg)

In [More Agile Testing](https://www.informit.com/articles/article.aspx?p=2253544), the author presented a new version with Micheal Hüttermann changes, with a circle of *Outside-in, Barrier-free, Collaborative in the middle of the quadrants*. The author believes that it will stretch beyond any quadrants. The main issue with this update is that it is a catchphrase and provides no information about the tools and techniques that may help.

![Agile Testing Quadrants next iteration](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/Agile_Testing_Quadrants_next_d74569f53c.jpg)

A few years back, [Lisa and Janet](https://agiletester.ca/applying-the-agile-testing-quadrants-to-continuous-delivery-and-devops-culture-part-1-working-towards-continuous-delivery/) also presented a version more in line with DevOps. This version adds a lot more techniques to each of the quadrants but also changes one side, from *Supporting Team to Guide Development*. While it may seem like a single name change, fortunately, the original name leads to many misunderstandings and the new label is much more descriptive. 

![RST Agile Testing Ecosystem 1.1](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/bb_ATQ_1fb44a3b67.png)


Michael Bolton and James Bach have also made the [above version](https://www.slideshare.net/HcmcStc/the-new-agile-testing-quadrants-bringing-skilled-testers-and-developers-together-james-bach) which is compatible with Rapide Software Testing. Most of these updates are only fine-tuning, adding elements that their author thinks are better in this context. So which one to choose? It is the author’s opinion that the original version holds well? It is still relevant, because:

## It is just a model.

“All models are wrong, but some are useful” is an old aphorism attributed to George Box. And it is important to bear it in mind in the case of Test Pyramid, Agile Testing Quadrants, etc. Models are what they are: models which need to help us think about a problem. **They are not the golden rules that should be followed in the letter**. They provide us with a good way of tackling the problem. But it is up to the user to decide if a specific model will work for a specific problem.

## Summary

The **Agile Testing Quadrant** is a model which is nearly 20 years old. Owing to its simplicity, it is still relevant. It is a great tool for teams and testers who have a problem with choosing the right testing method. For junior testers, it is a great place to look into different tools and approaches, and for more advanced testers it will be an excellent aid in building or improving the testing process.

large_beetle-g2d0bd3bb5_1280.jpg

medium_beetle-g2d0bd3bb5_1280.jpg

small_beetle-g2d0bd3bb5_1280.jpg

thumbnail_beetle-g2d0bd3bb5_1280.jpg

beetle-g2d0bd3bb5_1280.jpg

Agile Testing Quadrants

agile-testing-quadrants

A self-taught software QA Engineer working at an IT start-up, Dominika has experience in
web testing. Currently, she is focusing on developing her test automation skills with Python.
Apart from programming, she likes to make pottery and cook vegetarian delicacies.\

[linkedin.com/in/dominika-nykiel](https://linkedin.com/in/dominika-nykiel){rel="nofollow"}

medium_https://media-exp1.licdn.com/dms/image/C5603AQH6NBuKQg3UWw/profile-displayphoto-shrink_800_800/0/1637439945009?e=1646265600&v=beta&t=IwBmGAWmNdXvprRNzsLB-PbwuPFXBTKFfTLP612yEuA

small_https://media-exp1.licdn.com/dms/image/C5603AQH6NBuKQg3UWw/profile-displayphoto-shrink_800_800/0/1637439945009?e=1646265600&v=beta&t=IwBmGAWmNdXvprRNzsLB-PbwuPFXBTKFfTLP612yEuA

thumbnail_https://media-exp1.licdn.com/dms/image/C5603AQH6NBuKQg3UWw/profile-displayphoto-shrink_800_800/0/1637439945009?e=1646265600&v=beta&t=IwBmGAWmNdXvprRNzsLB-PbwuPFXBTKFfTLP612yEuA

https://media-exp1.licdn.com/dms/image/C5603AQH6NBuKQg3UWw/profile-displayphoto-shrink_800_800/0/1637439945009?e=1646265600&v=beta&t=IwBmGAWmNdXvprRNzsLB-PbwuPFXBTKFfTLP612yEuA

Dominika Nykiel

An aspiring technical writer with an interest in testing and a knack for learning languages.
Former journalist, translator, and editor, whose work was published in several magazines. She
speaks French and Norwegian and is looking forward to learning Spanish.\

[linkedin.com/in/aleksandra-spilkowska](https://linkedin.com/in/aleksandra-spilkowska){rel="nofollow"}

large_a_spilkowska_cropped.jpg

medium_a_spilkowska_cropped.jpg

small_a_spilkowska_cropped.jpg

thumbnail_a_spilkowska_cropped.jpg

a_spilkowska_cropped.jpg

Aleksandra Spilkowska

Ask any QA Engineer worth their salt about their Chrome extensions - they probably have a bunch of them ready to go at all times. And for a good reason! They can come in handy when you need to edit a screenshot, clear a browser cache, or test a website’s performance. Saving some clicks and double-checking for hidden issues means that a QA’s everyday life is much easier, and it leaves more time to focus on spotting all the bugs before the final product is released. Get ready to pimp your browser with some of the most useful Chrome extensions for testers!

[TABLE_OF_CONTENTS]

## [Monosnap](https://chrome.google.com/webstore/detail/monosnap/lhgadannddhdpkfhfjjgcpdjhpmjhkfm){rel="nofollow"}

![Monosnap](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/monosnap_a19cbb03a1.png)

A reliable recorder is a must-have in your virtual toolbelt. While most recorders are programs, Monosnap is an extension so there are extra advantages, especially if you tend to switch between devices a lot. You can capture a GIF or a video, a selected area of the screen, and of course a fullscreen. There is also a set of basic editing features such as drawing, creating marks, highlighting details, or hiding sensitive data. No watermark in sight! Monosnap works with both Windows and iOS.

## [Clear cache](https://chrome.google.com/webstore/detail/clear-cache/cppjkneekbjaeellbfkmgnhonkkjfpdn?hl=en){rel="nofollow"}

![Clear Cache](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/clearcache_ec20bd645f.png)

It does just what the name promises: clears your cache and browsing data. What's the catch? There isn't any! You can do it all with a single click. This extension allows you to choose how much data you want to clear and from which domains you want the cookies removed. Neat!

## [Bug Magnet](https://chrome.google.com/webstore/detail/bug-magnet/efhedldbjahpgjcneebmbolkalbhckfi){rel="nofollow"}

![Bug Magnet](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/bugmagnet_115b949485.png)

For many, this is a go-to extension for exploratory testing. Will it do all the work for you? Unfortunately, the answer is no, but it sure helps a lot. This exploratory testing assistant will offer you access to common boundaries and edge cases. It works on input fields, text areas, content editable DIVs, and more. This means that you don’t need to rack your brain to think of an email, postcode, or address anymore, Following selection of the value, Bug Magnet fills in the field you need. It’s an open-source solution and you can customize it to your liking.

## [PerfectPixel](https://chrome.google.com/webstore/detail/perfectpixel-by-welldonec/dkaagdgjmgdmbnecmcefdhjekcoceebi){rel="nofollow"}

![Perfect Pixel](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/perfectpixel_6d7ba283bd.png)

This extension is very convenient for frontend testing, especially when there is a need to compare a website design with frontend changes. It adds a semi-transparent overlay over the top of the developed HTML to make it perfect - down to the last pixel. The position, size, and opacity level are no longer your enemies with this handy extension.

## [Web Developer](https://chrome.google.com/webstore/detail/web-developer/bfbameneiokkgbdmiekhjnmfkcnldhhm){rel="nofollow"}

![WebDeveloper](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/webdeveloper_827b9ef3f7.png)

The Swiss Army knife of web developing tools, this extension will equip your browser with a toolbar full of functional items. Cookies? Managed. Information? Anchored. Web elements? Highlighted. CSS? Edited. Forms? Populated. Of course, some tools work better than others, and you aren’t going to use all of them equally, just like with a real Swiss Army knife (fish scaler, anyone?) but it’s still a quite useful quality multitool.

## [Lighthouse](https://chrome.google.com/webstore/detail/lighthouse/blipmdconlkpinefehnmjammfjpmpbjk?hl=en){rel="nofollow"}

![LightHouse](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/lighthouse_f51d97d7ce.png)

Yet another multi-use extension, it runs overall non-functional tests on websites. The metrics you can audit are performance, SEO, best practices, accessibility, PWA (Progressive Web Application). Lighthouse auto-generates a clear, simple report which lets you know how well your website performs for an average user. The audit will instantly highlight the areas that need improvement and will give you a general overview of the website’s functionality.  

## [Postman Interceptor](https://chrome.google.com/webstore/detail/postman-interceptor/aicmkgpgakddgnaphhhpliifpcfhicfo?hl=en){rel="nofollow"}

![Postman Interceptor](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/postman_interceptor_a0d0b62ce1.png)

Assuming you have heard of Postman (if you haven’t, [go check it out now](https://www.postman.com/){rel="nofollow"}!), you might be interested in further improving the workload with this practical extension. The Interceptor sends requests which use browser cookies to the Postman app without using a proxy. This is an advantage for testing APIs, as Interceptor makes it possible to send headers (typically restricted by Chrome).

## [SelectorsHub](https://chrome.google.com/webstore/detail/selectorshub-xpath-plugin/ndgimibanhlabgdgjcpbbndiehljcpfh){rel="nofollow"}

![SelectorsHub](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/selectors_hub_624439475b.png)

Whether you’re a seasoned tester or just starting your journey, this extension can be a worthy addition to your collection. SelectorsHub is easily the top-rated XPath & cssSelector plugin available now, and it lives up to the hype. It helps write and verify XPath and cssSelector in less than 5 seconds, auto-suggests attributes, text, etc. SelectorsHub not only supports the iframe and the SVG elements, it is also the only extension supporting shadowDOM (at least for now). And if you value a proper error message, you’ll get one for missing elements in the selectors. This impressive extension is indispensable not just in test automation but also to users of codeless, smart testing tools.

## Summary

That’s it! We hope you have enjoyed our pick. Of course, this list is in no way comprehensive, as new extensions are added to the Chrome web store all the time. While the choice might initially seem overwhelming, it is highly recommended to checkout various extensions and find the ones that suit your needs best. Hopefully, this article will be a good starting point for these explorations!


large_leaf-gfcaac0297_1280.jpg

medium_leaf-gfcaac0297_1280.jpg

small_leaf-gfcaac0297_1280.jpg

thumbnail_leaf-gfcaac0297_1280.jpg

leaf-gfcaac0297_1280.jpg

Best Chrome Extensions for Software Testers

best-chrome-extensions-for-software-testers

Professional Quality Evangelist, experienced in testing web and mobile applications.\

[linkedin.com/in/kinga-witko-b5bba659/](https://www.linkedin.com/in/kinga-witko-b5bba659/){ref="nofollow"}

thumbnail_https://pbs.twimg.com/profile_images/1410220490434744328/eqEl_DXL_400x400.jpg

https://pbs.twimg.com/profile_images/1410220490434744328/eqEl_DXL_400x400.jpg

Kinga Witko

As recent data show (https://gs.statcounter.com/browser-market-share), Chrome is the most used browser in the world, representing over 64% of the browser market.

The Chrome Developer Tools (DevTools for short) are a set of web authoring and debugging tools built into Google Chrome. The DevTools provide web developers with improved access to the internals of the browser and their web application. Did you know that they are quite useful for the Testers as well?

[TABLE_OF_CONTENTS]

## Give it a try in testing the web

There are multiple functions in which Dev Tools may support your testing, not only limited to:
- resizing (change user agents)
- screen rotation
- using Console: Debug JS – CSS, Logs
- network manipulation: Throttling, Performance
- security testing
- location change
- looking for elements (id)
- taking screenshots
- clearing cache
- test recording
- database checking (IndexedDB  - a transactional database system, like the SQL-based RDBMS)

## How to enter?
To open the developer console in Google Chrome, open the Chrome Menu in the upper-right-hand corner of the browser window and select **More Tools > Developer Tools**. 

![https://i.imgur.com/fIMFZgz.png](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/f_IMF_Zgz_bea99d61c0.png)

You can also take shortcuts:

**On Mac:** \
Command+Option+I 

**On Windows/Linux:** \
F12 or Control+Shift+ I


You can dock your DevTools window to the right of your viewport or at the bottom of the screen, depending on your preferences. By default, DevTools are docked to the right of your viewport. You can also dock to bottom, dock to left, or undock to DevTools to a separate bottom.

![https://i.imgur.com/13JgWaO.png](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/13_Jg_Wa_O_2e4ec2b59e.png)

It is possible to reorder the panels so they suit your needs best. Click and drag a panel tab to change its ordering. Your custom tab order persists across the DevTools sessions.


## How to start
Chrome DevTools can be used in web and mobile testing. Every Tester should take advantage of it because these tools are free and easily accessible. 
Here is a list of the most useful ideas for testing purposes. Maybe you'll come up with new ones?

### Resizing and Rotating
It is important to check the UI of your website on mobile browsers and multiple screen resolutions. For some people, it’s “Just the UI” testing but do remember that nowadays a lion’s share of the market (54.37%) belongs to mobile browsers ([stats here](https://gs.statcounter.com/platform-market-share/desktop-mobile-tablet)).

![Browser market stats](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/image7_6614995465.png)

Use the **Responsive** option in Chrome DevTools to make sure that a majority of different screen resolutions is covered.

You can select the desired resolution from the dropdown list of many different mobile devices to test against different screen sizes. DevTools tell us the screen dimensions of the device in the top menu as well. What is more, you are also able to rotate the screen. Screen rotation is a common cause of mobile bugs, as the mobile website layout changes when the screen is rotated, and page elements may need to reload. While it won't pretend to be the real device, it will give you a lot of hints on what other tests should be performed on your application.

![DevTools resizing and rotating](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/image6_02544f7f6a.png)

### Console - debug

Either a developer or a tester can debug a website on a mobile device.
In the JavaScript **Console** you can find all ranges of errors in the application.
As the complexity of JavaScript applications increases, developers need powerful debugging tools to help quickly discover the cause of an issue and fix it efficiently. The Chrome DevTools include several useful tools to make debugging JavaScript less painful.

![DevTools console](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/image9_aee0e7d98e.png)

### Network and Throttling
The **Network** option allows you to observe the order and time of loading elements. You are provided with insights into resources that are requested and downloaded over the network in real-time. It is of special importance in mobile testing as users expect mobile apps to work smoothly and fast.
1. Open DevTools (F12)
2. Click the ‘Network’ tab
3. Click the No Throttling dropdown
4. Select the type of connection you want to imitate
5. Reload the page to see the assets downloading at that connection speed.

The Network tab in Chrome DevTools has an option to faux throttle your network, so you can experience what your users might see visiting your website on 3G, Offline, or with your real speed connection. The **Throttling** option is also useful for visualizing how fonts are loaded. The lower the speed, the more end-user frustration on your website.

In this section, in the left upper corner, there is also a record button that allows you to preserve all the activities (logs) and compare them with other test results.

![DevTools network throttling](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/image8_9e4f1f589f.png)

### Elements
The Elements panel allows you to see everything in one DOM tree and fosters inspection and on-the-fly editing of DOM elements. It is also a useful option for automating tests.
You can always access this option for a specific element on the website:
1. Mouse-over the element (word, button, etc.)
2. Right-click on the mouse
3. Click Inspect

![DevTools Elements](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/image12_3c7cc49608.png)

In the Chrome DevTools panel, you’ll be directed to the very element in the DOM structure.
The **Elements** tab is extremely useful if you need to play around with CSS in your application. Sometimes you need to switch on and off some buttons/lists/checkboxes. It is possible in this section to select the desired elements in the DOM structure and edit them in the Styles sub-section. 

There is also a handy way of locating a node element in the DOM using a CSS selector. From the elements panel, use a keyboard shortcut (win: Ctrl+f, mac: Cmd+f) to open up the search input UI. Enter a CSS selector into the search input UI.

### Clear Site Data
Cache clearing is one of the activities that might be used daily in the testing exercise. Chrome DevTools can successfully help you out.
Press Control+Shift+P or Command+Shift+P (Mac) to open the Command Menu and then run the Clear Site Data command to clear all data related to the page, including Service workers, localStorage, sessionStorage, IndexedDB, Web SQL, Cookies, Cache, and Application Cache.
If you don't want to delete all site data, you can control what data are deleted from **Application >  Storage**.

![DevTools application storage](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/image10_3278490be8.png)

### Location
This is a fantastic functionality! It shows how to test an app that must work in different locations and presents a map for you - teleportation has not been invented yet! There is an option in DevTools that allows you to mimic location. The option is slightly hidden:
1. Open Dev Tools
2. Click the action bar in the top right corner of the screen
3. Select More Tools
4. Click Sensors

Now you can provide a website/application with your desired geo-coordinates and pretend you are there. Remember about reloading the page with every change to the coordinates.

![DevTools location](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/image11_d0b495da83.png)

### Performance monitor

In the same **More Tools** panel you can find the Performance Monitor. Use it to get a real-time view of various aspects of a page's load or runtime performance, including:
- CPU usage.
- JavaScript heap size.
- The total number of DOM nodes, JavaScript event listeners, documents, and frames on the page.
- Layouts and style recalculations per second.

There is also a separated **Performance** tab in the upper menu, which shows the Compute Intersections in the flame chart. These options can help you to identify the events and debug on its potential performance overheads.

![DevTools Performance](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/image13_7323c216c2.png)

### Screenshots
There are two ways of taking screenshots when working with DevTools:

**Screenshots of a portion of the viewport**

1. Click **Inspect** or press Command+Shift+C (Mac) or Control+Shift+C (Windows, Linux) to enter Inspect Element Mode.
2. Hold Command (Mac) or Control (Windows, Linux) and select the portion of the viewport if which you want to take a screenshot.
3. Release the mouse. DevTools downloads a screenshot of the selected portion.

**Screenshots of specific HTML nodes**

1. Select an element in the Elements tab
2. Open the Command menu
3. Start typing node and select Capture node screenshot. DevTools downloads a screenshot of the selected node.

## How to operate on mobile devices?
Last but not least, if you want to test the mobile option, it might be useful to know how to attach your device to Chrome DevTools.

1. Install the Chrome browser on your PC
2. Get hold of an Android smartphone
3. Find a USB cable to connect them
4. Plug your device in.
5. Type chrome://inspect in the Chrome browser.

On mobile devices, you can take advantage of Chrome Developer Tools in a very similar way to web testing. You can perform throttling, look for elements, or check the network connection. What is more, you are also able to check things in a database through Stetho (*debug bridge for Android applications*). 

If it doesn’t work right away - all you need to do is enable the developer options:
1. Open settings
2. Go to System> About phone>Version number
3. Tap on it 7 times (that’s right)

And there you go.
1. Open the developer options
2. Switch them on
3. Allow USB debugging

From now on the device should be visible in Chrome/inspect.
 
There is much more to Chrome DevTools - all you need is to explore the options. Each year, Chrome adds new functionalities to its extensions, including DevTools. The number of options expands and allows not only for better development but also better testing without multiple external tools or applications. For basic functional testing, Chrome DevTools is more than enough. They just require time to learn how to best use them and to benefit from all the resources they offer. 


large_weevil-g22fd4954a_1280.jpg

medium_weevil-g22fd4954a_1280.jpg

small_weevil-g22fd4954a_1280.jpg

thumbnail_weevil-g22fd4954a_1280.jpg

weevil-g22fd4954a_1280.jpg

Chrome DevTools introduction

chrome-dev-tools-introduction

[TABLE_OF_CONTENTS]

## Why Agile?

The word 'agile' became trendy years ago and the eagerness of using it in projects in organizations remains high even today.

We have Agile dev teams and Agile testers, Agile projects. Unfortunately, the term 'agile' has become a convenient replacement for lack of requirements, lack of documentation,  or, just basically,  a pure mess.

On the other hand, Agile means nothing more than a way of working well to reduce the cost of change and uncertainty.

## Traditional vs. Agile

Talking about methodologies is always good as it leads to spotting the difference between traditional - usually phased or gated (Waterfall) ways of making software,

![phased_or_gated](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/phased_or_gated_f4e60a9340.png)

as opposed to iterative, where every story is expanded, coded, and tested with a possible release after each iteration.

![agile.png](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/agile_4b0f766f35.png)

## The Agile change

Agile testing influences the whole process of making software, including:
- Whole Team Approach
- Coding and testing as a single process
- Feedback, collaboration
- TDD/ATDD practices
- Test-infected developers
- Better tools
- Better designed tests 

It also allows a tester to be an actual part of the team rather than a siloed unit in an organization. A tester is not a gatekeeper or quality police but he/she provides valuable input into each phase of development.

In real life, it means that an Agile tester takes part in all team meetings, provides the input, asks questions, is invited to pair programming sessions with developers, and takes the same responsibility for the code and product as any other member of the team. Of course, it requires from the Tester a larger skill set, great communications skills, and at least basic knowledge of coding and test automation.

## How to include testing?

There are many ways for extending Test Specialist responsibilities in an Agile team. They don’t necessarily include testing itself but also include a vast range of communication, presentation, and consultation activities.

### TDD
Traditional testing relies on ample documentation. Agile testing is an approach to a process that mimics the principles of agile software development, especially where:
- Testing is done more often,
- The whole team knows how and what to test, it is possible to share tasks
- Testing relies less on documentation and more on team member collaboration.

One of the great examples of how to include testing into an agile process is to start from Test-Driven Development. The TDD methodology is a software development process that emphasizes the need for good software quality. Traditionally, a developer has written the code, and then the testing department has examined it and reported bugs. Moreover, in TDD, it is the developers who write unit tests first before even writing any code that would complete a user story.

Of course, it is possible to pair in every task, so in most situations, it is possible for a developer to work together with a Quality Expert and think of possible test cases.

The tests initially fail until the developer writes a minimal sequencet of code to pass the tests. As the next step, the code is refactored to meet the quality requirements of the team.

It is possible to implement TDD into the Scrum or Kanban type of working. The point is to focus on testing and quality in general. Additionally, it makes testing more agile by moving most of the testing procedures to the early stages of the development lifecycle. When writing test scenarios at the beginning of each story, developers need to understand the requirements and cooperate with the other team members. This minimizes unnecessary code creation and also resolves any doubts at the beginning of the development cycle.

### Automate regression testing
Regression testing is yet another type of software testing to verify if the software, previously developed and tested, still performs correctly after changes or interface with other software. Changes may include software enhancements, patches, configuration changes, etc.

If regression testing in a software project is supposed to make any sense - it has to be performed as often as possible - at least on a sprint basis. Ideally - daily. Why? Because stable test suites would detect any unexpected system behaviors and react to a major change.

Small systems usually have checklists rather than test suites - large ones - even ‘monstrous’.
Deciding what should be automated is always tough because having automated test scripts may not be something that the team could present  to the customer. The question is rather: WHY would you want to automate anything?

The problem is that sometimes, automated suites are not reliable or trustworthy and the effort to keep them alive is quite hard. Test suites have to be regularly maintained and extended. Automation is great for this task but it is just another tool – not the purpose of the project.

Most importantly, even in the Agile environment, all tests need to be reviewed often and prioritized daily.

No one will benefit from extended long-running test suits. What is more, automated regression tests should be efficient and alert early in the development phase, to be of better quality and truly agile.

### Visualize testing activities
One of the great examples of working with a Scrum or Kanban board is to visualize all the tasks in the project, not only the developer’s ones.

It is a great idea for the team to select even small, repetitive manual test tasks as prospects for automation. All these ideas and tasks can be placed on a board and addressed accordingly. It doesn’t mean that a tester is responsible for doing it.

Besides, in cross-functional Agile teams, there is a great opportunity for the testers to learn more coding and test automation, and for the developers to pay more attention to code quality.

### Communicate
Irrespective if a project is handled in Scrum, Kanban, or another Agile manner – the key to success is (?) the communication in the team.

Also, there are useful respective tools which help  teams deliver faster and fight for better quality and efficiency. CI/CD build tools can be easily integrated with common tools for communication such as Slack, and can inform automatically that the build is done or if the build has failed. It can also be helpful in automated regression and getting the test result accessible to everyone.

### Avoid single used solutions
The responsible software development process requires planning and adjusting, not working “here and now” and then fixing production bugs. Single–used solutions in a code, in testing, unnecessary work done in the cutting-corners mode to satisfy the customer’s demands are the things that often  boomerang in the future.

When a manager  approaches you and asks for doing development quickly instead of properly – “because it’s Agile” – remember that this is nothing else but a start of technical debt.

Discuss, propose, think, but when you agree on a shortcut – for example skipping performance testing or postponing some exploratory sessions due to lack of time – be aware of the consequences in the future.

You’ll save some time now, but you’ll be in real trouble soon.

### Explore
As Elisabeth Hendrickson said in her book “Explore it!”:

 “No matter what your job title, you most likely find yourself testing regularly. Testing is an integral part of creating anything.”

This is so true in Agile. Exploratory testing is simultaneously designing and executing tests to learn about the system, and then using your insights from the last experiment to feed the next.

Exploratory testing involves scouting around areas that the net of other forms of testing doesn’t cover. A tester interacts with the implementation, designing, and executing tiny experiments in a rapid succession.

### Give and receive feedback
Every Agile Team uses feedback to improve. It does not matter whether feedback is provided daily or during longer sessions such as Retrospective or Planning. Testers are expert feedback providers, they usually have overall knowledge about the product and can advise at the beginning of brainstorming or at the end of the iteration to improve further work. Feedback lets the team make course corrections.

### Testing in Scrum
How to incorporate testing in Scrum? 
The Scrum Guide doesn't say anything about testing because it's "a framework for developing, delivering, and sustaining complex products". It transcends the type of product that the team is developing, and the types of testing that the team needs to do, or how testing fits into the process is different for producing different types of software products. It does not define a specific “Tester” role, as testing is the whole team’s responsibility.

Of course, there can be a person more focused on delivering high quality, measuring application performance or assuring application security, but this role is not defined by the book. It usually means that when a team decides to work according to Scrum Guide and at the same time, there is an organizatinal role of a “Software Tester” - an expectation might arise that this person should provide guidance for the entire team (or even organization) on how to test software most effectively and lead the others in the field.

## Summary
It is not the choice of a methodology or technology that limits us from raising the quality of the product or including test automation in a daily routine. It is rather a drive that comes from the team and helps everybody to get better at coding, testing, and taking care of the product.

The choice of the methodology is always a process, not a single decision. It requires work and effort from every team member, not only from the tester.

If you have anything to do with test management or quality assurance - this short exercise is great for practicing on a daily basis: start your working day from a short trip around different metrics - nightly builds, code coverage, cyclomatic complexity, etc. Try to explore and add a new flavor to the testing. Every day you can make an important discovery and improve the quality of your work.



large_gran-canaria-g1792ff18b_1280.jpg

medium_gran-canaria-g1792ff18b_1280.jpg

small_gran-canaria-g1792ff18b_1280.jpg

thumbnail_gran-canaria-g1792ff18b_1280.jpg

gran-canaria-g1792ff18b_1280.jpg

Agile testing – organized chaos or a planned way of improving your software?

agile-testing-organized-chaos-or-a-planned-way-of-improving-your-software

A QA engineer with over six years of experience in testing and test automation. Especially interested in backend technologies and reducing test flakiness. Neapolitan pizza and specialty coffee lover.\

[linkedin.com/in/mateuszslapinski/](https://www.linkedin.com/in/mateuszslapinski/){ref="nofollow"}

medium_MS (1).jpeg

small_MS (1).jpeg

thumbnail_MS (1).jpeg

MS (1).jpeg

Mateusz Słapiński

Every manager or project manager wants to deliver features as fast as possible. Testing and fixing bugs take a lot of time in the software development life cycle. People have tried  to reduce this time in many ways. One of them was to switch from Waterfall to Agile. Instead of sequential steps with testing as the last one (often after months of development), teams tried to deliver smaller parts of features in shorter sprints. The other idea was to invest in test automation. Tests executed by computers are faster than those performed by humans. Computers don’t make mistakes, they don’t suffer from fatigue after hours of work. We can use them to test on-demand, any time we want. That’s why many teams started automating their manual test cases - using the tools to click on the UI on their behalf.

[TABLE_OF_CONTENTS]

After the initial excitement about watching the tests executed by a machine, the teams realized that this approach also had some drawbacks. The more tests were automated, the more time their execution took. Testing in parallel didn’t help as it made some of the tests flaky - when run a few times in a row, their results differed without any apparent reason. Some of the tests were unstable even when run sequentially. Finally, the results were ignored as not trustworthy and the applications were tested manually. It looked like test automation didn't work. This model is called The Ice cream cone and it’s a well-known anti-pattern.

![the_ice_ceram_cone.png](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/the_ice_ceram_cone_86b5197f6e.png)


## What can be improved?
Fortunately, there are some ways to make test automation your friendrather than your enemy. One of them is the test pyramid. According to the [ISTQB glossary](https://glossary.istqb.org/en/term/test-pyramid){ref="nofollow"}, the testing pyramid is “A graphical model representing the relationship of the amount of testing per level, with more at the bottom than at the top.” Why does it matter? How does having more tests on the ground help? First of all, we need to define the levels of the pyramid, starting from the bottom.

![image3.png](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/image3_dd196f48f4.png)

## Unit tests
Unit tests check the smallest parts of the application code - in most cases the methods. They are isolated  which means that they are independent of other parts of the application or dependencies like databases. They are easy to write: you pass some parameters to the tested method and verify if the data returned by the method is as expected. When they are written properly, their execution takes very little time - milliseconds. This is because they can be run on a local machine right after the code has been compiled. That’s why they are used heavily in the Test-driven development technique (TDD) which consists of 3 steps repeated in a loop: 
A developer writes a test first, it should be failed, or even not compile.
A tested method is implemented, the test should be passed.
The tested method is refactored, the test should be passed.

Unit tests are also useful if you want to measure the quality of your code. You can calculate and track the test coverage - how many lines of code were checked by the tests. It's hard to reach 100% and it often does not make any sense to try. Howver, the more unit tests and the higher coverage the better. And that's why they are at the bottom of the pyramid. 

## Integration tests
As the name implies, integration tests check if the classes and methods or modules combined can work successfully. They might also verify if the application module can communicate successfully with external parts of a system (like the database, HTTP client, etc.). The application still doesn’t need to be deployed anywhere - integration tests can be run on a local machine or a CI system, but some of the application modules need to be set up together which might take more time in unit tests.

## End-to-end tests
Finally, we want to move from modules and parts of ane application to a combination of many applications. End-to-end tests, also called E2E tests, verify the whole working system, composed of many applications: backend, frontend, databases, message buses, etc. They check the system from the user's perspective and verify complex workflows. On the other hand, their stability depends on all the services in the test flow, which means they are flakier and their execution takes more time. Also, because of the test environment limitations, there might be queues of features to be tested, which increases the time of testing E2E even more. Due to these performance issues and the costly maintenance of these tests, their number should be limited. They should cover only the most critical business paths and that's why they are at the top of the pyramid.


## Back to the pyramid
Based on the testing pyramid, we should invest the most in the unit and integration tests, reducing the number of E2E tests. But why does it matter? The testing pyramid is about reducing the time and the cost, which are strictly connected. It is all about shortening the feedback loop. 


At the beginning, I mentioned that test automation has failed in many projects. Why did it go so bad? Let’s skip the bottom layers of the pyramid for a while and test everything on the E2E level. How long does it take to see if a change you have made in an application didn’t break anything? What needs to be done to test the application end-to-end?

1. Code review - in some projects the code change can’t be deployed to the testing environment before the change is approved (~hours).
2. Application build (~minutes).
3. Application deploy to the testing environment (~minutes-hours, there might be more people willing to deploy their changes causing the queue of changes to be tested)
4. Automated tests execution (~minutes to hours).
5. Results analysis - there are probably a few failed tests, not essentially caused by the change, but the test environment itself. Also, other teams might waste their time here, if your change has broken their testing (~minutes to hours).
6. Fixing a bug, if any, and going back to (1).

How many code changes can you test during a day? 

At the same time, if you invest the most in the tests at the bottom, verifying the change is quite straightforward:
1. Code compilation (~seconds).
2. Unit tests execution (~seconds).
3. Result analysis (~minutes if any failures, but in most cases, you can be sure that your change introduced the failures, not anyone else).
4. Fixing the bug, if any, and going back to (1).
 
You magically saved a few hours or even days, which means you can introduce many more changes and deliver more features.

## Does it mean you should write only unit tests?

![image2.png](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/image2_60fc5b87a5.png)

Probably you have seen a lot of pictures or GIFs like the one above. Two components can work perfectly fine in isolation, but when integrated with other parts of a system, they can cause some problems. Testing on higher levels of the pyramid helps to avoid these integration issues. These tests are essential for your CI/CD, however, when your components are properly unit tested, you can reduce their number and decrease the overall testing time.


## What is a rule of thumb?
You might be wondering how to decide on which pyramid level your test should be. My suggestion is to ask a few simple questions. 
1. *Can the feature be unit tested?* A unit test is a perfect solution. Test everything possible, including corner cases.
2. *How will it be combined with some other parts of the system?* Check it in an integration test (but only verify the integration, no need to check the details as they were unit tested).
3. *Will it be used in an end-to-end flow?* Check the happy paths to see if the system can be used properly by the users (corner cases of functionality and the integrations have been already covered).

You can stay with the ice cream cone model and keep complaining about the automated tests, or you can improve them with the pyramid model. Starting from the bottom of it will make your test automation successful. Stable and trustworthy tests will allow you to deliver faster and, more often, keeping everyone happy. The end users will receive high-quality features more frequently. The team members will be able to work on a new functionality instead of maintaining unstable tests.
The ice cream cone or the pyramid? The choice is yours.






large_grasshopper-g413872cef_1280.jpg

medium_grasshopper-g413872cef_1280.jpg

small_grasshopper-g413872cef_1280.jpg

thumbnail_grasshopper-g413872cef_1280.jpg

grasshopper-g413872cef_1280.jpg

Testing pyramid: A successful approach to test automation

testing-pyramid-a-successful-approach-to-test-automation

Features

Pricing

Roadmap

Case studies

Privacy policy

Terms of service

About us

Book a demo

Contact

Get more tips and product related content. Zero spam.

Don't miss any updates

End-to-end testing made simple! BugBug is a new, more intuitive tool for test automation of web apps. Get all the benefits of test automation without hiring new people or infrastructure changes.

BugBug.io | Intuitive and reliable test automation for startups

large_Open Graph 1200x630 3.png

medium_Open Graph 1200x630 3.png

small_Open Graph 1200x630 3.png

thumbnail_Open Graph 1200x630 3.png

Open Graph 1200x630 3.png

Blog about test automation

Guides to automation

small_pawel-about-512.png

thumbnail_pawel-about-512.png

pawel-about-512.png

Paweł Bylina

Are you a tech leader, CEO or CTO in a tech startup? You want to implement automation testing for your SaaS product? We've written this guide especially for you. It's packed with actionable tips on how to approach test automation in an agile way.

We're going to tackle the following questions:

-   What is the business goal of automation testing?

-   How to implement automation incrementally?

-   What resources do you need to develop automated tests?

-   What are typical problems and obstacles and how to overcome them?


The strategic approach to automation testing
--------------------------------------------

Automation testing is a very broad term. There is no single goal of automation. Depending on the level of automation you will get different benefits for your organization.

*Figure 1: Strategic matrix of different goals of test automation*
![bugbug-goals-of-automation-testing.png](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/bugbug_goals_of_automation_testing_9308b9cb1b.png)

See this strategic matrix above. You can achieve all of these goals, but not all at once. You should pick the right order of goals, starting with the ones that are low-effort and high-value. 

To achieve each of these goals, you need a different combination of resources and skills. We'll guide you through each of the goals, explaining the approach, time & resources and how to achieve it with BugBug.

Step 1: low cost & high impact goals
------------------------------------

### **Goal:**

-   Prevent loss of business due to production downtimes

### **Approach**

-   You don't need to hire new people, you can do it yourself

-   Create 5-10 automated tests and run them every hour to monitor your production, for things like login, landing pages, registration form, contact form, basic app navigation

### **Time & Resources**

-   2-4 hours of your own time

### **BugBug features that you should use**

-   [Recording tests](https://docs.bugbug.io/creating-tests/recording-the-tests)

-   [Cloud runs](https://docs.bugbug.io/running-tests/in-cloud-on-server)

-   [Schedules](https://docs.bugbug.io/running-tests/schedules)

-   [Notifications](https://docs.bugbug.io/collaboration/notifications)

-   [Re-recording test steps](https://docs.bugbug.io/recording-tests-steps/re-recording-steps)

Step 2: low hanging fruit
-------------------------

### **Goal:**

-   Improve team collaboration on quality assurance

### **Approach**

-   Give all the scrum teams access to the testing platform

-   Share direct links to failed tests in your tickets or on a chat

-   Configure who gets notifications

### **Time & Resources**

-   Inviting people only takes 5 - 10 minutes, but you should spend 2-4 hours on communicating its purpose

### **BugBug features that you should use**

-   [Invite users](https://docs.bugbug.io/collaboration/inviting-team-members)

-   [Project settings > Notifications](https://docs.bugbug.io/collaboration/notifications#choose-when-people-get-email-notifications)

Step 3: incremental development of automated tests
--------------------------------------------------

### **Goals**

-   Save time on manual regression testing

-   Release faster

-   Prevent cost of regression bugs

### **Approach**

-   Talk to the whole team about what you want to achieve and what's the incremental plan

-   Choose 1-2 people responsible for automated tests 

-   Create dedicated test user accounts that will be used for automation only

-   Run automation on some testing environment (staging)

-   Create separate test suites for staging and prod (run all tests on staging, but only essential smoke tests on prod)

-   Add more tests incrementally, 2-3 test cases per week (after 3 months you should have about 50 test cases automated)

-   Start adding [data-test-id attributes](https://docs.bugbug.io/preventing-failed-tests/selectors#use-custom-attributes-for-selectors) in your HTML

### **Time & Resources**

-   You will see the benefits after 4-8 weeks of ongoing work

-   You need to engage 1-2 people on the team. Make sure they have at least 2 hours per week to work on automation.

-   You need at least one technical person on the team who knows how selectors work.

### **BugBug features that you should use**

-   [New test from here](https://docs.bugbug.io/creating-tests/new-test-from-here)

-   [Duplicating tests](https://docs.bugbug.io/creating-tests/duplicating-tests)

-   [Components](https://docs.bugbug.io/editing-tests/wip-make-component)

-   [Variables](https://docs.bugbug.io/editing-tests/variables)

-   [Profiles](https://docs.bugbug.io/editing-tests/variables#profiles) to [test on different environments](https://docs.bugbug.io/editing-tests/variables#work-with-different-development-environments)

-   [Suites](https://docs.bugbug.io/organizing-tests/suites)

-   [Manually editing steps](https://docs.bugbug.io/editing-tests/manually-creating-the-test)

-   [Testing registration](https://docs.bugbug.io/editing-tests/variables#test-user-registration-and-login-using-variables)

-   [Custom javascript](https://docs.bugbug.io/editing-tests/custom-javascript-actions) might be needed

Step 4: integrate automation with CI/CD
---------------------------------------

### **Goal:**

-   Improve software development lifecycle, being sure that no code is merged without tests

### **Approach**

-   Integrate automation with your development pipelines

-   Trigger automated tests with every pull request

-   Show test results directly in your repo tool (Bitbucket, GitHub, GitLab, etc.)

-   Ask developers to run automated tests locally on the localhost environment before they commit code

### **Time & Resources**

-   You will see the benefits after 4-8 weeks of ongoing work

-   You need to engage 1-2 people on the team. Make sure they have at least 2 hours per week to work on automation.

-   You need at least one technical person on the team who knows how selectors work.

### **BugBug features that you should use**

-   [CLI](https://docs.bugbug.io/integrations/api)

-   [Overriding variables](https://docs.bugbug.io/integrations/api)

-   [Running on localhost](https://docs.bugbug.io/running-tests/test-in-your-browser)

Step 5: increase coverage
-------------------------

### **Goal:**

-   Get 100% test coverage, be confident that all the features, all the browsers, all the devices work correctly

### **Approach**

-   Pick your battles, choose what is the most essential to automate

-   We recommend you keep adding 1-2 test cases every week and don't try to achieve the impossible 100% coverage

### **Time & Resources**

-   To automate everything you would need an Infinite amount of time :)

-   To automate 2-3 new test cases per week you still need 2-3 hours weekly plus a bit more experience with tests and javascript.

-   Note that the more test cases you have, the longer the maintenance. The trick is to find the right balance.

### **BugBug features that you should use**

-   Project settings > Browser width (for [testing mobile screens](https://docs.bugbug.io/workflow-tips/mobile-version-testing))

-   Lots of custom Javascript

-   [Javascript variables](https://docs.bugbug.io/editing-tests/variables#javascript-variables)

-   [Javascript assertions](https://docs.bugbug.io/editing-tests/custom-javascript-actions#advanced-javascript-assertions)

-   [Parallel running](https://docs.bugbug.io/running-tests/parallel-mode)

Common problems and solutions in implementing automation testing:
-----------------------------------------------------------------

Here are some typical problems you may encounter when you want to start automating tests for a SaaS product. We've made a short list with possible solutions that worked for other [BugBug clients](https://bugbug.io/case-study/).

### **Captcha**

You need to [disable the captcha only for automation testing](https://docs.bugbug.io/troubleshooting/captcha-in-automation-testing). Unfortunately, it's not trivial and you need to read through captcha docs or ask a developer for help.

### **Date-based features, calendars** 

Is your app searching train timetables for tomorrow? It's difficult to automate because the date and the expected outcome can change every day.

Solution A: In the automated test, pick a date that's far in the future. Then update the tests regularly, changing this date and the expected outcome.

Solution B: create a dedicated environment for automation with a mocked-up hardcoded date on the backend

### **Dynamic lists such as search results, product listings**

It's hard to automate a test if some data is changing all the time. You need to create independent tests that are always sure what to expect, for example, if you're checking a list of promoted products on your homepage, your test should first go to the admin panel and mark such products as promoted.

You can also have a separate dedicated test automation environment where you have a hardcoded set of data.

### **Can't use the same user email twice**

Create a new user with each test - use the [built-it variable "testRunId"](https://docs.bugbug.io/editing-tests/variables#use-built-in-variables-for-dynamic-or-random-values) in combination with a [temporary email catcher](https://www.google.com/search?q=temporary+email&ei=ARVtY_X8IomKrwS_i4k4&ved=0ahUKEwi1gZ2u86P7AhUJxYsKHb9FAgcQ4dUDCA8&uact=5&oq=temporary+email&gs_lcp=Cgxnd3Mtd2l6LXNlcnAQAzIECAAQQzIFCAAQkQIyBQgAEJECMgUIABCABDIFCAAQgAQyBQgAEIAEMgUIABCABDIFCAAQgAQyBQgAEIAEMgUIABCABDoKCAAQRxDWBBCwA0oECE0YAUoECEEYAEoECEYYAFDhAVjhAWC1CmgCcAF4AIABa4gBa5IBAzAuMZgBAKABAcgBCMABAQ&sclient=gws-wiz-serp) always to run tests with a fresh new user. You can also delete these users at the end of the test by recording test steps in an admin panel (if you have one) Read more in how to test user registration.

### **Selectors are not reliable**

The only way to make sure your tests are stable and easy to update is to start using "data-testid" attributes in your HTML structure, [dedicated attributes for automation testing selectors](https://docs.bugbug.io/preventing-failed-tests/selectors#use-custom-attributes-for-selectors). You need to communicate that to the front-end developer and make sure you remember about it during the code review. No worries, adding additional attributes have no risk, it's just additional data for automation with no other function, it will not introduce new bugs.

### **Multilanguage**

Your page can run in different languages and you want to verify all of them. Try to assess the risk of having a bug only in one language without affecting the others. Perhaps it's not likely that such a bug would occur?

If you still need to test different languages, you have two options:

Option A: Record separate tests for each language. Hard to maintain on a bigger scale

Option B: Use variables & profiles to run the same test on different languages

-   Add data-testid to all of your app elements, so that the selectors are independent of the app language and the displayed test 

-   Add a way in your app to change the language via a URL, for example, "myapp.com/?lang=fr"

-   Create a variable "language code"

-   Create different profiles with different language codes

-   Replace the language code in the "Go to URL" step with the variable

-   Create separate suites for all the languages, overriding their profile with a chosen language

**Tests take long time to finish**

[Upgrade your BugBug plan](https://bugbug.io/pricing/) to be capable of [running cloud tests in parallel](https://docs.bugbug.io/running-tests/parallel-mode)

### **Downloading files**

Currently BugBug can't verify if the files have been download. Soon we'll add a new type of assertion for that, [talk to our support](https://bugbug.io/contact/) if you need it ASAP.

### **Drag & drop interactions**

BugBug support for drag&drop is in beta, you can achieve it by combining the "Mouse press", "Hover" and "Release mouse button" [actions](https://docs.bugbug.io/editing-tests/actions). Talk to our support for details.

### **A recorded selector doesn't work**

Sometimes it's not possible to record a reliable selector and you need to use your human intelligence and debug HTML a bit. 

Use our [No-code XPath Selector Builder](https://bugbug.io/xpath-selector-builder/) which makes it easy to craft a good selector manually and can teach you in-depth about [best practices in selectors](https://bugbug.io/xpath-selector-builder/#How-to-create-reliable-XPath-selectors-for-test-automation).

Success in test automation depends on the regularity of your progress
---------------------------------------------------------------------

Implementing automation testing in a SaaS company is a long-term process. You should do it step by step. The biggest challenge for tech leaders is lack of time to dedicate on QA area. The truth is you don't need a lot to push it forward. Better slow than never. [😉](https://emojipedia.org/winking-face/) Block yourself a slot in calendar every week to work on QA automation and keep the momentum going.

medium_bugbug-matrix-illustration.png

small_bugbug-matrix-illustration.png

thumbnail_bugbug-matrix-illustration.png

bugbug-matrix-illustration.png

Tech leader's guide to automation testing

tech-leaders-guide-to-automation-testing

We’ve prepared a short guide with essential recommendations on test automation that we created specifically for startups. We divided the guide into 3 levels:

[★☆☆ Level 1: get the most out of basic automation](https://bugbug.io/blog/software-testing/automation-testing-guide-for-startups-level-1/)

[★★☆ Level 2: automate hundreds of regression tests](https://bugbug.io/blog/software-testing/automation-testing-guide-for-startups-level-2/)

[★★★ Level 3: integrate with CI/CD builds](https://bugbug.io/blog/software-testing/automation-testing-guide-for-startups-level-3/)
  
Also see our guide dedicated to non-technical people: [beginner's tutorial to automation testing with BugBug](https://docs.bugbug.io/in-depth-guides/beginners-tutorial-to-automation-testing)
 
### What will you achieve on Level 1?
After finishing this level you will run 5-10 automated tests every hour and check if your core features are working properly. You will detect production bugs with big impact on users sooner than before.

### TLDR:
-   Start with short and simple tests that check your core features
-   Remember about recording “hover”
-   Create special test user accounts dedicated to automation
-   Schedule the tests to run every hour
-   Incrementally add more tests

### Focus on the most important features

![80-20 rule (1).png](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/80_20_rule_1_c2126d9815.png)

Many companies spend endless hours on a neverending pursuit to automate everything. This leads to very costly automation that requires a whole team to create and maintain. Don’t be like these companies and keep it simple. Agile startups should start with automated testing of the most essential functionality. 

> 👍 **Rule of thumb:**
> Usually, it just takes 5 to 10 automated tests to be sure that most users are able to achieve their goals.

What should you test? Think of your features. What percentage of users are using a particular feature? If the answer is higher than 80%, then this is a core feature. The goal of automating tests of such important features is to make sure that you can detect bugs that have a huge impact on your business as soon as possible (new user registration is for sure a core feature, but it’s a little bit more tricky to automate, which we will cover in the next episode)






### Start with short & atomic tests

When you [record your first tests](https://docs.bugbug.io/creating-tests/recording-the-tests) start with something very simple and atomic. Atomic means that the test is small, independent and checks just one thing.

A good idea is to start with testing the basic navigation of your landing page. Click “record” and create a test that just clicks one item of your navigation link in the menu. Add [assertion](https://docs.bugbug.io/recording-tests-steps/recording-assertions) for some text in a heading. Finish the test and run it. Awesome! You’ve created your first automated test that touches the most essential area of your business!

![Screenshot 2022-07-25 at 13.44.12 (1).png](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/Screenshot_2022_07_25_at_13_44_12_1_0ebde0ddbb.png)

Now create a separate test for each navigation link. Here are screenshots of how BugBug team set up their landing page tests.

![Screenshot 2022-07-25 at 13.44.15 (1).png](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/Screenshot_2022_07_25_at_13_44_15_1_74fb39e9de.png)


### Don’t forget about recording hovers

If your website navigation appears on hover, you need to tell BugBug that you want to record a mouse moving to a specific element. To do this you need to click “Hover” during the recording and then click an element. When running the test, BugBug will move the mouse cursor to this element.

![bugbug record hovers.png](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/bugbug_record_hovers_81639fbd30.png)


### Create special test users

Another idea is to test if login works. Create a separate test that only checks that one thing.

To achieve that you need to prepare a “test user” - a user account that is dedicated only to testing if login works properly

-   Manually register a new user with a new unique email, for example with john.smith+testautomation@email.com (you can use [disposable mail](https://www.google.com/search?q=disposable+mail) or [plus aliasing)](https://support.google.com/a/users/answer/9308648?hl=en)
-   Go to BugBug and record a test that clicks the login button, enters the user login and password and submits the form
-   Add an assertion for some text that confirms that the login was successful
-   Finish your test
    

  

You might be tempted to test other features in the same test, but don’t do this now. We want to be sure that the login test is atomic and independent from other features.

  

Remember that you can create as many test users as needed. For example, if your product offers multiple combinations of preferences, you can create a separate user for each combination.

  

> 👍 **Rule of thumb:**
> Always create new test users if this leads to less test steps

  

### Schedule all tests to run every hour

With BugBug you just need a couple of clicks to start running the tests every hour. Just go to “Schedules”, select the “Hourly” one and add “All tests”.

![hourly schedule (1).png](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/hourly_schedule_1_445d7fe266.png)


  

That’s it, now you will start getting emails every hour with the results of your test automation. Tip: create a filter on your email account so that you avoid inbox clutter or [configure notifications to only get an email on failures.](https://docs.bugbug.io/running-tests/schedules#configure-email-notifications)

![email notifications (1) (1).png](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/email_notifications_1_1_a23f96ba47.png)


  

### Incrementally add more tests

You’ve successfully created test automation that monitors your production! Now keep adding more short tests one by one. Don’t forget to focus on being cost-effective - if you have trouble automating a use case, skip it and go to the next one or find a different way of checking if it works. And remember: you can always [ask BugBug support to help you with any troubles](https://bugbug.io/contact).

Read next: [next level guide on how to easily maintain hundreds of automated tests ](https://bugbug.io/blog/software-testing/automation-testing-guide-for-startups-level-2/)
  
  

medium_illustration - guide - level1.png

small_illustration - guide - level1.png

thumbnail_illustration - guide - level1.png

illustration - guide - level1.png

Automation testing guide for startups - Level 1: get the most out of basic automation

automation-testing-guide-for-startups-level-1