Agile Testing Quadrants

Testing can be difficult, there are many questions and many issues we can face with every new feature and every new story. Every day, testers need to decide what to test, how to test, what to automate, and what risks to accept. These decisions are hard and require a lot of experience. Fortunately, there are techniques, heuristics, and models which can help with planning tests. One of the oldest and still very useful techniques is Agile Testing Quadrants (ATQ). It is a simple table that helps us select the tools we need to use, depending on the required outcome. In this article, we will look into the history of Agile Testing Quadrants, their uses, and limitations. We will also discuss the attempts at modernizing it.

Agile Testing Quadrants or Agile Testing Matrix?

First, we need to clear one thing: as almost everything in the IT sector, even this simple table might have many names! Chances are that you will come across it as an Agile Matrix, Testing Matrix, Testing Quadrants, or Agile Quadrants. All of them refer to the same thing. The only exception is the Agile Testing Matrix which may refer to both Agile Testing Quadrants and its precursor the Named Agile Testing Matrix.

The Story of Matrix

When speaking about Testing in Agile, there is probably not a single more important person to its development than Lisa Crispin and Janet Gregory. Their book, Agile Testing, has helped many testers. Testing Pyramid was born while Mike Cohn tried to explain the nuances of automation to Lisa’s team (Experience of Test Automation - chapter 1 Section 1.3.) This book, a true agile manifesto,  played a key role in the popularization of Agile Testing Quadrants. The creation of ATQ is often misattributed to this wonderful duo. As Lisa Crispin explained in her blog - they improved on the older concept by Brian Marick - aan Agile Testing Matrix created as part of his blog series presenting his vision of testing in Agile. So Agile Testing Quadrants have almost 20- year-long history and they are still one of the best models used for various types of testing.

What is the ATQ anyway?

Agile Testing Quadrants are a visual tool that allows to differentiate business facing tests and technology facing tests. The quadrants help to change your thinking about testing as a whole. The matrix consists of four basic quadrants and is able to support the team by better planning the testing effort as well as explaining the reasoning behind different types to a non-technical stakeholder.

Let's briefly go thru the 4 testing quadrants:

Quadrant 1

Unit tests and Component tests are those most commonly associated with automated testing. They are performed throughout your applications' lifecycle. The testing activities aim to give feedback to your developers with test cases focused for example on code quality.

Quadrant 2

The mix of automated testing and manual testing are focused more on the end client of your app. Functional testing and story testing will aim to check if your software meets the user requirements. Manual and automated tests focus on simulations as well.

**Quadrant 3 **

The third out of four quadrants holds all your manual tests. Not only the testing that needs to be done by your QA team, but the user alpha and beta testing as well. We mentioned before that allowing a small group of end users to critique the product is usually vital for success. The feedback they provide is amazing!

**Quadrant 4 **

Last but not least there are those tests that need all the fancy software, especially if your QA team is not that experienced. Load testing and performance testing are vital. All those technology based tests aim to improve your applications security.

How to use the Agile Testing Quadrants? Examples for different test types

Agile Testing Quadrants have two axes which split the matrix into 4 quadrants. Both axes pose the same questions, namely What is the goal of your testing? The major difference between the two axes is the flavor of the question: the horizontal axis asks what is currently more important: helping the team to deliver faster or ensuring that the product is as good as possible? The vertical axis asks if we are focusing on the aspects of technology, ensuring that everything is working well, or are we looking from the perspective of business and customers, making sure that our program is worth using? Obtaining both answers points to one of the Quadrants with a list of useful examples. This can be a little confusing, so let’s look at some of the examples.

Example 1: E-commerce Shop

Let’s say we have an e-commerce project that is going nicely. However, there is one enormous risk on the horizon, namely quite a lot of features. Each one is tested separately so nobody is sure how they will work in the real world. The team believes that the standard paths will work. However, nobody is sure what kind of edge test cases will come from the real world.

Looking at our matrix we can see that the problem is definitely business-facing, because we don’t know what the user will do. We also need to make sure that our product will handle it, ergo - Critique Product.

Looking at it we land in quadrant 3 where we have propositions of the following techniques:

• Exploratory Testing
• Scenarios
• Usability Testing
• User Acceptance Testing (UAT)
• Alpha/Beta Testing.

So our goal is to find edge cases. Looking at the list, all the techniques could prove helpful. But some may require too much investment so late in the development (Scenarios). Some are a poor fit for an e-commerce shop (alpha/beta testing, UAT). Which leaves us with Usability Testing and Exploratory testing. ATQ is not an extensive matrix, it contains only examples of techniques rather than a complete list.

For example, a different technique that could help is Dogfooding where company employees themselves use a product to satisfy their everyday needs.

Example 2: Messaging App

A company has created a very successful messaging app, but now they are facing the problem of unexpected success. Their small app, designed for a small community, is rapidly gaining popularity. The team feels that the app is not ready for all this level of attention. The situation is clear - from a business perspective, everything works well enough. The risk is technology, and again the problem lies in the critique of a product. This leads us to Q4 where we have, among others, Performance and Security testing. There is no doubt that we will need both. Lots of users means a load on an application and nothing will kill its momentum like failing servers. Security is also important. Attention also attracts bad actors who will try to exploit the system.

Example 3: Early Development

The team has a problem with poor code quality. In the case of some features, developers are reluctant to fix a familiar bug because of the high risk of breaking something else. One glance at the ATQ shows that we need something to support the development and its technology which leads us straight to Q1. Both Unit Test and Component Test would improve team productivity and code quality.

The above example may give a wrong impression that Agile Testing Quadrants are used to select one and only one tool. THIS IS NOT TRUE: we can use ATQ for each goal of our testing, for each need, for each problem we are trying to address. However, trying to improve too much at once may lead to too much-wasted effort and no result. Concentrate on one subject at a time.

Evolution

While the concept of AQT has changed little over the years, the techniques in each quadrant have evolved, for example:

In More Agile Testing, the author presented a new version with Micheal Hüttermann changes, with a circle of Outside-in, Barrier-free, Collaborative in the middle of the quadrants. The author believes that it will stretch beyond any quadrants. The main issue with this update is that it is a catchphrase and provides no information about the tools and techniques that may help.

A few years back, Lisa and Janet also presented a version more in line with DevOps. This version adds a lot more techniques to each of the quadrants but also changes one side, from Supporting Team to Guide Development. While it may seem like a single name change, fortunately, the original name leads to many misunderstandings and the new label is much more descriptive.

Michael Bolton and James Bach have also made the above version which is compatible with Rapide Software Testing. Most of these updates are only fine-tuning, adding elements that their author thinks are better in this context. So which one to choose? It is the author’s opinion that the original version holds well? It is still relevant, because:

It is just a model.

“All models are wrong, but some are useful” is an old aphorism attributed to George Box. And it is important to bear it in mind in the case of Test Pyramid, Agile Testing Quadrants, etc. Models are what they are: models which need to help us think about a problem. **They are not the golden rules that should be followed to the letter.**They provide us with a good way of tackling the problem. But it is up to the user to decide if a specific model will work for a specific problem.

Summary

The Agile Testing Quadrant is a model which is nearly 20 years old. Owing to its simplicity, it is still relevant. It is a great tool for teams and testers who have a problem with choosing the right testing method. For junior testers, it is a great place to look into different tools and approaches, and for more advanced testers it will be an excellent aid in building or improving the testing process.

Happy (automated) testing!