Agile Testing Quadrants

Testing can be difficult, it is hard to master, there are many questions and many issues we can face with each new feature, each new story. Every day, testers need to decide what to test, how to test, what to automate, and what risks to accept. These decisions are hard and require a lot of experience. Fortunately, there are techniques, heuristics, and models which can help with planning tests. One of the oldest and still very useful techniques is Agile Testing Quadrants (ATQ). it is a simple table that helps us select the tools we need to use, depending on the required outcome. In this article, we will look into the history of Agile Testing Quadrants, their uses, and limitations. We will also discuss the attempts at modernizing it.

Agile Testing Quadrants or Agile Testing Matrix?

First, we need to clear one thing: as with everything in IT, even this simple table needs to have many names! Chances are that you will come across it as an Agile Matrix, Testing Matrix, Testing Quadrants, or Agile quadrants. All of them refer to the same thing. The only exception is the Agile Testing Matrix which may refer to both Agile Testing Quadrants and its precursor the Named Agile Testing Matrix.

The Story of Matrix

When speaking about Testing in Agile, there is probably not a single more important person to its development than Lisa Crispin and Janet Gregory. Their book, Agile Testing, has helped many testers. Testing Pyramid was born while Mike Cohn tried to explain the nuances of automation to Lisa’s team (Experience of Test Automation - chapter 1 Section 1.3.) Agile Testing Book played a key role in the popularization of Agile Testing Quadrants, where the creation of ATQ is often misattributed to this wonderful duo. As Lisa Crispin explained in her blog - they improved on the older concept by Brian Marick - an Agile Testing Matrix created as part of his blog series presenting his vision of testing in Agile. So Agile Testing Quadrants have almost 20- year-long history and they are still one of the best models used for testing.

The Usefulness of Agile Testing Quadrants

Agile Testing Quadrants have two axes which split the matrix into 4 quadrants. Both axes pose the same questions, namely What is the goal of your testing? The major difference between the two axes is the flavor of the question: the horizontal axis asks what is currently more important: helping the team to deliver faster or ensuring that the product is as good as possible? the vertical axis asks if we are concentrating on the aspects of technology, ensuring that everything is working well, or are we looking from the perspective of business and customers, making sure that our program is worth using? Obtaining both answers points to one of the Quadrants with a list of useful examples. This can be a little confusing, so let’s look at some of the examples.

Example 1: E-commerce shop

Let’s say we have e-commerce shop development that is going nicely. However, there is one enormous risk on the horizon, namely quite a lot of features, each tested separately so no one is sure how they will work in the real world. The team believes that the standard paths will work. However, no one is sure what kind of edge cases will come from the real world.

Looking at our matrix we can see that problem is definitely business-facing - we don’t know what the user will do. We also need to make sure that our product will handle it, ergo - Critique Product. Looking at it we land in Q3 where we have propositions of the following techniques:

• Exploratory Testing
• Scenarios
• Usability Testing
• User Acceptance Testing (UAT)
• Alpha/Beta Testing.

So our goal is to find edge cases. Looking at the list, all the techniques could prove helpful. But some may require too much investment so late in the developments (Scenarios). Some are a poor fit for an e-commerce shop (alpha/beta testing, UAT). Which leaves us with Usability Testing and Exploratory testing. ATQ is not an extensive matrix, it contains only examples of techniques rather than a complete list. For example, a different technique that could help is Dogfooding where company employees themselves use a product to satisfy their everyday needs.

Example 2: Messaging App

A company has made a very successful messaging app, but now they are facing the problem of success. Their small app, designed for a small community, is gaining popularity fast. The team feels that the app is not ready for all this attention. The situation is clear - from a business perspective, everything works well enough. The risk is technology, and again the problem lies in the critique of a product. This leads us to Q4 where we have, among others, Performance and Security testing. There is no doubt that we will need both. Lots of users mean a load on an application and nothing will kill its momentum like failing servers. Security is also important. Attention also attracts bad actors who will try to exploit the system.

Example 3: Early Development.

The team suffers a problem of poor quality code. In the case of some features, developers are reluctant to fix a familiar bug because of the high risk of breaking something else. One glance at the ATQ shows that we need something to support the development and its technology which leads us straight to Q1. Both Unit Test and Component Test would improve team productivity and code quality.

The above example may give a wrong impression that Agile Testing Quadrants are used to select one and only one tool. THIS IS NOT TRUE: we can use ATQ for each goal of our testing, for each need, for each problem we are trying to address. However, trying to improve too much at once may lead to too much-wasted effort and no result. Concentrate on one subject at a time.

Evolution

While the concept of AQT has changed little over the years, the techniques in each quadrant have evolved, for example:

In More Agile Testing, the author presented a new version with Micheal Hüttermann changes, with a circle of Outside-in, Barrier-free, Collaborative in the middle of the quadrants. The author believes that it will stretch beyond any quadrants. The main issue with this update is that it is a catchphrase and provides no information about the tools and techniques that may help.

A few years back, Lisa and Janet also presented a version more in line with DevOps. This version adds a lot more techniques to each of the quadrants but also changes one side, from Supporting Team to Guide Development. While it may seem like a single name change, fortunately, the original name leads to many misunderstandings and the new label is much more descriptive.

Michael Bolton and James Bach have also made the above version which is compatible with Rapide Software Testing. Most of these updates are only fine-tuning, adding elements that their author thinks are better in this context. So which one to choose? It is the author’s opinion that the original version holds well? It is still relevant, because:

It is just a model.

“All models are wrong, but some are useful” is an old aphorism attributed to George Box. And it is important to bear it in mind in the case of Test Pyramid, Agile Testing Quadrants, etc. Models are what they are: models which need to help us think about a problem. They are not the golden rules that should be followed in the letter. They provide us with a good way of tackling the problem. But it is up to the user to decide if a specific model will work for a specific problem.

Summary

The Agile Testing Quadrant is a model which is nearly 20 years old. Owing to its simplicity, it is still relevant. It is a great tool for teams and testers who have a problem with choosing the right testing method. For junior testers, it is a great place to look into different tools and approaches, and for more advanced testers it will be an excellent aid in building or improving the testing process.