Testing in DevOps

As a movement, DevOps is over 10 years old, and it is one of the most dominating approaches to modern software development. In some companies, however, testing in DevOps has become a critical issue.

Lots of old methods and techniques are no longer good enough. DevOps requires changes in testers’ mindsets. Not everybody could adopt to this new era. Today we will try to help. We will look at some practices and techniques that actually work! We will also talk about quality mindset and look at automation. But first, we need to answer a few questions about DevOps and Testing, starting with:

What do we mean by DevOps?

As with everything, DevOps has become a generic term. Everybody understands it differently:

• for some, it is about tools like CI/CD,
• for others it is a role in a company - usually a cloud and a CI/CD expert.

We will use a definition by Gene Kim, author of books like Phoneix project and a DevOps handbook, one of the greatest evangelists of DevOps. In short: DevOps is a culture which helps to create learning organizations which can outperform competition, thanks to fast learning and experimentation enabled by fast delivery of the product.

In Beyond the Phoenix project, Gene Kim summed it up as 3 ways of DevOps:

• All about Flow,
• All about Feedback
• All about Learning

In short: You need to deliver a product fast to see how it works and learn what to improve, then repeat the cycle.

What About testing?

Testing is good about collecting feedback and giving information, so it can be helpful with the feedback part. But what about speed? Management Often sees testing as cost and a bottleneck. Some even say you can’t have a good quality product if you are going fast. As we learn later, this is not true, but first, let’s discuss another important topic:

What is quality?

I want you to answer one question. Which product is of better quality:

• both are doing similar things,
• both have been on the market for some time
• one is a popular product with millions of users and known bugs in hundreds.
• the other one has less than a thousand users but virtually no bugs.

So which is of better quality? At first glance, the other one feels as a better answer since it has fewer bugs. But commercially it is a failure, it barely has any users! Users chose the other one - maybe it was first on the market, maybe has had better marketing, maybe a better price. The fact remains that, despite its all flows, the product is good enough for the users. We call this shippable quality, a product that good enough for users doesn’t need to be perfect as long as it satisfies the users’ needs. The first thing you need to understand about Testing in DevOps is that shippable quality today is better than perfect quality tomorrow!

Quality at Speed in DevOps

There is also one interesting paradox: the faster and more often you deploy, the better your product quality. Why? In short:

1. First, you deploy frequently,
2. Each deployment contains less code, instead of months ita takes weeks, instead of weeks it takes days or even a single story.
3. Less code means fewer merging issues, fewer dependencies to check.
4. It also means smaller changes that are easier to test.
5. Resulting in fewer bugs.

It means you can also deploy quick fix if we find a serious issue.

The Goal of Testing in DevOps

Summing all the concepts, we add all this together. The goal of testing in DevOps is clear: Empower teams to achieve testing as fast as possible.

Ways of Testing in Devops

Shift Left

To put it simply, Shift left is about testing as early as possible - and it doesn’t mean testing the finished product as early as possible. It means that, if you can find a partial concept or even an idea that you can verify before everything is done, you should do it at first chance.

A few Examples: Start with the documentation. If the team has refinements, or there is a meeting where new features/stories are discussed, this is the place to put on the testing hat and do some lateral thinking. Pinpointing the issues here will save time later.

This is the time to ask a few important questions:

• How are we going to test it?
• What can we do to test it as fast as possible?

If the team is too big and the entire team meeting is out of the question, it is a good idea to have a 3 amigos meeting with the developer, the tester and the product owner/analyticto discuss how to develop a story and how to test it. The goal of this discussion should be to decide what needs to be automated and what to check manually.

Another example of Shift left is Dev Testing - instead of having a dedicated testing expert, developers with enough training can very well test their own code. It requires practice and discipline but canal in all, it saves a lot of time on the hand over imagine standard approach:

1. developers write a code;
2. the code waits for testing
3. the testers test and find bug
4. the bug waits for the developer to have the time to fix it
5. the developer fixes it
6. the bug waits for retest.

You get the idea. Now let’s look at a much shorter loop:

1. the developers write a code
2. they test it and fix issues immediately
3. Deployment

As The Goal has showed us, a physical product wastes a lot of time just waiting to be worked on by different stations. Our work is not so different. Of course developers need to be first taught how to test - and there are ways to do it: Dev-Test Pairing, or Session Based Testing with debriefings. There is a lot more to shift left, but in general you can describe it in one phase: test as soon as possible.

No code automation

Automation is a vast topic which we won’t cover completely here. Test automation should be a part of shift left, meaning: Automate as early as possible and only as much as you need. There are two tools that help in deciding what and how should be automated – a Test Pyramid and Testing Quadrants.

However, the basic rule of thumb isthat if something can be done as unit test simply it should be done.

Unfortunately, even using this rule, we still need some end-to-end testing - this type is difficult to write and hard to maintain. Flaky(unstable) test are the bone of the industry. Even the best companies, like Google, have problems with achieving 0% of flaky tests when creating end-to-end tests using tools like Selenium Web Driver or Cypress.

Usually, writing and stabilizing an end-to-end test takes a long time. Now imagine that you want to deploy often, but practically each new story requires fixing/changing the test automation, which takes a long time and slows you down.

Fortunately, the popularity of no-code leads to the rise of a new generation of Test Automation Tools like BugBug.io and Mabl. Both allow to create a stable test in minutes! Previously, the biggest issue related to this tool was that if a test was failed, your only option was to rerecord it. While fixing these test is much simpler, remember that Testing in DevOps is about speed!

Even if you would have to rerecord the test, it still will be faster than fixing a coded end to end test Written in Selenium Web Driver or Cypress.

Monitoring and Testing in production

Unit and integration testing is all very nice but your system testing is also important. You need to make sure that everything works nicely together. There is one problem: the test environments.

Usually they are not really on a par with production - they may have too low resources, run on different OS and many, many other issues. Even in the staging environment, often it is not a good representation of the production environment. Yes, in theory they should but in reality, if you want to test in an environment resembling production, there is no better place than production itself!

Ok, but how do you do that? There are lots of approaches. For example, you can do cannery releases of your application to see how it works in the wild.

If you have a/b testing - you can use these tools to hide all the new good things behind the a/b test and, after team ensures it is working, you can slowly start to show it to more and more users. Another approach is blue/green deployment.

However, first YOU NEED TO HAVE GOOD MONITORING - with good observability. especially of user activity. Remember what DevOps is about: “all about flow, all about feedback, and all about learning” - monitoring helps with both feedback and learning! With a good rollback strategy or feature flags, you can safely deploy a code to production. If something goes wrong, swiftly disable it.

If you want to learn more, Cindy Sridharan runs an amazing blog post series on this topic.

Quality Culture

We have talked about DevOps and we have talked about testing in it; there is one more issue to address as even the best tools and technical solutions won’t save the day if there is no culture promoting quality. In order to create this culture, we need good principles. Fortunately, Modern Testing comes to the rescue, trying to address the necessary cultural changes for testing in DevOps. The basic principles include:

1. Our priority is improving the business.
2. We accelerate the team, and use models like Lean Thinking and the Theory of Constraints to help identify, prioritize and mitigate bottlenecks from the system.
3. We are a force for continuous improvement, helping the team adapt and optimize in order to succeed, rather than providing a safety net to catch failures.
4. We care deeply about the quality culture of our team, and we coach, lead, and nurture the team towards a more mature quality culture.
5. We believe the customer is the only one capable of judging and evaluating the quality of our product
6. We use data extensively to deeply understand customer usage and then close the gaps between product hypotheses and business impact.
7. We expand testing abilities and know-how across the team; understanding that this may reduce (or eliminate) the need for a dedicated testing specialist.

Summary

Today we started with the same definition of DevOps - culture concentrating on flow, feedback and learning.

We have explained the goal of testing in DevOps - empowering teams to achieve shippable quality as fast as possible.

Then we looked at a different way to help us achieve this goal from shifting left, and no-code test automation, to testing on production, ending on testing culture.

Notably, at first many of the discussed issues were not directly related to testing.

Most of the technical solutions require changes either in the process the deployment strategy or the code structure.

This is another important aspect of Testing in DevOps - testing cannot be left to the end. Just like quality, it needs to be built into both the product and the process. Since the beginning, it cannot be an afterthought.

If there is one thing that can bring your team closer to excellent DevOps testing, start with asking these questions: How can we test it? When is the first opportunity we can test it?