Smoke Testing vs Regression Testing: Guide for QA

There are many types of software tests. Some of them overlap or are responsible for similar testing aspects. This can be confusing, especially for novice testers. In this article, we will show the differences and scopes of responsibility of smoke testing and regression testing. Both of them are used to verify if changes in the code are negatively influencing any existing functionality.

Let's not perform spoilers though! So dig thru the smoke and see if your testing knowledge did not regress recently.

Smoke Testing

Smoke testing is a type of test that aims to confirm whether the basic functionalities of your software are working as intended. It might be also known as build verification testing. We created a separate article on smoke testing process. It is performed at the very beginning of the software's life cycle.

Preferably on the very first software build. In general, performing smoke tests serves two goals.

First, it ensures that the most basic functionalities work properly. It does not have to be pretty, you do not have to move that logo 2 pixels left or right, it just has to do the job. If all currently existing features (even if its just logging and CRUD) work properly, you can move on to different areas of the software.

Secondly, it allows you to identify critical issues with your software.

Basically, it is an opposite scenario to the one above. Smoke testing helps to identify what should be working but fails to do so. There is no point in moving along with works if the basics fail to execute. Sticking to the logging example - you will not able to test the features available for the logged user if you can not log him/her in, right?

See our other article Smoke Testing vs Sanity Testing.

Pros and Cons of Smoke Tests

Smoke testing is like triage in a hospital. First thing that you do in triage is checking if the patient is dead or alive. And this is basically what the testing team does. Smoke testing is executed to estimate if we can proceed with the next stage of the software development cycle. The QA team needs to provide fast feedback and let everyone know if the app is stable enough to hold new functionalities. Early bug detection is another reason why smoke testing is done.

There are two sides of this coin actually. There is a certain level of randomness while this testing is performed. It may miss those issues, that are considered subtle. Of course the level of subtlety is defined test case after test case. One would consider a typo subtle, other will agree to SQL injection as subtle (please don't). The other key disadvantage would be very limited test coverage.

Regression Testing

Guess what? We have an article on that one as well - What is Regression Testing?.

Regression testing is a software testing method that aims to investigate and locate any possible flaws or bugs in software that has already been fully developed. Regression testing is a process that is performed to verify if the software is still functional after bug fixes or new features were implemented. It is especially important for those teams that work in an agile way (scrum especially) since in theory they implement new features every sprint.

This much more detailed testing approach is one of the main reasons for test automation. With every feature released, you need to check more and more dependencies so your testing phase gets longer and longer.

Automation saves you a lot of effort and money. Regression testing is performed to ensure the stability of your software after any change (even from the software/hardware standpoint).

Catching the so-called "regression bugs" is also a key reason why you perform regression testing. It is quite common in the software development world that some bugs emerge or even re-emerge whenever there are changes made in the code. The number of test cases here is way bigger than in smoke tests and it increases basically every release.

Pros and Cons of Regression Testing

Well first of all you have to mention the regression bugs again here. Imagine your marketing department spending a wagon of cash to advertise the "exciting new features" of your applications. Then zillion of customers download the update and the whole thing just crashes. Not the best case scenario, isn't it?

Regression testing is used to avoid such circumstances. Another reason why such testing takes place is to ensure that your software is reliable.

On the other hand regression testing should occur regularly after each update. This means that you launch the whole last testing scenario and then some. It is time and resource consuming. Those tests cannot really just scratch the surface. The testing tools have to work overtime for a really thorough testing. Also, such a rigorous testing might cause your testing scripts to require regular maintenance.

This is a good place to emphasize the importance of proper test documentation. If you fail to prepare it, your developers and testers will waste hours on checking "where on earth did we test that form".

Smoke Testing vs Regression Testing

Smoke and regression testing have a few things in common, but in general those are two separate testing types. Here are the key differences between both:

• Timing - smoke testing is performed rather early in the development process, it is basically surface-level testing. On the other hand, regression testing never occurs during the initial builds, but is used later in the software development life cycle.
• Scope - you use smoke testing to address potential bugs of core functionalities whereas the regression testing process has to have as large coverage as possible.
• Test cases - when it comes to smoke testing a relatively low number of simple test cases is used, but regression testing cases need to be very extensive, they have to cover all bases.
• Frequency - one of the most subtle differences between smoke testing vs regression testing is how often they are executed. Smoke testing occurs basically every build and regression testing as the more extensive testing takes place whenever changes are implemented in the app.
• Goal - those two types of testing focus on different stages and aspects. Smoke tests focus on new features and regression testing check if those (previously checked in separate tests) functionalities do not break the whole app altogether
• Environment - smoke tests are usually executed in a clean or separated environment. Regression testing is carried on a stable environment which should be as similar to production as possible

Difference Between Smoke Testing and Regression Testing

Both regression tests and smoke tests are a must-have for your testing strategy. Understanding the differences between them is also crucial. Smoke testing takes place earlier in the testing process and speaking in a non-technical way is similar to random bug hunting. Smoke testing reduces the risk of proceeding with development of a malfunctioning application.

Regression testing ensures that the already implemented functionality works properly with others, that were in place before. This is more advanced testing than smoke tests. Longer your development takes (more iterations), the more your app requires regression testing. Last but not least - regression testing is a great place to implement BugBug or another test automation software since this type of tests are very repetitive.

Smoke testing requires more manual work since you test different functionalities per release. The manual work pays off though, since much of it can later be introduced to regression test suites.

Happy (automated) testing!