Software Testing Knowledge

Professional Quality Evangelist, experienced in testing web and mobile applications.  
[linkedin.com/in/kinga-witko-b5bba659/](https://linkedin.com/in/kinga-witko-b5bba659/)

thumbnail_eq_El_DXL_400x400_b440e8a71d.jpg

eq_El_DXL_400x400_b440e8a71d.jpg

Kinga Witko

[TABLE_OF_CONTENTS]

The term “shift left” expresses movement. It can describe the movement of an entire team or individuals.
What does it do in a project or an entire development lifecycle? Putting it simply,  it reflects the position of testing on a timeline. 

**Shift Left means testing earlier in the development process.** It also means repositioning the engagement, development, and testing efforts towards the customer.


Typically, in a traditional (waterfall) model of software making, testing lands at the end of the pipeline or is one of the last phases of the process. Bringing testing at the very last stage usually causes many problems such as:

- delays in the deployment
- a lot of bugs at the end of the development lifecycle
- bugs discovered just before or after the release to production
- lack of collaboration between developers and testers through the entire process.

Poor software quality is one of the most harmful consequences of all of these pains listed above.

At first, the team members do not communicate with each other and they fail to communicate the problems early. 

It must not necessarily be a bug; sometimes there is some issue with the application setup, there is no testing environment, access to certain resources (such as specific tooling or external software providers), the task description is vague, or the team setup is incorrect. 

If communicated and fixed earlier, the entire process can be smoother and cause less frustration in the team. It also encourages people to focus more on the code quality and to deliver earlier and faster.

Most of those issues can be avoided by changing the style of working and shifting testing efforts at the beginning of the process.
->[Automate your tests for free](https://app.bugbug.io/sign-up/){.markdown-cta}<-


## Agile way of working

If you care about the quality of your product, your mindset has probably shifted left already.

This approach is strongly connected with the agile way of thinking and using agile work methods in software projects.


In this setup, testing is an integrated part of the team and accelerates software development at higher quality. As a result, testers can identify bugs sooner, saving time and effort. Notably, the earlier you test, the less expensive bugs are to fix.


[T The Seven Testing Principles (ISTQB)](https://www.istqb.org/downloads/syllabi/foundation-level-syllabus.html) tell that testing everything is not possible unless a system or functionality is very simple. In practice, however,  the testing efforts should be targeted appropriately at the risk analysis, techniques, and bugs prioritization rather than be attempted thorough testing. 


One of the good methods of approaching testing strategy for a specific application can be the guideline from [Agile Testing Quadrants](https://bugbug.io/blog/software-testing/agile-testing-quadrants/). The quadrants are a taxonomy to help teams plan the testing process and testing activities and make sure they have all the resources they need to accomplish it.


## Shift Left approach and DevOps and agile background

With the rise of the DevOps movement, there is an increased focus on delivering automated, continuous improvement. It holds true for development and testing alike.

What is more, Continuous Integration (CI) and Continuous Delivery (CD) are core software development disciplines integrated into the agile software development cycle. The point is to release fully-tested and stable builds early and as often as possible. Development, delivery, and test automation are crucial components of this process. 

## What it means to “Shift Left”

As mentioned at the beginning, Shift Left testing pushes testing to the “left”, to the earlier stages in the pipeline. This improves software quality and reduces time spent on resolving issues later in the pipeline.

Introducing testing early on the “left-side” of the development process, starting with unit tests and complementing with functional and integration tests is fundamental to making CI/CD happen. But that may not be enough.

The team can go even further: to eliminate unclear and contradicting requirements and improve the end-user experience, it is best to involve test engineers in the clarifying and prioritising application requirements. Test engineers can check the available or newly created requirements or user stories for clarity, consistency, testability, and traceability.

This also improves the shift towards client happiness. An early review of the requirements and spotting the flaws marks an early start of the communication with the end-user. 

The Shift Left approach introduces more care and thought exchange concerning the client. The team can be perceived as more proactive and more dedicated to the best quality standards of a project.
->[Automate your tests for free](https://app.bugbug.io/sign-up/){.markdown-cta}<-


## Benefits of Shift Left Testing for Agile and Devops teams

Agile teams benefit most from Shift Left Testing in terms of clearer specifications, smoother implementations, earlier testing, and reduced costs. However, Shift Right Testing has more benefits like the ability to incorporate user and system feedback based on real usage scenarios. 

In opposition to testing early, Shift Right can provide an additional context to the testing scenarios. Shifting right means testing at the end of the process, typically when the software is deployed to production (the post-production environment). 

It is performed by a group of experts or selected end-users. It can utilize testing methods such as API Monitoring, using Feature Toggles, or measuring the performance of a live application. 

This helps to understand the application better and discover some other flaws that could have been omitted earlier.

Nevertheless, when quality comes first:

- developers spend more time on coding because they discuss potentially good solutions - they do not investigate who pushes the code first into production without testing

- there are Unit Tests beside the application code, available and maintainable

- there is an infrastructure that allows testing before pushing to production, usually the entire team is responsible for the process health

- there is room for improvement and mistakes

- developers pair up to catch obvious errors on the fly and learn from one another

- teams estimate tasks together, taking quality and risks into account

- potential risks in the systems are spotted and minimized rather than left alone

And finally - the application is tested - not only by one automated or manual test  but by a set of different functional and non-functional scenarios. 

The tests are regularly reviewed, adjusted, and observed. This approach introduces a new way of working and highlights the value of testing.


## Benefits of the Shift-Left Testing approach

The idea of shift left testing is simple to understand but hard to implement in the development cycle. It is worth the effort though, especially when the team and the entire organization understands the main benefits of this approach, such as:

- Lower costs - as explained above, bugs detected earlier are less expensive to fix.

- Increased efficiency in the software development process.

- Improved product quality and reliability because more bugs are detected at earlier stages.

- Reduced time to market since the QA process covers the entire development cycle, not only its’ last phase.

- A competitive advantage gained by letting development teams move fast, build and defend their credibility by innovating in iterations.
->[Automate your tests for free](https://app.bugbug.io/sign-up/){.markdown-cta}<-


## Get started with “Shift Left” Testing in Agile

How to approach the change and switch to early software testing? Here is the basic list: 


1. **Plan**

Shape development in a way that is friendly to Shift Left testing. Once the requirements have been established, developers and testers should ideate how to shape the coding and software testing as early as possible. 


Depending on the product expectations as well as the limitations of the team, specific strategies, such as involving testers in the project since the planning phase or using the TDD approach, might have to be crafted to accommodate the Shift Left way of working.


2.** Review the requirements**

It is best to verify not only the existing code but also all the requirements necessary for the code to be written. Early verification can spot obvious mistakes.


3. **Use Static Code Analysis**

Static code analysis is the practice of checking the code without executing it. It runs through the basic code structure and ensures that the code is aligned with many standards and guidelines. By following the approach the team can early detect:

- Programming errors

- Violations of commonly followed coding standards

- Syntax anomalies

- Security issues

A static code analysis is best done via automation. The software should scan all untested codes in a project before it moves to the test phase.

4. **Collaborate**

As in any other [agile testing](https://bugbug.io/blog/software-testing/agile-testing-organized-chaos-or-a-planned-way-of-improving-your-software/) technique - early software testing requires a lot of collaboration with the team: Developers, Product Managers, Business Analysts, and other stakeholders responsible for the overall product. 

Everybody needs to understand the value of the Shift Left approach and spot the difference after some time of the new process introduction because as a result, the client benefits from better software quality and receives a lovable product.

## Conclusion for Left Shift Testing in test automation

There are a number of benefits to left shift testing in test automation. One of the primary benefits is that it helps to identify defects earlier in the development cycle. This means that defects can be caught and addressed before they become more complex and costly to fix. In addition, left shift testing can help to reduce the overall time and effort required for testing by identifying and addressing issues earlier in the process.

Another benefit of left shift testing in test automation is that it can help to improve the quality of the software being developed. By identifying and addressing defects earlier in the process, developers can ensure that the software they are creating is of higher quality and meets the requirements of the end-users.
->[Automate your tests for free](https://app.bugbug.io/sign-up/){.markdown-cta}<-


One of the keys to successful left shift testing in test automation is to incorporate testing into the development process **from the very beginning**. This means that testing is not an afterthought or an additional step at the end of the development process, but rather an integral part of the development process itself. Test automation tools and frameworks can be used to automate testing activities and make it easier to incorporate testing into the development process.

It is also important to involve testers in the development process as early as possible. This allows them to provide feedback and identify potential issues before they become more complex and difficult to fix. Testers can also work with developers to ensure that the code being developed is testable, and that the testing process is efficient and effective.

Overall, **left shift testing is an important approach in test automation that can help to improve the quality of software, reduce testing time and effort, and identify defects earlier in the development process**. By incorporating testing into the development process from the very beginning, developers and testers can work together to create software that is of higher quality and meets the needs of the end-users.

large_pexels-pixabay-40716.jpg

medium_pexels-pixabay-40716.jpg

small_pexels-pixabay-40716.jpg

thumbnail_pexels-pixabay-40716.jpg

pexels-pixabay-40716.jpg

Shift Left Testing Explained

shift-left-testing

A self-taught software QA Engineer, Dominika has experience in web testing. Currently, she is focusing on developing her test automation skills with Cypress. Apart from programming, she likes to make pottery and cook vegetarian delicacies.  
[linkedin.com/in/dominikanykiel](https://linkedin.com/in/dominikanykiel)

medium_1637439945009_e_1646265600_and_v_beta_and_t_Iw_Bm_GA_Wm_Nd_Xvpr_R_Nzs_LB_Pbwu_PFXBTK_Ff_TLP_612y_Eu_A_4d4900e8b2.jpg

small_1637439945009_e_1646265600_and_v_beta_and_t_Iw_Bm_GA_Wm_Nd_Xvpr_R_Nzs_LB_Pbwu_PFXBTK_Ff_TLP_612y_Eu_A_4d4900e8b2.jpg

thumbnail_1637439945009_e_1646265600_and_v_beta_and_t_Iw_Bm_GA_Wm_Nd_Xvpr_R_Nzs_LB_Pbwu_PFXBTK_Ff_TLP_612y_Eu_A_4d4900e8b2.jpg

1637439945009_e_1646265600_and_v_beta_and_t_Iw_Bm_GA_Wm_Nd_Xvpr_R_Nzs_LB_Pbwu_PFXBTK_Ff_TLP_612y_Eu_A_4d4900e8b2.jpg

Dominika Nykiel

I write about technology and help people who use it. I've always been fascinated by the details of things - how they work and what makes them tick. That's why I love writing about tech!  
[linkedin.com/in/aleksandra-spilkowska](https://linkedin.com/in/aleksandra-spilkowska)

large_a_spilkowska_cropped_c38f3487ad.jpg

medium_a_spilkowska_cropped_c38f3487ad.jpg

small_a_spilkowska_cropped_c38f3487ad.jpg

thumbnail_a_spilkowska_cropped_c38f3487ad.jpg

a_spilkowska_cropped_c38f3487ad.jpg

Aleksandra Spilkowska

[TABLE_OF_CONTENTS]

Why do we perform software testing? To find defects. That is the simple, or even simplistic, answer. We want software to run smoothly in a production environment and perform as expected. Seems pretty straightforward, doesn't it? But it's always worth double-checking when quality is at stake! There are crucial steps in a software development life cycle (SDLC) that, if followed, make the product reliable and more well-rounded. Before moving on to the importance of testing, let's take a quick look at different them.



## Different types of software testing

Typically, if testing is implemented in the SDLC, it would be divided into several layers. The first would be unit tests which are usually simple and fast, as they are only performed on a small module (unit) or a portion of the code. The advantages are speed and simplicity, as the developers are quickly alerted if a unit test fails so that they can review the part of the code that does not work correctly. 

Once individual units have been created and tested, it's time to perform integrated testing, which tests their interfaces. Individual modules can then be combined and tested as a group. This layer of testing determines how larger parts of the software work together and how they interact with third-party tools or APIs, if applicable. It may be simple or quite complex, depending on the software itself. It is advantageous if several developers are working on different units or changes were made to one of them. 
->[Automate your tests for free](https://app.bugbug.io/sign-up/){.markdown-cta}<-

The best practice is test along as more modules are created. If testing starts when the system is already practically integrated, and defects are found (and, let's face it - likely there are some), finding why and where they are in the software may be especially difficult due to the multitude of interactions and interfaces. 

Finally, [end-to-end testing](https://bugbug.io/blog/software-testing/what-is-end-to-end-testing/) focuses on the performance and capabilities of the software as a whole, including the entirety of tasks it can do. The goal is to simulate what actual use would be like from start to finish. It validates the main, as well as the interconnected systems. 

Interaction accuracy is ensured at this stage to ensure consistent user experience across multiple platforms, devices, and environments. Moreover, end-to-end testing increases test coverage, which helps build confidence in the success of the testing process. 

That is extremely important nowadays, as software becomes increasingly complex, often relying on an intricate network of subsystems and layers. In such cases, the system's stability depends on its parts' functionality. The definite downside of this approach is the cost. The all-encompassing premise of end-to-end testing and the overall monitoring required make it an extremely time-consuming test method. Tracing a bug can take even more time if found at this point, and tracing its cause can take even more time. 

After this brief overview of layers of testing, let's move on to what particular advantages it may bring.

## Importance of software testing and advantages

In today's fast-paced technological landscape, [software testing](https://bugbug.io/blog/software-testing/the-beginner-s-guide-to-software-testing-basics-types-and-methodologies/) has become an essential aspect of the software development process. It involves the evaluation of a software product or application to ensure that it meets the specified requirements, functions as intended, and is free of errors or bugs. In this section, we will explore the importance of software testing and the benefits of testing.

### Saving time and money 

Not only does money make the world go round, but it also tends to be one of the most important factors people consider when making decisions about purchasing or investing in developing software. If testing is neglected and there is a bug in the system (practically a certainty), it will need to go back to the developers to be fixed, which means more time and energy being put into the product. And, as the old aphorism says - time is money. Particularly in software development, more time undoubtedly equals extra cost. The faster the testing starts, the less likely it is possible that an evident defect will occur during end-to-end testing when it is considerably more expensive to fix. 


### Ensuring product quality 

Detecting defects and errors early dramatically reduces the risk of insufficient software quality. Naturally, the last thing one would want is for customers to find bugs in the software. Worse yet, the bugs can potentially make using the application – and thus making money on it – impossible. Testing, understood here as a synonym for quality assurance, relates to quality control through the SDLC and is meant to prevent such things. 

Testers can be involved in requirements reviews or clarifying user stories at the earliest stage. The stories help detect defects. Identifying and fixing these defects reduces the risk of producing incorrect or untestable software features. Close cooperation between testers and software developers at the design stage allows both parties to better understand the system design and how it should be tested. The better knowledge of the subject, the better the result. To ensure product quality, the tester must be aware of the risks of defects in the code and potential failures. 
->[Automate your tests for free](https://app.bugbug.io/sign-up/){.markdown-cta}<-

### Managing risk 

Risk in software testing is the possibility of an event that would have undesirable consequences in the future. The risk level is determined by estimating the probability of such an event occurring and its potential negative results. Analyzing risk can be used to plan testing. The goal is to identify the areas that need more attention—focusing on those reduces the probability of an undesirable event occurring (or at least helps manage its impact). The risk-based testing approach contributes to risk mitigation and provides stakeholders with valuable information on identified risks. They may then decide if the issues need immediate attention. When such an analysis is performed early, it can contribute to the entire project's success. 


### Customer satisfaction 

Product quality is intrinsically linked to customer satisfaction. Banking on cheap products barely serving their purpose is a short-lived business strategy. Quality is the way for those who want to establish a reputable brand and attract loyal software users. 

Anyone who has ever managed a large-scale project can attest that assessing a client's needs may save a lot of headaches later. Some products must fulfil a set of requirements based on agreements, laws, norms, and standards. Proper verification helps ensure that. Checking that those requirements are met prevents putting potentially unusable software into the user's hands, which can later prove costly, not to mention the loss of intangible assets, such as the client's trust. In the long run, testing doesn't just prevent failures and helps establish the product as secure, reliable, and easy to use, subsequently attracting more clients. 


### Security 

Last but not least, testing makes the product more secure. For this reason, security testing is a crucial aspect of software testing. It is meant to find weaknesses and vulnerabilities in the software to protect it from potential threats. Why is that so important? First of all, people are more likely to use trusted products. Secondly, security failures can damage the company's image and income. 


As the existing laws in the European Union protect personal and sensitive data, uncontrolled leaks can result in millions in fines and penalties. Security testing should ensure that the identity of those with access to personal information is adequately verified. Data should be available, correct, and safe from accidental leaks and nefarious attacks. In extreme cases, security breaches can lead to the whole software not working, which can mean no business continuity and high revenue losses. Like most types of testing, finding a defect typically costs more if done later and even more (fines, loss of credibility, etc.) if done improperly or skipped altogether. 



## Conclusion 

Considering its crucial role in ensuring flawless system performance, testing should be considered a necessity, not just an option. The negative consequences of passing faulty software to users are evident. Apart from their base task of finding defects, tests also have an instrumental significance for the SDLC, as they may contribute to improving software quality as a whole.

large_pexels-egor-kamelev-1126775.jpg

medium_pexels-egor-kamelev-1126775.jpg

small_pexels-egor-kamelev-1126775.jpg

thumbnail_pexels-egor-kamelev-1126775.jpg

pexels-egor-kamelev-1126775.jpg

Why is Software Testing Important?

why-is-software-testing-important

[TABLE_OF_CONTENTS]

# Scrum testing: QA on an Agile team

Testing is not typically the first thing that comes to mind when considering Scrum. Sure, the framework is usually adapted to work with software development processes, but testers may feel neglected.

The official Scrum Guide barely mentions [software testing](https://bugbug.io/blog/software-testing/the-beginner-s-guide-to-software-testing-basics-types-and-methodologies/) or scrum testing, even though it is a crucial part of the Software Development Life Cycle. Considering a relative lack of sources on [agile testing](https://bugbug.io/blog/software-testing/agile-testing-organized-chaos-or-a-planned-way-of-improving-your-software/), this article is, in a way, a venture on the uncharted seas as we explore how agile testing fits into Scrum methodology. 
->[Automate your tests for free](https://app.bugbug.io/sign-up/){.markdown-cta}<-


## What is Scrum? 

Scrum is a process framework designed to facilitate teamwork while managing projects. It can be used in any business where projects must be coordinated and planned, especially if immediate results are required.

It is often described as lightweight, as it has few elements. Some key roles, tools (known as artifacts), and meetings (called events or ceremonies) exist. The organization is simple because Scrum is meant to be uncomplicated and adaptative. 


## Difference between Agile and Scrum framework

Currently, Scrum is one of the most popular Agile frameworks, used primarily for managing software development. The key difference between Agile and Scrum is that the first is more of a project management philosophy. At the same time, the other is a specific framework that falls under the Agile umbrella and is used to practically improve workflow in a project. Effectively, it is a set of roles, tools, and meetings designed to help teams collectively manage their work. 


## Scrum Workflow 

The Scrum framework is based on dividing work into manageable, incremental releases, known as Sprints. Every Sprint lasts from one to four weeks. By the end of that time, a potentially shippable product had been created. "Potentially shippable" means meeting the acceptance criteria and definition of done. "Done" deliverables should work as expected. As mentioned before, the Sprint is loosely organized around the roles, artifacts, and ceremonies one must know about to understand the whole process. 

The three roles in Scrum process are: 

• **Product Owner** - a person who defines the features needed in the product and brings ideas to the table, 

• **Scrum Master** - a servant to the team, they are responsible for the process, running meetings, and generally keeping things going, 

• **Scrum Team Members** - consists of software developers, testers, technical writers, etc.. 


## Three scrum artifacts 

• **Product Backlog** - created by the Product Owner, it is a prioritized list of features, for example, User Stories, 

• **Sprint Backlog** - ranks User Stories, it is used to estimate and prioritize what needs to be done during the Sprint, contains requirements taken directly from the product backlog, 

• **The Increment** - the sum of the completed Product Backlog items during the current and all previous Sprints. 

The artifacts revolve mainly around User Stories, which serve to put oneself into the end user's shoes and anticipate their possible actions. User Stories are a set schema for this practice, containing one action each.

The schema goes as follows: as a [type of user], I need [the software to do a particular thing] so that [I can achieve my goal]. This practice may be beneficial for testers, who need to anticipate and test possible scenarios of the software's use. 


## Scrum events

• **Sprint Planning** - pretty self-explanatory - it is set up to discuss the User Stories and estimate their relative sizes.

• **Daily Scrum** - a brief stand-up meeting; the development team discusses what they have completed, what they are working on, and if they need any help in case they are stuck.

• **Sprint Review** - development team demonstrates the completed work to the Product Owner.

• **Sprint Retrospective** - discussion on what to improve on going forward. 

To summarize, Sprint's life cycle starts with the product backlog, from which the tasks are selected during Sprint Planning. Once the smaller Sprint Backlog is created, the Sprint itself is in full swing. During that time, everyone takes part in the daily stand-ups. They use the Burndown Chart to monitor progress.

After the Sprint is over, it's time for Sprint Retrospective to think about ways to improve future work. Lather, rinse, repeat. This cycle continues until the product is done, the budget is spent, or the project is closed. 
->[Automate your tests for free](https://app.bugbug.io/sign-up/){.markdown-cta}<-

## Why is Scrum effective? 

Ultimately, is Scrum just the good old "planning one's work" under a new fancy name? Well, not really. The scrum framework is often set against traditional methods, such as the Waterfall approach. It means planning everything (or at least most) in a project up-front, followed by executing the pre-planned tasks.

Testing in Waterfall and most traditional methods is typically relegated to the end of the SDLC. This is not the best approach in a rapidly changing modern environment, especially for large-scale projects. There are several reasons: 

• Modern customers might require immediate results to see how the work progresses and if the product is worth supporting. A tested product has a better chance of making a good impression! 

• The customer's needs might change rapidly, or the vision of the product the team has created might prove to be ineffective or impossible to fulfil. In such a case, planning everything and testing only once the plan is completed might not be the best approach, as it would require all the work to be redone. 

• There might be a need to scale the project rapidly. Again, that would mean re-arranging the whole painstakingly prepared plan of action. 


## Adapting testing process in Scrum methodology

As for the practice of testing in Scrum, the main difference compared to the traditional methods is that instead of finding bugs and defects when the product is done, the Agile tester is supposed to find them along the way.

The second crucial change is that every team member is responsible for the quality (development and testing), not just the testing team. Let's see how it works in the sprint cycle. 

For Sprint Planning, the obvious first steps are picking which User Stories should be tested and estimating their complexity. To speed things up, it is essential to use project management tools. Let's use JIRA as an example.

In Sprint Planning, you may add a "testing" column between "code review" and "done" (even though Scrum doesn't include it). One of Agile's staples is prioritizing working in software over documentation, but that doesn't mean you can't add a "How to test?" text field to the User Story. This practice helps keep everything in one place and ensures transparency and simplified communication. 

Before any deliverables are made by the developers (usually during the first few days of the Sprint), the testers prepare test cases for each User Story. The rule of thumb is to test User Stories as soon as they are completed and appear in JIRA's "testing" column. If a bug is found, it goes into the Sprint and has to be retested as soon as it gets fixed. After the deliverable has been produced, the testers may also perform acceptance testing. 

Speed and adaptability are keys to success. That is why automatization, including [regression testing](https://bugbug.io/blog/software-testing/everything-you-need-to-know-about-regression-testing/), is preferred over manual testing whenever possible. 

As for the Sprint Retrospective, it is so case-specific that it is the same for everyone - whatever your role is, at this point, your task is to reflect on the process, learn from mistakes and identify best practices for future Sprints. 
->[Automate your tests for free](https://app.bugbug.io/sign-up/){.markdown-cta}<-

## Summary of Scrum testing

By focusing on small parts of the production process, Scrum gives immediate results. The product is easily scalable and adaptable to changes. That makes the process Agile in its most basic sense - as adaptable as possible to changing customer needs.

The framework can make sense when applied to testing a small part of the product (such as a deliverable created during one Sprint) because the sooner bugs are found, the more cost-effective the method is. 

As mentioned before, a potentially shippable product in Scrum means a tested product that can be safely put into the hands of the client. This is one of the most significant advantages of Scrum.

During the project's first week, a deliverable can already be written, tested, improved, retested, and released to the customer. This is a crucial advantage in the context of QA work in Scrum, as testing takes place from the first moments of the project instead of at the end. 

The flip side is that adapting to Scrum may take some time in itself, and it is bound to be a trial-and-error process for those accustomed to traditional methods. Therefore, before diving into Scrum, consider whether this is the right approach for your team and the software you want to build - and test.

large_pexels-pixabay-35051.jpg

medium_pexels-pixabay-35051.jpg

small_pexels-pixabay-35051.jpg

thumbnail_pexels-pixabay-35051.jpg

pexels-pixabay-35051.jpg

Scrum Testing: QA on an Agile Team 

scrum-testing

Guides to automation

CEO of BugBug.io

[linkedin.com/in/pbylina/](https://www.linkedin.com/in/pbylina/)

medium_large_pawel_byline_profile_picture_0a4d9c7db1.jpg

small_large_pawel_byline_profile_picture_0a4d9c7db1.jpg

thumbnail_large_pawel_byline_profile_picture_0a4d9c7db1.jpg

large_pawel_byline_profile_picture_0a4d9c7db1.jpg

Paweł Bylina

Are you a tech leader, CEO or CTO in a tech startup? You want to implement automation testing for your SaaS product? We've written this guide especially for you. It's packed with actionable tips on how to approach test automation in an agile way.

We're going to tackle the following questions:

-   What is the business goal of automation testing?

-   How to implement automation incrementally?

-   What resources do you need to develop automated tests?

-   What are typical problems and obstacles and how to overcome them?


The strategic approach to automation testing
--------------------------------------------

Automation testing is a very broad term. There is no single goal of automation. Depending on the level of automation you will get different benefits for your organization.

*Figure 1: Strategic matrix of different goals of test automation*
![bugbug-goals-of-automation-testing.png](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/bugbug_goals_of_automation_testing_9308b9cb1b.png)

See this strategic matrix above. You can achieve all of these goals, but not all at once. You should pick the right order of goals, starting with the ones that are low-effort and high-value. 

To achieve each of these goals, you need a different combination of resources and skills. We'll guide you through each of the goals, explaining the approach, time & resources and how to achieve it with BugBug.
->[Automate your tests for free](https://app.bugbug.io/sign-up/){.markdown-cta}<-




Step 1: low cost & high impact goals
------------------------------------

### **Goal:**

-   Prevent loss of business due to production downtimes

### **Approach**

-   You don't need to hire new people, you can do it yourself

-   Create 5-10 automated tests and run them every hour to monitor your production, for things like login, landing pages, registration form, contact form, basic app navigation

### **Time & Resources**

-   2-4 hours of your own time

### **BugBug features that you should use**

-   [Recording tests](https://docs.bugbug.io/creating-tests/recording-the-tests)

-   [Cloud runs](https://docs.bugbug.io/running-tests/in-cloud-on-server)

-   [Schedules](https://docs.bugbug.io/running-tests/schedules)

-   [Notifications](https://docs.bugbug.io/collaboration/notifications)

-   [Re-recording test steps](https://docs.bugbug.io/recording-tests-steps/re-recording-steps)

Step 2: low hanging fruit
-------------------------

### **Goal:**

-   Improve team collaboration on quality assurance

### **Approach**

-   Give all the scrum teams access to the testing platform

-   Share direct links to failed tests in your tickets or on a chat

-   Configure who gets notifications

### **Time & Resources**

-   Inviting people only takes 5 - 10 minutes, but you should spend 2-4 hours on communicating its purpose

### **BugBug features that you should use**

-   [Invite users](https://docs.bugbug.io/collaboration/inviting-team-members)

-   [Project settings > Notifications](https://docs.bugbug.io/collaboration/notifications#choose-when-people-get-email-notifications)

Step 3: incremental development of automated tests
--------------------------------------------------

### **Goals**

-   Save time on manual regression testing

-   Release faster

-   Prevent cost of regression bugs

### **Approach**

-   Talk to the whole team about what you want to achieve and what's the incremental plan

-   Choose 1-2 people responsible for automated tests 

-   Create dedicated test user accounts that will be used for automation only

-   Run automation on some testing environment (staging)

-   Create separate test suites for staging and prod (run all tests on staging, but only essential smoke tests on prod)

-   Add more tests incrementally, 2-3 test cases per week (after 3 months you should have about 50 test cases automated)

-   Start adding [data-test-id attributes](https://docs.bugbug.io/preventing-failed-tests/selectors#use-custom-attributes-for-selectors) in your HTML

### **Time & Resources**

-   You will see the benefits after 4-8 weeks of ongoing work

-   You need to engage 1-2 people on the team. Make sure they have at least 2 hours per week to work on automation.

-   You need at least one technical person on the team who knows how selectors work.
->[Automate your tests for free](https://app.bugbug.io/sign-up/){.markdown-cta}<-


### **BugBug features that you should use**

-   [New test from here](https://docs.bugbug.io/creating-tests/new-test-from-here)

-   [Duplicating tests](https://docs.bugbug.io/creating-tests/duplicating-tests)

-   [Components](https://docs.bugbug.io/editing-tests/wip-make-component)

-   [Variables](https://docs.bugbug.io/editing-tests/variables)

-   [Profiles](https://docs.bugbug.io/editing-tests/variables#profiles) to [test on different environments](https://docs.bugbug.io/editing-tests/variables#work-with-different-development-environments)

-   [Suites](https://docs.bugbug.io/organizing-tests/suites)

-   [Manually editing steps](https://docs.bugbug.io/editing-tests/manually-creating-the-test)

-   [Testing registration](https://docs.bugbug.io/editing-tests/variables#test-user-registration-and-login-using-variables)

-   [Custom javascript](https://docs.bugbug.io/editing-tests/custom-javascript-actions) might be needed

Step 4: integrate automation with CI/CD
---------------------------------------

### **Goal:**

-   Improve software development lifecycle, being sure that no code is merged without tests

### **Approach**

-   Integrate automation with your development pipelines

-   Trigger automated tests with every pull request

-   Show test results directly in your repo tool (Bitbucket, GitHub, GitLab, etc.)

-   Ask developers to run automated tests locally on the localhost environment before they commit code

### **Time & Resources**

-   You will see the benefits after 4-8 weeks of ongoing work

-   You need to engage 1-2 people on the team. Make sure they have at least 2 hours per week to work on automation.

-   You need at least one technical person on the team who knows how selectors work.

### **BugBug features that you should use**

-   [CLI](https://docs.bugbug.io/integrations/api)

-   [Overriding variables](https://docs.bugbug.io/integrations/api)

-   [Running on localhost](https://docs.bugbug.io/running-tests/test-in-your-browser)

Step 5: increase coverage
-------------------------

### **Goal:**

-   Get 100% test coverage, be confident that all the features, all the browsers, all the devices work correctly

### **Approach**

-   Pick your battles, choose what is the most essential to automate

-   We recommend you keep adding 1-2 test cases every week and don't try to achieve the impossible 100% coverage

### **Time & Resources**

-   To automate everything you would need an Infinite amount of time :)

-   To automate 2-3 new test cases per week you still need 2-3 hours weekly plus a bit more experience with tests and javascript.

-   Note that the more test cases you have, the longer the maintenance. The trick is to find the right balance.

### **BugBug features that you should use**

-   Project settings > Browser width (for [testing mobile screens](https://docs.bugbug.io/workflow-tips/mobile-version-testing))

-   Lots of custom Javascript

-   [Javascript variables](https://docs.bugbug.io/editing-tests/variables#javascript-variables)

-   [Javascript assertions](https://docs.bugbug.io/editing-tests/custom-javascript-actions#advanced-javascript-assertions)

-   [Parallel running](https://docs.bugbug.io/running-tests/parallel-mode)

->[Automate your tests for free](https://app.bugbug.io/sign-up/){.markdown-cta}<-

Common problems and solutions in implementing automation testing:
-----------------------------------------------------------------

Here are some typical problems you may encounter when you want to start automating tests for a SaaS product. We've made a short list with possible solutions that worked for other [BugBug clients](https://bugbug.io/case-study/).

### **Captcha**

You need to [disable the captcha only for automation testing](https://docs.bugbug.io/troubleshooting/captcha-in-automation-testing). Unfortunately, it's not trivial and you need to read through captcha docs or ask a developer for help.

### **Date-based features, calendars** 

Is your app searching train timetables for tomorrow? It's difficult to automate because the date and the expected outcome can change every day.

Solution A: In the automated test, pick a date that's far in the future. Then update the tests regularly, changing this date and the expected outcome.

Solution B: create a dedicated environment for automation with a mocked-up hardcoded date on the backend

### **Dynamic lists such as search results, product listings**

It's hard to automate a test if some data is changing all the time. You need to create independent tests that are always sure what to expect, for example, if you're checking a list of promoted products on your homepage, your test should first go to the admin panel and mark such products as promoted.

You can also have a separate dedicated test automation environment where you have a hardcoded set of data.

### **Can't use the same user email twice**

Create a new user with each test - use the [built-it variable "testRunId"](https://docs.bugbug.io/editing-tests/variables#use-built-in-variables-for-dynamic-or-random-values) in combination with a [temporary email catcher](https://www.google.com/search?q=temporary+email&ei=ARVtY_X8IomKrwS_i4k4&ved=0ahUKEwi1gZ2u86P7AhUJxYsKHb9FAgcQ4dUDCA8&uact=5&oq=temporary+email&gs_lcp=Cgxnd3Mtd2l6LXNlcnAQAzIECAAQQzIFCAAQkQIyBQgAEJECMgUIABCABDIFCAAQgAQyBQgAEIAEMgUIABCABDIFCAAQgAQyBQgAEIAEMgUIABCABDoKCAAQRxDWBBCwA0oECE0YAUoECEEYAEoECEYYAFDhAVjhAWC1CmgCcAF4AIABa4gBa5IBAzAuMZgBAKABAcgBCMABAQ&sclient=gws-wiz-serp) always to run tests with a fresh new user. You can also delete these users at the end of the test by recording test steps in an admin panel (if you have one) Read more in how to test user registration.

### **Selectors are not reliable**

The only way to make sure your tests are stable and easy to update is to start using "data-testid" attributes in your HTML structure, [dedicated attributes for automation testing selectors](https://docs.bugbug.io/preventing-failed-tests/selectors#use-custom-attributes-for-selectors). You need to communicate that to the front-end developer and make sure you remember about it during the code review. No worries, adding additional attributes have no risk, it's just additional data for automation with no other function, it will not introduce new bugs.
->[Automate your tests for free](https://app.bugbug.io/sign-up/){.markdown-cta}<-


### **Multilanguage**

Your page can run in different languages and you want to verify all of them. Try to assess the risk of having a bug only in one language without affecting the others. Perhaps it's not likely that such a bug would occur?

If you still need to test different languages, you have two options:

Option A: Record separate tests for each language. Hard to maintain on a bigger scale

Option B: Use variables & profiles to run the same test on different languages

-   Add data-testid to all of your app elements, so that the selectors are independent of the app language and the displayed test 

-   Add a way in your app to change the language via a URL, for example, "myapp.com/?lang=fr"

-   Create a variable "language code"

-   Create different profiles with different language codes

-   Replace the language code in the "Go to URL" step with the variable

-   Create separate suites for all the languages, overriding their profile with a chosen language

**Tests take long time to finish**

[Upgrade your BugBug plan](https://bugbug.io/pricing/) to be capable of [running cloud tests in parallel](https://docs.bugbug.io/running-tests/parallel-mode)

### **Downloading files**

Currently BugBug can't verify if the files have been download. Soon we'll add a new type of assertion for that, [talk to our support](https://bugbug.io/contact/) if you need it ASAP.

### **Drag & drop interactions**

BugBug support for drag&drop is in beta, you can achieve it by combining the "Mouse press", "Hover" and "Release mouse button" [actions](https://docs.bugbug.io/editing-tests/actions). Talk to our support for details.

### **A recorded selector doesn't work**

Sometimes it's not possible to record a reliable selector and you need to use your human intelligence and debug HTML a bit. 

Use our [No-code XPath Selector Builder](https://bugbug.io/xpath-selector-builder/) which makes it easy to craft a good selector manually and can teach you in-depth about [best practices in selectors](https://bugbug.io/xpath-selector-builder/#How-to-create-reliable-XPath-selectors-for-test-automation).

Success in test automation depends on the regularity of your progress
---------------------------------------------------------------------

Implementing automation testing in a SaaS company is a long-term process. You should do it step by step. The biggest challenge for tech leaders is lack of time to dedicate on QA area. The truth is you don't need a lot to push it forward. Better slow than never. [😉](https://emojipedia.org/winking-face/) Block yourself a slot in calendar every week to work on QA automation and keep the momentum going.

medium_bugbug_matrix_illustration_cc881af8ed.png

small_bugbug_matrix_illustration_cc881af8ed.png

thumbnail_bugbug_matrix_illustration_cc881af8ed.png

bugbug_matrix_illustration_cc881af8ed.png

Tech Leader's Guide To Automation Testing

tech-leaders-guide-to-automation-testing

Test Automation Expert and Test Architect. He is a seasoned tester who can’t stay long in one place. He is looking for perfection and a place to challenge his skills. He gathered experience working for different companies with different working models, From small to big corporations, From Products via In-house development to software house. Thanks to that he has a wide perspective on testing quality and delivering value.  
[linkedin.com/in/wyrodek/](https://linkedin.com/in/wyrodek/)

medium_large_T1_A0194_6f5ba84b90.jpg

small_large_T1_A0194_6f5ba84b90.jpg

thumbnail_large_T1_A0194_6f5ba84b90.jpg

large_T1_A0194_6f5ba84b90.jpg

Maciej Wyrodek

Test automation, especially end-to-end, is currently enjoying a revival. A lot of new tools and approaches have emerged over the last few years. Selenium has finally launched version 4, Cypress and Playwright have brought coded automation to a new level and tools like [Bugbug.io](https://bugbug.io/), [Mabl](https://www.mabl.com/), [Testim.io](https://www.testim.io/), have revitalized the idea of record and play automation. Since a lot is happening, let's take a look at automation and compare the strengths and weaknesses of 3 approaches to Automation: Custom Made Frameworks, Out of the box framework and Record and play. We will look at the pros and cons of all the solutions and when they work best. However, first we need to discuss the reasons for using test automation.

Why we automate tests
-----------------------------------

If you ask typical testers why they automate tests, they can provide many reasons: from shortening the Regression time, improvement of test coverage, to providing faster feedback. Ultimately, if we look hard enough, all goes back to the one and only goal of companies, as stated by [Eliyahu M. Goldratt in The Goal](https://en.wikipedia.org/wiki/The_Goal_(novel)) - Making money. Testing by itself often is considered a cost - as it does not bring more money. But let's not forget that more money can be made not only by selling more - but by increasing the margin - either by making the product worth the price or by reducing the cost. Testing can help to protect the bottom line, by reducing the cost of mistakes and finding them more easily and faster. This also leads to reducing the cost of customer support, refunds, etc. But as [DevOps](https://itrevolution.com/the-devops-handbook/) and [Modern Testing](https://www.moderntesting.org/) state - testing can synergize with a great experimentation culture, leading to possible delivery of a quality product in a shorter time and cost-savings on the development cycles. But this cannot be achieved without robust test automation. And automation needs to follow a few rules. For that, we need to consider the properties of great automation.
->[Automate your tests for free](https://app.bugbug.io/sign-up/){.markdown-cta}<-

Traits of good automation
-------------------------

For automation to be useful in the DevOps world, it needs to be: 

-   Reliable 

-   Fast

-   Easy to write and maintain new test 

-   Easy to run

### Reliability

If test automation is not reliable, we won`t believe its results. As a result, we will still depend heavily on manual rechecking of everything and that is time-consuming. Or even worse, it can lead to wasting time as now no one will even bother to check the results.

### Fast

The development cycle is fast. If developers need to wait for the results of their work more than a coffee break, they will switch to the next task. Now the results will need to wait which leads to the cost of context switching. And longer [lead time of a task](https://kanbantool.com/kanban-library/analytics-and-metrics/kanban-definition-of-lead-time-and-cycle-time).

### Ease of Writing and Maintaining

If automation should be a regular part of a software development lifecycle, their benefit must justify the cost. If writing a test takes 4 days, it'd better be worth it - especially if developing  the tested feature takes less than that. The same holds true for maintenance: we need to be able to easily understand why a test has failed and whether this is a code issue or a test issue and how to fix it.

### Ease of running

This relates to being fast: anybody who needs to use a test to check a code should be able to do it without extra hoops, for example automatically running them locally or via CI/CD.
->[Automate your tests for free](https://app.bugbug.io/sign-up/){.markdown-cta}<-

How to choose the best automation approach
------------------------------------------

There are a lot of ways of choosing tools for a project. When looking for tools that resolve a problem, we need to choose a tool that contributes the coveted value. The list of questions requiring answers is long, but today we will look at only one aspect, namely automation development and maintenance cost. We will look at the 3 most popular approaches, starting from ones requiring most coding and maintenance effort, like "Custom Frameworks", through usage of "Out of the box" Frameworks and finishing with Record and Play tools. We will look at the pros and cons. As it is my belief, all these approaches have made a name for themselves on the market and suit different project needs.

**Pros**

-   Great Customizability

-   Ownership

-   "On-site support"

**Cons**

-   Huge Maintenance Cost

-   Time Consuming

-   Requires lots of skill to Maintain

When I was starting test automation, this was the way to go. Mainly because Selenium Web Driver was the tool of choice back then. And as its description stated, [Selenium automates browsers](https://web.archive.org/web/20120208115222/http://seleniumhq.org/). Selenium was just a tool that did exactly what it said it delivered.

It didn't provide any functions for running test, or making assertions. 

At that time, practically every company had to build its own framework.

Often it was the role of test automation testers. In most cases, they were just learning how to automate. Making lots of mistakes in the process. But some companies managed to make great frameworks and, as a result, decided to share them with others. Some of them are still in use and developed but other have not stood the test of time. [Ocaramba](https://github.com/ObjectivityLtd/Ocaramba),[Selino](https://github.com/TestStack/TestStack.Seleno), [Serenity](https://serenity-bdd.github.io/theserenitybook/latest/index.html),[Selendie](https://selenide.org/), [Protractor](https://www.protractortest.org/#/)

This approach is still used today. Why?

### Pros

The main advantage of this approach is control. It gives a company the greatest level of control in customization. If a specific feature for a project is required, it is relatively easy to find someone who can develop this functionality.

Additionally, it gives a company considerable independence; there is no risk that at one day, the tool support will end or [repo will be deleted](https://nakedsecurity.sophos.com/2022/01/11/javascript-developer-destroys-own-projects-in-supply-chain-lesson/), leading to massive issues.

What is more, there is someone who knows how the tool works on site and can teach new team members.

### Cons

The main drawback of this solution is the cost - maintaining internal test automation framework costs time and money. Many companies underestimate the cost of developing test automation.

Time is of essence here: this automation needs a long time to be developed,  from creating a framework to writing a test. And this is the aspect that can be most damning. As discussed before, for automation to bring value, it needs to provide fast feedback.

Creating a custom framework is as big a project as any and it requires a good dose of skills, ideas, and planning. This means a lot of expertise while typically, companies are not willing to spend precious resources on test automation.
->[Automate your tests for free](https://app.bugbug.io/sign-up/){.markdown-cta}<-

Out of the box frameworks: Pros & Cons
--------------------------------------

**Pros**

-   Shorter time needed to write first test

-   Much less need for framework maintenance

-   Lots of available online trainings and knowledge

**Cons**

-   Often only limited support from community

-   Writing and Test Maintenance takes time

-   Huge dependence on framework developers
As mentioned before, the growth of Selenium was accompanied by a lot of frameworks created on its base. Over time, the issues with the web driver became increasingly prominent, leading to the re-emergence of new automation frameworks not based on Selenium: [Cypress](https://www.cypress.io/), [Puppeter](https://github.com/puppeteer/puppeteer), [Playwright](https://playwright.dev/), to name a few most recent ones.

All these tools had one thing in common - they provided the user with out-of-the-box, ready options to write automation. No matter if this was Ocarmaba based on Selenium or Cypress: the goal of the solution was the minimalism needed for framework maintenance.

These tools usually provide built-in test runners, an extensive assertions library, boiler plates for starting writing project and tests, and - most importantly- a built-in mechanism to deal with test flakiness and general instability.

Additionally, these frameworks usually provide their own philosophy of writing test automation. As long as what we need to do is with accordance to their approach, everything goes well. Problems start when we need to go beyond what is suggested.

Until recently, the best example has been a test covering multiple domains in Cypress, but as of [version 9.6](https://docs.cypress.io/guides/references/changelog#9-6-0), it is no longer such a grievous case.

In another scenario, we need to rely heavily on framework developers, if they introduce breakthrough changes. Usually, we are left with two options: update and fix the code. Alternatively, don't update and hope that there won't be any issues in the framework that need fixing.

### Pros

Compared to the custom frameworks, we can start writing tests almost immediately without the need of spending weeks or months on writing the toolset first. Similarly, we don't need to wait whenever the operating system or browsers have an update. The tool maintainer will update the framework for us. And since most tools in this category are open-source ones, there is lots of information easily available online.

### Cons

As mentioned before, these tools have their own test philosophy, which is much more restrictive. Quite often support is limited - these tools are usually open-source and support in case of issues can be unavailable where we need to resort to forums and chats where we hope someone will answer a question; sometimes we get the answer in minutes but it may never come just as well.

However, the greatest risk of all is that we are at the mercy of framework developers - these tools become a part of our code base and if we want to keep support, sooner all later we will need to update even if we don't want to. A case in point is Cypress 10 - [it introduced breakthrough changes](https://docs.cypress.io/guides/references/changelog#10-0-0). As a result, many plugins and customer functionalities won't work without extensive updating.
->[Automate your tests for free](https://app.bugbug.io/sign-up/){.markdown-cta}<-

Record and Play Testing Tools: Pros & Cons
------------------------------------------

**Pros**

-   Fast Test Writing Time

-   Low Entry Level

-   Professional Customer Support

-   Easy Scalability

**Cons**

-   Usually cloud-based, requires access to test environments

-   Still young, "untested" technology

-   Requires a new mindset

-   Data management can be quite complicated

-   Usually a commercial paid tool


The idea of record and play automation is not new, even Selenium has Selenium IDE which, after a few years on developments lately, has won some users' recognition.

Microsoft had Coded UI Test as part of the Visual Studio from 2010 and only in [2019 announced it deprecation](https://devblogs.microsoft.com/devops/changes-to-coded-ui-test-in-visual-studio-2019/)

Back then this technology was long before its time, the frameworks were very unstable and the maintenance of this test was problematic.

But in the last 4-5 years, a lot has changed. Solutions like [Bugbug.io](https://bugbug.io/), [Mabl](https://www.mabl.com/), [Testim](https://www.testim.io/) have taken over what modern technology has to offer and reinvented the concept.

### Pros

Most coded automation requires great investment of time into writing and maintaining test - usually even a trivial change costs around half an hour. In case of Automation Records and Play, test are usually meant to be disposed of  after they  stop working, as it is much easier to re-record a test instead of fixing it. This is of special importance for startups and projects with a lot of changes in the UI layer. When a team is spread thin, it is hard to find the time to write a test. But usually, you can record a few tests while checking if what has been done even works.

On the other hand,e out-of-the-box coded frameworks offer usually an easier learning curve. Record And Play tools usually do not require coding knowledge - although in more complicated cases some JavaScript knowledge can prove very helpful.

Since most of the record and play tools are paid solutions, it also means that companies have professional Trainers and Customer Support to help when issues arise. Usually, it means that in the case of young projects, we can even influence their feature roadmap.

These modern record and play solutions come with their own cloud runners, making scalability of test automation unbelievingly easy. Typically, paralyzing a test takes a few clicks.

### Cons

Unfortunately, Cloud Running is also a huge Con. First and foremost, we need to give this tool access to our test environments. 

An additional issue related to the Cloud is Term of Service - not all projects and customers can afford cloud services.

Most of these Record & Play tools are relatively young and not battle-tested. For long-term projects, this uncertainty can be very risky.

In my opinion, the biggest issue is Data Management. So far, no Record and Play solution has provided an easy way to manage test data. And this can be a huge blocker. Consequently, we need to change our approach to E2E test automation, with a desire to cover all the important and easy paths as fast as possible, to make this automation possibly most valuable.

Lastly, there is the price - cloud solutions are not cheap and if we need to run a lot of tests, it may be quite expensive. But some tools like [BugBug](https://bugbug.io) are on the other hand very cheap and offer unlimited cloud runs for a fixed mohtly fee. Arguably, it is still less expensive than maintaining coded automation and its infrastructure.
->[Automate your tests for free](https://app.bugbug.io/sign-up/){.markdown-cta}<-

No clear-cut distinction
------------------------

Before the final thought, some clarification is required:

As much as we humans like clear categorizing, the real-word borders are much blurred.

For example: should Playwright be in the Custom frameworks or in the Out-of-the-box category? After all, its strength is the ability to use its own test runners, reporting services and easy building of the new features we need.

Similarly, it has the Record and Play functionally and so did Cypress (at the moment of writing this post, the latest version of Cypress is 10.3 and since 10 Cypress Studio, which allowed test recording, has been deactivated).

There are also tools that I like to refer to as combine-harvesters: basically, they do everything and claim that they can test anything, from mobile via web to desktop and many others -- the examples include TOSCA and Test Complete.

That's why we need to look at the numerous different dimensions of automation - and how much coding is just one of them.

Summary
-------

It is my belief that over time, both no-code development and record and play automation become the dominant players on the market. Yet there is a lot of room for improvement. All the types of approaches to automation, starting from custom frameworks, or using out-of-the-box coded solutions ending on Record and Play tools, have their uses on the market.

They answer to different needs and solve different problems. So, before selecting any tool, it is always good to ask yourselves: "What are we trying to gain?"

large_image_1_9678043200.jpg

medium_image_1_9678043200.jpg

small_image_1_9678043200.jpg

thumbnail_image_1_9678043200.jpg

image_1_9678043200.jpg

Record and Play vs Coded Automation Testing - How To Choose What Works For Your Team?

record-and-play-vs-coded-automation-testing

We’ve prepared a short guide with essential recommendations on test automation that we created specifically for startups. We divided the guide into 3 levels:

[★☆☆ Level 1: get the most out of basic automation](https://bugbug.io/blog/software-testing/automation-testing-guide-for-startups-level-1/)

[★★☆ Level 2: automate hundreds of regression tests](https://bugbug.io/blog/software-testing/automation-testing-guide-for-startups-level-2/)

[★★★ Level 3: integrate with CI/CD builds](https://bugbug.io/blog/software-testing/automation-testing-guide-for-startups-level-3/)
  
Also see our guide dedicated to non-technical people: [beginner's tutorial to automation testing with BugBug](https://docs.bugbug.io/in-depth-guides/beginners-tutorial-to-automation-testing)
 
### What will you achieve on Level 1?
After finishing this level you will run 5-10 automated tests every hour and check if your core features are working properly. You will detect production bugs with big impact on users sooner than before.

### TLDR:
-   Start with short and simple tests that check your core features
-   Remember about recording “hover”
-   Create special test user accounts dedicated to automation
-   Schedule the tests to run every hour
-   Incrementally add more tests

### Focus on the most important features

![80-20 rule (1).png](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/80_20_rule_1_c2126d9815.png)

Many companies spend endless hours on a neverending pursuit to automate everything. This leads to very costly automation that requires a whole team to create and maintain. Don’t be like these companies and keep it simple. Agile startups should start with automated testing of the most essential functionality. 

> 👍 **Rule of thumb:**
> Usually, it just takes 5 to 10 automated tests to be sure that most users are able to achieve their goals.

What should you test? Think of your features. What percentage of users are using a particular feature? If the answer is higher than 80%, then this is a core feature. The goal of automating tests of such important features is to make sure that you can detect bugs that have a huge impact on your business as soon as possible (new user registration is for sure a core feature, but it’s a little bit more tricky to automate, which we will cover in the next episode)

->[Automate your tests for free](https://app.bugbug.io/sign-up/){.markdown-cta}<-




### Start with short & atomic tests

When you [record your first tests](https://docs.bugbug.io/creating-tests/recording-the-tests) start with something very simple and atomic. Atomic means that the test is small, independent and checks just one thing.

A good idea is to start with testing the basic navigation of your landing page. Click “record” and create a test that just clicks one item of your navigation link in the menu. Add [assertion](https://docs.bugbug.io/recording-tests-steps/recording-assertions) for some text in a heading. Finish the test and run it. Awesome! You’ve created your first automated test that touches the most essential area of your business!

![Screenshot 2022-07-25 at 13.44.12 (1).png](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/Screenshot_2022_07_25_at_13_44_12_1_0ebde0ddbb.png)

Now create a separate test for each navigation link. Here are screenshots of how BugBug team set up their landing page tests.

![Screenshot 2022-07-25 at 13.44.15 (1).png](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/Screenshot_2022_07_25_at_13_44_15_1_74fb39e9de.png)


### Don’t forget about recording hovers

If your website navigation appears on hover, you need to tell BugBug that you want to record a mouse moving to a specific element. To do this you need to click “Hover” during the recording and then click an element. When running the test, BugBug will move the mouse cursor to this element.

![bugbug record hovers.png](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/bugbug_record_hovers_81639fbd30.png)


### Create special test users

Another idea is to test if login works. Create a separate test that only checks that one thing.

To achieve that you need to prepare a “test user” - a user account that is dedicated only to testing if login works properly

-   Manually register a new user with a new unique email, for example with john.smith+testautomation@email.com (you can use [disposable mail](https://www.google.com/search?q=disposable+mail) or [plus aliasing)](https://support.google.com/a/users/answer/9308648?hl=en)
-   Go to BugBug and record a test that clicks the login button, enters the user login and password and submits the form
-   Add an assertion for some text that confirms that the login was successful
-   Finish your test
    

  

You might be tempted to test other features in the same test, but don’t do this now. We want to be sure that the login test is atomic and independent from other features.

  

Remember that you can create as many test users as needed. For example, if your product offers multiple combinations of preferences, you can create a separate user for each combination.

  

> 👍 **Rule of thumb:**
> Always create new test users if this leads to less test steps

  ->[Automate your tests for free](https://app.bugbug.io/sign-up/){.markdown-cta}<-

### Schedule all tests to run every hour

With BugBug you just need a couple of clicks to start running the tests every hour. Just go to “Schedules”, select the “Hourly” one and add “All tests”.

![hourly schedule (1).png](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/hourly_schedule_1_445d7fe266.png)


  

That’s it, now you will start getting emails every hour with the results of your test automation. Tip: create a filter on your email account so that you avoid inbox clutter or [configure notifications to only get an email on failures.](https://docs.bugbug.io/running-tests/schedules#configure-email-notifications)

![email notifications (1) (1).png](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/email_notifications_1_1_a23f96ba47.png)


  

### Incrementally add more tests

You’ve successfully created test automation that monitors your production! Now keep adding more short tests one by one. Don’t forget to focus on being cost-effective - if you have trouble automating a use case, skip it and go to the next one or find a different way of checking if it works. And remember: you can always [ask BugBug support to help you with any troubles](https://bugbug.io/contact).

Read next: [next level guide on how to easily maintain hundreds of automated tests ](https://bugbug.io/blog/software-testing/automation-testing-guide-for-startups-level-2/)
  
  

medium_illustration_guide_level1_98f13902d2.png

small_illustration_guide_level1_98f13902d2.png

thumbnail_illustration_guide_level1_98f13902d2.png

illustration_guide_level1_98f13902d2.png

Automation testing guide for startups - Level 1: get the most out of basic automation

automation-testing-guide-for-startups-level-1

Blog about test automation

Features

Pricing

Roadmap

Integrations

Case studies

Privacy policy

Terms of service

About us

Careers

Book a demo

Contact

Get more tips and product related content. Zero spam.

Don't miss any updates

Automate web app testing easier than ever. Without excessive costs. Faster than coding. Free forever.

End-to-end testing made simple! BugBug is a new, more intuitive tool for test automation of web apps. Get all the benefits of test automation without hiring new people or infrastructure changes.

BugBug.io | Intuitive and reliable test automation for startups

large_opengraph_image.png

medium_opengraph_image.png

small_opengraph_image.png

thumbnail_opengraph_image.png

opengraph_image.png

Updates

Product

[TABLE_OF_CONTENTS]

Compatibility testing is non-functional testing, which means it focuses on such aspects as the reliability and usability of software applications. Thanks to focusing on the compatibility of the software, you can make sure that your app will work as expected on different hardware and software configurations.

You can say that there’s software compatibility when various modules and components of software work together and exchange information to produce expected results that satisfy the user of the software. In other words, if an app works with other apps, systems, and hardware doing what it was created for, it’s compatible with them.

## Software compatibility testing is important, but what is its purpose?

As a part of non-functional software testing, compatibility testing is a method that allows you to check if an application or software works correctly with different software and on various hardware configurations. The goal of compatibility testing is to ensure that there will be no problems when your app runs on a specific device, platform, or together with other apps. An incompatibility issues will result in bugs and problems that you can detect thanks to testing and fix early in the process.

## Key objectives of compatibility testing process

- The key objectives of performing a successful compatibility testing process are:

- Checking if the software is compatible across various platforms, devices, operating systems, or hardware configurations.

- Finding problems and fixing them before the software is released.

- Improving user experience since more users with different configurations can use an app that has good compatibility.


->[Automate your tests for free](https://app.bugbug.io/sign-up/){.markdown-cta}<-

## Types of compatibility testing

There are two main kinds of compatibility testing: backward and forward compatibility testing.

### Backward compatibility testing

It's when you check if your app works on older hardware or previous versions of operating systems (the latter is often referred to as operating system compatibility).

### Forward compatibility testing

It’s the process opposite to that of backward compatibility testing as it consists in ensuring that your app works correctly on new platforms and hardware.

Also, we can distinguish more specific subtypes of compatibility testing like the ones discussed below.

**1. Browser compatibility testing**

Making sure that your website or web app works correctly on various browsers is highly important given how many browsers there are. It usually involves checking the layout and features on such popular web browsers as Chrome, Edge, Safari, and Firefox.

**2. Network compatibility testing**

Most apps today use networks and connectivity to function properly. Even if your software doesn’t require an internet connection to run, some features, such as updates may be unavailable if the app isn’t connected to the internet. For this reason, it’s essential to check if your app is compatible with different network configurations and different conditions, for example, low download/upload speeds or latency issues.

**3. Device compatibility testing**

People will use your app on such devices as phones, tablets, and notebooks. Different devices have different operating systems, hardware configurations, and screens. If you offer your app for many types of devices, it’s vital to make sure that you have dedicated versions of your app that have been thoroughly tested on each platform.

**4. Mobile compatibility testing**

This kind of compatibility testing concentrates on mobile devices. As mobility is highly important for modern users, you should make it a priority too.

**5. Software compatibility testing**

Software interactions are not uncommon. For example, two different antivirus applications won’t get along too well. You should check if your software is compatible with other popular apps that people may regularly use.

**6. Version compatibility testing**

Apps get updated frequently these days. It’s important to know that after an update, all the features of your app will still be working. You’ll need to check, for example, libraries, app components, and other elements that might be affected by updates.

**7. Database compatibility testing**

As applications often store their data in databases, you’ll have to make sure that your software works well with various databases. The testing focuses on checking how your app handles queries and how it interacts with different databases.

->[Automate your tests for free](https://app.bugbug.io/sign-up/){.markdown-cta}<-

## Key components of compatibility testing

### Test environment setup

It’s important to make your test environment mimic that of production. To do that, you have to make sure that your hardware, software, and data are identical or at least similar to those used in the production environment.

### Test case design and execution

You’ll need to prepare a set of tests to check the compatibility of your software. First of all, you will need to list elements that you’ll have to test. Secondly, you’ll need to establish what the prerequisites for testing are. In other words, you’ll have to determine such conditions as hardware and software configurations. Finally, you’ll prepare the test cases for each defined testing condition. Once you’ve completed these steps, you’re ready for test execution which involves reporting, tracking, and managing the detected bugs as well as retesting and reporting after you’ve finalized the test execution stage.

### Test coverage considerations

Determining what test coverage should be is vital as it helps to make sure that you can identify all the compatibility problems and find solutions for them. It involves establishing what the target configuration will be (including hardware and software), specifying the requirements that need to be met, and understanding the software architecture and its implementation (because the way software was designed and implemented may lead to compatibility issues).

### Compatibility test automation

You can (and should!) use automation tools that will allow you to create, schedule, and run automated test tasks for you. Such an approach will give you:

- greater speed of testing since you can have tests running on many platforms at the same time.

- higher efficiency and accuracy because taking out the human factor allows for consistent and comprehensive execution of test cases.

- generating savings as not having to rely on manual testers can significantly reduce costs.

## Steps to follow if you want to perform compatibility testing

Once you've decided to perform compatibility testing, you can follow these steps:

1. Determine what the designated platform for your app is. 
2. Get your hands on the required hardware, operating system, and any software related to the designated platform that will be needed for testing your app.
3. Set up your app and configure it.
4. Run it on the designated platform and test how the app works.
5. Find, record, and report any issues with compatibility that you encounter.


## Example of compatibility testing

Imagine that a software company has built a Windows app. They will need to find out how their app behaves on different versions of Windows, for example Windows 11 or Windows 10, as well as on Windows computers with different configurations. That’s why they’ll have to test the app for compatibility. Also, the company will need to test the app under various conditions which includes compatibility testing in multiple network configurations or with different amounts of data that the app will have to handle.

->[Automate your tests for free](https://app.bugbug.io/sign-up/){.markdown-cta}<-

## Benefits of compatibility testing for QA teams

### Ensuring software functionality across multiple platforms and environments

The main goal of compatibility testing is to make sure that your applications works on various platforms, configurations, and under different conditions. The rule here is simple: the better the compatibility of your app, the more users will be using it.

### Mitigating risks associated with incompatibilities

Compatibility testing can help reduce risks of compatibility-related issues. Thanks to such testing, you can identify risks early before they turn into real issues for your users.

### Enhancing user experience and customer satisfaction

High compatibility of your product improves the satisfaction of your customers. Given how many users with different configurations there are, software that has a high compatibility level will satisfy more people and positively affect the image of your software development team and company.

### Supporting product scalability and market reach

Quality Assurance specialists can use compatibility testing to ensure that their software works correctly on various platforms and under different usage. And if the app can handle more data, bigger workloads, and a growing user base, it means that their product shows good scalability. It also results in a better market reach as more compatible products appeal to users of different devices and platforms so the company can reach a wider audience.



## Best practices to follow when compatibility testing is performed

### Comprehensive test planning and strategy development

By focusing on test planning and strategy, QA teams can effectively test their software for various compatibility issues. A good test strategy goes a long way and will help you avoid dealing with frustrated users who experience compatibility problems.

### Testing early and often

The sooner you start testing, the more compatibility issues you’re likely to identify and address. Test early and test often as it increases your chances of finding compatibility issues in the most bizarre configurations and scenarios that your creative users can come up with.

### Prioritizing most critical issues

First things first. Make the compatibility issues that are the most common and affect the greatest number of your users your top priority. You won’t be able to fix every tiny problem, that’s why it’s important to focus on compatibility scenarios that really matter. If you find some time after fixing your top compatibility priorities, you can always move on to other issues.

### Continuous monitoring and reporting of compatibility issues

Be vigilant and be on the lookout for any compatibility problems by establishing a good monitoring process. Monitor and report issues as soon as you spot them. Sometimes such a process will help you catch an issue before it’s noticed during regular testing.

## Tools for compatibility testing

When choosing a good compatibility testing tool, you should focus on what devices and browsers a specific solution supports as well as features that it offers, including testing methods or report generation.

### Overview of popular compatibility testing tools

When it comes to compatibility testing, you have a plenty of great tools at your disposal. Some good examples include:

**BrowserStack** - a cloud-based solution that offers cross-browser testing capabilities.

**LambdaTest** - another cloud-based platform that provides parallel testing and mobile testing features.

**TestComplete** - a commercial solution that you can use to test web apps as well as mobile and desktop applications.

->[Automate your tests for free](https://app.bugbug.io/sign-up/){.markdown-cta}<-

## Conclusion

Testers should dedicate their time to compatibility testing because that way, they can make sure that their software will work seamlessly on multiple different devices and platforms. Early detection of compatibility issues can help fix them before the new release hits the production environment and users start submitting support tickets. In other words, in the long run, compatibility testing can positively affect the company’s image and user experience.

## Key takeaways and recommendations for effective compatibility testing

The most important things that QA specialists and testers should remember about compatibility testing is that testing should be done often and early in the development process. Always test on multiple different configurations and platforms to detect as many compatibility issues as possible. Document and report all problems and keep on monitoring for compatibility problems all the time.

## Stay updated with emerging technologies in the field

New software versions and devices enter the market every day. As a tester, you should keep up with all tech novelties and approach from the compatibility standpoint. It’s important because many of your users may be early adopters of new technologies. They will also be the first ones to point out any compatibility issues, so create a good compatibility testing strategy, test the application that your team's been developing, and stay ahead of any compatibility problems!



small_Test plan template (5).png

thumbnail_Test plan template (5).png

Test plan template (5).png

What is Compatibility Testing?

what-is-compatibility-testing

Software developer currently working as a tech writer and PM.  
[linkedin.com/in/marcin-%C5%82ojewski-a4191518/](https://www.linkedin.com/in/marcin-%C5%82ojewski-a4191518/)

large_DSC04042-Edit-3.jpg

medium_DSC04042-Edit-3.jpg

small_DSC04042-Edit-3.jpg

thumbnail_DSC04042-Edit-3.jpg

DSC04042-Edit-3.jpg

Marcin Łojewski

[TABLE_OF_CONTENTS]

Recently we have covered the test plan and we did quite a good job doing it. The document itself covers the testing process quite thoroughly. We agreed that the core of the test plan are the test cases and test suites. Now let’s shift our focus a bit more on the test cases and test scenarios and as a bonus we will come back to the test suites for a moment or two. 

Both test cases and test scenarios are crucial components of the testing process. Unfortunately it is not uncommon that they are not defined and understood properly.  Also it happens that they are confused with each other. The ability to set them apart is very important for proper test execution. Each tester, QA, test manager should have this knowledge. We think a high-level understanding would be beneficial even for the business people as well. We will try to be professional but not too technical in this article so hop on board.

## Test scenario

Test scenario shows a specific use case of a given functionality. It can be written in a non technical way as it describes an expected (positive and negative) result of a high level action. What is also significant for a test scenario, is that it might touch multiple layers of your software. If we were to create a smooth definition, we would aim for something like the following.

**A test scenario is a described pattern of a positive or negative action performed by the software’s user.**

Remember our article on [test plans](https://bugbug.io/blog/software-testing/how-to-create-a-test-plan/)? Let’s use the same example app to create an example of a test scenario. 

*“Only an authenticated user can see the salary on a job posting subpage”.* 

This is a great example, because one simple sentence describes triggering multiple functionalities. This test scenario contains a basic action that the user may perform on the web application. There are multiple processes and actions involved. 

What does the user see instead of the salary? If the user is not logged in, does he/she see the login button or link? Does the authentication work properly? Finally, when the user does log in after moving from the job posting subpage, is he/she redirected back to the same subpage after authentication? 

We can go on and on about that one, but hopefully you get the point now. 

There is a few additional key information here. 

- Test scenarios can not be automated and are not really reusable. They cover multiple aspects of your software and may include many variables. 

- A test scenario may be prepared by a technical as well as non-technical person. Usually the business people will tell what the app should do, and then the developers will have to know how to achieve it. It is the same in testing.

- Test scenario covers the end to end functionality of the software that needs to be tested.

- Remember that test scenarios may come from outside your testing team, it is common that some scenarios are derived from use cases that happen after the app is complete. The final users are great in finding ways of "breaking" your app. Ways that you would have never thought of. Best practice alert! If it is sound from the business perspective - foresee a beta test period for your app.


Can you perform software testing just based on a test scenario? No, you need to be more thorough. That is why you need something more detailed. You need a...

## Test case

A test case lets the tester know how to test a given feature of your app. We have to emphasise the singular here. The rule of thumb is one feature - one test case. Test cases are derived from test scenarios. It will usually include some input and expected result. Most test cases are written by and for a technical person, so expect more programmers’ vocabulary than it was in the test scenario. Our definition of a test case follows:

**A test case is a set of instructions, inputs and expected outcomes that is needed to validate the proper functionality of one software feature.**

Some technical knowledge and understanding the subject matter of your software are necessary to write test cases. As stated in the definition, we do not only need to describe what needs to be tested. We also need to know what is the data we want to use and what is the expected outcome.

Let’s jump right in and try to produce a test case from the test scenario above.

*Test case #1 - verify if the salary is hidden in a job post*

*Actions:*

*- Go to “/ads/job/2” url*

*- Scroll to div with id = “salary”*

*- Expected outcome is “Please log in to see full details” with “log in” being a link to “/login” url*

*- Failure condition is “$100 per week”*

*- Report any different outcome*

As you can see there are a few different elements that make a test case. The case ID - just for clarity. The case name or title if you will - a brief description of what is being tested. Last, but not least there are the test steps for the test engineer.  

Such test cases are easy to reflect in test automation tools. Some of them, like BugBug, will even enable you to create many test cases with using little to no code with their drag and drop functionalities. 

->[Automate your tests for free](https://app.bugbug.io/sign-up/){.markdown-cta}<-

## The difference between Test case and Test scenario in software application testing

Trying to compare a test case and a test scenario is like trying to compare a book and a page. One could exist without the other but you have to agree that one page will never give you the whole context, right? It is the same for test scenario vs test case recognition.  

In software testing it is quite tricky to find actual similarities between scenarios and cases. They both serve a testing purpose, have a designed scope and purpose. The key is not to confuse them. A testing scenario describes user behaviour and may (should actually) be transformed into a list of technically defined single test cases.

Let's focus on the key differences to have a better understanding what a test scenario and test case is.

|                                            **Test scenario**                                           |                                    **Test case**                                    |
|:------------------------------------------------------------------------------------------------------:|:-----------------------------------------------------------------------------------:|
| Everyone related to the project can write test scenarios                                               | Technical personnel is responsible for creating test cases  and performing the test |
| A test scenario covers end-to-end behaviour of the user  what can go across multiple functionalities   | Test case contains one single functionality to be tested                            |
| Is created with no possibility of reuse or automation;  includes many variables and is not a procedure | Is reusable and easily adaptable for test automation                                |
| Informs what needs to be tested                                                                        | Informs how to perform every test                                                   |
| Strategically overviews how the users will be using  the software                                      | Gives some structure to the testing process                                         |



If we were to simplify the matter as much as we can, remember this. A test scenario is basically a set of test cases. There can be one or many test cases for a single test scenario. 

![Test scenario 1.png](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/Test_scenario_1_94473c7c69.png)

->[Automate your tests for free](https://app.bugbug.io/sign-up/){.markdown-cta}<-

## How to write Test cases and Test scenarios?

After the first few paragraphs we are sure that you are a person clearly understanding what a test case and a test scenario is. Creating test scenarios is easy for people with a high level overview of the software whereas test cases are easier for people with more experience in the software testing field. Whichever description suits you more, it is crucial to stick to the how / what division. 

You should not overthink while creating a test scenario. Take the easiest approach possible and become the user for a moment. Think of what you would like to do in your app and then write it down. It should be that easy. Try not to consider (and describe) what different functionalities would your scenario trigger. This is for the test cases to check.

When you are creating test cases for each scenario, your job will be a lot more thorough. We suggest the "from the general to the particular" approach. 

Once you have the test scenario, try thinking about what functionalities it triggers and list them down. Afterwards analyze how those functionalities should and should not behave under the circumstances given in the scenario. Then provide or mock the data that need to be used in the testing effort. Last but not least, define the positive and negative outcome of each test. 

For starters try using our template from this article. We do encourage you to experiment and adapt such templates for your testing environment. Remember that creating test cases is a one-time job, an investment if you will. They are the core of test automation and are easily reusable throughout the testing process. 

->[Automate your tests for free](https://app.bugbug.io/sign-up/){.markdown-cta}<-

## Test case vs Test scenario - summary

At the very beginning we promised adding the test suite into the equation. Thanks for sticking around for that one. In our last article we described the test suite as a set of test cases. Here we do the same for a test scenario. There is a key difference though. 

A test suite is a set of test cases relevant to each functionality and is an artificial entity that helps to manage the testing process. 
Whereas a test scenario can cover multiple functionalities and is an effect of natural user actions performed in the app. 

They both cover multiple test cases, but a test suite can be automated whereas test scenario can not.

We hope we were able to convince you that test scenario and test case are two completely different beings used during the testing of an application. The scenario focuses more on the "what" and the case focuses on the "how". Remember that test cases are derived from test scenarios and not the other way around. From the good practice perspective it is wise to encourage your users to beta test as they have much more creative ideas on how to break your app. 




small_Test plan template (4).png

thumbnail_Test plan template (4).png

Test plan template (4).png

Test Case vs Test Scenario: Compare, Contrast and How to 

test-case-vs-test-scenario

[TABLE_OF_CONTENTS]

We recently went through the whole theory on the purpose of such document. We hopefully proved to each tester that such a document is more than a list of features to be tested. A good plan vastly reduces the risks associated with the whole process. It is time to present you with our sample test plan document. In the following article you can read our test plan example as well as get a sample template so you can write a similar one on your own. 

Before we do that, we need an example app, so welcome do BestAdPortal! (disclaimer - I found no such portal so I'm calling "dibs" on the name). 

From software development standpoint it is a most basic app, that does the following:

- user account functionalities (create, store, password reset, authentication, limiting access to given pages)
- ad related functionalities (CRUD)
- internal message system with email notifications
- billing functionalities 

The list is just an example, we can focus our tests on.




## Sample test plan

### Test scope and objectives

This is the place where you put the comprehensiveness of the testing effort. In our case the test approach might be as follows.


*In a few sentences we were able to describe what we are going to test, and why we limit the testing efforts. On top of that let's move to the identification of the major testing goals.*

*We would consider the test deliverables as complete if all of the below occur:*

*- All CRUD operations on the User can be performed flawlessly*

*- The User authentication process is functional*

*- All user access levels are functional*

*- All CRUD operations on the Ad can be performed flawlessly*

*- The internal message system is functional*

*- The message system cooperates with external email tool*

*- The billing is done properly (i.e. no double billing occurs during the testing effort)*

->[Automate your tests for free](https://app.bugbug.io/sign-up/){.markdown-cta}<-



### Test environment

As you might remember from the last article - [how to create a test plan](https://bugbug.io/blog/software-testing/how-to-create-a-test-plan/), this is the place to describe the resources that will be used to test the designated app. We first describe the general info in bullets and then present the selected tool as well as other information on how and where does the testing activity take place.

*To perform testing activities the requirements are as follows:*

*- A test automation tool*

*- Designated communication channels: email address and Slack channel plus access to Jira and GIT*

*- Implementing the testing tool to our CD/CI pipeline*

*- Separate server space and a test environment setup accordingly to the CD/CI pipeline so the latest released app version is in place*

*- Dummy data creation functionality in place, at least 10 users (2 admins and 8 clients) and at least 50 ads created*

*The decision was made to use a low-code or no-code test automation tool. As we have no tester per say in the team, this is the best decision from the business perspective. After thorough research we decided to choose BugBug as it has low entry requirements, is easily integrated with communication clients as well as CD/CI pipelines.*

Providing the test environment allows a full understanding on what testing resource is needed to perform each part of testing services.  


### Test strategy and approach

We have already described the testing scope, objective and the necessary tools. It is time to let the recipients know what human resources are we going to utilize and what would be the high-level approach to their work.  Remember that it is not yet the time to define the schedule of testing. This chapter is also a good place to describe a high level situation of the app if it was not stated before. You can never foresee the changes in the team and what could be missed in the onboarding process. 

*In order to successfully complete the testing goals listed above, we need to consider our situation first. Given the agile approach to software delivery and MVP status we have decided that the testing efforts should be first and foremost iterative and highly automated. Each test milestone will cover the functionality provided in the latest development sprint.*

*The testing team will consist of the following:*

*- John Doe - test manager - major testing tasks and estimation*

*- Jane Doe - tester 1 - test execution, collecting reports for bugs encountered during the testing process*

*- Jeremy Doe - tester 2 - test execution and updating test scenarios*

*- Julia Doe - communication between the testing and the development team, assistant test project manager.*

*All of the above will be involved in testing for 20 hours per week.*

Remember that the business and finances people also must approve this plan. This is why the example covers also information that helps with the estimation of the time required. 

->[Automate your tests for free](https://app.bugbug.io/sign-up/){.markdown-cta}<-



### Test cases and test suites

The core of the document. This is where you write the test scripts that will be executed. The software backlog is something that will support the test suites and cases for you. Given that we have chosen the iterative approach it will be relatively easy to pull the functionalities from the milestones identified in the software planning. The following also allows you to define any additional test milestones. 

| ID | Test suite          | Test case                           | Description                                                                                                                                      |
|----|---------------------|-------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------|
| 1  | User creation       | Proper data in user creation form   | Insert proper and valid dummy data into the registration form. Check if the user was created correctly.                                          |
| 2  |                     | Invalid data in user creation form  | Insert invalid dummy data into each field of the registration form. Check if the user is prompted about the mistake and if the user was created. |
| 3  |                     | Post creation behaviour             | Check if the user is redirected to the profile page after successful creation                                                                    |
| 4  | User authentication | Proper data in the user login form  | Insert proper and valid dummy data into the login form. Check if the user was authenticated correctly.                                           |
| 5  |                     | Invalid data in the user login form | Insert invalid dummy data into each field of the login form. Check if the user is prompted about the mistake and if the user was created.        |
| 6  |                     | Check credentials for a new user.   | Check if a new user without admin rights can access the admin panel page.                                                                        |




This is just a brief, high level example to make it easier to create a test plan. Test cases and suites might not be put into a table for instance. The crucial part is - it has to reflect your needs and requirements. Maybe you need to place a column for test data, or just an "additional information" column will do just fine? Be creative in using this example. 




### Test execution schedule

This part of the test management has to consider the software project schedule as well. One thing you might reduce your effort with, is supplementing the table above with a "sprint number" column. That way you end up with a highly readable to do list. If you choose the more descriptive approach, try something like the following.

*The testing will be performed on a similar schedule to the one the application is being designed. After each iteration is released to the relevant environment (at the end of each sprint), the test team will test the designated groups of features. As each of the people involved in the test phase is available part time, test items might require increased managerial attention. As of now there is little possibility to estimate the time required for each case to be conducted. This is mostly because the team is not yet onboarded to the chosen tool.*

Remember that a good test plan foresees changes. Also note there is some flexibility in the schedule for each testing task, since you will deliver those on sprint basis. There is no need to plan the actions in high detail.


->[Automate your tests for free](https://app.bugbug.io/sign-up/){.markdown-cta}<-


### Defect reporting and tracking

As we have already planned the whole testing process, the final touch would be to take care of the “what if..” scenarios. Yes, the final purpose of the test plan is to find actual bugs in your software. The final part of the document should then outline the course of action after a bug is found. Remember, that proper setting up automated processes here is an investment that will easily and quickly pay off. Imagine sending out a hundred emails weekly just to initiate the fixing process.

*Thanks to multiple integration opportunities from the chosen tool we are able to notify the key people from both development and testing teams. To reduce unnecessary email communication, the reports on started and passed tests will be sent only via the designated Slack channel.*

*If a bug occurs, the following actions will be automatically undertaken:*

*- A message will be sent via the designated Slack channel*

*- An email will be sent to the Test manager and to the development teams’ PM*

*- A Jira ticket will be created with the description of the test case and error log*

*- A GIT branch will be created. It will be linked in the Jira ticket mentioned above*

*Such a flow will be efficient for the bug fixing process.*

Do note that not every test management tool allows you to easily integrate such functionalities, so choose wisely. In the test plan template you will be able to download, you can also find a flow chart for defect reporting.




## Sample test plan document

Here is one especially for you. After clicking the link below you can download sample test plan. If you are not the test manager, send it to someone responsible for providing the test documentation.  It contains the cases described above but put into one easy to read document. 


[Test Plan Example](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/Bug_Bug_TEST_PLAN_example_20b66d9113.pdf)




## Software test plan template 

We are not yet tired of giving away gifts. A sample test plan template will enable you to document the procedures for your app and meet the delivery date. And at the end of the day, what does it do? After reading the last article you should already know. It saves you money! Successful delivery of test items might and will reduce the man hours spent on bug fixing. Our template is universal so it will cover every field of software testing. 

[Test Plan Template](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/Bug_Bug_TEST_PLAN_template_3f4dd12d86.docx)




If you feel that this article does not consist enough information on how to create a test plan on your own, you are right. We created a separate article that includes much more knowledge. Check it out [here](https://bugbug.io/blog/software-testing/how-to-create-a-test-plan/).




small_Test plan template.png

thumbnail_Test plan template.png

Test plan template.png

Test Plan Template For Test Automation Process

test-plan-template

For many software development teams creating any sort of plan sounds dull and boring. Creating a plan for the testing process sounds.. Well… Pretty much Hermione Granger-ish. That is definitely not the case. Remember that the whole effort that you put into testing is there for a very important business reason - to save you time and money. Creating a test plan is the first step to achieve this goal.

**But how do you prepare a test plan for software testing?**



## What is a test plan document?

Stating the obvious - a test plan is a document describing the testing process of your web application. There is more into that though. This is a crucial document. It outlines the approach, objectives, scope, and resources required for testing a web application. It serves as a roadmap, guiding the entire testing effort and ensuring that all necessary activities are executed.

Creating it, is like answering to the “wh- questions” about the process:

- What will be tested 
- Why it will be tested
- When it will be tested 
- And how it will be tested (I always wondered why “how” is on the “wh- questions” list, but anyway)

A well-crafted test plan includes details such as the test objectives, tools, schedule, and roles and responsibilities of the testing team. It would be beneficial if It also identifies the test levels, entry and exit criteria, and the types of testing to be performed.

There are two important features of a good test plan. First - keep it a live document. Don’t set it in stone from the very beginning of the process. We all know that the only constant thing in software development is change. The test plan needs to react to those changes. Second - use the test plan. It is there for a reason. It will make the web application testing easier and more efficient. The worst thing you can do about it is to complete it and put it in a drawer. By doing so, you just wasted a lot of time and effort.




## What is the importance of a test plan anyway?

Well imagine going for a long hike without a map (optional North American sport parallel - imagine playing football without a playbook) - will work somehow, but do you really foresee success by doing so? Not really, huh?

It is crucial for web application testing that all team members to understand the key aspects of it. Especially now, when we live in times of fast software and business development, IT teams often spread geographically across different languages and timezones and flat company structures when a startup CEO may also work as the PM or tech lead. We all need to be on the same page. Even literally. 

This map (or playbook if you will) is vital for everyone involved to know what will be expected of them during testing and what should the pass/fail test criteria be for the application. 

->[Automate your tests for free](https://app.bugbug.io/sign-up/){.markdown-cta}<-

## When to create a test plan in software testing process?

Given what we have stated above, on the web app test plan being a live document, the answer would be - create and improve it constantly. When to start creating it, is a bit of a different topic.

The planning process depends on your situation and web development approach. Of course the test plan needs to be created prior to the first testing activity. You have the waterfall vs agile decision to make, which implicates not only the testing but the development as well. 

![Phases of Waterfall methodology](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/96588c07_0308_4689_915f_111b5eef1f14_1ee85a8819.png)

Waterfall gives you a bit more time to create the test plan, because the person who will actually create it may work parallelly to the development process. On the other hand there will not be much time left for modifying the plan after the tests start.

![Phases of Agile methodology](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/e8034c88_f92f_4f97_8f92_fa65b85594d2_5752e83cc8.png)

Agile approach is based on iteration which is also reflected in test planning. A few core parts of the plan need to be done right after the completion of the functional specification of the app. Those would be test plan objectives and scope, test environment and test strategy and approach. The rest (test case, test suite and test schedule) can be done in an iterative manner, basically side by side with the scrum masters’ backlog planning. For example if the first sprint of your web application creation aims to create the user authentication process, your test cases will include… yes, you are right - testing the user authentication process. 



## What are the crucial components of a test plan? 

You might get different answers to that question all over the web. If your team is experienced enough, the list of components may vary especially from project to project. The following list comes from our expertise and will be described more thoroughly below. 

In our opinion a good test plan has to include:

- Objectives and Scope
- Test Environment
- Test Strategy and Approach
- Test Cases and Test Suites
- Test Execution Schedule
- Defect Reporting and Tracking

 
## What should be placed in the test plan template?

**Objectives and scope** 


In this paragraph the creator needs to outline the specific goals the testing effort needs to fulfil plus the range of the tests themselves. It is wise to do a rather thorough description followed by a bullet point summary for readability. Also we recommend doing the positive / negative approach, meaning that first think and describe what you want to achieve, then reset and do the complete opposite - what are the limitations and boundaries and describe them as well. 

This should answer the “what” and “why”. Are we just testing the functional aspect of the app? If so, why are we not doing the security testing or the performance testing? There might be business reasoning behind that decision, for example we might have an external team being responsible for the security. 




**Test environment**

Remember our article on [the best testing tools in 2023](https://bugbug.io/blog/guides/testing-tools-for-web-application/)? This is where it comes in handy. In this chapter you need to describe all assets that will be used in the process. How many man-hours do you foresee using? Do you need any specific server / network / devices or maybe some fancy settings on those? And finally how are the tests going to be performed? Will your team code them by themselves? Or maybe you will use some no-code solution like BugBug?

This might sound less important from the testers’ standpoint, but trust me - it’s a must have from the business perspective since it enables “the money people” to do some cost estimations on the test process. 




**Test strategy and approach**

We have addressed the “what” and the “why”, it is time to address the “how” we need to test. The strategy part needs to take care of describing the overall approach and principles that will be followed during the tests. The key here is to make sure that all information is relevant to your specific project.

The test approach on the other hand describes the actual activities that will be done within the process. There are different testing methodologies and a brief description of the one(s) chosen needs to be fit here. Are you going [end-to-end](https://bugbug.io/blog/software-testing/what-is-end-to-end-testing/)? Do you require data-driven approach? Make sure you let all the team members know. Speaking of team members - it is wise to place the responsibilities description here, bringing us to a conclusion that we have another “wh-” question - who? 


->[Automate your tests for free](https://app.bugbug.io/sign-up/){.markdown-cta}<-

**Test cases and test suites**

This is the most crucial part of writing a test plan, because it covers the actual scope of testing. Single test case answers the question “what do I want to test?”, and the reply needs to be in singular. One test case covers one topic, but do note that it may be divided into a few separate scripts. For example one test case will cover the user login functionality, but separate scripts will cover logging in by using different web browsers. 

A test suite is a collection of test cases grouped for test execution purposes. Moving along with the example from above a test suite would cover the whole user authentication process, and logging in, password reset, logging out etc. would be separate test cases.




**Test execution schedule**

The schedule has to cover both the high-level and low-level planning of the test phase. The first defining moment comes with the decision on waterfall vs agile that gives you a basic idea on when the testing will happen. Is it going to be post-development or mid-development?  Remember that creating a schedule is not only putting the test procedure into a calendar, the harder part sometimes is to figure out in what order the testing will take place. 

For waterfall it is relatively easy since your whole test procedure takes place after the coding is done. You might consider what the testers are going to be doing during the development phase. For agile you have a bit more decisions to make, for example is there going to be a delay in testing? If so, how many sprints of a delay would you like to have. [Testing in agile environments](https://bugbug.io/blog/software-testing/agile-testing-organized-chaos-or-a-planned-way-of-improving-your-software/) deserves a completely different article since this is truly a different software testing life cycle.




**Defect reporting and tracking**

First of all do have a bug reporting and tracking system implemented, this is the foundation of your communication within the process. If you don’t have one yet, create one. Even if it is writing a post-it with a bug description and smacking it at someone’s desk. Fortunately we have more sophisticated methods of monitoring software defects.

There are multiple project management and ticketing systems available. BugBug enables integration with the most popular ones via Zapier. This will make life easy for the test manager, since the majority of communication will be automated and whenever a bug occurs, a ticket will be created, an e-mail sent or whatever you require.

![BugBug integration with Zapier](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/2d90e3ab_2727_4a1a_9eae_52d74916c8fc_79aac10fc3.png)


## How to write a good test plan?


**Define Test Plan Objectives and Scope**

The author needs to start creating a test plan by writing the objectives and scope. In majority of cases the objectives included in the test plan are basically stating the obvious - that we want to make sure that all our functionalities work, the app is secure and will withhold increased traffic if our business is a tremendous success.

The scope on the other hand requires a tad more attention. As an exercise try writing the scope as you would for a person, who does not work for you yet and has no clue whatsoever about your software. The scope of your testing needs to be relevant for your app from the business perspective but from the subject matter point of view as well. For example you will not require too much UI testing for a console application do you?

 ->[Automate your tests for free](https://app.bugbug.io/sign-up/){.markdown-cta}<-

**Identify the test environment**

After carefully analysing the market and choosing the best tool that suits your needs, you now need to reach out to people who take care of your network and servers. If there are any special requirements such as server structure, traffic limitations etc. those need to be put here since the test management strongly depends on those. 

One thing that is commonly missing in this chapter is the test data. Those need to be either prepared beforehand or you may end up using data automatically faked by some external solutions. Whichever route you would like to take, make sure you let the readers’ know. Remember that each software product will require different types and quantities data to start the test so make sure you plan it both time- and money-wise. 




 **Choose a Test Strategy and Approach**

A good place to start is choosing your goals. Those need to be relevant to the objectives and scope you defined earlier. Remember, that a good goal is a SMART goal. Then you move to the team and responsibilities. Define specific members of your team, or profiles if you need to hire some additional testers. After that just “drag and drop” different pieces of the testing cake to those people.

One of the most important strategic decisions comes next - what tests would you like to automate. Is the test automation tool relevant to your needs in that matter? The high level listing of “to be automated” aspects would be a good place to do some self-compliance. Last but not least comes the answer to the crucial question - when do we do [functional testing](https://bugbug.io/blog/software-testing/what-is-functional-testing-types-and-examples/). There is little point to define the test plan on security, UI etc if our app just does not do what it is supposed to.




**Develop Test Cases and Test Suites**

In theory the applications functional requirements are your best friend if you want to write a software test plan. Keep your fingers crossed that the requirements were created correctly and thoroughly. In such a case your work here is relatively easy even though it will be quite monotonous. Basically you create test cases by checking and adapting the whole list of functionalities, user actions, security and whatever falls into your test scope. Then after creating such a list, you divide it into test suites that cover different areas and groups of functionalities. 

Worse case scenario - your functional requirements are described on a very high level. In such case the test plan is typically a much more "live document" as one would desire. The test management process might be a bit harder, but this does not mean that it is not an effective test plan. You just have to come back to this part in a more iterative manner and implement changes to the test plan as soon as the development team completes a functionality. 




**Create test execution schedule** 

That is another part of the test plan that might be partially covered from a different source. If your team does agile development, you should have a backlog created in sprints. Nothing prevents you from using this backlog to your needs, there is no need to reinvent the wheel there. You end up having functional test deliverables just like that.

Other test scopes need to be covered here as well. Remember that a test plan is a detailed document that requires not only your knowledge and expertise, but lots and lots of inter- and intra-team communication. You have to understand for example if starting performance testing may happen only after completing the whole app, or are there any circumstances that will allow doing it sooner. There is no universal truth here, only project relevant stuff. Those are data, you have to collect and process on your own. If you never tried to create a test plan, you would have to learn how to write it. The first version of the test plan might be similar to a first pancake. It will be thrown away. Trust us, expertise, especially in scheduling the test plan comes with repetitions. 




**Define defect reporting and tracking mechanisms**

The main goal of a testing project is to report every test result. The dream is to have as few negative results as possible, but a comprehensive test plan (like the one we are describing here) needs to consider the “what if?” scenario.

In this final part you need to describe two things. First - which tools and integrations are needed for a proper bug reporting path. The idea if bugs are the only things worth reporting is also something to think about. Maybe it is a good thing to report the completion of a test phase as well. Next you write down who the information recipients should be. Use as much automation as possible. Thanks to multiple integrations a detected bug might immediately trigger an email to a bunch of recipients, a ticket or a task and even a creating a git branch for the issue to be resolved.


->[Automate your tests for free](https://app.bugbug.io/sign-up/){.markdown-cta}<-

## Executing a software test plan. How to work with changes to the test plan?

Ok, you have created a detailed document that outlines your plan for testing. It contains the success criteria for testing, describes many test cases and includes all steps to report bugs. And now what? Well, the testing obviously, but this is not the point here.  The key importance of a test plan in software testing is not to hide in the drawer. The test plan helps in all phases of software development process and all aspects of testing only if you use it properly. 

First things first. From the very beginning understand that even the best plan will need some changes and updates eventually. Every test plan is created with no possibility to foresee how many bugs the tests will uncover. Creating your test plan has to assume some time flexibility, both ways actually. Your test estimation will rarely be 100 percent accurate. Of course if your development team did a splendid job and there are virtually no bugs whatsoever, you need to make sure that your updated test plan contains it too. 

In addition to that, remember that the most important soft criteria for a test plan is readability. You need to make it easy to understand to all audiences. You are probably wondering why we put this into the “how to execute” part of our article. That's simple - all team members have to be aware that the test plan exists and how far we have progressed. Remember, they are the ones actually working with the plan, you have created. Without them, the test plan becomes just another document.  




-----------------------------------


We hope, that with this article we were able to emphasise the importance of test plan creation. There many types of test plans out there for you to check. If we were to suggest you one crucial thing it would be just to create one and do so as a team effort.

Make sure you check the next article, that will cover how do we at BugBug write the test plan. What are our steps to create a plan. It will include a template and an example as well! A good test project plan will make as happy as John Hannibal Smith always was.

![I love it when a plan comes together meme](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/7db975a4_659f_4a4a_9a0d_9b063aee1b40_e5f73ae706.png)



