Software Testing

Testing is an ever-present part of a software developer’s job, performed at every stage of the production process. There is a wide range of test types used for software testing, some performed at various points during production, and some performed after production to confirm that the software functions as intended.

Regression testing is performed on a product that has already been finished and released. How exactly is it performed? What are the objectives and how does the procedure look? If you’re keen to find out, keep on reading as we explain everything you should know about regression testing.

[TABLE_OF_CONTENTS]

## The definition of Regression Testing

**Regression testing** is a software testing method that aims to investigate and locate any possible flaws or bugs in software that has already been fully developed. Regression Testing is performed to find out whether your software has regressed (surprise, surprise) as you continue introducing new updates and bug fixes. Whenever a piece of software is changed in any way, new bugs might emerge and some of the older ones re-emerge. 

Most often, regression testing is performed immediately after introducing a new update, but it isn’t the only factor that might call for the test. If any software or hardware-related factor has changed, regression testing should be committed to find out whether everything performs as it has before the change. This can include any sort of bug fixes and software changes, but regression might also be caused by hardware configuration changes and electronic component degradation over time. Regression testing isn’t easy – it requires cooperation and diligence out of all of the team members, which makes it perfect for Agile Software Development practices.

## Regression Testing techniques

Regression testing can include various functional and non-functional tests in its process to thoroughly investigate the software in search of flaws. There are also many techniques with which developers and QA specialists perform regression testing, including:

- Retest All
- Regression Test Selection
- Test Case Prioritization
- Hybrid approach

## Retest All – an accurate yet expensive solution

The **Retest All** technique is the most accurate and all-around method of regression testing, but it is also the most expensive one. In Retest All, every test case is checked again on the software to confirm if all of its parts are working as intended. That’s all there is to Retest All – it might be simple in definition, but it is definitely the hardest and most costly technique to actually perform.

## Regression Test Selection – when you realize you might not have enough for a Retest All

**Regression Test Selection** has been designed as a solution to the expensive nature of the Retest All technique. In **RTS** we don’t run all of the tests in our test suite, but instead we divide all of the tests into three categories – reusable test cases, retestable test cases, and obsolete test cases. In addition, RTS might actually create new test cases that haven’t been in the suite before, but might be relevant for regression tests.

If you’re interested in finding out more about regression testing examples, we recommend taking a look at *Understanding Regression Testing Techniques* by Gaurav Duggal and Bharti Suri. The authors further divide RTS techniques, naming two most important categories – coverage techniques and minimization techniques, and explain what are safe techniques.

**Coverage techniques** feature specific coverage criteria, often provided by the client who wants their software regression tested. These techniques sift through all test cases, selecting only the ones that are relevant for the covered parts of your software. **Minimization techniques** work in a way similar to coverage techniques, but instead of selecting cases based on set criteria, you select a minimum set of cases that you think will do the job.

## Test Case Prioritization – a rational approach

**Test Case Prioritization** aims to maximize the rate of fault detection, creating the most thorough regression test for the lowest cost. Every test case is assigned a priority, after which all of them are performed in order, starting from the most important one and finishing on the least important one. Thanks to that, even if your resources aren’t enough to perform all of the tests in your suite, you will still perform the ones that should detect the most flaws.

**TSP** might be performed in two ways – the first one, called **General Prioritization**, selects test cases that should be effective (on average) on all versions of the software you’re working with. The other way, called **Version Specific Prioritization**, only takes into account a specific version of the software, without considering other existing or potential future versions.

## Hybrid approach – the best of both worlds

When choosing one technique of regression testing is impossible – why not combine the best features of Regression Test Selection and Test Case Prioritization? The **hybrid approach** is still being developed, and every researcher gives a personal spin on the idea. Since there isn’t an exact definition of what a hybrid approach actually is, there are tons of versions available – each one with its own upsides and downsides. 

## Conclusion – is Regression Testing necessary?

As you have already learned, regression tests are performed to keep the quality of your software as high as possible for as long as possible. Updates and changes to your software might introduce new problems, which is exactly why regression testing is so important – to keep user experience high throughout all iterations of your program. 

Not all projects require regression testing, of course. In projects that are already finished and won’t be further updated, regression testing is mostly irrelevant. However, if you are planning to continue development further – never forget to properly regression test your software before releasing an update.

large_regression-tests-blog.jpg

medium_regression-tests-blog.jpg

small_regression-tests-blog.jpg

thumbnail_regression-tests-blog.jpg

regression-tests-blog.jpg

Everything you need to know about Regression Testing

everything-you-need-to-know-about-regression-testing

When working on a piece of software, testing is one of the most crucial elements of the production process. It allows the development team to find out what works and what doesn’t, and doing it in a proper, standardized manner will give your entire work crew a better sense of communication. With proper testing practices, you can ensure your product stays at its highest quality, minimizing the number of bugs and flaws the users may experience and speeding up the development process.

Functional testing is one of the general software testing categories, meant to verify the functionality of an app or a website. Testing isn’t something done only at the very end of development – in fact, in the early planning stages of the project, you should create a list of functional specifications. Simply put, make a list of what your software is supposed to accomplish.

Implementing a proper functional testing phase can be very beneficial to your development process. How exactly to implement it properly? Let’s see what functional testing is all about with some examples to make it easier to understand.

## Who can perform functional testing?

Since functional testing is a black-box software testing technique, the tester doesn’t need any knowledge about the inner workings of your software. In fact, the tester doesn’t even need to know anything about programming – functional testing is all about user experience, so the tester shouldn’t need any more knowledge than an average user is privy to. In fact, functional tests are most effective when performed by someone without deeper knowledge of the system they are testing.

With functional testing, you can easily check the main functionality of your software, confirming whether all required features work as intended. It also gives you the ability to see how accessible the app is to standard users – as a developer and (most probably) a programming specialist, your knowledge allows you to instinctively use an app or a system in a way that a standard user wouldn’t be able to. Functional testing lets you see exactly how usable the software is for such users, letting you see whether the end-user can use the software comfortably and what exactly happens when an error occurs.

## What are the types of Functional Testing?

Since functional testing is more of a category of software tests than an individual testing methodology, there are a variety of types of functional tests. The most common types of functional testing are:

- Sanity Testing
- Smoke Testing
- Integration Testing
- System Testing
- Component Testing
- Unit Testing
- User Acceptance Testing

**Sanity Testing** is performed whenever your software gets updated with new bug fixes and changes. It verifies all of the newly introduced functionalities, making sure they work as intended. Most often, sanity testing is considered as a type of regression testing, and many programmers get it confused with smoke testing – let’s see why.

**Smoke Testing** is very similar to sanity testing, but there is a definitive difference. Smoke testing is performed on initial builds of the software, verifying that basic functionality works. This type of testing isn’t exhaustive – and it isn’t supposed to be. It aims to go over only the most fundamental components of your software. 

**Integration Testing** gives you the ability to confirm whether individual functionalities and components of your software perform well together. It is performed after all of the individual modules have been confirmed to work without flaws and is one of the most popular types of functional testing. Integration testing can be easily automated, and often is, making it a simple step in the QA process.

Once you are sure that all functionalities operate fine individually and when integrated together, it’s time for **System Testing**. System testing is performed not to confirm whether something works, but whether your software complies with specific requirements. These are most often requirements given to you by your client, and system testing gives you the ability to easily measure how well you have achieved what your client wanted of you.

**Component Testing** is performed before integration testing to verify the individual performance of every component of your software. Keep in mind that these components aren’t simply consisting of a single functionality. For example, a component test for the login component of a website might include testing the visibility of the UI, page loading, SQL performance, and end-user experience.Finally, **Unit Testing** is a very thorough test that directly examines the code of your software’s components to make sure it is doing exactly what it is supposed to do. Unit testing can provide valuable documentation of your code development as you progress with the software’s production, which can be extremely useful to communicate between teams and speed up the work. Without unit testing, failed tests can be much harder to explain, and finding error sources will take much more time.

## Conclusion – how to understand Functional Testing?

Since there are so many types of functional testing tests, it’s impossible to provide an example that would fit all of them. Instead, we want to demonstrate to you first a simple functional testing framework. In its simplest form, functional testing can be said to have three steps.

1. Prepare input values and categorize them into test cases
2. Verify your test cases
3. Compare the results to the software requirements and expected results

However, functional tests can have varying degrees of complexity. Let’s consider the example of a login functionality test. Unit testing would require dozens of test cases and many scenarios, including every part of the login process. In turn, a simple smoke test would require much fewer cases, only verifying the functionalities critical for the end-user to log in.

As you can see, functional testing is a fascinating and very comprehensive subject, with many software testing types that will surely be useful to any software developer. With proper testing practices, your software can achieve better quality much faster, fostering communication and cooperation between different teams working on the same project.

large_functional-testing.jpg

medium_functional-testing.jpg

small_functional-testing.jpg

thumbnail_functional-testing.jpg

functional-testing.jpg

What is Functional Testing? Types and examples.

what-is-functional-testing-types-and-examples

Before you release your product into the market, it is extremely important to properly test whether it’s functioning as intended. To do that, software developers utilize a variety of testing techniques that verify the software’s integrity. The IT industry makes use of dozens of various test suites – there are over 40 different testing methods, counting only the most popular ones! To thoroughly check their software, developers implement **End-to-End testing**, often simply called **E2E** testing – an elaborate technique that mimics end-user experience. 

There are many layers to End to End testing, as it is one of the most complex testing methods, requiring you to check every part of your software that the end-user interacts with in any imaginable way. 

With this guide, we want to show you the importance of E2E tests, as well as tell you precisely what they are, including a few specific examples.

[TABLE_OF_CONTENTS]

## What exactly is End-to-End testing?

**End-to-End testing** is one of the most useful and informative techniques for software testing, providing valuable data about the inner workings of the program from the end-user perspective. With E2E testing, developers can thoroughly verify the behavior of their software. During the test, you will be trying to replicate key user behavior scenarios when using your software. The actual list of things to do varies, depending on the type of software you’re testing – but in all cases, it is rather extensive.

When performing a software testing method, like an End-to-End test on a web application, you would start with opening your browser and navigating to the proper URL, and then proceeding to use the software as intended for the end-user.

Similarly, in the case of a regular desktop application or mobile application, you would open the app just like a user would, checking out every bit of functionality that is intended for the users to work with. Keep in mind that during E2E testing you won’t need to check developer-specific aspects of the software, only the end-user perspective if it is important to you.

During an E2E test, your goal is to verify all of the dependencies and data integrity that directly or indirectly influence the reception of the software by the end-user. This usually includes any external services, as well as databases used by the software and metrics, as well as any other system dependencies. E2E tests will ideally be performed in the production environment, but this isn’t always possible. The tests can, of course, be performed in an independent End-to-End testing environment, but you should make sure it is as close to the production environment as possible.

Within the SDLC process, E2E tests are performed in the Testing phase, which is situated after Coding and before Deployment. 

## Why is End-to-End testing so useful?

Is End-to-End testing necessary? It is true that more and more developers make sure to implement an E2E testing phase before the final deployment of their software. There are many benefits to E2E tests, but the most crucial ones are:

- The ability to verify software performance from the end-user perspective
- Testing not only the software itself but also any interrelated sub systems, increasing test coverage
- Validation of back-end systems
- Immediate end-user-like feedback on test failing

While tests such as unit testing or integration testing are very useful, allowing you to verify the working of each component and providing quick results, they don’t give you the whole picture. By not performing E2E tests, you won’t know how the experience of using your software will look like from the end-user perspective.

For example, individual component testing (integration tests) might show promising results and full functionality. However, when combined, something goes wrong and produces a bug. By only checking individual components, you wouldn’t be able to find out about the bug. With a proper End-to-End test, you can verify whether everything works correctly together as it should. Even integration testing doesn’t cover all of the system components of your software, and you might never know whether every aspect of your app clicks together.

E2E testing doesn’t necessarily have to discover bugs – it can also give you useful insight into the user scenario of your product. Overall, all types of tests play their respective roles in the whole testing process. Just because E2E tests give you a more complex understanding of your software as a whole doesn’t mean that other test types are less valuable. Before releasing your app, always make sure to proceed with a thought-through testing strategy. However, while they’re certainly extremely useful, keep in mind that E2E tests occupy the top of the testing pyramid, making them the most expensive and most difficult to perform.

End-to-End testing is also one of the few functional types of tests that can be performed manually, as well as automated - which is done especially often in Continuous Integration. In such a scenario, a chosen set of tests can be run automatically each time a new iteration is uploaded to the testing server.

## End-to-End Testing Framework

Alright, so now you know what End-to-End testing is – but how to exactly implement it? There is a wide variety of checks and individual tests that must be covered by your end-to-end testing plan, some of which include:

- Establishing test requirements in the preparation phase (including software and hardware requirements)
- Creating proper documentation
- Deciding what type of test environment will be used (production environment or independent test environment)
- Preparation of test cases and general E2E design
- Tracking of data input and output
- Test execution and evaluation

When executing E2E testing methods, developers usually follow an established E2E testing framework that guides you throughout the process, enumerating various tasks and responsibilities. The framework is relatively simple and consists of only three elements:

### Establish user functions

The User Functions relates to listing the features of your software, as well as all of its interconnected software systems. To properly measure and test software functionality, you need to first think about what you’re going to test. After you’re finished with listing user functions, remember to record everything relevant to each of them, including both the input and output data. Since E2E testing can be considered holistic, you also need to pay attention to all of the relations between every individual function – how they work together and whether there are any clashes.

### Come up with specific conditions

Now that you have established the user functions you’re going to test, you need to think about the testing conditions. Try to come up with as many quantifiable indicators that can be measured for each of the functions. These can range from response time to consistency of data flow, to simple convenience of use. Remember – you’re trying to imagine yourself as an end-user, so try to keep track of anything that might be an issue for a non-developer.

### Create individual test cases

Finally, before the E2E testing can commence, you need to prepare specific test cases for each of the conditions. At this point, you have prepared a list of user functions to test, as well as data conditions that will be measured during the test. Now, you need to create at least one test case for each of the conditions. By test cases, we mean user scenarios during which specific indicators will be able to be measured. 

And by test case preparation status, we mean the precise location of the test instances being developed, as compared to intended test cases.

## Examples of E2E tests

Congratulations, you’ve almost reached the end! At this point, you should pretty much be a walking fountain of knowledge about End-to-End tests. However, the raw theory is one thing, and implementing those types of tests for system testing in reality is another. To make the process a bit more digestible, consider this End-to-End testing example for a website with shopping functionality:

- Go to the specified URL
- Open a list of products
- Check whether the information on the list correlates with specific product pages
- Try to add the product to the cart and look for visible confirmation
- Navigate to the cart and check whether the product has been successfully added
- Double-check all of the relevant information (price, product info, etc.)
- Proceed with the check-out and verify payment options integration

See how simple it is in the end? Now you can implement E2E testing methodology in your software to ensure better end-user reception, increase your product quality, and build a positive brand image.

## Frequently Asked Questions

1. **What is automated testing?** \
Automated testing in software development is a form of software testing that uses the execution of predefined sets of operations to probe and validate the software system. The fundamental purpose of test automation in software development is to generate objective feedback about the quality of the product. One such tool is Selenium. It can run UI tests.
The ideal situation is to run E2E tests with both manual testing and automated testing approaches. If your application has a lot of complicated user interfaces, you'll need more end-to-end testing.

2. **What is software development lifecycle?** \
Software development lifecycle (SDLC) is the process used by developers to create, deploy and maintain software. This process includes several steps that are all closely connected with one another. These steps include requirements gathering, analysis and design, coding and testing.

3. **What is a testing pyramid?** \
In software development projects, we have limited time and resources. This is why it's important to consider our test efforts carefully and find the right balance between different types of testing.
Basically, there are 3 layers in the testing pyramid: Unit Tests, Integration Tests,  and E2E Tests.


large_testowanie-end-to-end.png

medium_testowanie-end-to-end.png

small_testowanie-end-to-end.png

thumbnail_testowanie-end-to-end.png

testowanie-end-to-end.png

What is End-to-End (E2E) testing?

what-is-end-to-end-testing

**Nowadays, the profession of a software tester has become a symbol of entering the IT world without elbow grease. Why can some people pull it off and some cannot? Is there a golden rule for securing a job as a tester? Let’s find out.**

A good software tester has not only undergone special training crowned by a respective diploma. If you want to do this job successfully, you will appreciate natural predispositions and skills that you can acquire single-handedly. You may wonder if pure skills will suffice. Well, it depends …

## Who can work as a software tester?

Speaking in most general terms: anyone. There are no age or “trade” limitations; for many people who want to branch out the profession of a tester marks the first step in a career in IT. Many candidates fall prey to a stereotype, thinking that this is a simple, easy and pleasant profession that anyone can practise. Here come the details that you need to be aware of if you are  contemplating the job of a tester. As it is the case with other professions, some predispositions are also welcome, not only personality-related. These predispositions include:

### Analytical thinking 
A good tester needs to be able to combine facts, view an application and its possible uses in the context of its product environment and to draw conclusions;

### Inquisitiveness


When, surfing the Internet, you ask questions like “why does it work this way?”, “I wonder why they used this solution”, you are on the right path. Even if people around you sometimes find your inquisitiveness irritating,  it will prove priceless in a tester’s work;

### Resilience to boredom

It comes in handy especially at the beginning of work as a manual tester, when you check how an application works and simply “click your way” through some software an unlimited number of times. A certain degree of repeatability is an inherent part of a tester’s job; 

### Openness to people 

Another stereotype worth debunking before stepping onto the path of software testing is an introvert hiding in his cubbyhole, a dead ringer of  a character from “IT Crowd”, avoiding people at all cost. The situation is quite the opposite! Yes, you spend some time on the testing itself but you need to communicate the results of your work to a developer, a project manager or an entire team. Sometimes you need to explain what is not working. Sometimes you need to ask why something is working the way it is ☺   This is not a job for people who would rather have their arm chopped off than talk to someone on the team.

### Creativity

It will come in handy on a higher level of advancement when you are responsible for creating testing scenarios. In some companies, testers cooperate closely with the entire project team and contribute to developing the assumptions behind a product which provides plenty of room for creativity.

The stereotype of a software tester has one “benefit”: people think that testers do not need to know programming. Yes and no – it all depends on your aspirations. 

## Do you want to be a manual tester?

At this stage,  you are typically in charge of “clicking your way” through an application and reporting possible errors. After several or over a dozen months of working as a tester you can also start to write scenarios and test cases. At this stage, in order to acquire new skills, you can also learn to make use of automation tools  or tools supporting manual testers. However, if you do not want to grow or learn the basis of a programming language – you hit the wall.   The same holds true for financial promotion – among testers, the manual ones are at the bottom of the pay brackets. 

## Do you want to be an automatic tester?

Knowing the basis of programming is imperative because in this job you write your own test-automatizing scripts and you fix possible operating errors. All the time, you supervise also the programmes’ operations and you enhance them, for example with new requirements that occurred in the process of creating a product. There is a reason why burnt-out developers or people who want to have a break in programming are good candidates for automatic testers. If you are literate in even one programming language (of course the more the better) you are good to go.

## How to start working as a programming tester

Ok, let’s assume that you remain firm in your decision to become a tester. You have all the predispositions and you want to grow in IT. Where do you start? Regular broadening of your horizons and self-learning are a must; it does not matter whether you  already know the job or are contemplating it. The Internet is a mine of knowledge; we recommend publications both in Polish and English  (there are many more materials in English). If you know some basis, you can apply right away for a manual testing internship programme. Junior positions are also within your reach – they are particularly attractive to people who branch out and cannot afford an unpaid or poorly paid internship. At the same time, you can try to take on single jobs on platforms like uTest – they will ensure an inflow of cash as well as experience, so valuable at the dawn of a professional career.

Read also: A manual tester: 10 reasons that make this job interesting

## Working as a software tester – with or without training?

As working in IT has become fashionable, training companies are flourishing. They offer not only trainings for developers but also testers. On the one hand, this is an interesting and developing option because a course poses an opportunity to acquire knowledge in a user-friendly way.  This is also a guarantee of regularity in learning which is of importance to less systematic people or ones having difficulties with self-motivation. On the other hand, a training is usually costly. Before you choose the school, take a good look of the market because some participants complain about the small number of practical classes and poor care on the part of the mentors. Whether you decide to embark on a  course or not, it is a good idea to find a mentor: a person who already works as a tester and who may lend a guiding hand in your search, recommend literature on the subject  or give you feedback on your work. Last but not least: even if you have the best teachers/mentors, without your determination and own work they will not be of great help. Acquisition of testing skills is, first and foremost,  a considerable effort and ensures regular enrichment of knowledge – a lot depends on you and your commitment.

## How to be an IT tester with a certificate

The schools mentioned here frequently offer their own diplomas but the market honours certificates issued by the International Software Testing Qualifications Board (ISTQB). While you can obtain certificates at any stage of professional development, a document on the basic level should be the first step to becoming a Certified Tester. If you are new to the industry, this item on your CV is bound to be welcome. On the other hand, some people are critical about the certificates and say that all you need to know as a tester can be learnt single-handedly.

## How to be a tester and earn real money 

This is a very important issue; after all, the time of poorly paid internship must come to an end. Big companies typically have a natural development and promotion path that you may follow after having worked for some time as an intern or an IT junior. However, continued development is key to promotion – if you are contemplating the position of an automatic tester, do not put off learning  programming. The more you know about testing and the industry the better your chances of becoming a QA (quality assurance) team member. It stands for a tester with considerable experience who is in charge of entire projects, also related to people management, writing testing scenarios, writing automatic tests and monitoring work within a project. Irrespective of the position you assume, when you work in IT you should know the methods of project management while words like “agile” and “waterfall” should definitely ring the bell. 

large_jak-zostac-testerem.png

medium_jak-zostac-testerem.png

small_jak-zostac-testerem.png

thumbnail_jak-zostac-testerem.png

jak-zostac-testerem.png

How you can become a software tester

how-you-can-become-a-software-tester

**It turns out that following the “golden rules” of testing does not always  bring about the desirable effects. Disregarding the rules can be equally disastrous as exemplified by companies and institutions that have become household names. One thing is for certain: software defects are expensive. Fortunately, there are ways to mitigate the damage.**

Imagine that, as a result of an defect in the medical care system,  the patients’ medical records which are the basis for writing out prescriptions, are not updated. As a result, physicians prescribe medications unsuitable for the reported ailments which may pose a threat to some people’s health or even lives. This is not a script of a science fiction film. This situation took place in 2018 in Europe’s second largest economy. What caused it? A bug in the system operating in National Health Service (NHS) in the United Kingdom. Back then, more than 10 thousand people received wrong prescriptions. Fortunately, adverse consequences for the health were not involved. It is difficult to evaluate the losses incurred by this type of defects because, on top of purely economic costs of rectifying these mistakes, the reputation-related consequences may be even more damaging (especially for companies from outside the public sector). **In the United States alone, the damage resulting from launching to the market poor quality  software was estimated in 2018 at $2.8 billion.**

## The results of testing defects

The trade experts are far from disregarding the importance of defects generated in the process of designing software. Following a survey carried out in 2019 by Micro Focus in the US,  it was possible to define how the three areas of software written by means of the Agile/DevOps methodologies: functional, performance- and security-related, affect the specific areas of a company’s operations. It was agreed that defects on all the three levels impact most a company’s revenues, brand reputation and have legal consequences.

![elements-contributing-to-negative-business-impacts.png](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/elements_contributing_to_negative_business_impacts_9e27ba5281.png)

Allegedly, the situation can be improved by the rapidly growing automatic testing market which (and this comes as a surprise) is also an area which poses considerable challenges to companies, as indicated in the latest “World Quality Report 2019-2020″:

![challenges-currently-faced-in-applying-testing-to-agile-developments.png](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/challenges_currently_faced_in_applying_testing_to_agile_developments_005725ae28.png)

The most typical challenges indicated in the survey included:

1. Lack of appropriate test environment and data.
2. Inability to apply test automation at appropriate levels.
3. Difficulty in identifying the right areas on which tests should focus.

Interestingly, testing was indicated as the biggest challenge accompanying actions aimed at preventing new defects in software:

![top-challenge-of-mitigating-defects.png](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/top_challenge_of_mitigating_defects_6c5a6e3a12.png)

Why is testing perceived as such a difficult task? This may stem either from lack of knowledge or inability to apply it. One solution is to audit the activities instigated in the company; the list below may prove useful.

## Poor testing strategies or how extra costs can be generated

**1. Underestimated testing scenarios.**

Testing scenarios should result from consultations held within the testing team working on the scenarios aligned with the previously collected business requirements. At the very early stage of talking with product owners, project managers and developers, detailed product documentation should be compiled. Together with other things, it will provide the basis for creating detailed action scenarios (the testers’ skills and experience play an important role, too). Ironically, a firm plan, implemented since the beginning of work on designing software in accordance with the shift-left testing idea (i.e. starting tests at the earliest stage of development) facilitates flexible adjustments to changes typical of the Agile approach. It is easier for testers to modify an existing plan than to work in chaos aggravated by subsequent sprints. A case in point is a small software company, described in a scientific work entitled  “’Good’ Organisational Reasons for ‘Bad’ Software Testing: An Ethnographic Study of Testing in a Small Software Company” by D. Martin, J. Rooksby, M. Rouncefield and I. Sommerville. For the benefit of the study, a company from the United Kingdom is referred to as W1REsys. It creates development environments which make it possible for customers to design mobile applications. W1REsys employs seven people including four software developers. Due to the lack of a dedicated team of testers, the requirements for new products or functionalities are laid down either with respect to the clients’ needs or ideas for the elements which potential new clients can find attractive. These ideas are provided by representatives of the marketing and sales departments while the scientists surveying the company decided that testing was based on suggestions of what needs to be tested rather than how to do it. Therefore, the automatic tests created in W1REsys are not a part of a structured plan and as such, they allow only selected functionalities to be developed. The ideas for enhancing them are originated also in the process of developers’ work; they result from discussions about the way the system is working. This approach incurs the risk of grave defects in production and, as the survey’s authors admit, these defects do occur  yet they are maintained on a level acceptable by the company.

**2. The testing process is not aligned with the way in which an organization operates.**

The example of W1REsys shows how important it is to adjust the principles of effective testing to an organisation’s capabilities. In larger companies which are in charge of complex projects, following the good practices guarantees success. In smaller companies, oftentimes coping with staff shortages, these rules need to be tailored to the employees’ actual capabilities. Otherwise, they will work under enormous pressure of time and frustration that they cannot accomplish all the goals towards which they have been working. As the Micro Focus research from 2019 suggests, the defects detected at the last stage of designing software (production) tend to be the most costly. This stems from research into both projects managed in Waterfall, now a less popular methodology, and Agile, a methodology enjoying great interest:

![effect-on-cost-by-development-stage-1024x523.png](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/effect_on_cost_by_development_stage_1024x523_09c6c9266d.png)

[According to Eric Elliot who has written an article on production defects for Medium.com](https://medium.com/javascript-scene/the-outrageous-cost-of-skipping-tdd-code-reviews-57887064c412), the cost of remedying a production defect can be 100 times higher than the cost of correcting it at the design stage, and 15 times higher that the cost of repair at the level of implementation.

Smaller companies, which cannot afford extended testing scenarios, should arrange their tasks from the most to the least important to a project. The mentioned software company W1REsys from the United Kingdom set a goal of testing the functionalities most frequently used by the clients. As a result, it can properly manage defects identified at the production stage. This is a good example of approaching  testing not like a way of excellent testing of the entire code which, in many cases, is simply impossible but rather, as a way of managing risk that can be mitigated when specific testing tasks are given priorities.

**3. Ignoring the importance of tests in the speed and efficiency of  mobile applications.**

As a result of focusing on functionality tests (working/not working), some teams ignore the importance of speed and efficiency tests. A survey conducted by Dimensional Research in 2015 informs about the actual costs of adopting this attitude. Users of mobile applications were asked for their opinions on the response time. As the diagram below shows, their expectations are very high with 2 seconds required for the optimum response time:

![how-fast-should-app-respond.png](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/how_fast_should_app_respond_0d7c860cce.png)

80% of the respondents admitted that in the case of problems with an application, they give it not more than three chances before they abandon it for the benefit of competitive solutions. Interestingly, in the process of choosing an application, the functionalities promised by the producers are as important as the reviews published by other users:

![which-of-the-following-most-influences-the-app-you-pick.png](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/which_of_the_following_most_influences_the_app_you_pick_76401d2bb5.png)

In 2017, Pinterest found out about the business benefits of rebuilding an application to make a faster and more responsive one; following its implementation, Pinterest enjoyed bigger revenues and traffic:

![comparing-old-mobile-web-to-new-mobile-web-1024x528.png](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/comparing_old_mobile_web_to_new_mobile_web_1024x528_6332cda00b.png)

**4. Giving up on Test Driven Development.**

This approach assumes designing automatic testing before works on writing the right code even commence. Of course, at the beginning the tests are failed; over time, as the subsequent lines of the code are written, at a certain stage they all end positive. As a result, the “trouble-free” section of the code grows; you can also think over and plan how the entire application works. Eric Elliot presented the following calculations as evidence of the  effectiveness of the method which daunts many IT managers due to the high costs involved at the initial stage of a project:

![source-medium.png](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/source_medium_e8b7b31c22.png)

In the fictitious example provided by Elliot, development without TDD was to cost 1,000 hours (the cost of maintaining the system excluded). TDD allowed to save 623 hours which is equivalent to a month of work of a team of four.

**5. Under- or overestimating automatic testing.**

The success of any testing team is based both on qualified employees and skilfully selected tools. Automatic testing significantly speeds up and fosters work thus releasing the testers who can focus on exploratory testing or a product analysis with respect to the users’ requirements or experience. Teams distrustful of automation tools or disappointed with older generation programmes may prefer manual tests. Frequently, this results in overloading the employees with work. They are also frustrated because they need to handle repetitive, mundane tasks. On the other hand, automatic testing cannot replace an employee who may come up with the right testing scenarios aligned with the business guidelines or even supplement them with suggestions of new and interesting functionalities. An optimum testing strategy helps to design the testers’ responsibilities to ensure that they can make the most of their automatic testing tools, at the same time taking advantage of the specific workers’ competence in carrying out manual exploratory testing.





large_zle-strategie-testowania.png

medium_zle-strategie-testowania.png

small_zle-strategie-testowania.png

thumbnail_zle-strategie-testowania.png

zle-strategie-testowania.png

Bad testing strategies or how to learn from other people’s mistakes and to minimise damage

bad-testing-strategies-or-how-to-learn-from-other-people-s-mistakes-and-to-minimise-damage

The work of a manual tester is not always as appreciated as it should be. Often, words like “testing” boil it down to (seemingly) boring activities following a scenario. This is a superficial opinion; check the 10 reasons why this profession can be recognised as surprisingly interesting:

**1. The job of a manual tester=creativity that many marketing guys could dream of.**

As a manual tester you get to know the product and create testing scenarios. This is the part of the work where you can unleash your creativity: you come up with the biggest possible number of ways in which the user can make use of your product and where an error can be detected. Even if you are a natural when it comes to looking for “weak points” in an application, you go beyond it and foresee the user’s behaviour and the operation of the entire software. Looking at the scenarios you have created, many people could say “there is no way this will happen!”. How about you? When your scenario shows an unexpected error, you simply flick an invisible speck of dust off your shoulder.

**2. Manual testing is heaven for an analytical mind.**

Analytical thinking and drawing conclusions will come in handy at every stage of your work: since the moment you get acquainted with the technical specification and the product’s application up to writing the mentioned scenarios. A good tester has the skill of inductive thinking i.e. drawing conclusions on the basis of pieces of information coupled with deductive thinking (from the general to the detailed). A tester’s role is not limited to checking how specific functionalities work; a tester also places them in a broader context – the product and its trade environment. Questions like “Is this the best solution? Does the design meet the standard? Can you simplify it?” are elements of daily work. A manual tester is not a person who follows the instructions blindly but rather, he/she creates them respecting the latest standards and sometimes even challenging them.

**3. A manual tester criticises as much as he/she likes.**

Did you dream as a child of being a literary, theatrical or film critic? Even if you did not, as a manual tester you have an opportunity to show the part of your personality that enjoys pinpointing mistakes and highlighting imperfections. There is more: in this job it will be more than welcome (with reference to the tested application, not your team mates, that is )!

**4. A manual tester stands for a “great communicator”.**

Did you know why Ronald Reagan was dubbed “the great communicator”? For his rhetoric skills in general and one talent in particular. It actually helped him to be elected president. In 1980, during a TV debate with Jimmy Carter, Reagan addressed all Americans directly and during his air time he looked straight into the camera, making an impression that he was speaking to each and every viewer individually. How does this connect with a tester’s responsibilities? As part of your work, you will also have many opportunities to look someone in the eye and say something  harsh, e.g. to show a code error to a developer or to ask a project manager for extra time for testing if the situation requires it. Rest assured that these will not always be nice conversations… It is good to know because many people perceive a tester as an individual who is knowledgeable about logs and only produces reports. However, this job requires a lot of interaction with people at various levels (especially when your company uses the scrum methodology) and here communication skills are much coveted. Assertiveness (ability to present your opinion without hurting other people’s feelings) is worth its weight in gold.

**5. Testing and… empathising.**

I do not mean only empathy with a developer who, following your remarks, needs to repair the bugs ;). As a tester, you empathise mainly with the user who will ultimately make use of the product you are testing. You try to best understand the user’s needs and train of thoughts. Your aim is to make the application as easy to handle as possible. This is yet another soft skill that you can use in your work.

**6. People keen on accuracy make good testers.**

If you are applying for a job of a manual tester, perfectionism will certainly be perceived as your strength during the job interview. Accuracy and care for the detail are prerequisites for being a tester; otherwise you might miss an important aspect of an application’s functioning. As a result of this omission, the company could lose its good reputation and considerable amounts of money. 

**7. As a manual tester you use your organisational skills.**

You do not have unlimited time for testing; in many cases, there is less time than you hope. Your planning skills and effective use of time come in handy. Time pressure is visible not only in Jira; it also affects the people managing a project.

**8. A manual tester’s continued growth.**

As a tester you need to keep up with the changing trends and technologies, also as part of your specialisation. There is no room for stoppages or boredom because every day, new information is disseminated, new courses and ideas originated. When implemented, sometimes they challenge the assumptions about reality. Every tester needs to know the product he/she handles and this is no mean feat: you need to know the rules of operation and find out the solutions offered by the competitors plus analyse in depth the technical documentation, trace the logs etc. A tester’s work requires regular updates.

**9. As a tester you are responsible for the product’s quality.**

By clicking on “Test passed” in Jira, you confirm that a specific functionality and, by extension, the entire product, meets the requirements. You are some sort of a quality controller and this is more than just checking if an application works as planned. As a result of your work, a product wins market trust while its effectiveness improves the corporate owner’s reliability.

**10. The position of a manual tester is the first step; there is always more**

Later on, you can branch out into automatic testing, QA or even programming. The more testing tools, technologies and programming languages you know the better. Consequently,  you strengthen your position on the job market. Your opportunities for growing, learning and enhancing your skills  are limitless.

large_dlaczego-praca-testera-moze-byc-ciekawa.png

medium_dlaczego-praca-testera-moze-byc-ciekawa.png

small_dlaczego-praca-testera-moze-byc-ciekawa.png

thumbnail_dlaczego-praca-testera-moze-byc-ciekawa.png

dlaczego-praca-testera-moze-byc-ciekawa.png

A manual tester: 10 reasons that make this job interesting

a-manual-tester-10-reasons-that-make-this-job-interesting

small_pawel-about-512.png

thumbnail_pawel-about-512.png

pawel-about-512.png

Paweł Bylina

We’ve prepared a short guide with essential recommendations on test automation that we created specifically for startups. We divided the guide into 3 levels:

[★☆☆ Level 1: get the most out of basic automation](https://bugbug.io/blog/software-testing/automation-testing-guide-for-startups-level-1/)

[★★☆ Level 2: automate hundreds of regression tests](https://bugbug.io/blog/software-testing/automation-testing-guide-for-startups-level-2/)

[★★★ Level 3: integrate with CI/CD builds](https://bugbug.io/blog/software-testing/automation-testing-guide-for-startups-level-3/)
  
Here comes the first part!
 
### What will you achieve on Level 1?
After finishing this level you will run 5-10 automated tests every hour and check if your core features are working properly. You will detect production bugs with big impact on users sooner than before.

### TLDR:
-   Start with short and simple tests that check your core features
-   Remember about recording “hover”
-   Create special test user accounts dedicated to automation
-   Schedule the tests to run every hour
-   Incrementally add more tests

### Focus on the most important features

![80-20 rule (1).png](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/80_20_rule_1_c2126d9815.png)

Many companies spend endless hours on a neverending pursuit to automate everything. This leads to very costly automation that requires a whole team to create and maintain. Don’t be like these companies and keep it simple. Agile startups should start with automated testing of the most essential functionality. 

> 👍 **Rule of thumb:**
> Usually, it just takes 5 to 10 automated tests to be sure that most users are able to achieve their goals.

What should you test? Think of your features. What percentage of users are using a particular feature? If the answer is higher than 80%, then this is a core feature. The goal of automating tests of such important features is to make sure that you can detect bugs that have a huge impact on your business as soon as possible (new user registration is for sure a core feature, but it’s a little bit more tricky to automate, which we will cover in the next episode)






### Start with short & atomic tests

When you [record your first tests](https://docs.bugbug.io/creating-tests/recording-the-tests) start with something very simple and atomic. Atomic means that the test is small, independent and checks just one thing.

A good idea is to start with testing the basic navigation of your landing page. Click “record” and create a test that just clicks one item of your navigation link in the menu. Add [assertion](https://docs.bugbug.io/recording-tests-steps/recording-assertions) for some text in a heading. Finish the test and run it. Awesome! You’ve created your first automated test that touches the most essential area of your business!

![Screenshot 2022-07-25 at 13.44.12 (1).png](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/Screenshot_2022_07_25_at_13_44_12_1_0ebde0ddbb.png)

Now create a separate test for each navigation link. Here are screenshots of how BugBug team set up their landing page tests.

![Screenshot 2022-07-25 at 13.44.15 (1).png](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/Screenshot_2022_07_25_at_13_44_15_1_74fb39e9de.png)


### Don’t forget about recording hovers

If your website navigation appears on hover, you need to tell BugBug that you want to record a mouse moving to a specific element. To do this you need to click “Hover” during the recording and then click an element. When running the test, BugBug will move the mouse cursor to this element.

![bugbug record hovers.png](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/bugbug_record_hovers_81639fbd30.png)


### Create special test users

Another idea is to test if login works. Create a separate test that only checks that one thing.

To achieve that you need to prepare a “test user” - a user account that is dedicated only to testing if login works properly

-   Manually register a new user with a new unique email, for example with john.smith+testautomation@email.com (you can use [disposable mail](https://www.google.com/search?q=disposable+mail) or [plus aliasing)](https://support.google.com/a/users/answer/9308648?hl=en)
-   Go to BugBug and record a test that clicks the login button, enters the user login and password and submits the form
-   Add an assertion for some text that confirms that the login was successful
-   Finish your test
    

  

You might be tempted to test other features in the same test, but don’t do this now. We want to be sure that the login test is atomic and independent from other features.

  

Remember that you can create as many test users as needed. For example, if your product offers multiple combinations of preferences, you can create a separate user for each combination.

  

> 👍 **Rule of thumb:**
> Always create new test users if this leads to less test steps

  

### Schedule all tests to run every hour

With BugBug you just need a couple of clicks to start running the tests every hour. Just go to “Schedules”, select the “Hourly” one and add “All tests”.

![hourly schedule (1).png](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/hourly_schedule_1_445d7fe266.png)


  

That’s it, now you will start getting emails every hour with the results of your test automation. Tip: create a filter on your email account so that you avoid inbox clutter or [configure notifications to only get an email on failures.](https://docs.bugbug.io/running-tests/schedules#configure-email-notifications)

![email notifications (1) (1).png](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/email_notifications_1_1_a23f96ba47.png)


  

### Incrementally add more tests

You’ve successfully created test automation that monitors your production! Now keep adding more short tests one by one. Don’t forget to focus on being cost-effective - if you have trouble automating a use case, skip it and go to the next one or find a different way of checking if it works. And remember: you can always [ask BugBug support to help you with any troubles](https://bugbug.io/contact).

Read next: [next level guide on how to easily maintain hundreds of automated tests ](https://bugbug.io/blog/software-testing/automation-testing-guide-for-startups-level-2/)
  
  

medium_illustration - guide - level1.png

small_illustration - guide - level1.png

thumbnail_illustration - guide - level1.png

illustration - guide - level1.png

Automation testing guide for startups - Level 1: get the most out of basic automation

Features

Pricing

Roadmap

Case studies

Privacy policy

Terms of service

About us

Book a demo

Contact

Get more tips and product related content. Zero spam.

Don't miss any updates

End-to-end testing made simple! BugBug is a new, more intuitive tool for test automation of web apps. Get all the benefits of test automation without hiring new people or infrastructure changes.

BugBug.io | Intuitive and reliable test automation for startups

large_Open Graph 1200x630 3.png

medium_Open Graph 1200x630 3.png

small_Open Graph 1200x630 3.png

thumbnail_Open Graph 1200x630 3.png

Open Graph 1200x630 3.png

Blog about test automation

Updates

Product

automation-testing-guide-for-startups-level-1

In the [first episode](https://bugbug.io/blog/software-testing/automation-testing-guide-for-startups-level-1/) of our 3-part guide, we covered the basics and described how to get the most business value from simple test automation. Now it’s time to learn some more advanced concepts and increase coverage of your automated tests. Here we go!
 
  
### What will you achieve on this level?

Once you finished this level you will:
-   have 100+ nicely structured tests
-   be able to update multiple tests at once when needed
-   run tests on different environments
-   automate testing of new user registration
    

  

### TLDR:

-   Organize your test steps in groups
-   Convert some groups into shared components
-   Duplicate tests efficiently with “new test from here”
-   Set up a URL variable to quickly swap between environments
-   Set up a testing email inbox to automate tests of new user registration
-   Use [built-in variables](https://docs.bugbug.io/editing-tests/variables#use-built-in-variables-for-dynamic-or-random-values) to generate random user email aliases
    
### Scaling the test automation

![illustration-number-of-tests.png](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/illustration_number_of_tests_7ee4d49d68.png)


In this episode, we will focus on scaling your test automation to a larger number of test cases so that your regression tests also cover corner cases and niche features. This is “Level 2” so things will get more complicated, but BugBug makes everything easy to achieve.

Example: in [the episode 1 of this guide](https://bugbug.io/blog/software-testing/automation-testing-guide-for-startups-level-1/) we’ve created just one test that checks if log-in works. Now we want to create several tests that check different outcomes of logging in, for example:

-   Log in - success
-   Log in - incorrect password
-   Log in - account blocked
-   Log in - password reset required
-   Log in - account doesn’t exist
    
You can see that we aim to increase the test coverage dramatically. Instead of 5-10 tests, we’re going to have hundreds of them.

### Keep tests nice and tidy

If you want to have more than 10 tests, you should start organizing the test steps and keep them in order. Why is it important? Imagine that you’ve created 100 tests, but later you changed something in the UI. All of your tests started to fail. If you haven’t organized test steps, you will need to go to each test individually, see what’s wrong and re-record the failed steps 100 times. Fortunately, BugBug allows you to easily avoid such situations.

### Group your test steps

Keep your steps tidy by grouping them. Give a clear name to each group, which will help you to navigate through long test cases. Move your cursor in between the steps to see a plus icon, then click “[split group here](https://docs.bugbug.io/editing-tests/grouping-steps)”.

![split-group-here.png](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/split_group_here_fce12649d5.png)


  

### Convert groups to shared components

Once you start grouping your test steps, soon you will realize that some groups are exactly the same in multiple tests. This makes them a perfect candidate for [components](https://docs.bugbug.io/editing-tests/wip-make-component). Components are magic groups: if you change it in one test, it will change in all other tests that have it.

![component-made (1).png](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/component_made_1_afa56da91d.png)
  

Start with creating a component for the first step that goes to your starting URL. Then go to each test and replace the normal group with the component by using “Insert existing component”.

![insert-existing (1).png](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/insert_existing_1_985f99b0c0.png)


  

### Duplicate the tests rather than create new ones

Since you already have components, you may not want to record each test case from scratch. A lot of your tests will start with the same components (for example: go to homepage, log in, enter a password, etc.). You can quickly create a new test “in the middle” of a different test. When you click “[new test from here](https://docs.bugbug.io/creating-tests/new-test-from-here)” you will create a duplicate of the test that keeps all the preceding test steps.
 
![new-test-from-here.png](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/new_test_from_here_5d79377e02.png)


  

If you use components in a smart way, you will be able to fix hundreds of tests with just one [re-recording](https://docs.bugbug.io/recording-tests-steps/re-recording-steps).

### Use data-testid attributes as selectors

Automated tests are most reliable if your [selectors](https://docs.bugbug.io/preventing-failed-tests/selectors) are clearly identifying an element. The best way is to clearly annotate such elements in your HTML structure. Ask your frontend developer to add `data-testid` HTML attributes to elements that you click in your automated tests. 

Why is it better? Let’s see an example. If you don’t use a dedicated attribute, your login button selector might look like `//SPAN[normalize-space(text())="Log in"]`. This selector points to any element with text "Log in". If you change the text in the button, such a selector will stop working and your test would fail. 

If you used an attribute, for example `[data-testid=”login-submit”]` your tests will be more resilient to changes. Even if you change the text in the button, you will not need to edit your automated test. What’s more, the developers will see this attribute in the code so they will be aware that this specific item is a subject of test automation. Once you have such attributes in your HTML, BugBug will automatically create selectors based on “data-testid” attributes during the recording. 

![data-testid.png](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/data_testid_669fc25baa.png)


### Work with different environments

If you are fully committed to testing automation, chances are you will need to run tests on different environments. Each environment can be accessed with different URLs, for example:

-   Production - acme.com
-   Staging - stage.acme-test.com
-   Local - localhost:3000

You should store these URLs in a variable with 3 [profiles](https://docs.bugbug.io/editing-tests/variables#profiles). After you set up more than one profile, a new dropdown menu will appear in each test, that lets you run tests with different environments. Learn more in our [detailed guide](https://docs.bugbug.io/editing-tests/variables#work-with-different-development-environments).

![profiles (1).png](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/profiles_1_e2d0e47b36.png)


  

### Automate tests for new user registration

To reliably automate user registration you need to use a combination of several advanced features

-   Third-party inbox that supports [plus-addressing](https://www.msoutlook.info/question/886#:~:text=Exchange%20Online%20(Microsoft%20365)%2C,also%20known%20as%20%E2%80%9Csubaddressing.%E2%80%9D) (for example Outlook.com)
    
-   Built-in “testRunId” variable for generating random e-mail address alias
    

  

Learn more in our separate step-by-step [guide on testing sign-up.](https://docs.bugbug.io/editing-tests/variables#test-user-registration-and-login-using-variables)

### Keep it simple

Don’t try to overcomplicate your test steps. Use components and variables only when it will actually save time for you in case of product changes. Remember that re-recording is a really fast way of fixing existing tests - sometimes it’s just faster to re-record steps in 5 tests than debugging a complex net of variables and dependencies. Keep it simple and if you have any problems feel free to [talk to our friendly customer support](https://bugbug.io/contact).  😊

[Read next: advanced guide with tips on CI/CD integration](https://bugbug.io/blog/software-testing/automation-testing-guide-for-startups-level-3/)

medium_illustration - guide level2.png

small_illustration - guide level2.png

thumbnail_illustration - guide level2.png

illustration - guide level2.png

Automation testing guide for startups - Level 2: automate hundreds of regression tests

automation-testing-guide-for-startups-level-2

If you followed our [level 1](https://bugbug.io/blog/software-testing/automation-testing-guide-for-startups-level-1/) and [level 2](https://bugbug.io/blog/software-testing/automation-testing-guide-for-startups-level-2/) of this startup guide, you already know how to create and maintain hundreds of automated tests for your web app. In this episode, we will show you how to introduce automated regression testing into your software development process. Integration with CI/CD is a professional way to achieve high-quality code and prevent bugs before they are released to customers

  

### What will you achieve on this level?

Once you finished this level you will:
-   collaborate with all team members on test automation
-   run automated regression tests with every build
-   see test results directly in your CI/CD tool

<br/>

--------------------

⚠️ **Technical skills required!**

To achieve this level you need to have software engineering knowledge and experience with development tools. If you’re not a technical person, ask your developers for help.

  --------------

### TLDR:

-   Use [BugBug CLI tool](https://docs.bugbug.io/integrations/api) to integrate with your CI/CD tool
-   Run tests every time an environment is deployed
-   Invite all your team members to BugBug, so that anyone can view test results
-   Copy & paste links to test results and share them with colleagues
    

### Run tests with every deployment

Your goal is to make sure that every time someone makes changes to the code, this new code is automatically checked against bugs before releasing to clients. If you use Github, Gitlab, Bitbucket or similar, you can configure these tools to run a “build” every time someone merges new code to a specific git branch. Our goal is to make sure that every time the build is run it will also trigger an automated regression tests suite. If the suite fails, the build also fails. If the build failed, you should not merge the code to master. This approach lowers the risk of introducing bugs to production.

### Check the ID of test suites

Simply go to “Suites”, and select “Run via CLI”. You will see terminal commands that you can copy and paste into your CI/CD scripts.

  

![pasted image 0 (2) (1).png](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/pasted_image_0_2_1_c03d47d839.png)


![pasted image 0 (3) (1).png](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/pasted_image_0_3_1_f5a9ebf1e6.png)


### Integrate test suites with your existing CI/CD pipeline

You probably already have a build script. It’s very easy to initiate BugBug tests from it. You just need to add a couple of lines of code that will trigger the tests and wait for the results. Learn more in the documentation of [BugBug CLI](https://docs.bugbug.io/integrations/api) and ask your DevOps for help if needed.

![api-example-script-safe.png](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/api_example_script_safe_cbb33eeae4.png)



  

Once you successfully integrate your build scripts with BugBug, you are able to see the tests results directly in your build status.

![pipeline-example-safe.png](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/pipeline_example_safe_88bb40c38a.png)



  

### Share links to test results

You noticed that a test failed and you wonder how can you send the report to your colleague? The easiest way is to simply copy the URL and send it via chat. Each item from the “Runs history” has a unique URL address.

![Screenshot 2022-07-26 at 15.37.47 (1).png](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/Screenshot_2022_07_26_at_15_37_47_1_b21615dcd5.png)


  

To access it the recipient needs to be invited to your [organization](https://docs.bugbug.io/collaboration/organization-settings). BugBug allows you to invite an unlimited number of people to your organization free of charge. Testing automation is more powerful if you make it available to the whole team!


### Don’t forget about "Level 2" recommendations

CI/CD integration doesn’t make sense if your tests are not resilient to changes and easy to maintain. See our previous tips in [the level 2 test automation guide, where we covered topics such as custom attributes, reusable components and variables.](https://bugbug.io/blog/software-testing/automation-testing-guide-for-startups-level-2/) 

medium_illustration - guide3b.png

small_illustration - guide3b.png

thumbnail_illustration - guide3b.png

illustration - guide3b.png

Automation testing guide for startups - Level 3:  integrate with your CI/CD

automation-testing-guide-for-startups-level-3

Test Automation Expert and Test Architect. He is a seasoned tester who can’t stay long in one place. He is looking for perfection and a place to challenge his skills.  He gathered experience working for different companies with different working models, From small to big corporations, From Products via In-house development to software house. Thanks to that he has a wide perspective on testing quality and delivering value.\

[linkedin.com/in/wyrodek/](https://linkedin.com/in/wyrodek/){ref="nofollow"}

large_https://i0.wp.com/wyrodek.pl/wp-content/uploads/2017/01/T1A0194.jpg

medium_https://i0.wp.com/wyrodek.pl/wp-content/uploads/2017/01/T1A0194.jpg

small_https://i0.wp.com/wyrodek.pl/wp-content/uploads/2017/01/T1A0194.jpg

thumbnail_https://i0.wp.com/wyrodek.pl/wp-content/uploads/2017/01/T1A0194.jpg

https://i0.wp.com/wyrodek.pl/wp-content/uploads/2017/01/T1A0194.jpg

Maciej Wyrodek

Testing can be difficult, it is hard to master, there are many questions and many issues we can face with each new feature, each new story. Every day, testers need to decide what to test, how to test, what to automate, and what risks to accept. These decisions are hard and require a lot of experience. Fortunately, there are techniques, heuristics, and models which can help with planning tests. One of the oldest and still very useful techniques is **Agile Testing Quadrants** (**ATQ**). it is a simple table that helps us select the tools we need to use, depending on the required outcome. In this article, we will look into the history of Agile Testing Quadrants, their uses, and limitations. We will also discuss the attempts at modernizing it.

![Agile-Testing-Quadrants](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/Agile_Testing_Quadrants_4b2d044860.png)

[TABLE_OF_CONTENTS]


## Agile Testing Quadrants or Agile Testing Matrix?

First, we need to clear one thing: as with everything in IT, even this simple table needs to have many names! Chances are that you will come across it as an Agile Matrix, Testing Matrix, Testing Quadrants, or Agile quadrants. All of them refer to the same thing. The only exception is the Agile Testing Matrix which may refer to both Agile Testing Quadrants and its precursor the Named Agile Testing Matrix. 

![Testing matrix](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/test_matrix_00ac277513.jpg)


## The Story of Matrix

When speaking about **Testing in Agile**, there is probably not a single more important person to its development than [Lisa Crispin](https://lisacrispin.com/) and [Janet Gregory](https://janetgregory.ca/). Their book, [Agile Testing](https://www.goodreads.com/book/show/5341009-agile-testing), has helped many testers. [Testing Pyramid ](https://bugbug.io/blog/software-testing/testing-pyramid-a-successful-approach-to-test-automation/) was born while [Mike Cohn](https://twitter.com/mikewcohn) tried to explain the nuances of automation to Lisa’s team (*Experience of Test Automation - chapter 1 Section 1.3.*) Agile Testing Book played a key role in the popularization of Agile Testing Quadrants, where the creation of ATQ is often misattributed to this wonderful duo. As Lisa Crispin explained in [her blog](https://lisacrispin.com/2011/11/08/using-the-agile-testing-quadrants/) - they improved on the older concept by [Brian Marick](http://www.exampler.com/old-blog/2003/08/22/#agile-testing-project-2) - an Agile Testing Matrix created as part of his blog series presenting his vision of testing in Agile. So Agile Testing Quadrants have almost 20- year-long history and they are still one of the best models used for testing.

## The Usefulness of Agile Testing Quadrants

Agile Testing Quadrants have two axes which split the matrix into 4 quadrants. Both axes pose the same questions, namely **What is the goal of your testing?** The major difference between the two axes is the flavor of the question: the horizontal axis asks what is currently more important: helping the team to deliver faster or ensuring that the product is as good as possible? the vertical axis asks if we are concentrating on the aspects of technology, ensuring that everything is working well, or are we looking from the perspective of business and customers, making sure that our program is worth using? Obtaining both answers points to one of the Quadrants with a list of useful examples. This can be a little confusing, so let’s look at some of the examples.

### Example 1: E-commerce shop

Let’s say we have e-commerce shop development that is going nicely. However, there is one enormous risk on the horizon, namely quite a lot of features, each tested separately so no one is sure how they will work in the real world. The team believes that the standard paths will work. However, no one is sure what kind of edge cases will come from the real world.

Looking at our matrix we can see that problem is definitely business-facing - we don’t know what the user will do. We also need to make sure that our product will handle it, ergo - Critique Product.
Looking at it we land in Q3  where we have propositions of the following techniques:
 - Exploratory Testing
 - Scenarios
 - Usability Testing
 - User Acceptance Testing (UAT)
 - Alpha/Beta Testing.

So our goal is to find edge cases. Looking at the list, all the techniques could prove helpful. But some may require too much investment so late in the developments (Scenarios). Some are a poor fit for an e-commerce shop (alpha/beta testing, UAT). Which leaves us with Usability Testing and Exploratory testing.
ATQ is not an extensive matrix, it contains only examples of techniques rather than a complete list. For example, a different technique that could help is [Dogfooding](https://en.wikipedia.org/wiki/Eating_your_own_dog_food) where company employees themselves use a product to satisfy their everyday needs.

### Example 2: Messaging App

A company has made a very successful messaging app, but now they are facing the problem of success. Their small app, designed for a small community, is gaining popularity fast. The team feels that the app is not ready for all this attention. The situation is clear - from a business perspective, everything works well enough. The risk is technology, and again the problem lies in the critique of a product. This leads us to Q4 where we have, among others, *Performance* and *Security* testing. There is no doubt that we will need both. Lots of users mean a load on an application and nothing will kill its momentum like failing servers. Security is also important. Attention also attracts bad actors who will try to exploit the system.

### Example 3: Early Development.

The team suffers a problem of poor quality code. In the case of some features, developers are reluctant to fix a familiar bug because of the high risk of breaking something else. One glance at the ATQ shows that we need something to support the development and its technology which leads us straight to Q1. Both Unit Test and Component Test would improve team productivity and code quality.

The above example may give a wrong impression that Agile Testing Quadrants are used to select one and only one tool. **THIS IS NOT TRUE**: we can use ATQ for each goal of our testing, for each need, for each problem we are trying to address. However, trying to improve too much at once may lead to too much-wasted effort and no result. Concentrate on one subject at a time.

## Evolution

While the concept of AQT has changed little over the years, the techniques in each quadrant have evolved, for example: 

![Updated Agile Testing Quadrants](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/updated_atq_06346e83b7.jpg)

In [More Agile Testing](https://www.informit.com/articles/article.aspx?p=2253544), the author presented a new version with Micheal Hüttermann changes, with a circle of *Outside-in, Barrier-free, Collaborative in the middle of the quadrants*. The author believes that it will stretch beyond any quadrants. The main issue with this update is that it is a catchphrase and provides no information about the tools and techniques that may help.

![Agile Testing Quadrants next iteration](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/Agile_Testing_Quadrants_next_d74569f53c.jpg)

A few years back, [Lisa and Janet](https://agiletester.ca/applying-the-agile-testing-quadrants-to-continuous-delivery-and-devops-culture-part-1-working-towards-continuous-delivery/) also presented a version more in line with DevOps. This version adds a lot more techniques to each of the quadrants but also changes one side, from *Supporting Team to Guide Development*. While it may seem like a single name change, fortunately, the original name leads to many misunderstandings and the new label is much more descriptive. 

![RST Agile Testing Ecosystem 1.1](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/bb_ATQ_1fb44a3b67.png)


Michael Bolton and James Bach have also made the [above version](https://www.slideshare.net/HcmcStc/the-new-agile-testing-quadrants-bringing-skilled-testers-and-developers-together-james-bach) which is compatible with Rapide Software Testing. Most of these updates are only fine-tuning, adding elements that their author thinks are better in this context. So which one to choose? It is the author’s opinion that the original version holds well? It is still relevant, because:

## It is just a model.

“All models are wrong, but some are useful” is an old aphorism attributed to George Box. And it is important to bear it in mind in the case of Test Pyramid, Agile Testing Quadrants, etc. Models are what they are: models which need to help us think about a problem. **They are not the golden rules that should be followed in the letter**. They provide us with a good way of tackling the problem. But it is up to the user to decide if a specific model will work for a specific problem.

## Summary

The **Agile Testing Quadrant** is a model which is nearly 20 years old. Owing to its simplicity, it is still relevant. It is a great tool for teams and testers who have a problem with choosing the right testing method. For junior testers, it is a great place to look into different tools and approaches, and for more advanced testers it will be an excellent aid in building or improving the testing process.