Software Testing Knowledge

Automated testing became an indispensable part of the software development business. It is also given specific business objectives to attain. Which trends will dominate automated testing in 2021 and determine who will get the biggest market share?

The rapid development of the Agile and DevOps methodology integrated automated testing into Agile (which also means accelerated) process of the code generation. One of the direct consequences of such a change was so called shift-left testing, which means moving the testing to the earlier phase of software development – it can be even gathering of the business requirements. Before “tha Agile era” testing could start only when the final effects of the developers’ work were available. It resulted in the higher costs of the maintenance and bug-fixing (because some of them were overseen in the earlier phases of the work). Not to mention huge losses of the company’s image and outflow of the customers disappointed with the product. It therefore comes as no surprise that the executive IT management surveyed in “The World Quality Report 2019-2020” recognised “contribution to business growth and business outcomes” as the most important objective for testing and QA in 2019:

Chart 1. Answers of the executive IT management concerning objectives for testing and QA in 2019:

![bb-blog-5-1.png](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/bb_blog_5_1_1081b97858.png)
Source: The World Quality Report 2019-2020

However, testing in 2013 (when Agile started to gain in popularity) and now are two different worlds. What will contribute to increasing business results after Agile became an indispensable methodology of work in the software development business?

## Automatization of the new generation

In recent years, the software testing market has been experiencing a real boom and – according to the latest analysis – it is only the beginning. As stated in the Techavio report “Software Testing Services Market by Product, Geography, and End-user – Forecast and Analysis 2020-2024”, it is expected to grow at an even faster pace – with CAGR at the level of 12,62%. Read more about business consequences of this phenomenon. The necessary element of this process is development of the automated tests, which supports changes taking place thanks to the implementation of the Agile and DevOps methodology. Currently, the tools which offer only simple functions such as recording the tests performed on the website or in the app, are insufficient. To keep up with the rapid changes in the development environment there is a need of the software which will enable for automatization of the most actions taken by the testers. Additionally, such a tool should be easy to handle for the person without any programming skills and offer the feature of quick identification of the most bug-prone parts of the code. Check out 10 characteristics that the new generation testing automation tool should offer. Automated testing ceased to be only the tool which supports testers in their work – it became an indispensable element in the end-to-end testing process.

## Artificial intelligence (AI) and machine learning (ML)

Although they became very popular and catchy slogans, the newest Techavio report “Software Testing Services Market by Product, Geography, and End-user – Forecast and Analysis 2020-2024” confirms that the testing sector has just started making use the benefits which stem from this technologies. The results are encouraging, and the forecasts concerning the pace of development of the testing market (according to the TechWiz its value in 2025 could reach 190,6 billion USD) confirm its great potential. As we read in the Techavio report, implementation of the artificial intelligence in the testing process is crucial in striving for the smart testing, which means ensuring maximum effectiveness of the testing from the earliest stages of the code creation. What is important, QA specialists and testers who will be introducing the artificial intelligence in the company should receive appropriate training – it is necessary to fully benefit from the possibilities provided by this technology. Companies which will be first to discover such a need would gain a huge competitive advantage. The experts quoted in “The World Quality Report 2019-2020” indicate the complementary role of machine learning in the process of enhancing quality of the testing with the utilisation of the AI. How could it happen? As stated in the report, “As organizations grow accustomed to mechine learning, they will increasingly understand its distinction from artificial intelligence. They will see that in order for AI testing to be effective, it needs access to the right data – and machine learning can provide this, by processing records in volume, thereby helping to identify and optimize test cases”.

## Big Data technology

It is another shining example how testing is intrinsically linked with the latest trends in the world of the software and new technologies, as well as with the e-commerce market. Big data is used in the development of the programmes for an automated testing (i.e. with the use of the ML described above), although the data sets also require testing and continuous monitoring. Big online shops or the software for health systems are great example here – such a huge data conglomerates require testing on each phase: introduction, processing, migration, performance and speed. Along with the progress of digitisation process, which concerns each line of business, the demand for such services will be growing.

## New challenge – internet of things

“The World Quality Report 2019-2020” mentions quick development of the testing in IoT in the following industries: car, energy and telecommunication. The latter would face a great challenge in the coming years: consequences of the 5G technology implementation. As piszą Darren Coupland (Sogeti) and Mandan Sundaraju (Capgemini) write, “This will create a new quality assurance (QA) dynamics. All these new use cases will need to be validated end to end. How so? Because the increase of internet of things (IoT) developments implicit in 5G has implications not just for IT applications, but also for the physical devices themselves. For example, it’s not just the app managing the traffic light that needs to be tested: it’s also the traffic light sensor itself”. Test coverage of the smart devices interconnected with the applications became a necessity in the times of the rapid expansion of the IoT market. As the Testing Whiz reports, in 2016 its worth was estimated for 6,4 billion USD, in 2020 – for over 20 billion USD.

## Blockchain technology as a new business model

Blockchain solutions are being used increasingly with regard to the processing and protection of the data. The report “5 Blockchain Trends for 2020”, prepared by Deloitte, reflects its growing popularity and scope of use:

Illustration 1. Conclusions for the coming year with regard to use of the blockchain technology in the business.

![bb-blog-5-2.png](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/bb_blog_5_2_09dd6a8b48.png)

In the coming 12 months implementation of the blockchain technology is planned to be increased in the following sectors:

- technology, media and telecommunication (from 45 to 49 pct.)
- energy (from 37 to 43 pct.)
- industry (from 40 to 43 pct.)
- life sciences and health care (from 36 to 41 pct.)
- finance (from 33 to 38 pct.)
- professional services (from 24 to 33 pct.)
- industrial and construction services (from 32 to 33 pct.).
- 
Testing plays crucial role in the blockchain-based products, because it monitors its security level and effectiveness of data-gathering. In these fields there is no room for error.

## The increased significance of the security testing

The statistics presented in “The World Quality Report 2019-2020” confirm this trend. The survey respondents indicated that matters of security and risks are the most important for the IT strategy adapted by the company. The security testing scale in the cloud computing environment increased from 42 to 57 pct., what reflects development of the securities and the way such tests are performed, as well as adaptation of this line of automated testing to the fast pace of work in Agile methodology, because testing in the cloud enables for quicker implementation of the changes and reaction to them. In order to ensure the maximum coverage of security testing, the authors of “The World Quality Report 2019-2020” recommend introducing it on the earlier phases of the software development and carry it out with the tools for automated testing. What is even more interesting, the level of automation in security testing is surprisingly low – it discharges only 13 pct. from this kind of tests. However, if you look at the weighting attached to the particular aspects of security in the process of the software development, the increased significance of security testing seems inevitable:

Chart 3. Ranking the importance of certain aspects of secure development strategies.

![bb-blog-5-3.png](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/bb_blog_5_3_36f89419ef.png)

## Performance engineering as an essential mind-set and way of working

The time when automated testing and software development consisted only of delivering the specific solutions, and when each team had been working on its task separately from the other, can be described as the programming stone age. Nowadays the measure of success is creation of the product that all teams had been working on together. All people involved in the project (business line, project managers, developers, QA, testers) from the very beginning get acquainted with the business requirements and the needs of the end-user. They ask questions and talk about possible solutions. Thanks to such a workflow it is possible to implement many changes at an early stage, which prevents from the irreversible (or highly costly) damages that can occur after the deployment. In this way the performance engineering perfectly dovetails into the Agile objectives, because it postulates to look at the product from the meta-level and requires all teams to closely cooperate with each other. The effects are astonishing: the products that were created in this approach have a great market value in terms of the software, hardware, security, UX, effectiveness and performance.

large_bb-blog-5.png

medium_bb-blog-5.png

small_bb-blog-5.png

thumbnail_bb-blog-5.png

bb-blog-5.png

7 trends in automated testing that will dominate in 2021

7-trends-in-automated-testing-that-will-dominate-in-2021

There are many test automation tools available on the market, but there is still a need for the “appropriate” ones. What does it mean? What features should the testing software have to meet the real needs of testing teams?

In the [World Quality Report 2019-2020 study](https://www.capgemini.com/pl-pl/raporty/world-quality-report-2019-20/) conducted by Capgemini, questions were asked about the greatest challenges related to achieving the level of test automation desired by testers and QA. It turns out that since 2018 the percentage of respondents who say that “They do not have the right testing software” (42 percent) has not changed. More than a third of them said they use too many automation tools. There are many voices on blogs and internet forums, which the software available on the market often endorses with the words: “Cool by design, just a pity that it doesn’t work” or “Why do I have to bother with functions from the previous era? We’ve previously mentioned some of these features in our post about testing. So what should not be missing from a tool of the 21st century?

![bb-7-1.png](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/bb_7_1_ddd248eaf2.png)

## Reusing the same steps in different tests

This postulate will be understood by anyone who recorded 150 test scenarios, which differed only in a few nuances in the final stages, but each time required a login path, registration or any other repetitive activities. This is a pure waste of time, unnecessary frustration and a burden on the team’s efficiency. A tool adequate to the needs should provide a module that allows the use of the same step (e.g. logging into the system) in various scenarios, without the need to record it each time. Such functionality has another additional benefit – in the event of an error in any of the steps, its repair will be visible in each test scenario where the step was used.

## Free edition of already recorded tests

It is also a problem for testers who use tools to record test scenarios. In the case of iterations, which are the order of the day in teams working in Agile methodology, it is necessary to re-record the test scenario from start to finish. However, there are several tools on the market that allow you to make changes without having to repeat the entire procedure from the beginning. A useful function is also “overwriting” steps to already existing scenarios.

## Lack of coding skills should not be an obstacle to the free use of the testing software

Many companies probably remember the pursuit of automation engineers, who were supposed to be a dream come true about test creativity and developer efficiency. Unfortunately, such people are real “specimens” on the market, and in times of increased demand for testing services, programming skills are not as important as the right approach and experience. The test automation market should also follow this trend and propose tools with built-in modules that previously required knowledge of programming languages, such as the Page Object Pattern described in point 1.

## Testing software: possibility to edit the code of test scenarios

The fact that a given tool can be used without programming skills does not mean that it should not allow interference in the code of test scenarios. Despite the most sincere intentions, the creators of any tool are not able to predict all the situations that the testers will face, so the possibility of editing the code of the scenarios prepared with a given tool should be indisputable.

## Applying appropriate conditions for the recorded scenarios

Test automation tools that can be safely called dinosaurs require the user to enter a pause between individual steps or have an automatically set waiting time between moving from one stage to the next. Intelligent tools record page reload or add conditions that allow you to proceed to the next step. It all happens automatically – without having to manually set the appropriate parameters after each step in each test scenario.

## The ability to use the testing software from any browser and in the cloud

[SaaS](https://www.oracle.com/pl/applications/what-is-saas/) solutions are valued for their flexibility and efficiency. Their use, however, does not exclude even more far-reaching facilities in the form of the possibility of launching a given tool from any web browser. Thanks to this, the entire testing process is incomparably faster and easier – you do not have to wait, for example, for all the reports in the cloud to load, and you can start running a given test scenario right away.

## Automatic reports

The tester should immediately be informed about the types of errors and their areas of occurrence, and should be able to sort by test scenario type / functionality / error type.

## Access to the bug map

The bug map is a functionality offered by the most modern test automation tools. Thanks to it, it is possible to precisely locate those areas of the code that can generate the most errors, which is very useful, among others. for regression testing. It replaces tedious exploratory tests of the entire product and allows you to focus on those areas that generate the greatest risk before the next implementation.

## Working in the CI/CD model

The increasing popularity of Agile and DevOps methodologies has imposed a fast pace of work not only on programmers – also testers should be able to quickly adapt to changes and maximize their efficiency. A test automation tool tailored to these needs should enable easy integration with any system via API and, consequently, quick test launch. Parallel testing should also be a standard.

## UI and UX design

Looking at test automation tools from 10 years ago, one can get the impression that they were created with the assumption that an aesthetic and useful interface is intended only for the end customer, not for the person who has to provide/test it. Fortunately, over time, the vendors of this software saw the business value of UX design in their own projects. The easier it is to navigate and the more intuitive the test tool is, the greater the efficiency of testing. One would like to say – the tester is also a human! Fortunately, more and more producers do not ignore this rule anymore.

large_bb-blog-4.png

medium_bb-blog-4.png

small_bb-blog-4.png

thumbnail_bb-blog-4.png

bb-blog-4.png

10 features every test automation tool should have

10-features-every-test-automation-tool-should-have

One often hears an opinion that role of the tester in the IT project is undervalued or marginalised. That developers treat him as the person who at all costs wants to criticise the code. Some projects managers complain on the delays connected with the amount of time that testing requires. These are – to a large extent – damaging stereotypes, which come from the times when role of the tester only started to develop. How does it look in the mature IT team?

## Role of the tester in the project

Phrase “in the project” is key here. Tester is not an island in the most distant part of the office. It is a person who actively participates in the meetings of the whole team, which consist of developers, analytics, project manager and product owner. Depending on the specifics of the work that needs to be done, in the same group you can also find graphic designer, ux designer and other roles that are in demand. Tester stays in contact with all these people and communicates with them on the daily basis; he also takes part in most of the meetings. An exception is so-called independent tester, hired to test application that has already been created. He is purposely “protected” from contacting the business team so that he can keep fresh perspective. In that way the team can obtain suggestions concerning the way the product works from the person who doesn’t know the requirements according to which it was created. Sometimes such a job is done by the whole team of independent testers. In this article we will focus on the work of the tester in the IT project employed in the software development company, who from the very beginning participates in the creation of the software. In this case very significant for the way the tester’s role is structured is methodology of the work that particular IT team embraced.

## Role of the tester in the scrum

[Scrum](https://www.scrum.org/resources/what-is-scrum), i.e. the so-called Agile approach to project management, is currently one of the most popular and frequently used methodologies in the modern IT teams. Its opposite is so-called waterfall. Simplistically, it aims at planning the work of IT team (which should effect in the product ready to use by the customer) so that from the very beginning each phase of work is clearly defined, without possibility of implementing major changes. In effect tester obtains product ready to go, already after the implementation phase. Scrum in the Agile approach divides work in the project into smaller phases (so called sprints), so that a product is created “in parts”. It enables i.e. for modification of requirements formulated in the beginning, when work on the product just started, and quick reaction to mistakes/bugs can be identified later. For tester the most obvious consequence of this approach is that he starts working not on the already created product, but on its part – i.e. specific functionality, which developers commited in sprint.

![bb-blog-3-1.png](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/bb_blog_3_1_42ba0a462d.png)

## Role of the tester in the IT project – what are his tasks?

What are the specific tasks assigned to tester? These are the very basic ones:

- writing the test scenarios,
- performing the tests according to the scenarios designed earlier,
- creating bug reports (and adding them to the backlog),
- preparation of the testing documentation,
- if automated tests are the case – writing them.

It would appear that these tasks could be easily performed by one person alone and our previous claim, that the teamwork is crucial for the tester, was highly exaggerated. Nothing could be more wrong. Of course, each company sets its own rules and has different approach as to how tester’s work should be organised. Although our long-term experience shows that the more testers are included in the work of the whole team, the better for the product that is created in this way. In the scrum testers get acquainted with the product documentation and closely cooperate with analysts, who gather requirements for the application from the client and present the way it should work. Sometimes already at this stage tester discovers a mistake or lack of precision in the documentation, which could later affect the way product is created.

## Tester in the project – renaissance man?

A little bit. A good tester will not only study diligently the technical documentation, but also conduct thorough market research (if he is not a specialist in his branch). Thanks to that he has wide knowledge about state-of-the art technological solutions and he can suggest viable changes concerning product design or broadly understood UI/UX design. It can happen during direct talks with developers and UX designers or 15-minute meetings of the whole team (characteristic to scrum), when each member of the team gives update on his work.

Based on his professional knowledge or bugs identified in the particular phase tester can inspire the product owner as well as developers to act. Contrary to prevailing stereotypes, developers often appreciate testers’ suggestions, because they enable them to objectively assess the work already done and take on a new perspective on solving the problem. These remarks are different than standard code review, which focuses more on the technical issues and is performed by another developer. Offered by tester approach: “how should it work” and “in this part this functionality can perform better, because later it generates errors such as…”, is also very important in programmer’s work. Such proactive, focused on finding new solutions attitude is required in projects conducted in scrum.

## Role of the tester in the project – why his time is so valuable?

In the Agile approach there is a popular saying: „Fail quickly. Fail often. Fail cheap”. This methodology aims at quick bug identification, what in a broader perspective enables for delivering a high class product. Its maintenance would not generate additional costs because all potential risks had been identified and addressed when product was created. That is why role of the tester is crucial here. The most frequently used types of tests in this job are:

## exploratory tests

Exploratory tests mean testing on “living organism”, without sufficient documentation. In this phase you get acquainted with the application in the testing environment. It is rather starting point for creation of the suitable testing scenarios, in which your knowledge and experience play very important role. That is why novices are almost never engaged in these tasks. The undoubted advantage of exploratory tests is that they offer plenty of potential for detecting bugs that were overseen while creating requirements for the product and new user stories.

## regression tests

They are performed after each deployment. They shall check whether after the implementation of the new functionality the whole application (including solutions added earlier) works properly. This group of tests also includes smoke testing (assessment if main functionalities of the application work seamlessly after the implementation) and sanity testing (tests performed after the application had been debugged). This type of tests has plenty of potential for wasting testers’ time, therefore it can be easily replaced with automated tests. Thanks to them tester in the IT project can focus on creating new user stories and searching for yet undiscovered bugs instead of rather tiring research of the “old” parts of the code. Check out first Polish tool for automated tests – BugBug

## system acceptance tests (SAT)

System acceptance tests it is a real cherry on the cake in tester’s work. Its aim is not discovering the bugs, but confirmation that product that has been created functions perfectly and in accordance with client’s requirements.

As you can see, in a well-planned project work tester cannot be absent. His role is very important – there is a saying in the scrum that every member of the team is a developer, and it is not without a reason! It means that each person from the team shares equal responsibility for delivering the product perfectly adapted to the assumptions. And without the tester it is simply unattainable!

large_bb-blog-3.png

medium_bb-blog-3.png

small_bb-blog-3.png

thumbnail_bb-blog-3.png

bb-blog-3.png

Role of the tester in the IT project – what does it entail?

role-of-the-tester-in-the-it-project-what-does-it-entail

Software testing industry has no choice – a new era is coming and it will determine development of the whole branch and its products such as web applications testing software. This new trend will also positively impact the management of IT departments. 2021 can be a breakout year in this respect.

For several years we have been witnessing a growing disproportion between development of project management (including programming) and the situation of testers in the IT industry. The latter are often excluded from the benefits that Agile, DevOps and CI/CD entail. It does not mean that testers do not work in these methodologies or that they are not part of the teams that implement them. According to the report „State of the Agile”, published in 2019, testing and web applications testing was placed at the top of the activities which are managed this way:

![bb-blog-2-1.png](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/bb_blog_2_1_5f1b995952.png)

Within 13 years Agile from enthusiastically received novelty has become a dominating style of IT project management. In 2019 it was used in 97% of the surveyed businesses. DevOps technology was combined with Agile (or it was intended in the next 12 months) in 73% companies. Among benefits that Agile implementation brought they mentioned, among others, acceleration of software delivery, enhanced ability to manage changing priorities and increased productivity:

![bb-blog-2-2.png](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/bb_blog_2_2_6630bd8462.png)

If you ask testers about the consequences of Agile implementation, they would probably agree with the above mentioned benefits concerning management of the workflow. However, in terms of their responsibilities, conclusions would not be so encouraging. And it is not the fault of Agile itself – it is a systemic problem, which was only exposed by this particular methodology.

## How changes in the IT industry influenced the situation of testers?

They caused them to have much more work with the limited resources. Increased productivity of programmers provided testers with much more code to check, which makes web applications testing longer. Concurrently, their staff as well as software tools resources have not changed. It is also widely known that tester/QA’s positions are significantly less occupied than developers’, what was confirmed by StackOverflow survey from 2019, illustrating employment in the IT sector:

![bb-blog-2-3.png](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/bb_blog_2_3_fa7dae83da.png)

Year by year the number of employed as testers/QA increased by 1,1%. It is not much in comparison with robust growth of the programmers job market. Unfortunately, this situation has a negative impact on the whole IT sector, which suffers from the lack of adequate test coverage known as “testing gap”. It was first described by Jason Arbon in 2017 in the article AI for software testing”. He pointed out that the complexity of code and new products grows exponentially, but the increase of test coverage grows linearly.

![bb-blog-2-4.png](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/bb_blog_2_4_02c5d2ecb0.png)

Source: Jason Arbon

In practice it means lack of adequate test coverage of newly implemented products or their selective testing (often with the use of exploratory tests). For some of the services/applications it could result in disappearance from the market or at least cause severe image losses. That was in the case of Uber company, which created an app for automatic car driving. Undetected bug cost the life of an USA citizen, what sparked a huge media storm.

Web application testing is starting to be set aside. According to the research conducted in the United Kingdom by Diffblue, as much as 42% developers resign from writing tests for the sake of faster product implementation. Experts from Softwaretestinggenius calculated that direct costs of testing constitute as much as 25% of the whole project budget. Indirect costs, including consequences of undetected bugs, double this figure!

## Better web application testing = better quality products. How to do it?

Invest in appropriate tools, which will take the burden off the testers and enable them to optimise their work. The good news is that internationally things are quite busy. In Techavio report “Software Testing Services Market by Product, Geography, and End-user – Forecast and Analysis 2020-2024” experts anticipate that testing industry will grow 12,62% per year. The lead in terms of pace of development should take Asian countries, South America and Australia:

![bb-blog-2-5.png](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/bb_blog_2_5_35f6065999.png)

Source: Techavio.com

Interestingly enough, North America should be responsible for 41% of growth in the whole sector. Its main driver would be the mobile app market, which for the sake of constant introduction of the new features and rapidly changing user requirements undergo continuous changes. This phenomenon is also visible in Poland, where the scale of mobile devices use is constantly increasing:

![bb-blog-2-6.png](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/bb_blog_2_6_4ecc0a5f6d.png)

## New trends in the testing industry – what quality would they bring?

We observe an increase in the amount of code provided for testing, as well as in demand for mobile products (not only – websites and e-commerce services are also thriving). Till now the common response to these challenges were automated tests. Although the tools that provide them are indispensable in the tester’s work, unfortunately many of them do not always correspond to rapidly changing customers’ expectations. Some of them were created several years ago and were a huge help in less dynamic circumstances – good examples could be commonly available in such software functions of recording tests. In a dynamically changing Agile environment, where new functions are often added with each sprint (or once established ones are modified), older tools do not enable for easy change of testing parameters. In practice it comes down to the necessity of recording each step from scratch, what loses the purpose of automation, that is diminishing the amount of work that has to be done manually…

## New trend no. 1: Page Object Pattern without programming

Page Object Pattern is a notion well known by test programmers. Very briefly, it allows to group elements in separate classes so that the change in one of them does not impact remaining parts of the website. An example can be a log-in page, where particular fields (e.g login, password, “log-in” button) constitute distinct components. Thanks to that in the case of the malfunction in any of them it is possible to quickly implement necessary adjustments without interference in the whole code of the website.

New tools for test automation would enable the implementation of solutions such as Page Object Pattern without necessity of programming. Each element of the website (e.g already mentioned “log-in” button) would be available as a separate graphic element, which can be freely utilized according to the needs of different tests. Referring to the log-in website: if the test scenario includes it, it could be quickly added using drag-and-drop function from the main panel. Such a solution has one big advantage: if any of the elements breaks down, it could be easily fixed in all scenarios at once just by implementing changes in one place, because update is made in all tests that this particular element was used. Hence we can officially announce the end of an era when tests had to be recorded from scratch each time errors occurred in any of their elements.

## New trend no.2: possibility of editing once recorded tests

Complete freedom and maximum speed are the wave of the future in creation and editing of tests. Till now modification of the already created scenario of automated tests was possible only after the test was recorded once again from the beginning after the end. New generation tools would enable not only for editing particular elements of the test that has already been recorded, but also for adding new parts to it at any time.

## New trend no.3: great UX/UI design

It is only a matter of time when we will reminisce about interfaces of the “old” programmes for test automation with the same tenderness as one that accompanied transition from Pegasus to Playstation. UX and UI design should not be overlooked in tools for people, whose task is to ensure the best experience for the end-user! This paradox is indeed an old reality: for several years testers have been working on programmes, which graphic design epitomises trends from the last decade… It also influences the quality of their work and it is great news that there are tools on the market that aim at changing this situation.

## New trend no.4: launching highly-specialised tests from the web browser

Cloud solution does not exclude fast access to the tool from the web browser – this motto would guide authors of state-of-the art programs for test automation. Tester does not have to acquire advanced knowledge about programming nor technical skills of administrators: installation of the web browser plugin should suffice for effective work. The same applies to test launching: new generation tools would enable its start, test recording and its instant modification exclusively from the web browser. Testers no longer have to wait for test launch and reports from the cloud – they can act immediately at the minimum technical requirements.

## New trend no. 5: real, not fake, reliability and stability

There are many tools that promise a lot, but in practice are disappointing and give testers a feeling of wasted time. Users that felt let down wrote several articles about such tools – here is an example». The future belongs to these test automation tools, which will give testers feeling of control and freedom in terms of taking action, concurrently delivering what they have promised in specification and marketing materials.

![bb-blog-2-7.png](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/bb_blog_2_7_6bd4ef461b.png)


Our app is a [modern software tool for testers and software developers](https://bugbug.io/) that allows recording, editinng and running reliable end-to-end tests for web applications.





large_bb-blog-2.png

medium_bb-blog-2.png

small_bb-blog-2.png

thumbnail_bb-blog-2.png

bb-blog-2.png

New era in testing industry – which trends will dominate in 2021?

new-era-in-testing-industry-which-trends-will-dominate-in-2021

At the first glance the IT industry may seem to be the main winner in coronavirus pandemic: it fully embraced remote work long ago, easily adapts to changes in accordance with Agile methodology and in the age of digitisation programmers are still in demand. But there is a “but”.

Demand for the IT services in Poland was growing before the outbreak of a pandemic – according to the S[tepStone report for Business Insider Polska](https://businessinsider.com.pl/rozwoj-osobisty/kariera/pracownicy-poszukiwani-w-2020-r/hsjbb7f) published in December 2019, the most sought-for profession in 2020 were predicted to be an IT specialist/programmer/developer. IT industry dominated the whole ranking, because the second place took  an IT project management leader/project manager and the fourth – an analyst. Demand for such qualifications can be considered to be global, because “technological” positions also dominated the annual LinkedIn report with projections for 2020. First came an artificial intelligence specialist, followed by (amongst others) the full stack developers, IT security specialists and developers.

## IT industry and coronavirus pandemic – additional incentive for growth?

The coronavirus pandemic negatively impacted the labour market in each country that was affected by it. However, the IT industry belongs to those which emerged from it unscathed. You can even say that it won. Tyler Cowen, Professor of Economics at George Mason University, who regularly publishes his opinions on the Bloomberg website, has written in April 2020 that “Technology is indispensable to lead a normal life”. Nowadays (which means in the era of the coronavirus pandemic), according to Cowen United States of America would be forced to introduce „track and trace” system. One of the conceptions describing its implementation is the smartphone application making use of geolocalisation, which would send information that its user is nearby the person infected with COVID-19. Such ideas, backed up by real needs, would be more and more prevalent, because the pandemic almost overnight changed the way we live:

- The necessity to introduce remote work forces to do so even those companies, which so far have been sceptical towards this solution.

The global giants were prepared for such a scenario: Twitter, Google or JPMorgan, as reported by CNBC, have been testing the possibility of implementing the remote work in advance. Atlassian and Slack obliged all their employees to work remotely. In this context the title of the GitLab report from March 2020: “The future of the work is remote” does not seem accidental. In the study took part more than three thousand workers representing different branches. Here is the portrait of the remote worker that came out of it:

![bb-blog-1-1.png](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/bb_blog_1_1_1859ebbe4a.png)

Source: GitLab.com
Advantages of the remote were also emphasised in the analysis prepared by FlexJobs and Global Workplace Analytics. It shows that in the last ten years popularity of the remote work in the United States has increased by 91%. Moreover, 85% of the surveyed companies claimed that introduction of the remote work boosted efficiency of theirs employees. As many as 57% respondents working remotely reported increased job satisfaction.

The landscape of the Polish labour market is slightly different: in the Homejob survey, conducted before the pandemic outbreak, only 36% workers declared that they have possibility to work remotely. The changes that we witness now have already modified this statistics. According to the respondents of Devire survey, in the situation of pandemic 7 from 10 companies which earlier had not wanted to enable remote work, changed their mind. The IT industry would play a big role in this transformation: the main task would be creating and implementing appropriate systems for remote work and ensuring security of the companies’ data.

- E-learning would be developing alongside the telemedicine; events industry would see the chance to bounce back from coronavirus crisis in the videoconference tools.

IT supports and for sure would be supporting transfer of the medicine and education to the internet. Here also the boundary between the giants, excellently prepared for such a transition (i.e. Harvard), and “others”, which would have to take accelerated lesson about new technologies, is visible. The events industry, which worth is estimated for trillions of dollars, is also looking for an ideal way to tackle a decline. Many high profile events moved to the internet – i.e. Adobe Summit, SAS Global Forum or Google Cloud Next’20, which is also called Digital Connect and would be free of charge. The organisers of these events want to make the digital experience to feel as close as possible to the “real” one. It could be done only with an appropriate technical support. Online streaming or teleconferences which shall substitute for traditional networking would be a challenging task for not only for the developers responsible for building proper infrastructure (and its maintenance), but also for the organisers and participants of the events.

- „Pandemic is the mother of invention” – in this unprecedented situation IT industry is asked to create new systems and applications, as well as to increase efficiency and capacity of the ones that are already in use.

[The machine learning model helps to develop efficient therapy for COVID-19 infection](https://www.bbva.com/en/technology-against-coronavirus/). New programmes, which enable for providing medical services from a distance, are created. In Israel there is ongoing work on the Diagnostics.ai – the tool for automatization of the coronavirus tests, which can double their daily number. In Madrid special application named „Corona Madrid” was implemented. It was designed for the people who suspect that they could be infected – it helps to identify the symptoms and suggests the places where person can get help. WHO launched the chatbot which provide the information about coronavirus and special module in social media named „Coronavirus information center”. There is no doubt that we are witnessing the beginning of the new era.

- In the face of economic crisis many companies close stationary shops and invest in online services.

As a consequence new websites, CRM-s and databases are in a great demand. According to the International Monetary Fund, in 2020 growth in the world economy will fall by 3%. It is expected to be the worst economic crisis since the 30s of the XX century. Projections for Poland also are not optimistic – according to the IMF our PKB this year will shrink by 4,6%. This means wave of redundancies, but also transfer of the battle for consumers into the Internet.

## Threats for the IT industry in the coronavirus era

Theoretically, all signs in heaven and on earth indicate growing demand for this type of services, what should result in increasing returns. In my opinion external factors are not the ones that can pose potential threats for the IT industry – we should take a closer look at the internal risks. Paradoxically, they are the effect of the rapid development of the branch –increase of productivity in the teams that implemented scrum methodology (according to McKinsey it was 27% growth in comparison to the projects managed in waterfall) and the fast pace of software development. The phenomenon that can cause deterioration of the quality of the software that is produced as well as prolongation of the delivery period is so called testing gap, which means lack of the appropriate test coverage.

These risks were already pointed out in 2018, in the article „Situation of the testers on the market”. Unfortunately, there has been relatively little change in this area:

„Complexity of the tests increases exponentially with the appearance of the new functions, and test coverage grows linearly, because the tests can be added only individually. In effect the amount of code produced is bigger than testers’ capacity, what leads to the gap in testing coverage (so called testing gap).”

There is also serious labour shortage, as the results of the „Developer Survey 2019” indicate. It was conducted among more than 80 thousands IT workers from all over the world. In testing specialised only… 7,8% of the respondents – developers are the overwhelming majority here:

![bb-blog-1-2.png](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/bb_blog_1_2_0da6894e8f.png)

Source: Stackoverflow.com

Additionally, the complexity of the applications created by developers, as well as the amount of data that should be processed, is constantly growing. Because [software testing professions](https://bugbug.io/blog/end-to-end-tests/role-of-the-tester-in-the-it-project-what-does-it-entail/) do not enjoy the same level of popularity as the positions of the developers, recruitment and training of the new employees do not seem to be enough to address it. The solution to this problem could be freeing resources and making the best use of the knowledge and competencies that are already available on the market. How can you do that?

## Implement the appropriate software testing automation tool

“Appropriate” means one that can not only enable you to speed up testing process, but also would be a solution that offers higher quality than the others advertised on the market. What requirements it should meet?

- Availability to all testers, including those who do not know a thing about programming.
- Reliability and effectiveness, which exclude non-deterministic tests (so called flaky tests).
- Support for the tester in terms of bug analysis and identification of the causes of their appearance.
- Easy configuration and handling thanks to appropriate interface design (in accordance with the newest UI/UX recommendations).
- Recording of the tests.
- Ability for auto-recovery of the tests.
- Ready-to-use templates of the single steps in test scenarios, which you can group and configure in any way.
- Tests modelling displayed on the graph, what enables you to quickly estimate the phase that the particular test scenario is in.
- Easy creation of even the most complicated test scenarios thanks to the fact that your programme operates on many different tabs, browser windows and frames.


large_bb-blog-1.png

medium_bb-blog-1.png

small_bb-blog-1.png

thumbnail_bb-blog-1.png

bb-blog-1.png

IT industry in the coronavirus era not always wins. What risks should you be aware of?

it-industry-in-the-coronavirus-era-not-always-wins-what-risks-should-you-be-aware-of

Test Automation Expert and Test Architect. He is a seasoned tester who can’t stay long in one place. He is looking for perfection and a place to challenge his skills.  He gathered experience working for different companies with different working models, From small to big corporations, From Products via In-house development to software house. Thanks to that he has a wide perspective on testing quality and delivering value.\

[linkedin.com/in/wyrodek/](https://linkedin.com/in/wyrodek/){ref="nofollow"}

large_https://i0.wp.com/wyrodek.pl/wp-content/uploads/2017/01/T1A0194.jpg

medium_https://i0.wp.com/wyrodek.pl/wp-content/uploads/2017/01/T1A0194.jpg

small_https://i0.wp.com/wyrodek.pl/wp-content/uploads/2017/01/T1A0194.jpg

thumbnail_https://i0.wp.com/wyrodek.pl/wp-content/uploads/2017/01/T1A0194.jpg

https://i0.wp.com/wyrodek.pl/wp-content/uploads/2017/01/T1A0194.jpg

Maciej Wyrodek

Test automation, especially end-to-end, is currently enjoying a revival. A lot of new tools and approaches have emerged over the last few years. Selenium has finally launched version 4, Cypress and Playwright have brought coded automation to a new level and tools like [Bugbug.io](https://bugbug.io/), [Mabl](https://www.mabl.com/), [Testim.io](https://www.testim.io/), have revitalized the idea of record and play automation. Since a lot is happening, let's take a look at automation and compare the strengths and weaknesses of 3 approaches to Automation: Custom Made Frameworks, Out of the box framework and Record and play. We will look at the pros and cons of all the solutions and when they work best. However, first we need to discuss the reasons for using test automation.

Why we automate tests
-----------------------------------

If you ask typical testers why they automate tests, they can provide many reasons: from shortening the Regression time, improvement of test coverage, to providing faster feedback. Ultimately, if we look hard enough, all goes back to the one and only goal of companies, as stated by [Eliyahu M. Goldratt in The Goal](https://en.wikipedia.org/wiki/The_Goal_(novel)) - Making money. Testing by itself often is considered a cost - as it does not bring more money. But let's not forget that more money can be made not only by selling more - but by increasing the margin - either by making the product worth the price or by reducing the cost. Testing can help to protect the bottom line, by reducing the cost of mistakes and finding them more easily and faster. This also leads to reducing the cost of customer support, refunds, etc. But as [DevOps](https://itrevolution.com/the-devops-handbook/) and [Modern Testing](https://www.moderntesting.org/) state - testing can synergize with a great experimentation culture, leading to possible delivery of a quality product in a shorter time and cost-savings on the development cycles. But this cannot be achieved without robust test automation. And automation needs to follow a few rules. For that, we need to consider the properties of great automation.

Traits of good automation
-------------------------

For automation to be useful in the DevOps world, it needs to be: 

-   Reliable 

-   Fast

-   Easy to write and maintain new test 

-   Easy to run

### Reliability

If test automation is not reliable, we won`t believe its results. As a result, we will still depend heavily on manual rechecking of everything and that is time-consuming. Or even worse, it can lead to wasting time as now no one will even bother to check the results.

### Fast

The development cycle is fast. If developers need to wait for the results of their work more than a coffee break, they will switch to the next task. Now the results will need to wait which leads to the cost of context switching. And longer [lead time of a task](https://kanbantool.com/kanban-library/analytics-and-metrics/kanban-definition-of-lead-time-and-cycle-time).

### Ease of Writing and Maintaining

If automation should be a regular part of a software development lifecycle, their benefit must justify the cost. If writing a test takes 4 days, it'd better be worth it - especially if developing  the tested feature takes less than that. The same holds true for maintenance: we need to be able to easily understand why a test has failed and whether this is a code issue or a test issue and how to fix it.

### Ease of running

This relates to being fast: anybody who needs to use a test to check a code should be able to do it without extra hoops, for example automatically running them locally or via CI/CD.

How to choose the best automation approach
------------------------------------------

There are a lot of ways of choosing tools for a project. When looking for tools that resolve a problem, we need to choose a tool that contributes the coveted value. The list of questions requiring answers is long, but today we will look at only one aspect, namely automation development and maintenance cost. We will look at the 3 most popular approaches, starting from ones requiring most coding and maintenance effort, like "Custom Frameworks", through usage of "Out of the box" Frameworks and finishing with Record and Play tools. We will look at the pros and cons. As it is my belief, all these approaches have made a name for themselves on the market and suit different project needs.

**Pros**

-   Great Customizability

-   Ownership

-   "On-site support"

**Cons**

-   Huge Maintenance Cost

-   Time Consuming

-   Requires lots of skill to Maintain

When I was starting test automation, this was the way to go. Mainly because Selenium Web Driver was the tool of choice back then. And as its description stated, [Selenium automates browsers](https://web.archive.org/web/20120208115222/http://seleniumhq.org/). Selenium was just a tool that did exactly what it said it delivered.

It didn't provide any functions for running test, or making assertions. 

At that time, practically every company had to build its own framework.

Often it was the role of test automation testers. In most cases, they were just learning how to automate. Making lots of mistakes in the process. But some companies managed to make great frameworks and, as a result, decided to share them with others. Some of them are still in use and developed but other have not stood the test of time. [Ocaramba](https://github.com/ObjectivityLtd/Ocaramba),[Selino](https://github.com/TestStack/TestStack.Seleno), [Serenity](https://serenity-bdd.github.io/theserenitybook/latest/index.html),[Selendie](https://selenide.org/), [Protractor](https://www.protractortest.org/#/)

This approach is still used today. Why?

### Pros

The main advantage of this approach is control. It gives a company the greatest level of control in customization. If a specific feature for a project is required, it is relatively easy to find someone who can develop this functionality.

Additionally, it gives a company considerable independence; there is no risk that at one day, the tool support will end or [repo will be deleted](https://nakedsecurity.sophos.com/2022/01/11/javascript-developer-destroys-own-projects-in-supply-chain-lesson/), leading to massive issues.

What is more, there is someone who knows how the tool works on site and can teach new team members.

### Cons

The main drawback of this solution is the cost - maintaining internal test automation framework costs time and money. Many companies underestimate the cost of developing test automation.

Time is of essence here: this automation needs a long time to be developed,  from creating a framework to writing a test. And this is the aspect that can be most damning. As discussed before, for automation to bring value, it needs to provide fast feedback.

Creating a custom framework is as big a project as any and it requires a good dose of skills, ideas, and planning. This means a lot of expertise while typically, companies are not willing to spend precious resources on test automation.

Out of the box frameworks: Pros & Cons
--------------------------------------

**Pros**

-   Shorter time needed to write first test

-   Much less need for framework maintenance

-   Lots of available online trainings and knowledge

**Cons**

-   Often only limited support from community

-   Writing and Test Maintenance takes time

-   Huge dependence on framework developers
As mentioned before, the growth of Selenium was accompanied by a lot of frameworks created on its base. Over time, the issues with the web driver became increasingly prominent, leading to the re-emergence of new automation frameworks not based on Selenium: [Cypress](https://www.cypress.io/), [Puppeter](https://github.com/puppeteer/puppeteer), [Playwright](https://playwright.dev/), to name a few most recent ones.

All these tools had one thing in common - they provided the user with out-of-the-box, ready options to write automation. No matter if this was Ocarmaba based on Selenium or Cypress: the goal of the solution was the minimalism needed for framework maintenance.

These tools usually provide built-in test runners, an extensive assertions library, boiler plates for starting writing project and tests, and - most importantly- a built-in mechanism to deal with test flakiness and general instability.

Additionally, these frameworks usually provide their own philosophy of writing test automation. As long as what we need to do is with accordance to their approach, everything goes well. Problems start when we need to go beyond what is suggested.

Until recently, the best example has been a test covering multiple domains in Cypress, but as of [version 9.6](https://docs.cypress.io/guides/references/changelog#9-6-0), it is no longer such a grievous case.

In another scenario, we need to rely heavily on framework developers, if they introduce breakthrough changes. Usually, we are left with two options: update and fix the code. Alternatively, don't update and hope that there won't be any issues in the framework that need fixing.

### Pros

Compared to the custom frameworks, we can start writing tests almost immediately without the need of spending weeks or months on writing the toolset first. Similarly, we don't need to wait whenever the operating system or browsers have an update. The tool maintainer will update the framework for us. And since most tools in this category are open-source ones, there is lots of information easily available online.

### Cons

As mentioned before, these tools have their own test philosophy, which is much more restrictive. Quite often support is limited - these tools are usually open-source and support in case of issues can be unavailable where we need to resort to forums and chats where we hope someone will answer a question; sometimes we get the answer in minutes but it may never come just as well.

However, the greatest risk of all is that we are at the mercy of framework developers - these tools become a part of our code base and if we want to keep support, sooner all later we will need to update even if we don't want to. A case in point is Cypress 10 - [it introduced breakthrough changes](https://docs.cypress.io/guides/references/changelog#10-0-0). As a result, many plugins and customer functionalities won't work without extensive updating.

Record and Play Testing Tools: Pros & Cons
------------------------------------------

**Pros**

-   Fast Test Writing Time

-   Low Entry Level

-   Professional Customer Support

-   Easy Scalability

**Cons**

-   Usually cloud-based, requires access to test environments

-   Still young, "untested" technology

-   Requires a new mindset

-   Data management can be quite complicated

-   Usually a commercial paid tool


The idea of record and play automation is not new, even Selenium has Selenium IDE which, after a few years on developments lately, has won some users' recognition.

Microsoft had Coded UI Test as part of the Visual Studio from 2010 and only in [2019 announced it deprecation](https://devblogs.microsoft.com/devops/changes-to-coded-ui-test-in-visual-studio-2019/)

Back then this technology was long before its time, the frameworks were very unstable and the maintenance of this test was problematic.

But in the last 4-5 years, a lot has changed. Solutions like [Bugbug.io](https://bugbug.io/), [Mabl](https://www.mabl.com/), [Testim](https://www.testim.io/) have taken over what modern technology has to offer and reinvented the concept.

### Pros

Most coded automation requires great investment of time into writing and maintaining test - usually even a trivial change costs around half an hour. In case of Automation Records and Play, test are usually meant to be disposed of  after they  stop working, as it is much easier to re-record a test instead of fixing it. This is of special importance for startups and projects with a lot of changes in the UI layer. When a team is spread thin, it is hard to find the time to write a test. But usually, you can record a few tests while checking if what has been done even works.

On the other hand,e out-of-the-box coded frameworks offer usually an easier learning curve. Record And Play tools usually do not require coding knowledge - although in more complicated cases some JavaScript knowledge can prove very helpful.

Since most of the record and play tools are paid solutions, it also means that companies have professional Trainers and Customer Support to help when issues arise. Usually, it means that in the case of young projects, we can even influence their feature roadmap.

These modern record and play solutions come with their own cloud runners, making scalability of test automation unbelievingly easy. Typically, paralyzing a test takes a few clicks.

### Cons

Unfortunately, Cloud Running is also a huge Con. First and foremost, we need to give this tool access to our test environments. 

An additional issue related to the Cloud is Term of Service - not all projects and customers can afford cloud services.

Most of these Record & Play tools are relatively young and not battle-tested. For long-term projects, this uncertainty can be very risky.

In my opinion, the biggest issue is Data Management. So far, no Record and Play solution has provided an easy way to manage test data. And this can be a huge blocker. Consequently, we need to change our approach to E2E test automation, with a desire to cover all the important and easy paths as fast as possible, to make this automation possibly most valuable.

Lastly, there is the price - cloud solutions are not cheap and if we need to run a lot of tests, it may be quite expensive. But some tools like [BugBug](https://bugbug.io) are on the other hand very cheap and offer unlimited cloud runs for a fixed mohtly fee. Arguably, it is still less expensive than maintaining coded automation and its infrastructure.

No clear-cut distinction
------------------------

Before the final thought, some clarification is required:

As much as we humans like clear categorizing, the real-word borders are much blurred.

For example: should Playwright be in the Custom frameworks or in the Out-of-the-box category? After all, its strength is the ability to use its own test runners, reporting services and easy building of the new features we need.

Similarly, it has the Record and Play functionally and so did Cypress (at the moment of writing this post, the latest version of Cypress is 10.3 and since 10 Cypress Studio, which allowed test recording, has been deactivated).

There are also tools that I like to refer to as combine-harvesters: basically, they do everything and claim that they can test anything, from mobile via web to desktop and many others -- the examples include TOSCA and Test Complete.

That's why we need to look at the numerous different dimensions of automation - and how much coding is just one of them.

Summary
-------

It is my belief that over time, both no-code development and record and play automation become the dominant players on the market. Yet there is a lot of room for improvement. All the types of approaches to automation, starting from custom frameworks, or using out-of-the-box coded solutions ending on Record and Play tools, have their uses on the market.

They answer to different needs and solve different problems. So, before selecting any tool, it is always good to ask yourselves: "What are we trying to gain?"

large_image (3).jpg

medium_image (3).jpg

small_image (3).jpg

thumbnail_image (3).jpg

image (3).jpg

Record and Play vs Coded Automation testing - how to choose what works for your team

Features

Pricing

Roadmap

Case studies

Privacy policy

Terms of service

About us

Book a demo

Contact

Get more tips and product related content. Zero spam.

Don't miss any updates

End-to-end testing made simple! BugBug is a new, more intuitive tool for test automation of web apps. Get all the benefits of test automation without hiring new people or infrastructure changes.

BugBug.io | Intuitive and reliable test automation for startups

large_Open Graph 1200x630 3.png

medium_Open Graph 1200x630 3.png

small_Open Graph 1200x630 3.png

thumbnail_Open Graph 1200x630 3.png

Open Graph 1200x630 3.png

Blog about test automation

Updates

Product

Guides to automation

small_pawel-about-512.png

thumbnail_pawel-about-512.png

pawel-about-512.png

Paweł Bylina

Are you a tech leader, CEO or CTO in a tech startup? You want to implement automation testing for your SaaS product? We've written this guide especially for you. It's packed with actionable tips on how to approach test automation in an agile way.

We're going to tackle the following questions:

-   What is the business goal of automation testing?

-   How to implement automation incrementally?

-   What resources do you need to develop automated tests?

-   What are typical problems and obstacles and how to overcome them?


The strategic approach to automation testing
--------------------------------------------

Automation testing is a very broad term. There is no single goal of automation. Depending on the level of automation you will get different benefits for your organization.

*Figure 1: Strategic matrix of different goals of test automation*
![bugbug-goals-of-automation-testing.png](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/bugbug_goals_of_automation_testing_9308b9cb1b.png)

See this strategic matrix above. You can achieve all of these goals, but not all at once. You should pick the right order of goals, starting with the ones that are low-effort and high-value. 

To achieve each of these goals, you need a different combination of resources and skills. We'll guide you through each of the goals, explaining the approach, time & resources and how to achieve it with BugBug.

Step 1: low cost & high impact goals
------------------------------------

### **Goal:**

-   Prevent loss of business due to production downtimes

### **Approach**

-   You don't need to hire new people, you can do it yourself

-   Create 5-10 automated tests and run them every hour to monitor your production, for things like login, landing pages, registration form, contact form, basic app navigation

### **Time & Resources**

-   2-4 hours of your own time

### **BugBug features that you should use**

-   [Recording tests](https://docs.bugbug.io/creating-tests/recording-the-tests)

-   [Cloud runs](https://docs.bugbug.io/running-tests/in-cloud-on-server)

-   [Schedules](https://docs.bugbug.io/running-tests/schedules)

-   [Notifications](https://docs.bugbug.io/collaboration/notifications)

-   [Re-recording test steps](https://docs.bugbug.io/recording-tests-steps/re-recording-steps)

Step 2: low hanging fruit
-------------------------

### **Goal:**

-   Improve team collaboration on quality assurance

### **Approach**

-   Give all the scrum teams access to the testing platform

-   Share direct links to failed tests in your tickets or on a chat

-   Configure who gets notifications

### **Time & Resources**

-   Inviting people only takes 5 - 10 minutes, but you should spend 2-4 hours on communicating its purpose

### **BugBug features that you should use**

-   [Invite users](https://docs.bugbug.io/collaboration/inviting-team-members)

-   [Project settings > Notifications](https://docs.bugbug.io/collaboration/notifications#choose-when-people-get-email-notifications)

Step 3: incremental development of automated tests
--------------------------------------------------

### **Goals**

-   Save time on manual regression testing

-   Release faster

-   Prevent cost of regression bugs

### **Approach**

-   Talk to the whole team about what you want to achieve and what's the incremental plan

-   Choose 1-2 people responsible for automated tests 

-   Create dedicated test user accounts that will be used for automation only

-   Run automation on some testing environment (staging)

-   Create separate test suites for staging and prod (run all tests on staging, but only essential smoke tests on prod)

-   Add more tests incrementally, 2-3 test cases per week (after 3 months you should have about 50 test cases automated)

-   Start adding [data-test-id attributes](https://docs.bugbug.io/preventing-failed-tests/selectors#use-custom-attributes-for-selectors) in your HTML

### **Time & Resources**

-   You will see the benefits after 4-8 weeks of ongoing work

-   You need to engage 1-2 people on the team. Make sure they have at least 2 hours per week to work on automation.

-   You need at least one technical person on the team who knows how selectors work.

### **BugBug features that you should use**

-   [New test from here](https://docs.bugbug.io/creating-tests/new-test-from-here)

-   [Duplicating tests](https://docs.bugbug.io/creating-tests/duplicating-tests)

-   [Components](https://docs.bugbug.io/editing-tests/wip-make-component)

-   [Variables](https://docs.bugbug.io/editing-tests/variables)

-   [Profiles](https://docs.bugbug.io/editing-tests/variables#profiles) to [test on different environments](https://docs.bugbug.io/editing-tests/variables#work-with-different-development-environments)

-   [Suites](https://docs.bugbug.io/organizing-tests/suites)

-   [Manually editing steps](https://docs.bugbug.io/editing-tests/manually-creating-the-test)

-   [Testing registration](https://docs.bugbug.io/editing-tests/variables#test-user-registration-and-login-using-variables)

-   [Custom javascript](https://docs.bugbug.io/editing-tests/custom-javascript-actions) might be needed

Step 4: integrate automation with CI/CD
---------------------------------------

### **Goal:**

-   Improve software development lifecycle, being sure that no code is merged without tests

### **Approach**

-   Integrate automation with your development pipelines

-   Trigger automated tests with every pull request

-   Show test results directly in your repo tool (Bitbucket, GitHub, GitLab, etc.)

-   Ask developers to run automated tests locally on the localhost environment before they commit code

### **Time & Resources**

-   You will see the benefits after 4-8 weeks of ongoing work

-   You need to engage 1-2 people on the team. Make sure they have at least 2 hours per week to work on automation.

-   You need at least one technical person on the team who knows how selectors work.

### **BugBug features that you should use**

-   [CLI](https://docs.bugbug.io/integrations/api)

-   [Overriding variables](https://docs.bugbug.io/integrations/api)

-   [Running on localhost](https://docs.bugbug.io/running-tests/test-in-your-browser)

Step 5: increase coverage
-------------------------

### **Goal:**

-   Get 100% test coverage, be confident that all the features, all the browsers, all the devices work correctly

### **Approach**

-   Pick your battles, choose what is the most essential to automate

-   We recommend you keep adding 1-2 test cases every week and don't try to achieve the impossible 100% coverage

### **Time & Resources**

-   To automate everything you would need an Infinite amount of time :)

-   To automate 2-3 new test cases per week you still need 2-3 hours weekly plus a bit more experience with tests and javascript.

-   Note that the more test cases you have, the longer the maintenance. The trick is to find the right balance.

### **BugBug features that you should use**

-   Project settings > Browser width (for [testing mobile screens](https://docs.bugbug.io/workflow-tips/mobile-version-testing))

-   Lots of custom Javascript

-   [Javascript variables](https://docs.bugbug.io/editing-tests/variables#javascript-variables)

-   [Javascript assertions](https://docs.bugbug.io/editing-tests/custom-javascript-actions#advanced-javascript-assertions)

-   [Parallel running](https://docs.bugbug.io/running-tests/parallel-mode)

Common problems and solutions in implementing automation testing:
-----------------------------------------------------------------

Here are some typical problems you may encounter when you want to start automating tests for a SaaS product. We've made a short list with possible solutions that worked for other [BugBug clients](https://bugbug.io/case-study/).

### **Captcha**

You need to [disable the captcha only for automation testing](https://docs.bugbug.io/troubleshooting/captcha-in-automation-testing). Unfortunately, it's not trivial and you need to read through captcha docs or ask a developer for help.

### **Date-based features, calendars** 

Is your app searching train timetables for tomorrow? It's difficult to automate because the date and the expected outcome can change every day.

Solution A: In the automated test, pick a date that's far in the future. Then update the tests regularly, changing this date and the expected outcome.

Solution B: create a dedicated environment for automation with a mocked-up hardcoded date on the backend

### **Dynamic lists such as search results, product listings**

It's hard to automate a test if some data is changing all the time. You need to create independent tests that are always sure what to expect, for example, if you're checking a list of promoted products on your homepage, your test should first go to the admin panel and mark such products as promoted.

You can also have a separate dedicated test automation environment where you have a hardcoded set of data.

### **Can't use the same user email twice**

Create a new user with each test - use the [built-it variable "testRunId"](https://docs.bugbug.io/editing-tests/variables#use-built-in-variables-for-dynamic-or-random-values) in combination with a [temporary email catcher](https://www.google.com/search?q=temporary+email&ei=ARVtY_X8IomKrwS_i4k4&ved=0ahUKEwi1gZ2u86P7AhUJxYsKHb9FAgcQ4dUDCA8&uact=5&oq=temporary+email&gs_lcp=Cgxnd3Mtd2l6LXNlcnAQAzIECAAQQzIFCAAQkQIyBQgAEJECMgUIABCABDIFCAAQgAQyBQgAEIAEMgUIABCABDIFCAAQgAQyBQgAEIAEMgUIABCABDoKCAAQRxDWBBCwA0oECE0YAUoECEEYAEoECEYYAFDhAVjhAWC1CmgCcAF4AIABa4gBa5IBAzAuMZgBAKABAcgBCMABAQ&sclient=gws-wiz-serp) always to run tests with a fresh new user. You can also delete these users at the end of the test by recording test steps in an admin panel (if you have one) Read more in how to test user registration.

### **Selectors are not reliable**

The only way to make sure your tests are stable and easy to update is to start using "data-testid" attributes in your HTML structure, [dedicated attributes for automation testing selectors](https://docs.bugbug.io/preventing-failed-tests/selectors#use-custom-attributes-for-selectors). You need to communicate that to the front-end developer and make sure you remember about it during the code review. No worries, adding additional attributes have no risk, it's just additional data for automation with no other function, it will not introduce new bugs.

### **Multilanguage**

Your page can run in different languages and you want to verify all of them. Try to assess the risk of having a bug only in one language without affecting the others. Perhaps it's not likely that such a bug would occur?

If you still need to test different languages, you have two options:

Option A: Record separate tests for each language. Hard to maintain on a bigger scale

Option B: Use variables & profiles to run the same test on different languages

-   Add data-testid to all of your app elements, so that the selectors are independent of the app language and the displayed test 

-   Add a way in your app to change the language via a URL, for example, "myapp.com/?lang=fr"

-   Create a variable "language code"

-   Create different profiles with different language codes

-   Replace the language code in the "Go to URL" step with the variable

-   Create separate suites for all the languages, overriding their profile with a chosen language

**Tests take long time to finish**

[Upgrade your BugBug plan](https://bugbug.io/pricing/) to be capable of [running cloud tests in parallel](https://docs.bugbug.io/running-tests/parallel-mode)

### **Downloading files**

Currently BugBug can't verify if the files have been download. Soon we'll add a new type of assertion for that, [talk to our support](https://bugbug.io/contact/) if you need it ASAP.

### **Drag & drop interactions**

BugBug support for drag&drop is in beta, you can achieve it by combining the "Mouse press", "Hover" and "Release mouse button" [actions](https://docs.bugbug.io/editing-tests/actions). Talk to our support for details.

### **A recorded selector doesn't work**

Sometimes it's not possible to record a reliable selector and you need to use your human intelligence and debug HTML a bit. 

Use our [No-code XPath Selector Builder](https://bugbug.io/xpath-selector-builder/) which makes it easy to craft a good selector manually and can teach you in-depth about [best practices in selectors](https://bugbug.io/xpath-selector-builder/#How-to-create-reliable-XPath-selectors-for-test-automation).

Success in test automation depends on the regularity of your progress
---------------------------------------------------------------------

Implementing automation testing in a SaaS company is a long-term process. You should do it step by step. The biggest challenge for tech leaders is lack of time to dedicate on QA area. The truth is you don't need a lot to push it forward. Better slow than never. [😉](https://emojipedia.org/winking-face/) Block yourself a slot in calendar every week to work on QA automation and keep the momentum going.

medium_bugbug-matrix-illustration.png

small_bugbug-matrix-illustration.png

thumbnail_bugbug-matrix-illustration.png

bugbug-matrix-illustration.png

Tech leader's guide to automation testing

tech-leaders-guide-to-automation-testing

record-and-play-vs-coded-automation-testing

We’ve prepared a short guide with essential recommendations on test automation that we created specifically for startups. We divided the guide into 3 levels:

[★☆☆ Level 1: get the most out of basic automation](https://bugbug.io/blog/software-testing/automation-testing-guide-for-startups-level-1/)

[★★☆ Level 2: automate hundreds of regression tests](https://bugbug.io/blog/software-testing/automation-testing-guide-for-startups-level-2/)

[★★★ Level 3: integrate with CI/CD builds](https://bugbug.io/blog/software-testing/automation-testing-guide-for-startups-level-3/)
  
Also see our guide dedicated to non-technical people: [beginner's tutorial to automation testing with BugBug](https://docs.bugbug.io/in-depth-guides/beginners-tutorial-to-automation-testing)
 
### What will you achieve on Level 1?
After finishing this level you will run 5-10 automated tests every hour and check if your core features are working properly. You will detect production bugs with big impact on users sooner than before.

### TLDR:
-   Start with short and simple tests that check your core features
-   Remember about recording “hover”
-   Create special test user accounts dedicated to automation
-   Schedule the tests to run every hour
-   Incrementally add more tests

### Focus on the most important features

![80-20 rule (1).png](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/80_20_rule_1_c2126d9815.png)

Many companies spend endless hours on a neverending pursuit to automate everything. This leads to very costly automation that requires a whole team to create and maintain. Don’t be like these companies and keep it simple. Agile startups should start with automated testing of the most essential functionality. 

> 👍 **Rule of thumb:**
> Usually, it just takes 5 to 10 automated tests to be sure that most users are able to achieve their goals.

What should you test? Think of your features. What percentage of users are using a particular feature? If the answer is higher than 80%, then this is a core feature. The goal of automating tests of such important features is to make sure that you can detect bugs that have a huge impact on your business as soon as possible (new user registration is for sure a core feature, but it’s a little bit more tricky to automate, which we will cover in the next episode)






### Start with short & atomic tests

When you [record your first tests](https://docs.bugbug.io/creating-tests/recording-the-tests) start with something very simple and atomic. Atomic means that the test is small, independent and checks just one thing.

A good idea is to start with testing the basic navigation of your landing page. Click “record” and create a test that just clicks one item of your navigation link in the menu. Add [assertion](https://docs.bugbug.io/recording-tests-steps/recording-assertions) for some text in a heading. Finish the test and run it. Awesome! You’ve created your first automated test that touches the most essential area of your business!

![Screenshot 2022-07-25 at 13.44.12 (1).png](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/Screenshot_2022_07_25_at_13_44_12_1_0ebde0ddbb.png)

Now create a separate test for each navigation link. Here are screenshots of how BugBug team set up their landing page tests.

![Screenshot 2022-07-25 at 13.44.15 (1).png](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/Screenshot_2022_07_25_at_13_44_15_1_74fb39e9de.png)


### Don’t forget about recording hovers

If your website navigation appears on hover, you need to tell BugBug that you want to record a mouse moving to a specific element. To do this you need to click “Hover” during the recording and then click an element. When running the test, BugBug will move the mouse cursor to this element.

![bugbug record hovers.png](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/bugbug_record_hovers_81639fbd30.png)


### Create special test users

Another idea is to test if login works. Create a separate test that only checks that one thing.

To achieve that you need to prepare a “test user” - a user account that is dedicated only to testing if login works properly

-   Manually register a new user with a new unique email, for example with john.smith+testautomation@email.com (you can use [disposable mail](https://www.google.com/search?q=disposable+mail) or [plus aliasing)](https://support.google.com/a/users/answer/9308648?hl=en)
-   Go to BugBug and record a test that clicks the login button, enters the user login and password and submits the form
-   Add an assertion for some text that confirms that the login was successful
-   Finish your test
    

  

You might be tempted to test other features in the same test, but don’t do this now. We want to be sure that the login test is atomic and independent from other features.

  

Remember that you can create as many test users as needed. For example, if your product offers multiple combinations of preferences, you can create a separate user for each combination.

  

> 👍 **Rule of thumb:**
> Always create new test users if this leads to less test steps

  

### Schedule all tests to run every hour

With BugBug you just need a couple of clicks to start running the tests every hour. Just go to “Schedules”, select the “Hourly” one and add “All tests”.

![hourly schedule (1).png](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/hourly_schedule_1_445d7fe266.png)


  

That’s it, now you will start getting emails every hour with the results of your test automation. Tip: create a filter on your email account so that you avoid inbox clutter or [configure notifications to only get an email on failures.](https://docs.bugbug.io/running-tests/schedules#configure-email-notifications)

![email notifications (1) (1).png](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/email_notifications_1_1_a23f96ba47.png)


  

### Incrementally add more tests

You’ve successfully created test automation that monitors your production! Now keep adding more short tests one by one. Don’t forget to focus on being cost-effective - if you have trouble automating a use case, skip it and go to the next one or find a different way of checking if it works. And remember: you can always [ask BugBug support to help you with any troubles](https://bugbug.io/contact).

Read next: [next level guide on how to easily maintain hundreds of automated tests ](https://bugbug.io/blog/software-testing/automation-testing-guide-for-startups-level-2/)
  
  

medium_illustration - guide - level1.png

small_illustration - guide - level1.png

thumbnail_illustration - guide - level1.png

illustration - guide - level1.png

Automation testing guide for startups - Level 1: get the most out of basic automation

automation-testing-guide-for-startups-level-1

In the [first episode](https://bugbug.io/blog/software-testing/automation-testing-guide-for-startups-level-1/) of our 3-part guide, we covered the basics and described how to get the most business value from simple test automation. Now it’s time to learn some more advanced concepts and increase coverage of your automated tests. Here we go!
 
  
### What will you achieve on this level?

Once you finished this level you will:
-   have 100+ nicely structured tests
-   be able to update multiple tests at once when needed
-   run tests on different environments
-   automate testing of new user registration
    

  

### TLDR:

-   Organize your test steps in groups
-   Convert some groups into shared components
-   Duplicate tests efficiently with “new test from here”
-   Set up a URL variable to quickly swap between environments
-   Set up a testing email inbox to automate tests of new user registration
-   Use [built-in variables](https://docs.bugbug.io/editing-tests/variables#use-built-in-variables-for-dynamic-or-random-values) to generate random user email aliases
    
### Scaling the test automation

![illustration-number-of-tests.png](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/illustration_number_of_tests_7ee4d49d68.png)


In this episode, we will focus on scaling your test automation to a larger number of test cases so that your regression tests also cover corner cases and niche features. This is “Level 2” so things will get more complicated, but BugBug makes everything easy to achieve.

Example: in [the episode 1 of this guide](https://bugbug.io/blog/software-testing/automation-testing-guide-for-startups-level-1/) we’ve created just one test that checks if log-in works. Now we want to create several tests that check different outcomes of logging in, for example:

-   Log in - success
-   Log in - incorrect password
-   Log in - account blocked
-   Log in - password reset required
-   Log in - account doesn’t exist
    
You can see that we aim to increase the test coverage dramatically. Instead of 5-10 tests, we’re going to have hundreds of them.

### Keep tests nice and tidy

If you want to have more than 10 tests, you should start organizing the test steps and keep them in order. Why is it important? Imagine that you’ve created 100 tests, but later you changed something in the UI. All of your tests started to fail. If you haven’t organized test steps, you will need to go to each test individually, see what’s wrong and re-record the failed steps 100 times. Fortunately, BugBug allows you to easily avoid such situations.

### Group your test steps

Keep your steps tidy by grouping them. Give a clear name to each group, which will help you to navigate through long test cases. Move your cursor in between the steps to see a plus icon, then click “[split group here](https://docs.bugbug.io/editing-tests/grouping-steps)”.

![split-group-here.png](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/split_group_here_fce12649d5.png)


  

### Convert groups to shared components

Once you start grouping your test steps, soon you will realize that some groups are exactly the same in multiple tests. This makes them a perfect candidate for [components](https://docs.bugbug.io/editing-tests/wip-make-component). Components are magic groups: if you change it in one test, it will change in all other tests that have it.

![component-made (1).png](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/component_made_1_afa56da91d.png)
  

Start with creating a component for the first step that goes to your starting URL. Then go to each test and replace the normal group with the component by using “Insert existing component”.

![insert-existing (1).png](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/insert_existing_1_985f99b0c0.png)


  

### Duplicate the tests rather than create new ones

Since you already have components, you may not want to record each test case from scratch. A lot of your tests will start with the same components (for example: go to homepage, log in, enter a password, etc.). You can quickly create a new test “in the middle” of a different test. When you click “[new test from here](https://docs.bugbug.io/creating-tests/new-test-from-here)” you will create a duplicate of the test that keeps all the preceding test steps.
 
![new-test-from-here.png](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/new_test_from_here_5d79377e02.png)


  

If you use components in a smart way, you will be able to fix hundreds of tests with just one [re-recording](https://docs.bugbug.io/recording-tests-steps/re-recording-steps).

### Use data-testid attributes as selectors

Automated tests are most reliable if your [selectors](https://docs.bugbug.io/preventing-failed-tests/selectors) are clearly identifying an element. The best way is to clearly annotate such elements in your HTML structure. Ask your frontend developer to add `data-testid` HTML attributes to elements that you click in your automated tests. 

Why is it better? Let’s see an example. If you don’t use a dedicated attribute, your login button selector might look like `//SPAN[normalize-space(text())="Log in"]`. This selector points to any element with text "Log in". If you change the text in the button, such a selector will stop working and your test would fail. 

If you used an attribute, for example `[data-testid=”login-submit”]` your tests will be more resilient to changes. Even if you change the text in the button, you will not need to edit your automated test. What’s more, the developers will see this attribute in the code so they will be aware that this specific item is a subject of test automation. Once you have such attributes in your HTML, BugBug will automatically create selectors based on “data-testid” attributes during the recording. 

![data-testid.png](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/data_testid_669fc25baa.png)

### Build robust selectors without coding

Sometimes it's not possible to create a reliable selector with recording. Dynamic lists, tables and hover actions are notorious for being difficult to record. You just need to use your human intelligence and use a more complex XPath selector. We created a dedicated tool for that with an in-depth guide - see [No-code XPath Builder](https://bugbug.io/xpath-selector-builder/).

![no-code-selector-builder.png](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/no_code_selector_builder_b90d46b1c1.png)

### Work with different environments

If you are fully committed to testing automation, chances are you will need to run tests on different environments. Each environment can be accessed with different URLs, for example:

-   Production - acme.com
-   Staging - stage.acme-test.com
-   Local - localhost:3000

You should store these URLs in a variable with 3 [profiles](https://docs.bugbug.io/editing-tests/variables#profiles). After you set up more than one profile, a new dropdown menu will appear in each test, that lets you run tests with different environments. Learn more in our [detailed guide](https://docs.bugbug.io/editing-tests/variables#work-with-different-development-environments).

![profiles (1).png](https://bugbug-homepage.s3.eu-central-1.amazonaws.com/profiles_1_e2d0e47b36.png)


  

### Automate tests for new user registration

To reliably automate user registration you need to use a combination of several advanced features

-   Third-party inbox that supports [plus-addressing](https://www.msoutlook.info/question/886#:~:text=Exchange%20Online%20(Microsoft%20365)%2C,also%20known%20as%20%E2%80%9Csubaddressing.%E2%80%9D) (for example Outlook.com)
    
-   Built-in “testRunId” variable for generating random e-mail address alias
    

  

Learn more in our separate step-by-step [guide on testing sign-up.](https://docs.bugbug.io/editing-tests/variables#test-user-registration-and-login-using-variables)

### Keep it simple

Don’t try to overcomplicate your test steps. Use components and variables only when it will actually save time for you in case of product changes. Remember that re-recording is a really fast way of fixing existing tests - sometimes it’s just faster to re-record steps in 5 tests than debugging a complex net of variables and dependencies. Keep it simple and if you have any problems feel free to [talk to our friendly customer support](https://bugbug.io/contact).  😊

[Read next: advanced guide with tips on CI/CD integration](https://bugbug.io/blog/software-testing/automation-testing-guide-for-startups-level-3/)

medium_illustration - guide level2.png

small_illustration - guide level2.png

thumbnail_illustration - guide level2.png

illustration - guide level2.png

Automation testing guide for startups - Level 2: automate hundreds of regression tests

automation-testing-guide-for-startups-level-2

Tools for automation