Investing in Right Test Automation

We are in a world where software can do anything. So why not make software test software? That way, one computer can become an army of virtual testers!

Test Automation is a powerful technique and an idea that holds tremendous possibilities in terms of efficiency, better utilisation of QA Teams, saving repeated effort and last, but not least, money! Here, in a graphic, is what Test Automation can do for you:

But be cautious. Automation can open the doorway to great promises, but it can distract you and waste resources. Just as we invest our money to get the best returns, we need to invest in Test Automation, too, to get the best out of it and advance the mission of delivering quality software.

So, what then is Right Automation? My experience has led me to these mantras that have helped me cut the distractions:

• Have an organised testing process before automating it.

• Automate the tests that complement the development process by ensuring healthy test builds.

• Create tools to generate exhaustive test input data.

• Identify the critical scale, load, performance, and endurance tests.

• Follow the rigour of the Software Development process for your Automation

Now, let us understand how to achieve the right and valuable Automation.

Have an organised testing process before automating

If you do not have an organised test process or a good and clear test design, that is how your automation will also be. It will just be a faster version of bad tests with unclear or false-positive test results.

To develop automation that gives you the best results, you first need to establish a good test process and design good tests. Only after that you can identify which tests to automate.

Thinking about and creating a good test suite before automating it will let you find good defects and reliable results.

Automate tests that ensure healthy test builds

Automations that help speed the development process saves cost and project delivery times. Such Automations usually provide the following benefits:

• They quickly detect destabilising changes in builds

• They expose regressions and unintended consequences

• They help identify problems quickly, thus, make fixing them easier

Automation of Smoke and Unit Tests are two such use cases to help you reap these benefits. Also, these are the types of tests that can be executed by anyone, anytime. They can be executed even as part of the build process before integrating and passing the build to the Test team.

Create tools to generate exhaustive test input data

While working on the Warehouse Management Systems of a few e-Commerce applications, I often found myself in the situation where I needed hundreds of orders, each at a different stage of the order lifecycle. Sometimes, I needed multiple types of promotions or the submission of a registration form with data of a thousand unique users.

All these situations made me realise:

• The amount of time spent on manual Test Data creation was far more than the time required by the actual test.

• Feature Testing Automations usually does not cater for this kind of use case.

• This kind of test data is always reusable. We will always need this kind of Test Data for different release cycles.

Manually generating data for such use cases is tedious, error-prone and time-consuming. These use cases are prime candidates for Automation since they can save a lot of time for testers over multiple release cycles.

Creating a large set of test inputs, heavy files, generation of random test data, creation of testbeds are a few examples of Test Activities that require laborious efforts by the test and Development Teams. In our eagerness to automate the actual testing, we often miss out on investing in test input data creation.

Identify the scale, load, performance, and endurance tests

Consider a scenario where we have two essential acceptance criteria for our product. The first one is that users should be able to log in to the mobile app, see product displays correctly, and place orders. And, the second one is about how it behaves at scale. It should perform well even when 4000 users are placing orders simultaneously.

The first criterion can be verified manually. But for the second criterion, the only option is Automation.

Usually, when we talk about Automation, our focus goes towards functional or regression tests. While they are good candidates for Automation, some tests like scale, load, performance, endurance, and tests for race conditions can only be done with the help of Automation.

All of us indeed have QA teams responsible for testing all product dimensions in limited time and resource bandwidth. I found that automating the non-functional (like load and performance) proved to be a better investment compared to functional (like the order has to be displayed correctly) because:

• Instead of repeating the same feature tests, again and again, we create new non-functional Test Cases.

• It leads to testing the product from non-functional approaches and discovering new bugs, contributing to better Test Coverage.

• If not automated, these problems will only be found by angry users.

Follow the rigour of Software Development processes

Test Automation can prove to be expensive in terms of maintenance and scaling if the discipline of the software development process is not followed. Automation suites are also software, so let us treat them the same. These are the rules and sequences I like to be followed:

• Plan your project milestones and key deliverables

• Use source control system for automation code and tests

• Design tests, review them before coding them

• Please test your test automation

• Track automation bugs in a bug-tracking tool

Conclusion

I have always felt that test automation is an investment. You have to invest in the right things to get the best returns. For test automation, the right thing may not always be the easiest. I hope my mantras help other QA and automation experts identify the right things.