The Downfall of Manual Testing

This week, I want to share another post I shared on the test IO blog. This one focuses on manual testing; yes, all testing is human, and all testing can be described as manual (thank you, Michael Bolton), but for the sake of addressing the below, let's define "manual testing" as exploratory real-world, human testing. Thank you, Joe Colantonio, for the interesting thoughts.

[You can read the follow up to this post here!]

. . .

On his blog, Joe Colantonio, posits a few reasons as to why manual testing may in fact be "problematic" and entail a variety of "pitfalls." Although Joe presents a fantastic, in-depth introduction to automation testing, we would like to qualify his (arguably anachronistic) statement on manual testing.

He states that manual testing:

1. Uses a lot of resources

2. Is time consuming

3. Sometimes lacks proper coverage

4. Is often of a repetitive nature, and testers may become bored and make mistakes while testing

While certainly holding true in the past, his sum up of why manual testing may fall short can be addressed by the technology we have at our fingertips today.

Our response to the above four points:

1. Uses a lot of resources

Traditional manual testing can require a lot of resources when constrained by having to hire in-house to perform this testing. However, if you can share the personnel burden, or negate it all together, the immediate resource cost diminishes drastically. 

For example, one of our customers quoted our services at "8 full-time hires but at 1/10 the cost." Given the average cost of hiring a full-time QA professional in the Bay Area, that's a nearly $1 million per-year service that we provide at 1/10 of that cost.

2. Is time consuming

It definitely can be... When you’re relying on a finite number of in-house team members to perform the manual checks. What if you extend your in-house team? For every additional team member, the total amount of time spent testing before satisfactory results may diminish, but at an additional resource cost. Now, what if you could maintain the speed of doing so without this cost? With crowdtesting, you’re able to reap the rewards of an extended team without the extended cost.

In 2018 alone, over 180,000 hours were spent testing for our customers, equating to over 22,000 workdays of testing! This means that on average, our customers tested for more than 900 hours, or 1/3 the yearly workload of a full-time hire, with our most active customers testing up to 8x that amount.

3. Sometimes lacks proper coverage

In terms of device coverage, there’s definitely a drawback to manual testing, as maintaining a device farm can be pricey and time consuming. You could use simulated environments to help deal with this issue, which is more cost effective, but you’d miss out on testing in a real environment more representative to what your users will experience. By leveraging the crowd, you can test your software on nearly unlimited device and browser combinations. The size of the crowd, for example, ensures that all device-browser-update matrixes are covered, while also ensuring accessibility to the newest devices out there.

Sure, having one or two individuals manually follow test cases or perform exploratory testing may take a lot longer than running a tool like Selenium; but, more often than not, with automation you’re limiting yourself to very specific user flows or functionalities (you may miss something a real user might experience in a live setting).

With crowdtesting, you can gain the insights of real people on real devices. And with us specifically, you can take advantage of guided exploratory testing to receive detailed bug reports, complete with screencasts under real-world conditions.

4. Is often of a repetitive nature, and testers may become bored and make mistakes while testing

We completely agree with this point; this is why having a more diverse and, when necessary, dynamic audience to perform your testing can help to keep testers’ perspectives fresh and unbiased. While mistakes are still possible, leveraging a higher volume of testers enables you to catch these mistakes, and anything else that might go unnoticed by a smaller team, much more quickly.

Overall, there’s nothing wrong or incorrect about Joe’s initial statements, but as software testing continues to evolve, there may be more alternatives than initially thought.

If you're thinking about giving crowdtesting a try, we recommend you read this short checklist first. Otherwise, if you want to learn more about what we do at test IO, click here.