Walking through the Drawbacks of Having Technical QA

In software development organizations everywhere there is a tension between shipping product out to users and the quality bar developers need to hit and maintain to ensure their users are happy with the final product. This tension leads to a number of very different approaches on how to handle testing and what role if any a Quality Assurance discipline fills. In some companies this involves pushing all testing activities onto the development team, in others it is a focus on DevOps, while many focus almost all of their efforts on manual testing. This article continues the focus on the role Technical QA can fill within an overall testing effort.

Building a strong arm of Technical QA within any testing organization has a variety of well established benefits, with the most obvious and clear examples being their ability to build more tools and automation and understand the code at a deeper level than their less technical brethren. While the benefits of Technical QA are relatively well known, with some key arguments written about them earlier (https://www.garudax.id/pulse/exploring-benefits-having-technical-qa-benjamin-seifert/), I want to delve into some of the unintended and unexpected consequences that many companies run into when developing a Technical Quality Assurance arm in their organization. I will continue to use the definition of Technical QA having experience in test and experience programming to a minimum level where they can perform white-box testing and write scripts or code in at least one language when walking through these consequences.

Technical QA are expensive in both time and money.

Technical QA have a much higher cost per person than both entry and highly-skilled manual QA. The skillset required to be effective as a Technical QA professional is a combination of programming and testing skills, and it is often quite difficult to find the opportunity to progress along both axes at the same time. This somewhat divergent skillset takes longer to develop and often runs counter to what is expected for most people in their career progression. This specialized commitment and scarcity of people following this track lets them demand much higher salaries than their less technical peers. In addition to the monetary costs, Technical QA often require much longer to deliver on their purpose than their less technical peers. Building strong automation systems and tools requires time to understand the systems already in place, time to map out the requirements, and time to build the solutions your product needs. Investing in Technical QA now may mean you won't see meaningful returns for months, when less technical QA could be contributing meaningfully in a small number of weeks. They are absolutely an investment in the future, but they won’t solve your testing needs right now.

Automation systems and Tooling gives a false sense of thoroughness in testing.

For teams that rely heavily on Technical QA, the testing efforts will focus primarily on automation and tooling in an effort to identify all serious defects. When we automate large degrees of test coverage it is very easy to say that machines don’t make the same mistakes people do, they run the same tests the same way every time and we can rely on their results in a way we can’t with manual testing efforts. We assume these tests cover all the ways user interact with our software and function exactly the same way they will experience any function.

While it is certainly possible to approach 100% test or code coverage with automation, it is time consuming and it doesn’t translate directly into finding 100% of defects or even 100% of serious defects. The sheer quantity of tests required across unit, integration, and system levels to achieve near 100% coverage also suffers from the same flaws of any human designed system: areas will be missed, and you won’t know where. In addition to any missed requirements, any continuous delivery development studio will add fragility to those automation systems with regular software updates, and running every test developed on every check-in will make developers pull out their hair as they wait for the tests to run. Regardless of the reasoning, the tests that do run appropriately will not cover all of the permutations of interactions users will experience, even if you can say you have 100% code or test coverage and that leaves you with a false sense of security.

Technical QA use what skills they know best, and that is usually coding

With any split skillset most people eventually specialize in one direction or another. For Technical QA that direction is usually to lean more heavily into programming. Technical QA focus on using the tools they know how to use (coding) sometimes at the expense of other options. Everyone loves to talk about automation, and can pressure Technical QA towards that as an end all solution, but Automation primarily works via defect detection, which should only be part of a QA specialist's skillset. The most effective quality assurance professionals looks to mitigate risk, prevent defects, assess process health, and defect defects. Leaning super heavily into Technical QA without the appropriate support structures can lead them to focusing heavily on defect detect at various stages of development instead of other areas where their expertise is also highly valuable, but less understood.

Developers feel less pressure to write good code in the first place, increasing tech debt

Potentially controversial, but I’ve seen that the stronger the safety net, the less engineers feel the need to write good code in the first place. With a robust Technical QA staff, it is very easy to have most developer errors caught fairly quickly. This often leads to less due diligence in the first place as “QA will catch the bugs”. While often true(ish), the technical debt incurred with less optimal code slowly impacts overall product health, and because Technical QA helps developers find defects sooner, the developers will often feel less pain on average for each bug fix compared to a team that needs to rely on manual testing efforts, or developers self-policing their own work. Over time this compounds into a system that needs Technical QA to function to support the tech debt that is already accrued.

As a counter to my previous article, this one focuses only on the often unexpected negatives involved with a strong Technical QA presence. When merging these two different perspectives it hopefully becomes clear that there isn’t the same right answer for everyone. Depending on budget, timelines, product priorities, and scope of work building a highly technical testing organization may or may not be the right approach for you and your company. I think when you are building any organization to support multiple products using a continuous delivery model, having a robust Technical QA discipline is extremely important. It provides a level of sustainability of testing that is impractical without the investment. However, Technical QA is rarely the most efficient and effective way to cover all of your testing needs. Finding the right blend of QA professionals with various skills is always the challenge. 

At Riot Games, we are actively working on solving this problem and within our teams we have different ideas on how to move forward. A number of us will be at GDC wandering the floors, going to talks, and attending a QA event hosted at the Alise hotel in San Francisco on March 22nd if you are interested in talking about these or any of the other problem spaces within Quality Assurance.