Exploring the Benefits of Having Technical QA

Throughout the games and software industry, there are number of divergent approaches on how to handle testing, and the role Quality Assurance traditionally fills. In some companies this involves pushing all testing activities onto the development team, in others it is a focus on DevOps, while many focus almost all of their efforts on manual testing.

At Riot Games, we are working to push as much accountability for quality onto the development teams as possible, while also providing subject matter expertise to evaluate the difficult to find defects that always occur in game development. This requires substantial specialization across a number of axes. One area of this specialization is the broad category of Technical Quality Assurance. Almost everyone I’ve spoken with feels being more technical is always better, but there are different opinions on what Technical QA should be and why they are important. While Technical QA are extremely important they aren’t the only specialization necessary for thorough testing of software.

I’ll present 2 arguments around Technical Quality Assurance based on what I’ve learned working at Riot Games, with the first part detailed below explaining why Technical QA is awesome and how it really helps game development and the second part arriving next week explaining the inherent pitfalls of building up a Technical QA subdiscipline. Technical QA as described in this article will be defined as having experience in test and experience programming to a minimum level where they can perform white-box testing and write scripts or code in at least one language.

Technical QA are excellent Quality Leaders.

Leaders in Quality Assurance are expected to shift their perspectives on a regular basis from a zoomed in highly detail oriented view to a more macro view and answer a variety of questions, from “why did this specific function break”, to “explain how this testing process will evolve over time”. Possessing a high degree of technical skill gives you an inherent advantage when trying to answer these questions and solve a number of quality problems. Technical QA are more inclined to understand how core systems work together to make the whole game engine run and can more proactively ask the right types of questions early on in development, leading to better rates of defect prevention. They can more easily tie everything together from early planning through late stage defect detection and reporting as they can produce the tools necessary to make testing and metrics collection easier and more sustainable.

Technical QA provide more opportunities for testing value throughout each development cycle.

Their technical background lets them easily understand system specs and provide insight into what types of systems could break. This improves their early risk analysis and sets appropriate acceptance criteria on a user story level, beyond just user facing expectations of functionality.

Throughout their skill in white box testing, technical qa help developers more easily walk through defects at the code or scripting levels allowing fixes to be inputted much more easily and rapidly, reducing the time sink that is going back through your code to find what went wrong.

Teams with Technical QA build more tools and automation.

Technical QA will build tools allowing for developers and other QA to more rapidly and reliably test different components of your systems. These systems run the gamut from system level automation suites, to prebuilding testing scenarios, to tools allowing artists to see their work in client without having to wait for a build to finish. Across the board these efforts all lead to finding and reproducing difficult to find bugs more quickly and easily than manual efforts alone. This focus on tooling and automation often lead to reduced overall headcount therefore reduce overall cost and keeping the benefits associated with smaller teams.

All of these arguments are strongly in favor of technical QA as part of a comprehensive testing organization, and for most teams they should be an integral part of the testing process. Technical QA possess enough programming expertise to be able to walk through code with engineers and evaluate what isn’t working; a skill that you would typically have to hire an additional engineer for. Technical QA do have their own set of limitations and problems, highlighted in a follow up article. Regardless, at Riot we haven’t always done the best job prioritizing their needs appropriate and are actively looking to improve how we work. I'm always interested in hearing other QA professionals opinions either in the comments below or at GDC next month where I'll be wandering the floor and attending some talks looking for people to chat up about these types of problems.