QA in the Scrum Process

Today, nearly every software development organization will describe itself as some variant of Agile, ranging from disciplined teams with Scrum masters, ceremonies, and the Agile manifesto posted on the wall, to those who've only adopted lingo, dividing work into sprint cycles, and the credo: “We’re agile. We break all rules.” And the first rules tossed in the dustbin are all about QA.

QA, in my opinion, is the aspect most overlooked by the Scrum/Agile approach to software development. Which is a shame, because these new methodologies can be the most exciting changes to transform how we test.

Agile/Scrum Mistakes in the Industry

There are a few experiences I have had in various projects which reflect trends in the industry I think we should avoid. 

We don’t need testers. Devs can do their own testing.

This is very shortsighted thinking. Developers can and should write their own unit tests and be involved in test. Quality is everybody’s business. But test automation frameworks require expertise and multi-sprint efforts to build, and it is not reasonable to expect all developers to do this. It is often hard enough just to get everyone writing unit tests.

Developers have areas of expertise, such as database, front end JavaScript, or middle tier. An SDET is a developer with expertise in test automation and tools. Frameworks for performance and UI automation are not trivial. Performance, load, and capacity testing is a discipline itself. So is security / penetration testing. It is not reasonable to expect all developers to be experts in every field.

Embedded testers are assigned as assistants to developers

Embedded testers have resulted in great working relationships 1:1 between developers and testers and produced some good results, often catching bugs before they even appear in a build.

It also enforces a secondary role to QA in development, enshrining that testers are junior members answerable to dev leadership, reducing their ability to provide a check and balance. Companies such as Microsoft have pushed it farther, eliminating all FTE testing positions in favor of contracting and offshore testing. Without a career path, the experienced testers will leave taking with them both the domain knowledge and experience building things like frameworks needed to reduce the time regression testing and test performance.

There has always been a certain number of SDETs who view it as a stepping stone to becoming a developer. This number has been increased by the elimination of FTE positions, which only reinforces in the corporate mindset that SDETs are junior developers, because when those developers were SDETs, they were junior.

All the benefits of embedding testers described above, end even more, are seen when they are embedded as full partners in a Scrum team as QA experts, empowered to be cross-functional and participate in all planning and design decisions.

Cross-Functional is enforced. Everybody must be trained to do everything and substitute for everybody else.

I love cross-functional. I find it liberating and empowering, allowing the individual to take his/her career in any direction. The mistake is when team members feel forced to learn every role. This is simply not practical. Cross functional does not mean everyone does everything. It means anyone can choose to do anything for which they have an interest, an aptitude, and for which the team has a need.

Cross functional in this sense means there is no waiting for one group to throw their work over the fence. At the beginning of the sprint when there are more product code tasks, the SDETs can jump in and help out. Towards the end there will be more testing tasks, and the developers can help with that. The db expert will still take the lion’s share of db tasks, the UI dev will tackle most of the JavaScript tasks, and the SDET will still do most of the automation work. But any one of us may expand out, and we all will pitch in to do whatever is needed to complete the sprint. Cross functional means my expertise doesn’t limit me; it enables me to advocate best practices and to ask or offer help when needed.

Lock the Managers out of the product planning sessions

As an individual contributor SDET, I found this idea nuts. The intention is to allow team members to talk freely without intimidation from management. But I knew my manager supported me and had my back advocating for QA. He was locked out of the room yet somehow still responsible to the stakeholders for quality in the product.

Trust works both ways. One of the operating principles of Scrum is that management trusts the Scrum teams to make the right decisions. The team needs to trust that management is there to empower them and remove obstacles.

“Good enough” means we test only our scrum team’s sprint items

Nobody owns regression testing. But we intend one day to automate it.

No one owns inter-component testing. No one even thinks about cross product testing.

Or they do, but only to think gee, I wish there was a way we could do that.

A Better Way

Basic Tenants

Quality is the Most Important Feature We Ship. Everybody owns it.

Teams are self-managing. We trust the teams to make the right decisions on completing their tasks. The team works together to move tasks off the board.

Your job title does not define what you do; it describes your expertise. We do what needs to be done.

Cross-Functional Empowerment, not Enforcement.

Teams and Tasks

QA Epics and User Stories are written by test leadership and prioritized on the product backlog along with everything else. QA user stories are assigned to the sprint team and broken down into tasks.

Scrum teams are 6 to 8 person self-managing teams with embedded test.

Change stand ups to Task Based

Instead of round the circle “justify what you did yesterday”, stand up focuses on moving the tasks on the board. The person to whom the task is assigned – or anyone else with pertinent information – speaks up as the discussion moves from task to task.

Mangers/leads trust the teams to assign their own tasks and to make the right decisions. Leadership is able to be responsible back to the stakeholders for outcomes by:

• Managing the stories and the product backlog
• Lead product planning meetings
• Monitoring the sprint burn down
• Managing the bugs
• Manage multi-sprint efforts for features and QA frameworks
• Coordinate cross team integration, such as cross product testing
• Mentor/coach their direct reports
• Provide goal setting and performance reviews

But managers don’t assign tasks. The teams themselves do that. Managers only step in if there is a breakdown in the process, such as when a team needs coaching on how to constructively deal with conflict.

Definition of Done includes:

• Unit tests written for all new code meeting a code coverage standard set for the product (or a good explanation why it was not needed, e.g., third party tools and constructors).
• Completed code review includes review of unit tests, security, and performance.
• The test strategy is written into every user story. This is a couple paragraphs explaining what we are going to test and how we are going to test it, not a formal test plan in the waterfall sense.
• Test case review completed, including for unit tests. Code Coverage can be gamed.  That’s not a reason to ignore it, but it isn’t enough. We must apply the same good testing pracitices to unit tests as we do any other type of automation, e.g., equivalence classes, boundary conditions, error testing, etc.
• Automated tests written for new features when possible, or a good explanation why it was not possible (such as missing infrastructure; insufficient time is not a good reason) and the missing test written up and added to our technical debt.
• When not technically possible to complete an automated test, we at least buy down our technical debt completing automation framework tasks directly related to the incomplete test and wrote what we could, such as an abstraction layer for a UI test, API/integration tests that can be used in load testing, etc.
• Automation framework tasks completed.
• Every sprint we buy down a percentage of technical debt regarding code written in the past that does not have sufficient code coverage.

Test Leadership

For simplicity’s sake, I’ll use “test leadership” generically. These tasks could assigned to a test lead, SDET lead, QA manager, Senior/Principle SDETs, etc. Some of these tasks are shared with PM and Dev Leads.

The role of test leadership is:

• Advocate QA
• Coordinate multi-sprint efforts such as automation frameworks and the buy down of technical debt as QA goals, written up as QA user stories
• Coordinate cross-product and cross-team integration test effort
• Attend all scrum team meetings for his/her products
• Shepherd the QA user stories
• Monitor the QA task burn down
• Manage the bug db
• Call triage meetings
• Call QA strategy meetings – everyone invited, not just STE/SDET
• Create test plans and call test plan review
• Mentor and coach testers and SDETs
• Evaluate performance for reviews

SDET

SDETs are both developers and testers. They are developers with expertise in test automation and tools. They are embedded on the Scrum team and their job is:

• To advocate the best automation practices.
• To take the lead in automating tests for the sprint.
• To take their turn writing product code. The better their understanding of the product, the better their testing will be.
• To make sure QA and automation tasks are moving on the board, pairing with other teammates as needed.
• To pitch in with other tasks, such as planning, product code, and manual testing, as needed.
• To report back to their team and lead any blocking issues.

STE/SDET

• To advocate the best QA practices.
• Complete manual testing as needed.
• File bugs and shepherd them through triage and resolution.
• Participate in product planning.
• Participate in spec reviews.
• Participate in code reviews – yes, non-coders, too. If the code is not organized well enough that the developer can explain what it does, then it needs to be refactored.

No Leads Does Not Mean No Leadership

Many companies have adopted with Scrum a flatter reporting structure: self-managing teams, no leads, and managers that are more like mentors and coaches. Test leadership, therefore, must be taken up by those knowledgeable enough about test methodology, assertive enough to advocate for it, able to be convincing, diplomatic enough to get along with others, sometimes in endless process improvement committees, and experienced enough to be taken seriously. Without a good sprinkling of senior QA – along with equally experienced dev and BA – teams will need much more hands-on guidance and mentoring.

Responsibility for leadership can be taken up by anybody. That isn’t to say that everybody has equal aptitude, but that anyone who sees the need can make the attempt to fill it. It’s often the case that leadership will be distributed. One is good at articulating a vision, but is so passionate he bites everyone’s heads off in a committee. Another is the diplomat, able to encourage others to the best of themselves, but has very few original ideas. Here again, Scrum can shine, if you let it. Kanban your leadership tasks and let everyone take up what each does best.

Scrum is very salient because it seems so democratic. But it requires constant cultivation.  Without responsible people stepping up Scrum will break down, QA will break down, the development process itself will break down. But if we are mindful it is an exciting time to be in software development. It is empowering because one of the areas in which we can be cross-functional is, in fact, leadership.