Did we test enough?

Who did not experience this Gate-meeting to decide on whether we can release our software? Test dashboards are proudly presented, showing green results and when the Quality Manager asks whether sufficient tests were executed, a Code Coverage and Requirements Coverage of 100% are shown. No open Bugs with severity Blocker or Critical are reported and only a few Major open Bugs are reported as known Bugs. With full confidence the release is approved!

Only a few days after deployment into production, the software fails and a critical bug surfaces, resulting in an angry and disappointed customer. How could this happen, we had so good metrics shown in the Gate-meeting?

Engineering teams, but especially management love metrics. Coverage percentages, pass/fail charts, defect counts—they all feel reassuringly objective. They turn the messy, uncertain world of software testing into tidy dashboards and green checkmarks based on which it is decided whether we can release or not.

But here’s the uncomfortable truth:

You can measure everything measurable and still have no idea whether your system will hold up in the real world.

That gap exists because there are two types of quality at play: Modeled Quality and Transcendent Quality. Until you understand both, you’ll keep mistaking activity for assurance.

Modeled Quality: What We Can Measure

Modeled quality is the quality that fits inside a spreadsheet. It reflects the aspects of testing that can be defined, instrumented, tracked, and automated. Modeled Quality is objective.

Typical examples include:

• Code coverage
• Number of executed test cases
• Branch/condition coverage
• Open Bug counts
• Test pass/fail ratios

These metrics serve a purpose. They help teams stay disciplined, consistent, and predictable. They allow for automation and reporting. They uncover obvious blind spots.

But they answer only one narrow question: “What did we measure about our testing?” They don’t tell you what actually matters: whether the product behaves reliably, safely, and meaningfully under real conditions. To answer this second question we need to understand what Transcendent Quality is.

Transcendent Quality: The Part You Can’t Put on a Dashboard

Transcendent quality is the felt part of testing; the expert judgment that signals whether the system is truly understood. Transcendent Quality is intuitive and subjective, like beauty and love.

It shows up in questions like:

• Did we explore realistic workflows—or just exercise code paths?
• Do we understand how the system fails?
• Have we challenged our assumptions?
• Did we test what matters most to users?
• Are our tests meaningful, not just present?

As an example, Code Coverage tells you what code ran. It does not tell you:

• how good the tests are
• whether critical risks were covered
• whether the system behaves correctly under stress, scale, or weird timing
• whether the testers applied domain knowledge
• whether anyone questioned the happy path

This is why two systems can both have 95% coverage, yet one is robust and the other collapses under real-world pressure.

Transcendent Quality is not arbitrary. It’s the accumulated craft, intuition, and contextual understanding that cannot be quantified but absolutely determines real-world reliability.

The Real Question: Did We Understand the System Enough?

A better framing for “Have we tested enough?” is:

Modeled Quality tells us how much of the system was touched, not how well it is touched. Transcendent Quality tells us how deeply the system was understood.

This implies that we need both:

• Modeled Quality ensures structure, repeatability, and transparency.
• Transcendent Quality ensures insight, relevance, and confidence.

One without the other leads to brittle systems, false security, and expensive surprises.

Conclusion

If your testing strategy focuses only on what can be measured, you will get exactly that: measured activity. But great engineering requires more. It requires the holistic, intuitive, expert-driven assessment that numbers alone can never provide.

The teams that thrive are the ones that know the difference.

More about Software Quality: https://cloudtsoftwareconsulting.com/the-book/