Test coverage vs test quality: what a green build never tells you

Every test was green. Yet, we still lost an afternoon figuring out why production was down.

That afternoon changed how I think about software quality. All test cases were passing. Coverage looked healthy. And none of that helped, because the logs on that path told us something broke, but not what or where. A bug that should have taken ten minutes to locate turned into hours of guessing, adding print statements to a live system and running the same procedure over and over.

Nobody had written a bad test. The problem was that we never counted logging as quality work, so nobody paid attention to it.

This article is about that gap. When most developers hear "quality in software engineering," they think about tests. Tests matter, but they are one dimension of a much larger picture. If you want to build software that a team can actually maintain properly, quality has to include everything that happens after the tests pass.

Why we reduce quality to test coverage

There is a good reason tests became the face of software quality: they are the part of quality you can actually see.

Tests are measurable. They produce a number. They turn a build green or red. You can put coverage in a dashboard and watch it go up. That visibility is what makes tests easy to talk about and easy to require.

The other dimensions of quality have none of that. Nobody gets a passing checkmark for writing a clear runbook. There is no red build when a fix ships without anyone telling the support team. Observability, documentation and communication are real quality work, but they are invisible on the usual dashboard. So they get treated as optional.

The result is a familiar pattern. Teams push coverage toward some target number and feel confident. Then incidents drag on, and support keeps escalating problems that engineering thought were solved.

Coverage tells them the code does what they asked. It never promises anything about what happens when something they did not anticipate goes wrong.

A better definition of quality

This is the definition I prefer:

Quality isn't just whether the code does what you wanted. It is how quickly and confidently your team can find, fix and explain what happens when it does not.

Tests mostly cover the first half. They check intended behavior. But most of the real cost of software shows up in the second half: the unintended behavior, the edge case nobody wrote a test for, the queue that quietly stopped consuming.

The DORA research program, which studies what makes software teams effective, has consistently found that failed deployment recovery time, how fast a team gets back to working software after a bad release, matters at least as much as change failure rate, how often releases break in the first place. Everyone ships bugs. What separates strong teams is how fast they recover. And recovery speed comes from observability, runbooks and deployment tooling. Not from how many tests passed before the release.

DORA has also added a metric for rework, the deployment rework rate: the share of your deployments that are unplanned work to fix bugs. That metric exists because counting tests and features misses something: how much of your team's time goes into cleaning up after work that was technically "done."

If a good part of the cost sits in recovery and rework, then quality has to cover what makes recovery faster and rework less common.

Why this matters more in the age of AI

If you write software with AI assistance today, all of this gets more urgent.

AI made generating code much faster. That is genuinely useful. But generation was never the bottleneck for most teams, and speeding it up does not touch the quality work that surrounds it. AI will happily produce a feature and a passing test for it. What it won't do, if not specifically asked to, is add a useful log line to the failure path, update the document explaining this subsystem or write the note that tells everyone else what changed.

As producing code gets cheaper, everything around it gets more expensive by comparison. The bottleneck moves downstream, to running, understanding and explaining all the code you can now generate in a fraction of the time. You ship more surface area with the same number of people who have to support it. It is the same judgment problem as knowing when not to accept an AI suggestion, moved one step later in the process.

What decides how much you get out of AI is whether your quality practices can handle that extra volume. Observability, documentation, communication. Generating more code does not help if none of that keeps up.

The three kinds of quality work developers skip

In the teams I've worked with, the missing quality work pretty much always falls into three categories. None of them are tests. All of them fail silently, which is what makes them easy to skip.

1. Observability

This is the one that cost me that afternoon. Tests confirm behavior in a controlled environment. Observability is what you have when the environment is not controlled and something breaks in production.

A green suite tells you nothing about whether the failure path emits a useful log line, whether requests carry a trace ID you can follow across services, or whether your metrics would show the incident at all.

So, the question I ask myself now: for any important failure path, "if this breaks in production, what will I actually see?" If the honest answer is "a generic 500 and no context," that is a quality gap, no matter how good the coverage number is.

2. Documentation and guidelines

Every test can pass and a new engineer still needs two weeks to get productive, because nothing is written down. How to run it locally, why that strange workaround exists, what "done" means for a change in this codebase, which environment variables you actually need, who to ask when the deployment fails.

When knowledge lives only in people's heads, those people become the bottleneck. Every question routes through them, which is why they get called on vacation or feel the urge to check emails when they are sick.

None of this is hard to write. It just has to be seen as part of the job.

A how-to that saves the next person an hour is quality work. A short guideline that prevents a recurring mistake is quality work. A README that gets someone running locally in ten minutes instead of half a day is quality work. It just doesn't feel like engineering. That's why it's the first thing to go when the sprint gets tight.

3. Communication and visibility

A bug gets reported, and a week later it's fixed. The code is correct, and the test is green. But nobody tells the support team, so they keep reopening tickets for it. Customers hear "we are looking into it" for a problem that no longer exists.

The engineering work was done. The quality work was not: telling everyone else that the bug is gone.

The same thing happens towards sales and support. A known limitation that is never documented where they can see it becomes a recurring problem. Sales oversells a feature because nobody told them the limitations. Support gets the same escalation every month because the workaround lives in a Slack thread from six months ago.

This is the cheapest category to fix, yet the easiest to forget. A one line note in the right channel, or a status update on the ticket, already solve most of it. It costs a minute and it saves hours downstream.

Making the real state of the product visible to the people who represent it is quality work. It reduces rework and escalations. And what matters most: it builds trust with customers. It just happens to have nothing to do with a test runner.

What a green build tells you, and what it doesn't

It helps to see these side by side with what a green build actually guarantees.

The full scope of quality is easier to picture as a loop:

Tests live at the very start of that loop. Everything after them is where most of the cost sits, and where most of the quality work gets skipped.

How to put this into practice

There is no framework to adopt here. Just treat these three categories as real work and give them the same attention you give tests.

A few concrete starting points:

• For every meaningful failure path, ask what you would see in production if it broke. If the answer is "nothing useful," write the ticket.
• When a fix is deployed for a reported problem, make telling everyone else part of "done". Remember the ticket status, the support channel and the customer who reported it.
• While the context is still loaded in Claude, or whatever tool you use, write the short document. The runbook, the "why this workaround exists" note, the local setup guide. Future you and every new hire will recover the time.
• Make known limitations visible to the people who represent your product, before sales and support discover them the hard way.

None of these produce a green checkmark. What they reduce is the time your team spends recovering and reworking. That is where the real cost of low quality shows up.

The part that costs the most

That afternoon we lost to a green test suite was not a testing failure. It was an observability failure, hiding behind a passing build.

Tests are necessary. Treating them as the only thing that counts as quality is what leaves you confident right up until production breaks and you have no way to see inside it.

Real quality is what a team can do when the code does something nobody intended. Find it, fix it, explain it. Tests cover a slice of that. Observability, documentation and communication cover the rest, and that is the part that usually costs the most.

Common questions

Is test coverage a good measure of software quality?

Coverage measures how much of your code a test suite executes, not whether the assertions are meaningful or the failure paths are debuggable. It is a useful map of what is untested. It is a poor target, because a team can raise the number without improving anything about how fast they recover from a real incident.

What is the difference between test coverage and test quality?

Coverage is a percentage of lines or branches touched during a run. Test quality is whether those tests would actually catch a regression, fail for the right reason, and leave you able to diagnose the problem. Two suites with identical coverage can differ completely on all three.

Why does recovery time matter more than test coverage?

Because every team ships bugs. DORA's research consistently finds that failed deployment recovery time separates strong teams from weak ones more reliably than how often changes fail. Coverage only speaks to intended behavior before release. Recovery speed comes from observability, runbooks and tooling after it.