Does Higher Test Coverage Mean Fewer Bugs?
I worked on a platform that watched network devices and alerted the customer when one had a problem. That was the entire promise.
Devices didn't agree on how to encode "false." Some sent 0. Some sent an actual boolean. Some sent the string "false" or "FALSE", and every new device model was a chance to introduce one more variant nobody had planned for. One of those variants swallowed a real alert: a device reported a problem, the parsing logic read its false value the wrong way, and the customer got no warning.
The code that did this parsing was covered. A test ran it on every build. It never checked that one encoding, so it stayed green while the bug was live. Nothing in the suite caught it. Not CI, not review. A customer complaint did.
More coverage wouldn't have caught this. Better assertions would have.
What "coverage went up" actually claims
When coverage goes up, most teams read it as the suite getting safer. It's worth looking at what the number actually counts.
What does code coverage actually measure?
Code coverage is an execution count: the share of lines or branches that ran while the tests ran. It records nothing about whether an assertion checked the result. A line counts as covered the moment a test passes through it.
Take a branch that decides whether to fire an alert. A test can run every line in it and only check that the function didn't throw. The tool marks the branch covered either way, whether the assertion checked the actual alert or checked pretty much nothing at all.
A rising number still reads as a safety claim. It looks like "more of the code is protected now." What it actually says is "more of the code got run during a test," which is a much smaller claim. The gap between those two is where the alert in the story above went missing.
Those two sound close enough that most teams never separate them. There's no second number on the dashboard that says whether anything got checked, so the one that's there gets read that way. Nobody decided to trust the wrong number.
One large study checked the number against real bugs
Kochhar, Lo and colleagues (2017) studied 100 large open-source Java projects, comparing each project's coverage against the bugs logged after one stable release. Their paper states the finding plainly: coverage has an insignificant correlation with post-release bugs at the project level, and no correlation at all at the file level. Coverage across those projects ranged from 0.1% to 93%, and projects with higher coverage didn't have fewer post-release bugs. The data was collected in 2013, so the code is old. But the question the study answers hasn't changed.
The study doesn't say coverage is worthless. A module sitting at 0% coverage is genuinely a blind spot. What it doesn't support is the belief that a number climbing from 60% to 80% means the codebase got measurably safer. Nothing in those 100 projects backs that up.
These were mature projects. Established, open source, every bug in a public tracker. If coverage doesn't predict bugs there, I wouldn't expect it to in a private codebase shipping under deadline pressure.
Why the number can rise while the tests don't get better at catching anything
There are two common ways to raise coverage without catching more bugs. The first is the tautological test: it runs a line and asserts something that can only ever be true. Something like checking that a function "was called," without checking what it was called with. The second is over-mocking: the test mocks the very thing it's supposed to check, so the real logic never runs.
That parsing test from the start was the first kind. Nobody set out to write a bad test. Its assertion was just loose enough to pass on almost anything. Who writes a test like that doesn't matter: a developer under a deadline, or an AI in a few seconds. The coverage tool cannot tell the two apart, and neither can a dashboard that only ever shows the percentage.
Either way, the coverage number goes up, but the suite isn't actually more likely to notice when something breaks. The line that ran is not the line that got checked.
Code that runs in a test but never gets checked shows up in real bugs, too. Petrović, Ivanković, Fraser and Just (2021) looked at 1,502 high-priority bugs at Google that had already been fixed, and went back to the changes that introduced them. All of those changes were covered by existing tests. By the coverage number, every one of them was fine. And Google doesn't take testing lightly: it runs 500 million tests a day.
Then the researchers asked whether mutation testing would have flagged the change before it merged. Mutation testing means breaking the code on purpose in small ways and checking whether any test notices. For 70% of the bugs it would have. In other words, you could break that code on purpose and the tests would still pass. They ran it, but they probably never checked the behavior that broke later.
The authors sum it up in one line: coverage "only determines whether code has been executed, regardless of how well its behavior has been checked." The tool works as designed. It only turns into a problem once a team reads the number as a stand-in for something it never counted.
What this actually costs, and who pays it
In our case, the customer paid first. They missed an alert, and they were the ones who told us.
The postmortem didn't mention the tests at all. We fixed the parsing and moved on. At some point we found out that the tests around that code weren't really testing it properly. That was a lesson for us.
Everyone trusted the coverage gate. If a change got through it, we assumed the tests were good enough. But the gate only counts the lines that ran. It can't tell you whether anything got checked.
Review didn't catch it either. We looked at the code change, saw that it came with tests, saw that they were green and that the gate had passed. Whether those tests asserted the right thing, we probably didn't check. Not because anyone was lazy, we just didn't know better. A green test next to a change looked like someone had already done the checking.
Now that AI writes a lot of the code and the tests, it's even easier to trust them without looking. I've caught myself in a review thinking "this will be fine, Claude wrote it." I wrote about that trap in when not to use GitHub Copilot. If the tests came from the same prompt, AI wrote the code and the tests. Maybe no person ever checked either of them. A green build on top of that doesn't tell you much.
Common questions
Is code coverage a good measure of test quality? Not on its own. Coverage tells you which lines ran, not whether the assertions around those lines would catch a real defect. A module can sit at 95% coverage with tests that don't actually check anything, like asserts that pass no matter what.
What is a good code coverage percentage? There isn't a target number worth chasing. The percentage doesn't reliably predict defect rates, and in the Google study every bug-introducing change was already covered when the bug went in. A more useful question than "what percentage" is whether the tests behind that percentage would actually fail if the behavior they cover broke. If you pick 80% because it's round or because the tool suggests it, you'll hit it. It just won't tell you much.
Why did our coverage go up and our bugs stay the same? The Kochhar study looked at pretty much this question. Projects with more coverage didn't have fewer bugs after release. Most likely your suite grew in a way that ran more code without checking more of it. That's normal, and worth a look.
What is the difference between code coverage and test coverage? In practice the two terms get used interchangeably, and both usually mean the same thing: the percentage of lines, branches or statements a test suite executes. Some teams draw a finer line, using "code coverage" for the raw line or branch count and "test coverage" for a broader sense of which requirements or user scenarios are tested. Either way, neither term says anything about whether the test that ran the code actually checked what it does.
What to check this week
I wrote about the related mistake, treating a green build as proof rather than a starting point, in test coverage vs test quality. The number in this essay explains why that green build proves less than it looks like it does.
Pick the module with the biggest coverage jump this year and ask one question: if this broke, which test would fail? If it takes the team more than a minute to answer, that's where to look first. Not the module with the lowest number.
Get the checklist I use to audit an AI-written test suite. No email needed.
Want the occasional email?
New articles, more free resources, and what I’m working on. Usually once or twice a month, often less.
What happens with your address
It goes to me, Thomas Langhorst, and to Brevo, who send the emails. It is used for the occasional emails about testing and code quality you just asked for, and for nothing else. Whether you open those emails and what you click in them is measured, which is what the second sentence of the box is about. You can unsubscribe at any time with one click, which is as easy as subscribing was. Your signup and your confirmation click are logged, and that record is kept for three years after you unsubscribe so the consent can still be proven. Nothing is shared with anyone else and nothing is sold. Full detail in the privacy policy.
You can object to advertising at any time, and the emails stop.