Zero Errors, Green CI, AI-Written Code: Who Actually Thought It Through?

A Pull Request of four hundred lines, green pipeline, every test passing, and an architect who rejects it anyway. What did the architect see that the CI doesn't? What do a study from Anthropic and GitClear's data say about AI-written code? And what checks do I run before that code gets into the repository?

It’s Friday afternoon. A developer on your team opens a Pull Request on the invoicing module of your business application: four hundred new lines. The CI is green, the compiler gives no warnings, all the unit tests pass, the names are tidy and there’s a comment above every method. The ticket said two days of work, and it’s ready in an hour and a half.

You scroll through the lines and find a transaction left open in the middle of a loop. You call the developer and ask one question: “If the database times out on the third line of the invoice, what happens here?”

He looks at the screen and says: “Claude wrote it for me, all the tests pass.” You press “reject”.

The scene is made up, but anyone who has done a few code reviews recognizes it in two lines. This article is the written version of episode 11 of the podcast (in Italian). To write it I went back to the sources, and two numbers I gave in the episode were wrong: I corrected them here, and there’s a correction note on the episode page.

What Kernighan says

Brian Kernighan and P. J. Plauger, in The Elements of Programming Style, write that debugging is twice as hard as writing the code. If you write it using all your intelligence, by definition you aren’t smart enough to fix it. That’s my paraphrase, so you should check the exact wording in the book.

Kernighan was talking about hand-written code. If you squeeze your brain to the limit to write a function, the day it breaks in production you need a version of yourself smarter than the one who wrote it. There isn’t one.

Now take out the part where you squeeze your brain: you hit enter, the model writes, you paste. At three in the morning, with the server hung, you have to understand code you never had to think through.

The source file is only what’s left of a line of reasoning. The rest is the map you built while sorting out the special cases, the database constraints, the dependencies on the other modules. If you skip the effort of writing, you skip that map too. The code ends up in git, compiles and goes to production, but in the team’s heads it’s a black box.

The Anthropic study

Judy Hanwen Shen and Alex Tamkin published How AI Impacts Skill Formation on arXiv (arXiv:2601.20245, January 2026). The setup:

• 52 programmers, 26 per group, randomly assigned
• two tasks with Trio, a Python library for asynchronous concurrency that was new to the participants
• one group with the documentation only, the other with a built-in AI assistant (GPT-4o)
• 35 minutes at most for the tasks, then a quiz on the code they had just written: reading the code, concepts, debugging

The quiz had 14 questions for 27 points. The AI group scored 4.15 points lower, which is 17% less, a statistically significant difference (p = 0.010). The authors find no significant difference in completion time.

The study doesn’t only say uncomfortable things about AI. In the second task, 4 participants out of 26 without the assistant didn’t finish in 35 minutes, while everyone in the AI group did.

The screen recordings interest me most. The authors distinguish six ways of using the assistant. The three that preserved the most learning are the ones where the participant asks conceptual questions or asks for explanations, instead of having the code handed over. People who delegated all the writing saved a little time and learned the library less well. The authors attribute the control group’s advantage to the errors they ran into and solved on their own.

One limitation, and it matters: these were people recruited online who were learning a new library. The study says what happens when you learn something with the assistant. It doesn’t say what happens to a team that has known its own code for fifteen years, and it shouldn’t be read as if it did.

GitClear’s data

GitClear is a company that analyzes repositories. Its 2026 report The Maintainability Gap covers 623 million changes made between 2023 and 2026. These are the measures I care about:

The reuse figure explains what happens in your business application. The model works well on the files it has in context, but it doesn’t know the history of a project that has been growing for fifteen or twenty years. You ask it to calculate a customer discount or to validate a VAT number, and it doesn’t know that three folders away there’s already a unit that has been battle-tested for ten years, with all the tax exceptions. It writes you a new one. It compiles, it passes the test on the base case, and now the project has a second version of the same rule.

With ten developers and hundreds of commits a month, after a year you have the same rule in eight places, with small differences from one copy to the next. When the law changes you update the main class, and the other seven copies keep calculating with the old rule.

Two caveats when reading this table. GitClear sells an analysis tool, and the report doesn’t separate AI-written code from hand-written code: it measures the period in which assistants became common. It’s a correlation, not proof that AI is the cause. The “doubling of churn” I quoted in the episode isn’t in this report either: it was a projection for 2024, compared with 2021, from the previous report.

Why tests aren’t enough

A green CI says that the code does what the tests check. The happy path is the world where the data arrives clean, the database answers in two milliseconds, the network holds and nobody else changes your row while you’re working on it. AI does great in that world.

A business application spends its day outside that world: network timeouts, deadlocks, transactions left hanging halfway, files imported with the wrong encoding, users who double-click the same button. When you write by hand, that’s where the effort goes: every three lines you ask yourself what happens if the connection drops or the query comes back empty.

There’s also a simpler problem. The same model often wrote the tests, on the same assumptions it used to write the function. They check the case it had in mind. Nobody asked what happens when the database times out, and you find out on real data.

Three checks before approving

I’m not throwing AI away: on repetitive code, on supporting tests and on first drafts it saves a lot of time. I change what the code has to get through before it enters the main repository. The three checks are my method, not the result of a study.

1. Mandatory requirements. The Pull Request must meet criteria agreed beforehand: functional validation, minimum test coverage, security and performance constraints. If one is missing, I reject it right away.
2. A review by another model. I pass the Pull Request to a second model, different from the one that generated the code, for a first analysis of requirements, edge cases and dependencies. If the report is positive, I go ahead. It doesn’t replace the human review: it clears out the obvious things before I look at it myself.
3. The final review, done by a person. Me or the colleague I designate, with a checklist: architecture, consistency with the project guidelines, reuse of what already exists. For AI-written code I also check that the tests cover the edge cases and not just the happy path.

When I write the code myself, I do the opposite: I ask the AI to play the adversary. I tell it to look for edge cases, error conditions, unusual inputs, everything outside the happy path.

What about the Friday Pull Request?

Rejecting it looks like a waste of time: the code is there and the tests are green. But two hours spent on Friday rewriting it with the right classes from the project, understanding every step, cost less than three days of emergency work six months later, with the module running on the customers’ real data.

The next step I’d suggest is to try the second check on your next Pull Request: have a second model review the change before you look at it. If you work with Delphi and DelphiMVCFramework, the post on delphi-ai-skills explains how I teach agents the framework’s real API. The episode where I talked about the debt that builds up in code is Perché una modifica semplice ti costa tre mesi (in Italian, “Why a simple change costs you three months”).

If your company has complex business applications or legacy architectures to evolve, and you want to give your team a concrete method for using AI without losing control of the code, write to professionals@bittime.it: at bit Time Professionals we work alongside technical teams with architectural consulting and hands-on training.

Sources

• Shen, J. H., Tamkin, A., How AI Impacts Skill Formation, arXiv:2601.20245. Tasks and annotated transcripts in the study’s repository.
• GitClear, The Maintainability Gap: AI Code Quality in 2026, gitclear.com. For the 2024 churn projection: Coding on Copilot.
• Kernighan, B. W., Plauger, P. J., The Elements of Programming Style, 2nd ed., McGraw-Hill, 1978.