I’ve been building a tool where several AI models work on the same codebase. One makes the changes; the others review them.
About ten weeks in, the part I keep coming back to is what a reviewer’s approval is actually worth.
Early on, I’d get a “looks good” from the second model and feel more confident about the change. But reading back through the transcripts, I often couldn’t tell what it had checked. Had it opened the surrounding code? Run the tests? Or just read the diff and found nothing obvious?
That made the approvals hard to use, so I changed a few things:
- Reviewers can inspect files and run builds and tests on the same machine, with restrictions on what commands they can use.
- The tool records the commands that actually ran, their exit codes, and their output. A model saying “tests passed” doesn’t count as evidence by itself.
- An approval without any checks is marked “unverified.” If a reviewer approves while a check is failing, that gets flagged too.
There are still gaps. Passing tests don’t help much if the tests miss the bug. I’m wary of calling something “verified” when all we really know is that a particular set of checks passed.
The command restrictions also limit what reviewers can confirm, especially when a change needs writes or a deployment to test properly.
And I haven’t benchmarked this against ordinary review, so I don’t know how many additional bugs it catches. So far, the benefit is that I can see what an approval is based on.
How do you decide an AI-generated change is done? Do you rerun the tests yourself, trust the agent’s report, or have another model review it? If that second model approves without running anything, does its approval carry any weight for you?
Disclosure: I’m building LetThemBuild, by GPULogy AI. It’s free to try. I’ve left the link out of the post, but can share it in the comments if anyone asks.
