Review your AI's Code - A Simple Process for Building More Robust Systems Faster with AI

DISCLOSURE: If you buy through affiliate links, I may earn a small commission. (disclosures)

I've been using AI to help me code for the past several months and have noticed a pattern that seems to set those who use it effectively apart from those who use it haphazardly.

The pattern seems to be that those who review their AI's code, understand it, and change it accordingly build more robust systems faster than those who don't.

AI is good but it's not great

AI generated code and references are incredibly good. But they're not perfect.

The reason they're not perfect can be many-fold:

Their training data is different
They don't hold the whole codebase in context
They don't know the best practices of your org or your subjective taste
You haven't given them enough context to actually do what you wanted, the way you wanted

So every time you do a generation, it might be a little bit off.

Misalignments compound

This means that even good generations are often off by some % from the ideal - maybe it's just 1% or maybe it's 10%.

The problem is that these misalignments compound. It might not be 2 or 3 generations from now but give it 10 or 20 and those small misalignments in what we hoped to achieve have now turned into large misalignments in how the system has grown.

99% twice leads to 98% (0.99 ^ 2)
99% ten times leads to 90% (0.99 ^ 10)
99% one hundred times leads to 36% (0.99 ^ 100)

This is often why vibe coding projects turn into big balls of mud after a few days / weeks. They start out working but as you layer small misalignments onto each other, even the AI starts to get confused about what's going on.

These misalignments happen in software engineering teams too but the key is the speed at which the system's trajectory goes off course. This is mostly because effective software engineering teams have processes in place to regularly reconcile these misalignments.

Code reviews are simple, effective tools for reconciling misalignment

The way we resolve this in software engineering teams at scale is RFCs for large pieces of work and code reviews with peers for each change to reconcile those small % differences at implementation time.

Some people are against code reviews as they "slow people down" but I think they're one of the most effective practices we have for improving system quality and knowledge transfer. I also think this view of slowing down is short-sighted as it's a small cost short-term to avoid huge costs long-term if you need to fix a misalignment later.

The code is implemented so we can talk specifics, not nebulous theory
Can course correct early before the code reaches prod and additional systems are built on top of it
You can point out new / different ways of doing things people may not have seen before
The review can serve as a long-lasting form of documentation for what we did, why we did it that way, and alternatives considered
Spreads the knowledge about how things are implemented so more people are aware and can build off that new context

Best practices for AI code reviews

So hopefully we're aligned that AI code is good but not great, misalignments compound, and code reviews are an effective tool for reconciling misalignments.

Here I want to share a few practices I've found work well for turning this AI code into a production-ready package.

Review your AI's code - Review it like you would another engineer's. Understand it. Ask questions, run the tests, and make changes to better align with the goal.
Ask an AI to review the code - Each AI comes in with a different perspective based on their training data and the context you provide. This means they can serve as another pair of eyes to offer suggestions for things to change. I like to ask it for the top 3 things I could change about a PR to make it better and for specific reviews on areas I feel could be improved but can't think of alternatives.
Clean up the PR and get it to a state where you would approve it - You should never publish AI generated code for review before reviewing it yourself. That's basically as bad as writing code and making someone review it before you've run the tests. It might even be worse as AI is pretty good at generating code that looks like it makes sense and then writing tests that inconspicuously use fake data to pass so a reviewer might just LGTM and land broken code. It's a waste of time for you and the reviewer as there's almost always something obvious that should be improved so a whole cycle is wasted on them pointing out the obvious. So you should first improve the code until it's good enough that you would've approved it coming from someone else before submitting it to another human to look at. (You should be doing this even on code you've written yourself. Similar to writing, edit your work before you force someone else to read it.)
Put it up for human review - If you're working on a team / at a company you should still get a human review. Humans with domain knowledge are still going to far outperform AIs on reviews for the simple fact that we can hold more context in memory - about the technologies, about the company, and about the business case this code is meant to solve for. This is the ultimate alignment reconciliation so while it's expensive time and effort wise, it's well worth the returns. Human reviews are usually required for both practical and compliance reasons but it's worth calling out explicitly. If you're just working on personal side projects then you can probably skip this step.

So yeah, review your AI's code. Your system will be better for it and you'll likely improve as an engineer from it. Plus I find the more active approach to using AI is more fulfilling - we're building together, I'm not just pressing the slot machine button over and over.

If you liked this post you might also like:

Want more like this?