Like I said, I've got a pile of projects I'm building with AI, and I'm going to write about all of them here. But let me start with the most obvious one: this blog.
Publishing here is automatic, and that's on purpose. I write a draft. The AI revises it and expands it into something better written than what I handed over. Then a service takes over: it rebuilds the whole website and puts the new version online. I never press a publish button.
That service, of course, I also built with AI.
And at some point the AI added a small safety rule to it. In plain terms: don't try to publish if the access keys are missing. Reasonable thing to want. It looked like every other rule of that kind I've seen, so I said yes and moved on.
Then this blog quietly stopped publishing, and nothing told me.
The rule had been put somewhere that this particular kind of rule isn't allowed. Nothing reported an error, though. The instruction file just became invalid, and an invalid file doesn't run at all. So there was no failure, because nothing ran in order to fail. No warning, no email, no red mark anywhere. I kept writing, kept saving, and the site kept showing the old version, patiently, for as long as I let it.
In fairness to the rule, it was extremely safe. Nothing ever goes wrong with a publish that doesn't happen.
It sounds the same when it's wrong
I've worked with people for many years, and people show doubt. They leave a note saying "this should work". They explain the easy part in great detail and go strangely quiet about the hard one. They slow down. They ask a question first. After enough years you learn to hear all of it.
An AI gives you none of that. It'll describe a real thing and a thing it invented ten seconds ago in the same tone, at the same speed, with the same easy confidence. Nothing in the writing separates the two.
So I couldn't have caught that broken rule by reading it more carefully. It didn't look wrong. It looked completely normal, which is exactly how the correct version would have looked too.
There's a line about this from Edsger Dijkstra, one of the people who worked out how software should be built, in a lecture he gave in 1972. Testing, he said, can show that problems are present. It can never show they're absent. He was talking about work done by careful humans. It hits harder when your collaborator writes a thousand lines before lunch and has never once sounded unsure about any of them.
So I stopped trying to judge the work by reading it, and started checking it instead.
Give the checks to the AI, not just to yourself
When the work is code, the useful checks already exist, and they're all the boring mechanical kind. Does the project still assemble correctly? Does anything break the rules the project set for itself? Do the automated tests still pass? None of that needs intelligence or taste. A computer answers all of it in under a minute, and the answer is a plain yes or no.
What actually improved my results wasn't adding those checks. It was handing them to the AI and letting it run them while it works, instead of running them myself at the end.
That sounds like a small difference, and it really isn't. An AI that can run the checks itself finds and fixes most of its own mistakes before I ever see them. An AI that can't will write me a calm, well-argued paragraph explaining why the error doesn't really matter.
So now every save runs those checks automatically, and nothing reaches the live site unless they all pass. A check like that has no opinion about how confident the AI sounded while writing. It's the only reviewer on this project that can't be talked into anything, and I've grown very fond of it.
Passing all the checks doesn't mean it's any good
While all of that was working perfectly, the front page of this blog was serving a 2.4 megabyte image in order to display a picture roughly the size of a postage stamp.
Every check passed. On my computer, testing locally, the page looked flawless. Nothing was broken in any way a program could measure. But anyone arriving on a phone, on mobile data, was slowly and painfully downloading a large file for absolutely no reason.
None of my checks had an opinion about that, so none of them said a word. That's the real limit of automatic checking. It tells you something works. It can't tell you it's any good. The fix took a few minutes once I finally noticed, which is the annoying part.
Two things help with the rest, the part that needs actual judgment.
The first is turning judgment into rules, whenever a rule will hold still. This blog has a written style guide, which is really just a list of decisions I've already made. How images are prepared. How sections are structured. What the writing should sound like. The site enforces some of them by itself now, so breaking them isn't even possible. Every decision I write down is one I never have to pay an AI to make again, and unlike an opinion, it doesn't drift.
The second is asking a different AI to review the work. Not the one that produced it. Ask an AI to review its own output and it will approve it, warmly, immediately. So I use a separate one with a separate instruction. For code, I tell it to read the changes as though its job is to break the site. For writing, I ask it to hunt down the claims I haven't actually supported.
There's an obvious trap here that I still manage to forget. The reviewer is also an AI, so it sounds equally certain whether it's right or wrong. It has the exact problem I started this post with, except now it's pointed at my quality standards. So every so often I read its judgments myself and check they still make sense. Otherwise I haven't fixed anything. I've just moved the problem somewhere I never look.
This post went through all of that, which is only fair. I wrote a draft. The AI expanded it and made it read a lot better. A second one went looking for anything overstated. Then I read the whole thing myself, slowly, because the last check is always mine.
Does a blog post need all this?
No. What I just described takes about half an hour, and half an hour is about right for a blog post.
Another part of this project needs a lot more. Somewhere in here is the description of the actual machinery the site runs on: the storage where it lives, the network that delivers it around the world, the security certificate that makes the padlock appear in your browser, the settings that point the domain name at all of it. That's defined in code too, which is normally an excellent thing.
That system also keeps a memory file, recording what it has already built. At some point, that memory file was lost. Which meant the next time anyone ran it, it would have looked around, decided nothing existed yet, and started confidently building a second copy of everything. Another delivery network. Another certificate. Then a crash on a storage name that was already taken. All of it live, all of it costing real money, and all of it thoroughly miserable to untangle.
So that part gets treated with a lot more suspicion. Explicit instructions to adopt what already exists instead of rebuilding it. A memory file kept somewhere it can't get lost again. A process that shows me exactly what it plans to do before it touches anything, and I read that list every single time.
Same project, same AI writing both. The difference isn't how much I trust it. The difference is what the mistake costs to undo.
People go wrong in the other direction too, and it's just as easy. You end up with an enormous checking system wrapped around something tiny. A committee of AIs solemnly debating a minor edit. A review process so slow that you quietly stop bothering with it. A check you skip protects nothing at all.
This part is still my job
The broken rule is gone. Publishing happens on every save now, and when it fails, it fails loudly, somewhere I'll actually see it. That's all I need from it.
I'm not trying to make the AI always right. That's not a realistic goal, and I wouldn't know how to aim for it anyway. I'm trying to find out quickly when it's wrong, which is very achievable.
None of this is new, and honestly none of it is really about AI. Confidence was never proof. Not from a contractor quoting for work, not from a colleague who's sure it'll be ready on Friday, and not from me at two in the morning, absolutely certain about something I hadn't checked. An AI is just the first thing I've ever worked with that manages it every single time, instantly, and in a very pleasant tone of voice.
"Because the AI said so" is not an answer. It only sounds like one.