Notes · · 5 minute read
How I check AI agent output.
An AI workflow is only as good as what happens on the day it gets something wrong. These are the five rules I build every automation around, for clients and for my own studio.
AI fails quietly
A broken script stops. A wrong answer from a language model usually does not. It arrives in fluent sentences, in the right format, at the right time. The failures that matter in business automation are rarely crashes; they are confident outputs that are slightly wrong: a lead marked as spam, a total copied from the wrong row, a reply that promises something you do not offer.
That is why I do not judge a workflow by how good its best output looks. I judge it by what happens to its worst output.
The loop
Every project I make, whether a website, a documentary, a game or an automation, goes through the same four steps: brief, build, check, ship. For an AI workflow, those steps look like this.
| Step | In an AI workflow |
|---|---|
| Brief | Write down what correct output looks like, and what is out of scope |
| Build | Agents do the work, inside a budget |
| Check | A separate step tests every output against the brief |
| Ship | Passing work goes out; failing work goes to a person; every run is logged |
1. Define correct before you build
Without a definition, checking is an opinion. Before I build a workflow I write down what a correct result contains, what it must never contain, and which cases go straight to a person. For lead intake that might be: every enquiry gets a category from a fixed list, a contact detail that was really in the message, and no price that is not on your price list.
2. The checker is not the agent
Nobody should mark their own work. The step that checks the output is separate from the step that produced it. Some checks are plain code: is the category on the list, does the number match the source, is the email address real text from the message. Some use a second model with a narrow question and no stake in the first answer. Either way, the agent's output is treated as unverified until it passes.
3. Failures go to a person
An output that fails its check is retried once with the reason attached. If it fails again, it goes to a named person with everything they need to decide. It is never sent on silently, and it is never dropped.
4. Every run leaves a record
Each run keeps its input, its output, which checks passed or failed, who handled the exceptions and what it cost. When something goes wrong, the question is not "what did the AI do?" but "which step let it through?", and the log answers it.
5. A hard stop on cost
Every workflow has a budget per run and per day, like a circuit breaker. If it is reached, the workflow pauses and tells a person, rather than spending its way through a loop.
What this means for a business
None of this makes a workflow clever. It makes it safe to rely on. The useful question for any business automating a process is not whether the AI can do the task, but what happens on the day it gets the task wrong. If the answer is "a person sees it, with the reason, before a client does", the workflow is ready.
The habit comes from eight years in engineering quality control, where nothing was finished until it had been checked. It is the standard I bring to every AI automation project.