2026-09-12
The honest test
An agent earns its complexity when a workflow has three properties: the inputs vary too much for deterministic automation, the volume is high enough that humans are the bottleneck, and mistakes are detectable and recoverable. Miss any one of the three and a plain script, a better UI, or a checklist will outperform an LLM loop at a fraction of the cost.
We run this test with clients before writing a single prompt. Roughly half the 'we need AI' conversations end with a cheaper, more reliable recommendation — and that honesty is why the other half succeed in production.
Where agents consistently win
Triage and routing: reading high-volume inbound work (tickets, documents, leads), classifying it, drafting responses, and escalating the novel cases. Retrieval-grounded answering over private knowledge. And multi-step research and data-entry pipelines where every step is verifiable after the fact.
Where they consistently disappoint: open-ended creative judgment with no ground truth, workflows with zero tolerance for a wrong output, and anything where you cannot measure quality. If you can't evaluate it, you can't ship it.
Start with evaluation, not prompts
The teams that succeed build an evaluation set from real historical cases first — even 50 labeled examples beat intuition. Every prototype change is then measured, not vibes-reviewed. This single practice separates the agents that survive production from the demos that don't.