AI
When an AI agent is the wrong answer
Not every support queue, every workflow, every question needs a model behind it. Here is how we tell.
- AI
- 6 min read
We build AI agents for a living, so it might seem strange to publish a post about when not to build one. But the fastest way to make someone distrust AI permanently is to hand them a bad implementation of it — a chatbot that confidently invents a policy that does not exist, or an "automation" that replaces a two-minute human task with a worse five-minute AI-mediated one. Most of the damage AI has done to its own reputation is self-inflicted, and it is avoidable by being honest about fit before writing a line of code.
Good fit: high volume, low stakes per answer, patterned
AI agents earn their keep on questions that arrive constantly, follow a recognisable pattern, and where a wrong answer is inconvenient rather than dangerous. "What are your opening hours," "where is my order," "do you ship to New Zealand" — high volume, patterned, low individual stakes even if occasionally wrong, because the fix is a follow-up message, not a real problem. This is the sweet spot, and it is genuinely valuable: a well-trained agent handling the majority of first-contact questions frees a real person for the ones that need one.
Poor fit: rare, high-stakes, or genuinely novel
Three situations where we actively talk clients out of an AI agent, even when they arrive asking for one:
The question is rare
If a specific question comes up twice a year, training and maintaining a model to handle it costs more than just answering it by hand when it happens. AI agents pay for themselves on volume. Without volume, you are maintaining infrastructure for a problem a person could solve faster than the agent could be kept accurate.
A wrong answer actually matters
Medical guidance, legal interpretation, anything touching a refund or a contractual commitment — the cost of a confidently wrong answer is not "mildly annoying," it is a real problem for the person on the other end and a liability for you. These need a human, or at minimum a human reviewing every response before it goes out, which usually erases most of the speed benefit anyway.
The situation is genuinely novel each time
If every case requires understanding something specific and non-repeating about that customer's situation — not a variation on a known pattern, but something new — a model trained on past patterns has nothing reliable to pattern-match against. This is where agents most often produce a confident, plausible, wrong answer: not because the model is bad, but because the problem was never a good fit for pattern matching in the first place.
The rule we actually build against
An agent should say "I don't know, let me get someone" more readily than it should guess. Every agent we ship is built with an explicit confidence threshold: below it, escalate; above it, answer. Getting that threshold right takes real transcripts, not a demo — which is why we run a staging period before an agent talks to an actual customer, watching a week of real questions before it goes live unsupervised.
The honest version of "AI can help your business" is narrower than most of the marketing around it suggests: it is very good at absorbing volume on patterned, low-stakes questions, and a poor tool for rare, high-stakes, or genuinely novel ones. Knowing which category a given problem falls into, before building anything, is most of the job.
Ready when you are.
Tell us what you're building and we'll map the shortest route there.
Keep reading