How to deploy an AI agent for customer support without it going wrong
By Mariam Raouf. Published 2026-07-14. 8 min read.
In 2025 Anthropic let Claude run an actual shop in their office. It lost money. It sold below cost, invented a Venmo account that did not exist and told customers to pay into it, and gave away a tungsten cube to someone who asked nicely on Slack.
Then they published all of it. Which is why it is the best case study in existence for anyone working out how to deploy AI agents for customer support, and why I would rather point you at Anthropic's failures than a vendor's demo.
Here is the practical version: what to automate first, what to keep human, what scaffolding changed the outcome, and a rollout sequence you can follow without betting your customers on a demo.
What to automate first
There is a shape of task current agents genuinely do well, and it has four properties. It is bounded, with a clear finish line. It is verifiable, so you can tell from the output whether it worked. It is tool mediated, so the agent reads and writes through an API rather than guessing. And a wrong answer is cheap, because a human sees it first. In a support queue, that is a short list.
- Classifying and routing incoming tickets
- Drafting a first reply for a person to approve and send
- Pulling order, subscription or account status into the ticket
- Summarising a long thread before a human picks it up
- Answering documented policy questions with a citation to the source
That is not a coincidence. It is what US businesses already use AI for. The Census Bureau's April 2026 working paper found AI in worker tasks at 23% of firms, dominated by writing, document analysis and information search, with two thirds using it for three or fewer tasks.
What to keep human
Refunds, credits, discounts and anything else that moves money out of the business. Cancellation and retention conversations. Escalated complaints. Anything touching a legal or regulatory promise. Anything where the answer is no.
That last one is the failure mode nobody expects. Anthropic's phase two writeup lists, among the remaining problems, the agent's "inability to refuse unreasonable demands" as a consequence of its helpfulness training. Models are trained to be agreeable, and your support queue contains people who will use that.
Read this before you deploy AI agents for customer support
Project Vend, published June 2025, gave Claude a real shop, a real budget and real customers who were Anthropic employees. It ran the business autonomously. The pattern in the results is what matters.
It priced below cost. It set prices without researching what things cost to buy.
It hallucinated a payment account. It invented a Venmo account and told customers to send money to it. Not a wrong tone of voice. A wrong bank detail, confidently delivered.
It folded under social pressure. Employees talked it into discount codes and giveaways over Slack, including a tungsten cube. Offered $100 for a six pack of Irn Bru available online for about $15, it said only that it would "keep the request in mind".
It raised a price exactly once across the entire run.
It did not learn. After acknowledging that its discount logic was losing money, it "returned to offering [discount codes] within days".
Map those onto a support queue. Confident wrong information, susceptibility to a persuasive customer, no memory of yesterday's lesson. That is the risk profile, published by the lab with the most to gain from you thinking otherwise.
What changed in phase two, and why it is the whole lesson
Phase two, published December 2025, reached stabilized profitability across three locations. Same capability. What changed was everything around it:
- A CRM, so it had a record of who it was dealing with
- Inventory tooling with cost visibility, so it saw what it paid before setting a price
- Web price comparison, so its pricing had an external reference
- Payment link generation, so payments went through a real system
- An oversight agent checking the primary agent's decisions
Read that list again. Almost none of it is model capability. It is a CRM, a costed inventory, a price check and a supervisor. The difference between losing money and making money was the scaffolding, not the agent.
The remaining failures still matter: naive compliance with an unrealistic contract, inadequate theft protection, susceptibility to social engineering, and that inability to say no. Scaffolding moved it from failing to working. It did not make it safe to leave alone.
Why a good demo tells you almost nothing
Here is the most useful number in this article, and one you will not see in a sales deck. The tau bench paper measures agents on realistic tool use with a simulated user. Leading function calling agents "succeed on <50% of the tasks, and are quite inconsistent", with pass^8 under 25% in retail.
pass^8 means: does the agent succeed at the same task all eight times you run it. Under a quarter of the time.
That gap between single attempt success and repeated success is the entire risk of a support deployment. A demo is one attempt. Your queue is thousands a month. An agent right 70% of the time on the first try, and inconsistent across repeats, produces a steady trickle of confidently wrong answers to customers who have no way of knowing.
Broader benchmark data says the same thing more gently:
| Benchmark | Best model | Human baseline |
|---|---|---|
| OSWorld, real computer tasks | Claude Opus 4.5, 66.3% | 72.35% |
| GAIA, multistep real world questions | 74.5% | 92% |
| WebArena, web navigation | 74.3% | 78.24% |
| tau bench, tool use and multiturn | Claude Opus 4.5, 70.2% pass@1 | no model above 71% |
Source: Stanford HAI, AI Index 2026, Chapter 2. The AI Index says agents "advanced from answering questions to completing tasks in 2025, though they still fail roughly one in three attempts on structured benchmarks", with the worst results on multiturn interactions needing tool coordination and policy compliance. A precise description of customer support.
How to deploy AI agents for customer support, stage by stage
Stage 0, two weeks. Instrument first. Categorise your last 500 tickets. You need your real mix before automating any of it, and most teams find four categories cover 70% of volume.
Stage 1, two to four weeks. Classification only. The agent reads and tags, and sends nothing. Compare its tags against your team's. If it cannot route a ticket, it cannot answer one.
Stage 2, four weeks. Drafts behind a human. The agent writes a reply, a person presses send. Log the edit rate. If your team rewrites more than a third of drafts, your knowledge base is the problem.
Stage 3. Autonomous on one narrow category. The most repetitive, lowest risk one from Stage 0. Order status, opening hours, a documented policy. Nothing that moves money. Keep the human path one click away.
Stage 4. Widen slowly, with a supervisor. Add categories one at a time, plus an oversight step reviewing a sample of autonomous responses, as phase two of Project Vend did.
The guardrails that matter throughout:
| Guardrail | What it prevents |
|---|---|
| Tool mediated data access only | Hallucinated account details, invented policy |
| Hard stop on money movement | Discounts and refunds granted under pressure |
| Confidence threshold with human handoff | Confident wrong answers on unusual tickets |
| Full logging plus sampled review | Silent drift once launch attention fades |
What this actually costs
Running cost is smaller than people expect. Microsoft Copilot Studio bills $0.01 per credit with an agent action at 5 credits and a generative answer at 2, so roughly 7 cents per useful interaction, or about $700 a month at 10,000 interactions, before model costs. The build is where the money goes, and Vend tells you why: you pay for the CRM integration, costed data access, logging and the oversight step, not for the model. Our breakdown of AI agent development and running cost has the arithmetic.
The honest summary
The conclusion from the best documented agent deployment we have is that the model was never the hard part. A frontier model with no scaffolding lost money and invented a payment account. The same model with a CRM, costed inventory, a price check and a supervisor turned a profit. Everything in between was plumbing.
So if a vendor demo is very impressive and very short, that is the tell. Ask what happens on the eight hundredth ticket, ask to see the logs, and ask what the agent does when a customer pushes back. Those three questions sort real products from rebranded chatbots.
At Calpir we build AI agents for support and operations, and a fair amount of that work is telling people which parts of their queue should stay human for now. If you want a look at your own ticket mix first, get in touch.
Frequently asked questions
How long does it take to deploy AI agents for customer support?
Budget three to four months from instrumenting your ticket data to running one category autonomously. The model work is days. Integration, guardrails and observation are the rest. Teams that skip the observation weeks roll it back later.
What should an AI support agent never be allowed to do?
Move money, promise anything contractual, or handle an escalated complaint. Project Vend showed an agent talked into discounts and giveaways over chat, and Anthropic's phase two writeup still lists social engineering as unresolved. Put a hard technical stop on those actions rather than instructing the agent to avoid them.
Do AI agents actually reduce support costs?
For common, repetitive tickets, plausibly. Gartner predicts agentic AI will [autonomously resolve 80% of **common** customer service issues by 2029](https://www.gartner.com/en/newsroom/press-releases/2025-03-05-gartner-predicts-agentic-ai-will-autonomously-resolve-80-percent-of-common-customer-service-issues-without-human-intervention-by-20290) with a 30% cost reduction, and the word common is doing real work there. Complex, emotional and policy heavy tickets are not in that number, and should not be in your business case.
Why did my agent work in testing and fail in production?
Because single attempt success is a bad proxy for reliability. The tau bench paper found leading agents succeeding on under half of realistic tasks and scoring under 25% on pass^8, consistency across eight repeated runs. A demo tests one attempt. Production tests thousands, and the inconsistency shows up as a trickle of wrong answers.