An AI voice agent answers and places real phone calls, understands speech as it arrives, looks information up mid-conversation, and takes action — booking a slot, updating a record, processing a request — before handing to a human when needed. It works by chaining speech recognition, a reasoning model with tool access, and speech synthesis inside a strict latency budget.
Voice is the hardest agent to build well and the easiest to demo badly. In a scripted demo you control the questions, the accent, the background noise and the network. On a real inbound line you control none of that, and every weakness shows up in the first hour.
The stack, and where the time goes
A real-time voice agent is a pipeline with a stopwatch on it. Audio arrives, gets turned into text, a model decides what to do, possibly calls a tool, produces a reply, and the reply is turned back into audio — and the whole round trip has to fit inside the pause a human will tolerate.
| Stage | Typical budget | What blows it up |
|---|---|---|
| Telephony transport | 50–120 ms | Poor carrier routing, no regional media server |
| Speech to text (streaming) | 100–200 ms | Batch instead of streaming recognition |
| Model reasoning | 200–400 ms | Oversized prompts, no streaming, a model chosen for benchmarks not speed |
| Tool call | 0–300 ms | A slow CRM API, no cache, a synchronous call that should have been deferred |
| Text to speech (streaming) | 80–150 ms to first audio | Waiting for the full sentence before synthesising |
Two design decisions do most of the work here. First, everything streams — the agent starts speaking the beginning of its answer while the rest is still being produced. Second, slow tools get handled conversationally: the agent says it is checking, then speaks the result, instead of going silent for two seconds.
Interruptions are the real test
People interrupt. They say the account number before the agent finishes asking, they change their mind halfway through, they talk over a confirmation. A voice agent that cannot be interrupted feels like an IVR with a nicer voice, and callers treat it the same way.
- Barge-in: the agent stops speaking the instant the caller starts, and discards the rest of the utterance it was about to say.
- Endpointing: knowing the caller has finished rather than paused. Tuned too tight it cuts people off; too loose and every turn feels sluggish.
- Backchannel handling: an mm-hm from the caller is not a turn, and treating it as one derails the conversation.
- Recovery: when two people talk at once, the agent yields and re-asks, briefly, rather than pretending it heard.
Doing things, not just talking
The value is not the conversation. It is what happens because of it. A voice agent that can only answer questions is a very expensive FAQ; one with tools closes the loop while the caller is still on the line.
- Look up the caller by number and greet them with their order or booking already in context.
- Check live availability and book the slot, then confirm it in the same breath.
- Write the call outcome, summary and next step into the CRM before hanging up — the same write path covered in CRM and workflow automation.
- Take a payment or a return through a compliant flow, or hand to a human where regulation requires one.
- Trigger the follow-up — the confirmation email, the ticket, the callback task.
This is where a voice agent stops being a novelty and becomes part of operations, and where it stops being a separate product: the caller it greets is the same person the sales agent emailed yesterday, and the policy it quotes should be the one the support agent quotes too. It is also where the integration work lives — the conversation is a week of tuning, the tools are the rest of the project.
Inbound and outbound are different products
Teams often ask for both and assume it is one build. It is not, and the difference is mostly about who holds the initiative.
| Inbound | Outbound | |
|---|---|---|
| Caller intent | Unknown — the agent must classify it fast | Known — the agent owns the agenda |
| Hardest part | Coverage of everything people might ask | Opening seconds, and consent |
| Failure mode | Cannot help, transfers badly | Sounds like a robocall and gets hung up on |
| Compliance load | Disclosure, recording consent | Disclosure, calling hours, do-not-call, opt-out |
Outbound carries genuine regulatory weight depending on where you and your customers are: consent requirements, permitted calling hours, do-not-call registries and recording rules all vary by jurisdiction. This is a legal question before it is an engineering one, and it should be answered with your counsel rather than inferred from a vendor page.
The handoff decides whether customers forgive it
Every voice agent will meet a call it cannot finish. What happens next determines whether the customer remembers a useful assistant or a wall.
- Detect the limit early — repeated misunderstanding, an out-of-scope request, an audibly frustrated caller, or an explicit ask for a person.
- Say what is happening: it is connecting them to someone, and why.
- Pass context to the human — transcript, caller identity, what was tried, what is still needed.
- Stay on until the human picks up, so nobody is dropped into hold music mid-sentence.
- Log the escalation reason, so the pattern shows up in next month's improvements.
Measure escalation rate as a quality signal, not a failure count. An agent escalating twelve percent of calls with clean handoffs is in far better shape than one escalating three percent by bluffing through the other nine.
What to measure once it is live
- Containment rate — calls fully resolved without a human, segmented by intent rather than averaged into one number.
- Time to first word, at the ninety-fifth percentile. Averages hide the calls that felt broken.
- Interruption recovery — how often a barge-in leads to a confused turn.
- Outcome accuracy — sampled transcripts checked against what should have happened, weekly.
- Cost per resolved call, not cost per minute. A cheap agent that fails and transfers costs more than an expensive one that finishes.
Voice rewards patience. The first two weeks after launch produce more useful signal than any amount of pre-launch scripting, because real callers immediately find the paths nobody thought to write down.