Evaluating a text agent is mostly about whether the answer was right. Evaluating a voice agent is about that too — plus whether it heard the caller correctly, answered fast enough to feel human, and sounded like it should have. A voice agent can produce a perfect response and still fail the call, because it transcribed "can't" as "can," or took four seconds to start talking, or delivered a refund denial in a chirpy tone.
The core mistake teams make is evaluating a voice agent like a chatbot with a microphone bolted on. It isn't. It's a pipeline, and every stage can fail independently.
A voice agent is three systems in a trench coat
Almost every production voice agent is a pipeline: speech-to-text (ASR) turns the caller's audio into a transcript, an LLM decides what to say and which tools to call, and text-to-speech (TTS) turns the response back into audio. Some newer stacks collapse this into a single speech-to-speech model, but the failure modes are the same — they're just harder to isolate.
You have to evaluate each stage, because a bad call can originate in any of them:
- ASR mishears a word, an account number, or a "no" as a "yes." Everything downstream is now reasoning over a corrupted input.
- The LLM does the wrong thing — hallucinates a policy, calls the refund tool without confirmation, loses the thread across turns.
- TTS mispronounces a name, flattens a question into a statement, or reads a number as digits when it should be a year.
Score only the end-to-end transcript and you'll see that the call failed but not where. Per-stage tracing is the difference between "the agent is bad" and "our ASR model drops the last word when the caller talks over the prompt."
Latency is a quality metric, not an ops metric
In text, latency is a nice-to-have. In voice, it is quality. A caller experiences a two-second gap before the agent starts speaking as the agent being confused, or the line being dead. So the metrics that would live on an infra dashboard for a chatbot are first-class eval metrics for a voice agent:
- Time to first audio — how long after the caller stops talking before they hear a response begin.
- Turn latency across the full STT → LLM → TTS round trip, tracked at p50 and p95, not just the average.
- Barge-in / interruption handling — when the caller cuts in, does the agent stop and listen, or keep talking over them?
A correct answer delivered too late is a bad answer. Your eval has to encode that.
What to actually measure
Beyond latency, the metrics that matter for a voice agent:
- Task success / containment rate — did the call accomplish what the caller wanted without escalating to a human? This is the top-line number.
- Word error rate (WER) on the ASR stage, ideally weighted so that critical tokens (numbers, names, yes/no) count more than filler.
- Turn-level correctness — was each response factually right and policy-compliant, judged against the corrected transcript so you're not double-penalizing ASR errors.
- Sentiment and tone — did the caller's sentiment trend up or down over the call, and did the agent's delivery match the situation.
- Interruption and dead-air rate — how often the agent talked over the caller or left an awkward silence.
- Escalation / hang-up rate — where in the flow callers give up or demand a human.
Offline: replay recorded calls before you ship
Build a dataset from real recorded calls — audio, the corrected transcript, and the known outcome. Then replay that audio through your agent and score the result. This is your regression suite: when you swap the LLM, change a prompt, or upgrade the ASR model, you rerun the same calls and diff the scores.
The reason to anchor on recorded audio rather than clean text is that half your failures live in the audio itself — accents, cross-talk, background noise, the caller mumbling a confirmation code. A text-only test set can't surface any of it. This is the offline half of the loop: fixed inputs, run before every deploy.
Online: post-call QA on production traffic
No offline set covers the live phone lines. So the other half is post-call quality assurance — scoring production calls after they happen. Sample real calls, run LLM judges over the transcripts for task success, correctness, and tone, and surface the low scorers.
Two things make voice QA different from text QA:
- You can't score everything with a judge alone. Tone, escalation-worthiness, and "would a customer be satisfied" are exactly the judgments where an LLM judge is weakest and a human reviewer is strongest. Route the ambiguous and high-stakes calls to people; let the judge triage the volume.
- The failures you find are gold. Every production call where the agent mishandled an interruption or misread a number is a case your offline set didn't have. Promote it into the recorded-call suite so your CI gate defends against it forever.
Multilingual and IVR wrinkles
If your agent handles more than one language, add pronunciation accuracy and code-switching (a caller who switches languages mid-sentence) to the rubric — and evaluate each language separately, because an aggregate score hides a stage that's fine in English and broken in Spanish. For IVR-style agents that navigate menus and route calls, treat the navigation path itself as a scored trace: did the agent get the caller to the right place in the fewest turns?
The short version
Evaluating a voice agent is evaluating a pipeline against the clock:
- Trace and score every stage — ASR, LLM, TTS — not just the final answer.
- Treat latency as quality, measured at p95, not just p50.
- Anchor offline evals on recorded audio so acoustic failures show up before you ship.
- Run post-call QA online, with humans on the calls that judges can't reliably grade.
- Feed production failures back into the offline suite.
The same offline/online loop that governs any AI product applies here — voice just adds three more places for it to break, and a stopwatch.