Test the voice agent the way a customer meets it. On the phone.
EvaliQA places the call, holds the conversation with a persona and a goal, records both sides, transcribes them, and scores the transcript against the same metrics your text evals use.
What is voice agent evaluation?
Voice agent evaluation is testing a phone or voice bot over real calls rather than typed transcripts: placing or answering the call, holding a conversation, recording both sides, transcribing them and scoring the transcript against metrics. It catches what only breaks on the phone: talking over the caller, long silences, and answers that were right but unintelligible.
EvaliQA runs the call through your Twilio account with a caller persona and a goal, keeps the recording and the turn-by-turn transcript, and scores the transcript with the same catalogue your text evals use plus call-only metrics such as response time.
A real call, not a transcript you pasted in
Voice evaluation is not text evaluation with an audio file attached. EvaliQA orchestrates an actual phone call through Twilio: our agent speaks, your bot answers, speech is transcribed as it arrives, replies are synthesised, and the call ends when the conversation does or when the duration cap hits.
That means the things that only break on the phone get caught: an agent that talks over the caller, one that leaves four seconds of silence before answering, one whose answer was right but whose speech was unintelligible.
- Outbound we place one call per dataset row to the number you give us.
- Inbound your bot calls your Twilio number and our agent answers, one row per incoming call.
- Per-call recording played back in the browser as a waveform, with the transcript beside it.
- Turn-by-turn transcript who said what, in order, with the scoring attached to the exchange.
- Caller agentA persona with a goal, on our side of the call.
- Speech synthesisThe caller's turn is spoken aloud (TTS).
- Phone callA live call carries the audio, both ways.
- AI systemunder testThe voice agent answers, turn by turn.
- TranscriptionBoth sides are written down as they speak (STT).
- ScoringThe transcript is scored against your metrics.
Loops each turn until the call ends.
Everything else reads like a normal eval
The dataset, the metrics, the run list and the run detail page are the ones you already know, with a transcript layer grafted on. A voice plan carries a phone number column, and the rest of the wizard behaves exactly as it does for text.
Which means the comparison you care about still works: two runs side by side, a metric history across releases, and a written report at the end.
Metrics that only make sense on a call
Alongside the full catalogue, which scores a voice transcript exactly as it scores a chat answer.
| Metric | What it measures |
|---|---|
voice_response_time | Time from the caller stopping to the agent starting to speak. |
voice_transcript_correctness | Judge scores the transcribed answer. |
voice_intent_match | Did the agent fulfil what the caller actually called about? |
answer_relevancy | The same judge your text evals use, on the transcript. |
Everything around the call, included
Set up your number
Point a run at a phone number and you are ready. EvaliQA places the call, or answers when the call comes in, one call per dataset row.
A persona makes the call
The caller is a model with a goal and a temperament, from a cooperative customer to one who interrupts and changes its mind.
Recorded and transcribed
Both sides of every call are kept and written down as they speak, so a run has audio to play back and a transcript to read.
Scored like any other run
The transcript is judged against your metrics, multi-turn by default, because a phone call is a conversation and a one-shot test rarely tells you anything.
Frequently asked questions
How do you test a voice AI agent?
You test it by calling it, not by scoring a transcript you typed. EvaliQA places an outbound call to your number, or answers when the agent calls in, then drives the conversation with a persona and a goal. It records both channels, transcribes them as the call runs, and scores the transcript. One dataset row is one call, and up to five run in parallel, so a set of scenarios finishes in minutes instead of one call at a time.
How is testing a voice agent different from testing a chatbot?
A chatbot test scores text you send; a voice test has to survive the phone. Latency before the agent speaks, talking over the caller, dead silence, speech-to-text errors and answers that were right but unintelligible only show up on a live call. EvaliQA runs the actual call and keeps the audio, then scores the transcript with the same metric catalogue your text evals use plus call-only metrics, so a voice run reads like every other run.
What metrics measure voice agent quality?
Two kinds: the metrics that only exist on a call, and the ones you already use on text. Call-only metrics cover response time from the caller stopping to the agent speaking, transcript correctness, and intent match, whether the agent handled what the caller actually called about. On top of those, the full catalogue scores a voice transcript exactly as it scores a chat answer, including relevancy, groundedness and any custom judge you have written.
How do you measure a voice agent's response latency?
By timing the gap between the caller finishing and the agent starting to speak, on every turn of a real call. EvaliQA records that as a response-time metric per row, with a threshold you set, so a build that got slower fails a number instead of being called laggy by feel. Because the timing comes from an actual call over the line, it includes the speech-to-text and text-to-speech legs, not just the model.
Can you evaluate a voice agent without making real phone calls?
You can score a saved transcript with the text metrics, but you will miss everything that only happens on the line. Latency, interruptions, silence and speech the caller could not understand never appear in a clean transcript someone typed up. That is why EvaliQA places or answers a real call and keeps the recording next to the transcript, so the score reflects the call your customer would actually hear.
How do you run voice regression tests before a release?
Keep a dataset of reference calls and run it against every build, scoring each call on the same metrics with a passing threshold, then compare the new run to the last one. Scripted mode replays your exact user turns byte for byte, which is what makes a regression test stable; simulation and adaptive modes cover the messier paths. Open the row whose score dropped and read the judge's reasoning, rather than re-listening to every call.
How does the caller behave during a simulated call?
A model plays the caller, with a goal and a temperament rather than a fixed script. There are three modes: scripted, where you write the turns and they replay byte for byte; simulation, which uses deterministic seeds for predictable coverage; and adaptive, where the caller improvises from what your agent just said. So one row can be a cooperative customer and the next one who interrupts, changes its mind and asks for a human.
What do you need to set up voice evaluation, and what does it cost?
A phone number, a Twilio account, and your speech-to-text and text-to-speech keys. Calls are placed on your own account and speech runs through your own providers, so nothing is resold and the numbers stay yours. Free includes 30 call minutes a month and Team has no cap. The rest of the wizard is the one you already use for text plans, with a phone number column added to the dataset.
Call your own voice agent and read the transcript
Point a run at a number, pick a persona, and listen to what happened.