Voice eval page

The Voice eval page at /voice-eval is the workspace-wide list of voice evaluations over real phone calls. Each row is one run: EvaliQA's AI agent either placed calls to a target voice bot (outbound) or answered the calls that bot made to your Twilio number (inbound), recorded audio and transcripts, and scored the transcripts against the plan's metrics.

Voice eval is fundamentally different from text eval. Instead of HTTP requests over a connector, EvaliQA orchestrates real phone calls via Twilio, transcribes the target's speech via STT, synthesises the AI agent's replies via TTS, and hangs up when the call ends or the max-duration cap hits. Everything else (metrics, dataset structure, run detail page) reads like a normal eval, with a transcript layer grafted on.

This page is the read/manage surface. The per-run detail page (/voice-eval/[id], not covered here) is where you replay transcripts, listen to recordings, and see per-turn scoring.

For methodology on when voice eval fits into your pipeline, and the persona / scenario choices that matter for voice specifically, see Personas and escalation styles and Evaluation, multi-turn.

When to open this page

• To start a call run for a test plan with channel = voice. The New voice eval button in the toolbar opens the drawer, where you pick the direction: outbound (we call the bot, one call per dataset row) or inbound (the bot calls your Twilio number and our agent answers, one dataset row per incoming call).
• To monitor a call in progress. Rows update on reload (see Freshness); the run detail page is where live audio and transcript stream in real time.
• Audit trail. Recordings and transcripts are stored per run, this list is the entry point.

What this page is NOT for. Setting up Twilio, STT and TTS, that's in /credentials. The bot's phone number is entered on the run itself (a target number) or comes from the dataset's optional phone_number column.

Header

• Icon and title. Mic, "Voice eval".
• Subtitle. "Voice evaluations over real phone calls: our AI agent calls your bot, or answers when your bot calls it. Every call is recorded, transcribed and scored."
• No header-level feature flag on this route today. Access is implicit: without a voice-channel test plan and voice credentials, the empty state points you to create them, and the New Voice Eval sheet blocks submission.
• No header actions, everything lives in the toolbar.

Empty state

Card with Mic icon.

• Title. "No voice eval runs yet".
• Description. "Create a test plan with the «Voice, outbound call» mode, add voice credentials, and start a run."
• Primary CTA. "New voice eval" opens the creation sheet.
• Secondary CTA. "Read the docs" → /docs.

Loading state: "Loading voice runs...". Errors: red inline banner above the table.

Toolbar

Search input

Placeholder "Search by id or model". Matches run.id and run.llm_model case-insensitively. Plan name isn't in the search index today, filter by Status or Model instead.

Filters popover

Simpler than the text eval-runs page. Two columns:

Column	Kind	Operators
Status	status enum	equals (Queued / Running / Completed / Failed)
Model	text	contains, equals

Rules combine with AND. Blank values are skipped.

Columns popover

Run column is required and can't be turned off. Optional columns:

• Status (default visible).
• Progress (default visible).
• Model (default visible).
• Cost (default visible).
• Created (default visible).
• Duration (default hidden). See below.
• ID (default hidden).

Footer: "X of 8 selected".

Bulk-delete button

Red "Delete voice runs (N)", disabled when nothing selected.

+ New voice eval

Text link with plus icon, opens the creation sheet (see New Voice Eval sheet).

Row card

Floating card row. Whole row is a role="button"; click or Enter/Space navigates to /voice-eval/{id}.

• Left checkbox for bulk selection.
• Brand accent stripe on the Run column that saturates on hover.
• Run cell. Monospaced short id linked to the detail page.
• Status pill. Queued / Running / Completed / Failed. If run.phase is set, the pill appends "· " for extra detail (e.g. "Running · dialing", "Running · transcribing", "Running · scoring"). Phase is the granular sub-status the worker publishes as the call moves through its stages.
• Progress. Bar plus "current/total" text from run.progress_current / run.progress_total. Reflects completed calls, not seconds.
• Model. Monospaced target-model string, truncated.
• Cost. $X.XXXX, or "-" when zero.
• Duration. Computed from started_at → finished_at (or now if still running), formatted MM:SS or HH:MM:SS.
• Created. Relative time.
• ID. Monospaced, truncated.

No per-row menu, no trash icon on the row. Deletion flows only through the bulk-select checkbox and the toolbar button. This is deliberate: individual voice-run actions (cancel, regenerate transcript, replay audio) live on the detail page.

Status pill

Same shape as the text eval-runs list, four states:

• Queued (muted grey). Waiting for a worker slot.
• Running (brand blue). A call is being placed or scored. The optional · {phase} suffix gives sub-status.
• Completed (pass green). All calls in the run finished and were scored.
• Failed (destructive red). A systemic error prevented completion (Twilio rejected the calls, target unreachable, STT/TTS provider outage).

Bulk delete

Selection is per row; the bulk button opens the standard BulkDeleteSheet (entity "voice run" / "voice runs").

Cascade summary. "the run itself and every call it placed (recording, transcript, scores)".

Queued and running runs can't be deleted. The backend returns 409 on those; per-row failures surface in the error toast and the partial-success message reads ". N deleted successfully.". Successfully-deleted ids drop out of the selection; the list refreshes either way.

Cancel a run first (from the run detail page) if you need to delete it.

New Voice Eval sheet

The "+ New voice eval" button opens a right-side drawer.

• Title. "New voice eval".
• Description. "One real phone call per dataset row. The conversation is recorded, transcribed and scored with the plan's metrics."

Fields, in order

Direction. Two cards. Outbound: our agent calls the bot under test. Inbound: the bot calls your Twilio number and our agent answers in the row's persona. Inbound shows a note explaining that the credential's number is used and that its Voice webhook is pointed at EvaliQA for the run and restored afterwards.

Test plan. Select of plans whose channel === "voice" in a mode the voice runner can play: eval_single_turn (one question per call), eval_multi_turn (a conversation; simulation, adaptive or scripted) or redteam_single_turn (one attack per call, judged by the platform's vulnerability judges). Multi-turn redteam is text-only for now, the wizard disables it on the voice channel. The description below the picker shows the mode and strategy. Empty state: "No voice test plans found. Create a test plan with Voice channel first.".

Dataset. Datasets attached to the chosen plan. Voice datasets always include a phone_number column plus the strategy-specific columns (see below).

Voice credentials. Three separate pickers, filtered by provider category:

• Twilio (twilio provider). Helper: "Twilio account for making phone calls."
• STT (deepgram or whisper). Helper: "Speech-to-text for transcribing the bot's voice."
• TTS (elevenlabs or openai_tts). Helper: "Text-to- speech for our agent's voice."

Each empty state links back with "Add ... in Credentials → Voice tab" so you can wire the missing provider without losing your place.

Column mapping. Heading flips between "Column mapping (scripted)" and "Column mapping (simulation)" based on the plan's strategy. Fields auto-pick columns by conventional name:

• Always required. Phone number column (auto-picks phone_number or phone). This is the number EvaliQA will call.
• Simulation required: Intent (auto-picks intent), Persona (persona), Success criteria (success_criteria). Optional: Max turns (max_turns).
• Scripted required: User turns (user_turns). Optional: Expected outcome (expected_outcome).

Auto-pick means you can create a dataset with the conventional column names and this sheet fills the pickers correctly; if you used different names, override manually.

Max call duration. Slider, 10 to 600 seconds in steps of 10, default 120 s. Cap on how long any single call can run before EvaliQA hangs up. Voice bots that gather info can take several minutes; support-flow bots usually resolve inside 60-90 seconds. Set generously the first time, tune down as you learn your target's response times.

Metrics summary. Read-only chip list of metrics bound to the plan. If none: "No metrics bound, transcripts won't be scored.". Voice metrics come from the same catalog as text metrics, most Agent metrics that read turns also work on voice transcripts (Task Success Rate, Goal Achievement, Role Adherence). See Agent metrics.

Validation

Submit is disabled until every required field is filled:

• Voice test plan picked.
• Dataset picked.
• All three credentials set (Twilio, STT, TTS).
• Outbound only: a phone-number column, or a target phone number dialled for every row (rows with a number in the column win).
• Mode-specific required columns picked: input for single question per call, intent / persona / success_criteria for simulation, user_turns for scripted.

Column mapping by plan mode

• Single question per call (eval_single_turn): input column, optional expected-output column. Outbound: the agent dials, says the input, records the bot's answer and hangs up. Inbound: the agent picks up with "Hello?", lets the bot say its piece, asks the input, records the answer.
• Simulation / adaptive: intent, persona, success criteria, optional max-turns and expected-outcome columns. The agent improvises from the persona; inbound uses a callee prompt (the bot leads, the agent reacts).
• Scripted: user-turns column (lines spoken one by one), optional expected-outcome column.
• Red team attack (redteam_single_turn): attack-prompt and vulnerability columns, optional vulnerability-type, attack-type and expected-refusal columns. The agent says the attack verbatim; the bot's answer is judged against the vulnerability by eval-engine's red-team judge on the plan's LLM credential, exactly as a text red-team row would be.

Parallel calls. How many calls run at the same time, 1 to 5. Outbound: parallel dials. Inbound: that many slots wait for the bot at once. Every call is still its own recording and transcript.

Max wait per call (inbound only). How long each dataset row waits for the bot to dial in before the call is marked failed, 60 to 3600 seconds, default 900.

Barge-in

While our agent speaks, the pipeline keeps listening. The moment the bot talks over it, playback is cut and the agent's line is marked interrupted in the transcript, the way a real caller would stop mid-sentence. Set barge_in: false in the plan's generation config to mute listening during playback instead, which is safer on a handset with acoustic echo.

Scoring and cost

Transcripts are scored by eval-engine, the same service that scores text runs: plan metrics for evaluation plans, the vulnerability judge for red-team plans. Each scored call counts as one evaluated case on the workspace's allowance. Every started minute of every call counts against the plan's voice minutes; when the allowance runs out the run finishes with the calls made so far and says so in its banner.

Each call shows an estimated cost split: telephony and recording (Twilio list price by direction), speech-to-text and text-to-speech (by provider) and the exact LLM spend reported by the gateway. Your own Twilio, STT and TTS accounts are billed by those providers; the figure here is an estimate for comparing runs.

On submit

Calls createVoiceRun (POST /voice/voice-runs), closes the sheet, refreshes the list, and navigates to /voice-eval/{id} where you can watch the call progress live.

Prerequisites

Everything the run needs is surfaced inline in the New Voice Eval sheet. If a prerequisite is missing, the corresponding picker's empty state points you to fix it:

• No voice plan → "Create a test plan with Voice channel first." Go create a text or voice plan under a project.
• No credential in a category → "Add ... in Credentials → Voice tab." Go add Twilio / STT / TTS credentials via /credentials.

Beyond the UI: Twilio needs a phone number attached to your account (subject to your regional regulations). Outbound: the number must be voice-enabled for outgoing calls and the target bot has to accept calls from it. Inbound: the number receives the calls; EvaliQA sets its Voice webhook for the duration of the run (using your Twilio credential, nothing platform-owned) and restores the previous value when the run finishes or is cancelled. The webhook URL is shown on the run detail page in case you want to check or set it by hand.

Every Twilio webhook is signature-checked with your credential's auth token; nothing else can feed audio into a run.

Cancelling a run

A queued or running run has a Cancel run button on its detail page. The current call is hung up, remaining rows are skipped, and the run ends with status Cancelled; inbound runs restore the number's webhook. Runs interrupted by a service restart are closed as Failed with a message saying so, so they never stay stuck in Running.

Freshness

Same behavior as the text eval-runs list:

• Data fetch on mount + after any delete. No polling on this list page.
• Running rows do NOT auto-update here. Progress and phase are snapshot values from the last fetch.
• The run detail page polls every couple of seconds for live updates, that's where you watch a call in progress.

Interpreting the list

• Queued for a long time. No worker picked the run up. If it's more than a few minutes, ask an admin to check the worker queue; voice runs need a specific worker type.
• Running · dialing then Failed. Twilio couldn't place the call. Check the phone number format (E.164, +1234567890) in the dataset.
• Running · transcribing then Failed. STT provider responded with an error. Check the credential is still valid and the audio format is one your STT supports.
• Cost is much higher than expected. Voice runs cost = Twilio minutes + STT seconds + TTS characters + judge tokens + target tokens. Long calls compound fast; tune Max call duration and scenario length.
• Duration >> expected. The call ran to the cap without the simulator recognising a resolution. Look at the transcript, the target might have gone silent or the simulator's success criteria might be over-specific.

Tips and nuances

• Voice modes bypass the text connector layer entirely. The Connector tab in the Playground won't help you debug a voice bot, use a text-mode connector to that same backend service if you need HTTP debugging.
• Phone numbers must be E.164. +14155551234, not 415-555-1234. Import validation is loose; call placement is strict.
• Max call duration is a hard cap, not a soft target. Long calls get abruptly hung up, which the metric may score as "did not resolve". Set the cap deliberately.
• The · {phase} suffix on Running rows is worth monitoring. It's the fastest way to see whether the worker is stuck in one phase (dialing, transcribing, scoring, all can hang).
• Voice datasets multiply row cost. Every row is a whole phone call + STT + TTS + LLM per turn. A 20-row run at 5 turns each with a strong judge can be several dollars, plan accordingly.
• Recordings and transcripts stay stored with the run. Delete the run and both are gone (permanent). Download recordings from the run detail page if you need them for training or audit.
• Auto-pick of dataset columns relies on conventional names (phone_number, intent, persona, etc.). If your dataset came from an import with different names, retype the pickers once and re-save the plan; auto-pick isn't sticky per plan.

Related

• Methodology. Evaluation, multi-turn, most voice plans use the multi-turn simulation / adaptive / scripted split. Personas and escalation styles, the persona catalogue that drives simulated voice callers. Agent metrics, metrics like Task Success Rate and Goal Achievement that fit voice transcripts.
• Adjacent surfaces. Test plans page, where voice plans are created (channel = voice). Datasets page, the voice templates (voice-outbound-simulation, voice-outbound-scripted) live there. Add your first LLM credential, covers Twilio / Deepgram / Whisper / ElevenLabs / OpenAI TTS credential setup.
• Text counterpart. Eval runs page, the text-side eval history.