Evaluation, single-turn

eval_single_turn is the workhorse mode. One dataset row = one input = one call to your system = one scored response. No conversation state, no follow-up messages, no attacker persona, just "you gave input X, here's what you produced, was it good?".

Every project should start here. If your system can't score well on a solid single-turn plan, everything else is noise.

When to use it

• You want release-gate signal for CI. Single-turn plans run in seconds to minutes and give you a defensible pass rate to block merges on.
• You're comparing models or prompts. The clean input → output loop makes side-by-side comparisons meaningful, every difference is attributable to the change you made.
• The product is stateless per interaction. Search assistants, summarizers, classifiers, single-question RAG, all naturally single-turn.
• You're debugging a specific failure. Reproduce the failing input as a one-row dataset, iterate on the prompt, run, repeat.

When not to use it. If the whole point of your product is multi-turn conversation (a support chatbot that gathers info over N turns, a coach that adapts to the user), single-turn evaluation misses the interaction quality. Score it too, but don't stop there.

The shape of a row

Minimum row for eval_single_turn:

Column	Purpose
input	What the user says / asks.
expected_output	The gold answer, for correctness-style metrics.
context	(RAG) The passages the model should ground on.
system_prompt	(Optional) Override the connector's default.
metadata / tag / category	Freeform, for slicing later.

Beyond that, add whatever columns your metrics need. Faithfulness needs retrieval_context; Answer Precision and Contextual Recall need expected_output; Toxicity and PII Leakage need neither.

Which metrics fit

Best fits for eval_single_turn, straight from EvaliQA's catalog:

• Answer Precision: LLM judge scores the actual output against expected_output. The closest thing to a general-purpose "correctness" metric and the one to start with for supervised datasets.
• Answer Relevancy: is the answer actually about the question? Cheap to add on top of Answer Precision; needs no expected_output.
• Faithfulness (RAG), is every claim in the answer supported by the retrieval_context? The RAG-specific grounding metric, this is how you catch hallucinations against the passages the model was given.
• Contextual Relevancy / Contextual Precision / Contextual Recall (RAG), did retrieval bring back the right passages and in the right order?
• Bias and Toxicity: flag biased framing or hateful / insulting content in the answer.
• Restricted Refusal: heuristic check (no LLM judge) that the model correctly declines out-of-scope requests.
• Security-side metrics: PII Leakage, Harmful Content, and the two Resistance metrics (Prompt Injection, Jailbreak) work on single-turn rows too and are worth mixing in even in an eval plan.
• Deterministic checks: Regex Match, JSON Schema, Length Check, Contains: cheap, LLM-free assertions for structured outputs.
• Latency / cost: always-on KPIs on the run page, not metrics you attach.

If your product has open-ended creative outputs, mix in a G-Eval or Custom Eval with a rubric, LLM judges do well on "does this answer follow the tone / structure I want?" style rubrics, and both metrics can be attached multiple times if you want several rubrics on the same plan.

Dataset composition

For a single-turn plan targeting general quality, aim for a mix along the lines of:

• ~50 % Happy path
• ~20 % Edge cases
• ~10 % Metamorphic
• ~10 % Forbidden topics
• ~10 % Inappropriate usage

See Evaluation dataset types for the reasoning behind those ratios. If you only care about one axis (e.g. correctness against a knowledge base for a RAG system), a 100 %-happy-path dataset is fine, just don't confuse it for a robustness signal.

Sizing

• CI smoke test: 20–50 rows. Should complete in under a minute.
• Release qualification: 100–300 rows across a balanced mix.
• Exploration / coverage: 500–2000 rows, mostly generated + promoted from production traces.

Cost per row scales linearly. A 100-row run with 3 LLM metrics on GPT-4o-mini as target + GPT-4o as judge is under $1; the same run on GPT-4o as target + GPT-4 as judge is ~$5–10.

What "good" looks like

• Answer Precision pass rate ≥ 90 % on happy-path rows for a release-ready system. Under 80 % means functional gaps.
• Faithfulness ≥ 95 % for RAG systems. Hallucinated grounding is a bug, not a fuzziness knob.
• Latency p95 within your SLA. The tile shows average, check distribution in the row-level view.
• Pass rate ~= across categories. A 20 pp gap between two tag categories points at a specific class of bug.

Failure patterns and what they mean

• Answer Precision ok, Answer Relevancy bad. The model is technically right but talking around the question. Usually a prompt problem.
• Answer Precision bad, Contextual Precision / Recall ok. RAG retrieved the right passage but the model still fabricated. Try a stronger model or a more explicit prompt.
• Answer Precision ok, Contextual Precision / Recall bad. The model is guessing correctly. Sooner or later it'll be wrong on a row where guessing doesn't land, fix retrieval anyway.
• Cost creeping up run to run. Prompt or context bloat. Check avg_input_tokens on the per-row token usage.

Common configuration

• Wizard Step 2, Mode: pick "Evaluation, single-turn". No sub- strategy to pick.
• Wizard Step 3, LLM Judge: pick a strong judge and keep it fixed across runs (see hub page).
• Wizard Step 4, Metrics: start with Answer Precision plus one other (Answer Relevancy is the natural companion; Faithfulness for RAG). Add more only when the first ones aren't giving you enough signal.
• Wizard Step 5, Params: thresholds arrive filled from the catalog (Answer Precision default 0.8, Answer Relevancy 0.6, Faithfulness 0.7). Tune them based on what falls out of your first run, a metric that passes 100 % of rows probably has its threshold set too low for you.
• Wizard Step 6, Docs: optional. Uploaded PDFs / Markdown seed dataset generation. Only single-turn plans use this.
• Wizard Step 7, Dataset: Generate new for a first pass; Attach existing to reuse the golden set from another plan.
• Wizard Step 8, Generate: Row count 20–100 for the first pass, temperature 0.3, one dataset type (usually Happy path).
• Wizard Step 9, Review → Save & generate.

Tips and pitfalls

• Don't skip expected_output even if the metric is unsupervised. Future you (or a different metric) will thank you.
• Pin the target model too. A "single-turn eval" that used gpt-4o-mini yesterday and gpt-4o today is not comparable.
• Watch for retrieved context leaking the answer. If the RAG passage literally contains the gold answer verbatim, correctness becomes trivial and you're measuring nothing.
• Judge disagreements aren't always bugs. If two runs give slightly different scores on the same row, that's judge variance (temperature ≥ 0 on the judge). Average over dataset size, don't chase per-row wobble.
• One dataset per type. If you want correctness AND refusal-rate numbers, split into two datasets. Mixing gives you a meaningless average.

Related

• Test plans: hub
• Evaluation dataset types
• Metamorphic testing
• Metrics
• Run your first evaluation