RAG and general LLM-judge metrics

The RAG category in the wizard is misnamed a bit. It does contain the RAG-specific metrics (Faithfulness, the three Contextual metrics), but it also houses the general-purpose response-quality metrics you'll use on any single-turn plan, RAG or not: Answer Relevancy, Answer Precision, Bias, Toxicity. Treat this category as "everything that scores one response".

Every metric here (except Restricted Refusal) uses the plan's judge model. Pin the judge and don't rotate it between runs you want to compare.

Answer Relevancy

What it scores. How directly the response addresses the user's question. 1.0 means "answers exactly what was asked"; 0.0 means "talks about something else".

Columns required. input, actual_output. No expected output needed, this is the metric of choice when you don't have a gold answer.

Default threshold. 0.6.

When to use. Any single-turn plan where the response should be on-topic. Free-form Q&A, chat, RAG assistants. Also useful as a companion to Answer Precision: an answer that scores well on precision but poorly on relevancy is technically correct but wandering off.

When not to use. Multi-turn conversations (the metric reads one input and one output, not a whole turn history). Classification or structured- output tasks (relevancy isn't the right question there, use JSON Schema or Contains instead).

Answer Precision

What it scores. How closely the actual response matches the expected_output. Multi-component similarity: numeric agreement with tolerance, semantic overlap, factual consistency. 1.0 is a strong match; 0.0 is a mismatch.

Columns required. actual_output, expected_output.

Default threshold. 0.8. Higher than most metrics on purpose: this is the "correctness" metric and you want it strict.

When to use. The primary metric on any supervised dataset where you know what the right answer looks like. First choice for Q&A, factual lookups, structured-answer tasks. Combine with Answer Relevancy for a two-metric quality baseline.

When not to use. Open-ended creative tasks where "right" is a range of acceptable answers. Reach for G-Eval or Custom Eval with a rubric instead: describe what makes an answer acceptable and let the judge score it.

Faithfulness

What it scores. Whether every claim in the response is supported by the retrieval_context. 1.0 means all claims are grounded; anything less means at least one claim was fabricated (or brought in from outside the provided context). This is the RAG-specific hallucination metric.

Columns required. input, actual_output, retrieval_context.

Default threshold. 0.7. Consider raising to 0.9-0.95 for compliance- sensitive verticals (medical, legal, financial) where any ungrounded claim is a bug.

When to use. Every RAG plan. If the product retrieves passages and answers based on them, Faithfulness tells you whether it actually stayed grounded. Pair with Contextual Precision / Recall to separate "retrieval was bad" from "retrieval was fine, generation drifted".

When not to use. Non-RAG plans (no context to check against). Also skip if the answer intentionally goes beyond the retrieved passages (summarisation + commentary), Faithfulness will over-penalise that.

Contextual Relevancy

What it scores. Whether the retrieved context is on-topic for the user's question. Judges the passages themselves: did retrieval bring back material that could plausibly answer the question, or is it noise?

Columns required. input, retrieval_context.

Default threshold. 0.6.

When to use. RAG plans where you suspect retrieval is bringing back irrelevant passages. If Contextual Relevancy is low but Answer Precision is high, the model is answering correctly despite bad retrieval (guessing well). Fix retrieval anyway.

When not to use. Non-RAG plans. Also skip if you're already running Contextual Precision and Contextual Recall, they overlap.

Contextual Precision

What it scores. Precision@k weighted by relevance. Does the retriever rank the most useful chunks first? 1.0 means the top chunks are the most relevant; 0.0 means retrieval order is inverted.

Columns required. actual_output, retrieval_context.

Default threshold. 0.7.

When to use. RAG plans where retrieval order matters (top-k truncation downstream, LLM attention bias toward earlier context). This is the retrieval quality metric for teams working on the retriever itself.

When not to use. Products where you feed the LLM every retrieved chunk without ranking. Skip if retrieval is a black box you don't own, you'll just get noise.

Parameters. Optional top_k caps the number of context chunks evaluated. Useful when you retrieve 20 but only feed the top 5 to the LLM: cap at 5 so the metric reflects what the model actually saw.

Contextual Recall

What it scores. Proportion of claims in the expected_output that are supported by the retrieved context. 1.0 means retrieval brought back everything needed to answer; anything less means retrieval missed critical info.

Columns required. input, expected_output, retrieval_context.

Default threshold. 0.7.

When to use. RAG plans where you have a gold answer and care about whether retrieval could have supported it. If Contextual Recall is high but Answer Precision is low, retrieval did its job and generation dropped the ball. If Contextual Recall is low, no amount of prompt work will fix the answer, fix retrieval first.

When not to use. RAG plans without an expected_output (the metric needs it to know what "everything needed" is). Use Faithfulness or Contextual Relevancy instead.

Bias

What it scores. Presence of biased framing in the response: gender, race, political, religious, socio-economic, etc. 1.0 = no bias detected; 0.0 = strongly biased.

Columns required. input, actual_output.

Default threshold. 0.8.

When to use. Any customer-facing product, especially in regulated verticals (hiring, credit, healthcare) or products serving a broad demographic. Add it to a happy-path plan as a low-cost guardrail even if you also run a full red-team pass.

When not to use. Skip in narrow-scope products (internal tools, technical assistants) where user-facing bias isn't a real risk vector. Not a substitute for a proper red-team plan on high-stakes products.

Toxicity

What it scores. Presence of toxic, hateful, or insulting language in the response. 1.0 = clean; 0.0 = strongly toxic.

Columns required. input, actual_output.

Default threshold. 0.7.

When to use. Any product that interacts with real users, always. Especially valuable on rows where the user input was itself toxic or provocative: does the model take the bait?

When not to use. Nowhere. It's cheap enough to keep on by default.

Restricted Refusal

What it scores. Whether the model correctly refuses out-of-scope requests. Heuristic-based, no LLM judge required, so it's the one fast/cheap metric in this category.

Columns required. input, actual_output.

Default threshold. 0.5.

When to use. Guardrail evaluation of out-of-scope requests (forbidden-topics datasets). Pair with a refusal-quality custom rubric if you also want to check how well the model refused (did it point the user to the right resource, did it stay polite).

When not to use. Doesn't work as a general quality metric, only scores refusal patterns. On rows that should not be refused, this metric doesn't add signal.

How to pick a starting set

For a first evaluation plan on a new project:

• Non-RAG single-turn Q&A: Answer Precision + Answer Relevancy + Toxicity. Three metrics, covers correctness / on-topic / safety.
• RAG assistant: Answer Precision + Faithfulness + Contextual Recall
  • Toxicity. Splits generation quality from retrieval quality; catches the two most common RAG bugs (hallucination against context, incomplete retrieval).
• Compliance-sensitive product: add Bias and a G-Eval rubric for your compliance policy on top of the RAG baseline.

Add more once your first-round numbers show a specific gap. Don't attach all nine at once, the wizard will let you, but the pass-rate signal becomes noise.

Tips and pitfalls

• Faithfulness only sees the retrieval_context column. If your retrieval pipeline sends context via some other mechanism (embedded in the system prompt, streamed via a tool), Faithfulness sees nothing and scores as if there was no grounding. Make sure the column is populated.
• Contextual Precision needs ranked context. If your dataset stores retrieval as an unordered set, the metric can't tell "top" from "bottom" and produces noise. Fix the dataset to preserve order.
• Answer Precision penalises paraphrasing. For open-ended answers, it can score a semantically-equivalent answer lower than a verbatim match. Reach for G-Eval when the "right answer" is a range.
• Toxicity thresholds are strict on purpose. A 0.7 threshold means even light rudeness fails. Lower it (say to 0.5) only if you want to allow spicy tone; keep strict for support / medical / children's products.

Related

• Metrics hub
• Agent metrics
• Security metrics
• Custom metrics
• Evaluation, single-turn