RAG evaluation metrics

The RAG category holds the metrics that score one response: the RAG-specific ones (Faithfulness, the three Contextual metrics) and the general ones you'll use on any single-turn plan, Answer Relevancy and Answer Precision.

Every metric here except Answer Precision uses the plan's judge model. Answer Precision compares text and makes no judge call. On your own key, pin the judge and don't rotate it between runs you want to compare; on platform credits it is kept fixed for you.

Answer Relevancy

What it scores. How directly the response addresses the user's question. 1.0 means "answers exactly what was asked"; 0.0 means "talks about something else".

Columns required. input, actual_output. No expected output needed, this is the metric of choice when you don't have a gold answer.

Default threshold. 0.6.

When to use. Any single-turn plan where the response should be on-topic. Free-form Q&A, chat, RAG assistants. Also useful as a companion to Answer Precision: an answer that scores well on precision but poorly on relevancy is technically correct but wandering off.

When not to use. Multi-turn conversations (the metric reads one input and one output, not a whole turn history). Classification or structured- output tasks (relevancy isn't the right question there, use JSON Schema or Contains instead).

Answer Precision

What it scores. How closely the actual response matches the expected_output as text: exact and normalised match, character and token overlap, and numbers compared with a tolerance. It does not read for meaning and makes no judge call, so it costs no credits. 1.0 is a match; 0.0 is a mismatch.

Columns required. actual_output, expected_output.

Default threshold. 0.8. That is strict for anything longer than a short phrase: a correct answer worded differently from the expected output scores well below it.

When to use. Short factual answers whose wording is predictable: numbers, names, codes, dates, yes or no, fixed phrases.

When not to use. Free-text answers. A correct answer in other words scores low, because the metric compares text, not meaning. To check a free-text answer against a gold answer, use Semantic Similarity (meaning, no judge call) or a judge metric: G-Eval or Custom Eval with a rubric that describes what makes an answer acceptable.

Faithfulness

What it scores. Whether every claim in the response is supported by the retrieval_context. 1.0 means all claims are grounded; anything less means at least one claim was fabricated (or brought in from outside the provided context). This is the RAG-specific hallucination metric.

Columns required. input, actual_output, retrieval context (from your system or the retrieval_context column).

Default threshold. 0.7. Consider raising to 0.9-0.95 for compliance- sensitive verticals (medical, legal, financial) where any ungrounded claim is a bug.

When to use. Every RAG plan. If the product retrieves passages and answers based on them, Faithfulness tells you whether it actually stayed grounded. Pair with Contextual Precision / Recall to separate "retrieval was bad" from "retrieval was fine, generation drifted".

When not to use. Non-RAG plans (no context to check against). Also skip if the answer intentionally goes beyond the retrieved passages (summarisation + commentary), Faithfulness will over-penalise that.

Contextual Relevancy

What it scores. Whether the retrieved context is on-topic for the user's question. Judges the passages themselves: did retrieval bring back material that could plausibly answer the question, or is it noise?

Columns required. input, retrieval context (from your system or the retrieval_context column).

Default threshold. 0.6.

When to use. RAG plans where you suspect retrieval is bringing back irrelevant passages. If Contextual Relevancy is low but Answer Precision is high, the model is answering correctly despite bad retrieval (guessing well). Fix retrieval anyway.

When not to use. Non-RAG plans. Also skip if you're already running Contextual Precision and Contextual Recall, they overlap.

Contextual Precision

What it scores. Precision@k weighted by relevance. Does the retriever rank the most useful chunks first? 1.0 means the top chunks are the most relevant; 0.0 means retrieval order is inverted.

Columns required. actual_output, retrieval context (from your system or the retrieval_context column).

Default threshold. 0.7.

When to use. RAG plans where retrieval order matters (top-k truncation downstream, LLM attention bias toward earlier context). This is the retrieval quality metric for teams working on the retriever itself.

When not to use. Products where you feed the LLM every retrieved chunk without ranking. Skip if retrieval is a black box you don't own, you'll just get noise.

Parameters. Optional top_k caps the number of context chunks evaluated. Useful when you retrieve 20 but only feed the top 5 to the LLM: cap at 5 so the metric reflects what the model actually saw.

Contextual Recall

What it scores. Proportion of claims in the expected_output that are supported by the retrieved context. 1.0 means retrieval brought back everything needed to answer; anything less means retrieval missed critical info.

Columns required. input, expected_output, retrieval context (from your system or the retrieval_context column).

Default threshold. 0.7.

When to use. RAG plans where you have a gold answer and care about whether retrieval could have supported it. If Contextual Recall is high but Answer Precision is low, retrieval did its job and generation dropped the ball. If Contextual Recall is low, no amount of prompt work will fix the answer, fix retrieval first.

When not to use. RAG plans without an expected_output (the metric needs it to know what "everything needed" is). Use Faithfulness or Contextual Relevancy instead.

How to pick a starting set

For a first evaluation plan on a new project:

• Non-RAG single-turn Q&A: Answer Relevancy plus a correctness metric: Answer Precision for short factual answers, Semantic Similarity or G-Eval for free text.
• RAG assistant: Faithfulness + Contextual Recall, plus the same correctness metric. Splits generation quality from retrieval quality; catches the two most common RAG bugs (hallucination against context, incomplete retrieval).
• Compliance-sensitive product: add a G-Eval rubric for your compliance policy on top of the RAG baseline.

Add more once your first-round numbers show a specific gap. Don't attach all six at once, the wizard will let you, but the pass-rate signal becomes noise.

Tips and pitfalls

• Faithfulness only sees the retrieval context. It takes the passages from your system's response through the connector's Retrieval context path, or from the dataset's retrieval_context column when the response has none. If your pipeline passes context some other way (inside the system prompt, through a tool), map it with the path or copy the passages into the column, or Faithfulness scores as if there was no grounding.
• Contextual Precision needs ranked context. If your dataset stores retrieval as an unordered set, the metric can't tell "top" from "bottom" and produces noise. Fix the dataset to preserve order.
• Answer Precision penalises paraphrasing. It compares text, so a correct answer in other words can fail. Keep it for short factual answers; use Semantic Similarity or G-Eval for free text.

Related

• Metrics hub
• Agent metrics
• Security and Safety metrics
• Custom metrics
• Evaluation, single-turn