Metrics catalog

Built-in metrics ship with EvaliQA via the eval-ai-library kernel. They're categorised by what they measure. Every metric can be attached to a plan directly or wrapped in a custom preset with your own prompt and threshold.

Correctness

Metric	What it does
answer_correctness	LLM-judge scores how correct the answer is vs. the expected output.
exact_match	Deterministic: actual == expected.
contains	Deterministic: expected in actual.
bleu / rouge	Classic NLP overlap metrics.
semantic_similarity	Cosine similarity of embeddings.

Relevance and faithfulness

Metric	What it does
answer_relevancy	LLM-judge scores how on-topic the response is vs. the input.
faithfulness	LLM-judge scores whether the answer is supported by the retrieval context.
hallucination	LLM-judge scores presence of unsupported claims. Lower is better.
contextual_precision	Judge scores relevance of retrieved chunks.
contextual_recall	Judge scores coverage of the expected answer by the retrieval.

Safety and compliance

Metric	What it does
toxicity	LLM-judge scores toxic content. Lower is better.
bias	LLM-judge scores demographic bias.
contains_pii	Deterministic PII detection (regex + entity extraction).
pii_leakage	LLM-judge assesses whether the response leaks PII from the input.
prompt_injection_resistance	Judge scores whether the agent held its instructions under attack.

Redteam

Metric	What it does
redteam_attack_success	Was the adversarial goal achieved? Binary.
redteam_refusal_quality	Did the agent refuse cleanly, or refuse-then-comply?

Performance

Metric	What it does
latency_ms	Wall-clock time per row.
total_tokens	Prompt + completion tokens.
token_cost_usd	Estimated USD cost per row.

Voice-specific

Metric	What it does
voice_response_time	Time from caller stopping to agent starting to speak.
voice_transcript_correctness	LLM-judge on the STT transcript.
voice_intent_match	Did the agent fulfil the caller's intent?

Related

• Metrics concept
• Custom metrics