Evaluating AI Agents
A practical guide to grading an agent's path, not just its answer: tools and MCP, the trajectory, multi-turn conversation, trace analysis, and safety when the agent can act.
Get the guideTesting covers the inputs you chose. This covers the ones your users chose.
Your agent reports what it did. EvaliQA stores it, groups it into sessions, scores it against the same metric catalogue your test runs use, and tells somebody when a rule breaks.
39Built-in metricsBound to any trace, the same catalogue as your tests
10Problem typesThe AI Analyst names reading a whole session
6AI frameworksOpenAI, Anthropic, LangChain, LlamaIndex, AutoGen, CrewAI
2Ways inPython SDK or any OTLP exporter
What is online evaluation for LLM applications?
Online evaluation is scoring the traffic your AI application actually receives in production, as it arrives, rather than a test set you chose in advance. Each trace is stored, grouped into a session, scored against metrics and watched by alert rules, so a regression in the wild is found by a score rather than by a user complaint.
- Traces inYour app posts one trace per invocation, over OTLP or the Python SDK.
- SessionsTraces sharing a session id become one conversation, the unit scoring works on.
- ScoringThree layers on a session: deterministic detectors, the metrics you bound, and an AI Analyst.
- AlertA broken rule fires to Slack, Discord, Teams, Telegram, email or a webhook.
- RegressionA bad session is pulled back into a dataset, so the next test set covers it.
Runs on live traffic, and feeds a bad session back to your tests.
Two doors in, and they land in the same place
If your stack already emits OpenTelemetry, point an OTLP/HTTP exporter at EvaliQA and you are done. If it does not, the Python SDK wraps your agent's entry point, records each step as a span and posts the finished trace as JSON. Pick by what your app already has, not by what it might need later. Traces from either door are stored, grouped into sessions, and attributed to users identically.
- One trace per invocation with the spans underneath it, so you can read the path and not just the answer.
- Sessions traces sharing a session id become one conversation, which is the unit scoring works on.
- Users attribute a session to the person who had it, and find every session they had.
- Nothing runs in your app scoring happens on our side, on your workspace's key.
Three layers of scoring, from free to thorough
Not every session deserves a strong model reading it end to end, and not every problem needs one to be found. So scoring is layered, and you choose how deep it goes per project.
- Deterministic detectors sub-second checks that need no model at all, run on everything.
- Bound metrics the metrics you attached, scored per trace and per conversation once the dialogue goes quiet.
- The AI Analyst a strong model reading the whole session and writing findings, when you want it.
Switch on scoring for live traffic
Under a minute: turn online evaluation on for a project, and production traffic starts getting scored with the same metrics the release was gated on.
Everything around the session, included
Trace explorer
Every trace, its spans, its inputs and outputs, filtered by project, session, user or time, with the scores attached.
Alert rules
One rule, one project, one thing to watch, fanned out to Slack, Discord, Teams, Telegram, email or any webhook.
Feed the datasets
A real session that went wrong is the best test row you will ever write. Pull it back into a dataset and it becomes a regression test.
Retention you control
Traces expire on your plan's window, 14 days on Free and 180 on Team. Scores and datasets are never deleted.
Guides that go deeper
The method behind the product, written down in full. Free to read and keep.
Evaluating RAG Systems
A practical guide to measuring retrieval and generation separately, building datasets you can trust, and turning one-off checks into a regression process.
Get the guideFrequently asked questions
What is the difference between offline and online evaluation?
Offline evaluation runs a test plan against a dataset you wrote before release. Online evaluation scores the conversations your users actually have after it. EvaliQA keeps the two on separate rails, so a noisy day in production never moves a release-gate number, and they meet only at the metric catalogue, so a score means the same thing whether it came from a test run or from live traffic.
What is LLM observability, and how is it different from evaluation?
Observability shows what your AI application did per request: the trace and its spans, the inputs and outputs, latency, tokens and cost, grouped into sessions. Evaluation adds a score on top, so you can tell whether what happened was actually good. A logging tool answers what happened; online evaluation answers whether it was right, and tells you when it stops being.
How do you evaluate an AI agent in production?
You score the traces it produces as they arrive, instead of a test set you chose in advance. Your agent posts one trace per request, EvaliQA groups the traces into sessions and scores each session against the metrics you bound, and a rule tells someone when a score drops. Nothing runs on your eval path: the scoring happens on our side, on your workspace's key.
How do you send traces to EvaliQA?
Two ways. If your stack already emits OpenTelemetry, point an OTLP/HTTP exporter at EvaliQA and you are done, GenAI span conventions included. If it does not, the Python SDK wraps your agent's entry point, records each step as a span and posts the finished trace as one HTTP request. No daemon, no long-lived connection, and nothing runs on the request path.
How do you catch hallucinations and quality regressions in production?
Bind the metrics that measure them, faithfulness and groundedness for a hallucination, relevancy and your own judges for quality, and they score live traffic exactly as they score a test run. Deterministic detectors flag errors and retry loops in under a second, and the AI Analyst reads a whole session for reasoning and tool-use problems. An alert fires when a pass rate slips, not only when latency spikes.
How do alerts on live traffic work?
A rule watches one thing on one project: a metric's pass rate, its mean score, or findings above a severity. When a freshly scored session crosses the line, the alert fires once and reaches the channels you picked, Slack, Discord, Teams, Telegram, email or any webhook. Because it watches evaluation signals, it catches a quality drop, not just a 500 or a slow response.
Can a bad production session become a test case?
Yes, and it is the point. A real session that went wrong is the best test row you will ever write, so you pull it back into a dataset and it becomes a regression test on the offline rail. The next release is scored against the failure you already saw in the wild, so the same problem cannot ship to users twice.
How long are traces kept?
Traces expire on your plan's retention window, 14 days on Free and 180 days on Team. The scores computed from them, and any dataset rows you pulled from a session, are never deleted, so your history and your regression tests outlive the raw traces they came from.
Watch the traffic you did not write the test for
Send one trace, see it scored, and set the rule that would have told you.