How production monitoring works
The loop from a live trace to a scored session to an alert, and which page owns each step.
Last updated 2026-09-07
Testing tells you how your agent behaves on inputs you chose. Production monitoring tells you how it behaves on inputs real users chose. EvaliQA keeps the two on separate rails, so a noisy day in production never moves a release-gate number, and editing a test plan never changes what you watch live. This page is the map; every other page in this section is one stop on it.
Two roads, one metric library
A test run starts from a dataset and calls your app. Online evaluation starts from your app and calls nothing: it reads what your app reports. The metrics in the middle are the same catalogue, so a number on the online-evaluation Overview tab means the same thing as the number on an eval run.
- Slack, Discord, Teams, Telegram, email, webhook
- Session and trace pages with findings
- Promote a bad trace into a dataset
The loop, in five steps
- Connect. Your agent reports traces. Two doors: the Python SDK
(
eval-ai-library) or any OTLP/HTTP exporter. One API key, bound to one project, works for both. Runtime tracing. - Read. Traces appear on Runtime eval within seconds, grouped into sessions and users, with a gantt timeline per session and a span tree per trace. The project's Observability tab aggregates volume, errors, cost, and latency. Explore sessions and traces.
- Score. Turn on Runtime auto-eval for the project and bind the metrics that matter. Each trace is scored as it arrives; each session is analysed 60 seconds after its last trace: deterministic detectors, conversation metrics, and optionally the AI Analyst. Online evaluation.
- Alert. Rules watch pass rates, mean scores, and finding severities per session and post to your channels, throttled so a bad deploy is one message, not forty. Alerts.
- Promote. The trace behind the alert becomes a dataset row with Add trace to dataset, and the next test run covers it. That is the only place the two roads meet, and it's on purpose.
What happens to one trace
Time from the moment your app finishes an invocation:
| When | What |
|---|---|
| Immediately | The SDK POSTs the trace (one request, retried on 5xx and timeouts). An OTLP exporter ships its spans in batches; EvaliQA assembles them into one trace when the root span arrives, or after 20 quiet seconds if it never does. |
| Within seconds | The trace is on the Traces tab. With a session_id it's also on Sessions, and with a user_id on Users. |
| Immediately, if auto-eval is on | Every bound single-turn metric whose scope matches scores this trace. Verdicts show on the trace page under Trace metrics. |
| 60 s after the session's last trace | Detectors, conversation metrics, and the AI Analyst run over the whole session. Findings show on the session page. Alert rules are checked. |
| Same moment | Matching rules post to their channels and the event lands in Alerts → History. |
Two things follow from the table. A trace without a session_id is
visible but never analysed, so set it. And a long conversation is scored
once, at the end, not once per turn, because every new trace pushes the
60-second timer forward.
Which page owns what
| You want to | Go to |
|---|---|
| Create or revoke an API key, switch auto-eval on for a project | Project → Tracing tab |
| See volume, error rate, cost, latency, top tools over time | Project → Observability tab |
| Read one session's timeline and findings | /runtime-eval/sessions |
| Read one trace's span tree, metric scores, judge reasoning | /runtime-eval/traces |
| Find the heaviest or most failing users | /runtime-eval/users |
| Bind metrics, set thresholds, watch pass rate drift | /online-evaluation |
| Set up channels and rules, audit what fired | /alerts |
| Turn a live trace into a test case | Trace page → Add trace to dataset |
What it costs
Storing traces counts against your plan's runtime-trace allowance. Scoring is model calls on your own Platform AI agent, billed by your provider: one judge call per trace per bound metric, one call per session for the AI Analyst, none for the detectors. Online scoring also spends evaluated cases from the same pool as test runs. The Online evaluation page has the levers.
Related
- Evaluation pipeline, the testing road in the same depth.
- Datasets, why promoted traces are the best rows you'll ever add.