Evaluation pipeline

Datasets, test plans, metrics, and runs are the pieces. This page is about how they fit together into a working evaluation practice, something that runs on a cadence, blocks a release when it should, and feeds itself with new coverage from production. That's the evaluation pipeline.

A pipeline isn't one thing you build once. It's a maturity ladder you climb one rung at a time. Most teams start with manual runs, add a CI-blocking gate, then add nightly heartbeats, then bring production traces into the loop. Each rung adds one more feedback signal and takes days, not weeks, to wire up.

This page is the methodology: what each rung looks like, how to decide when to climb the next one, and how to wire EvaliQA to the tools you already have.

The maturity ladder

Five rungs, roughly. Skip ahead if the earlier ones don't apply, but almost every team benefits from having them all in place eventually.

Rung 1, Manual runs on demand

The starting point. You have a project, a credential, a test plan, a dataset. When you change something, you open the UI and click Run eval.

Signal you get. Point-in-time pass rate whenever someone remembers to check.

When it's enough. During the first week of a project, before you know what "good" looks like. Also for one-off experiments.

When it's not. As soon as more than one person is iterating on the product. Manual runs get skipped under pressure exactly when they matter most.

How to climb. Save your first test plan against a small golden set. Aim for a plan that runs in under a minute so you can put it in CI next.

Rung 2, PR-level gate (CI-blocked)

Every pull request kicks off a run against a small golden set. Merge is blocked if the pass rate drops.

Signal you get. Whether this change moves quality up, down, or sideways. Blocks obvious regressions before they hit main.

When it's enough. Small team, single product, weekly release cadence. This rung alone catches most quality regressions cheaply.

When it's not. As soon as you have production traffic worth watching or a red-team surface worth testing.

How to wire it. See How to wire CI below.

Rung 3, Nightly heartbeat

A scheduled run against the golden set (and typically a larger living dataset) fires every night. You look at it in the morning.

Signal you get. Drift you couldn't have caused: provider model updates, retrieval index changes, external data quality issues.

When it's enough. Mid-size product, weekly / monthly release, active development.

When it's not. As soon as your product is customer-facing and you need security signal on a shorter cadence than "monthly".

How to wire it. Same trigger endpoint as CI, called on a cron schedule from your infra (or from EvaliQA's built-in scheduled runs, if enabled in your workspace).

Rung 4, Production traces feeding the pipeline

Add the SDK to your production agent. Traces stream into EvaliQA. Two things become possible:

• Online evaluation. A sample of live traces gets scored automatically against the same metrics you use offline. You see live quality in near-real-time.
• Trace promotion. Interesting production conversations get promoted into the golden set weekly.

Signal you get. The gap between what the golden set tests and what production actually does. This gap is where every uncaught regression lives.

When it's enough. Fully-instrumented product with active production traffic and a team that reviews traces at least weekly.

When it's not. Never, from here. Production trace ingestion is a permanent capability.

How to wire it. See Runtime tracing SDK setup and Online evaluation.

Rung 5, Full pipeline with red-team + scheduled deep sweeps

The full shape. Every rung above plus:

• Weekly red-team pass on a redteam_single_turn plan.
• Monthly red-team pass on a redteam_multi_turn plan.
• Quarterly deep evaluation on a large living dataset, with a formal report shared with stakeholders.

Signal you get. Continuous quality signal + continuous safety signal + strategic direction for what to work on next.

When it's enough. For most commercial products. Beyond this, you're tuning the pipeline you have.

How to wire it. Combination of scheduled runs (rung 3 approach) and calendar-driven human reviews.

The full cadence, once you're on rung 5

Cadence	What runs	Where
Per PR	eval_single_turn on the golden set, 20-50 rows	CI
Nightly	eval_single_turn on the full golden set (100-300 rows) + eval_multi_turn on a smoke set	Scheduled
Weekly	redteam_single_turn on the standard vulnerability set	Scheduled
Monthly	redteam_multi_turn (full pass) + eval_multi_turn (full sweep)	Scheduled
Quarterly	Deep review: full dataset, full metric coverage, human read-through, quarterly report	Human-triggered
On model swap	Every plan, on the new model. Compare against last baseline.	Human-triggered
On production incident	The scenario that broke gets a new golden-set row (a permanent regression alarm)	Human-triggered

The numbers scale down for smaller projects and up for high-stakes ones, but the shape is the same.

The data flow

The most important integration isn't with CI, it's the loop between production traces and the golden set. This is what keeps evaluation relevant.

              production traffic
                     |
                     v
              SDK traces to EvaliQA
                     |
        +------------+------------+
        v                         v
   online evaluation      trace review (weekly)
   (auto-score sample)          |
        |                       v
        v              promote interesting rows
   alerts on               to golden set
   metric drops                 |
                                v
                         golden set grows
                                |
                                v
                      nightly / PR runs
                      catch regressions
                                |
                                v
                        product improves

Without the promotion arrow, the golden set stagnates: you keep scoring against the same 60 rows while production surfaces new failure modes you never test. Without the online evaluation arrow, you miss regressions between nightly runs.

How to wire CI

Every EvaliQA workspace can trigger runs via an API endpoint. The integration shape:

1. Create an API token in /integrations with the workspace scope that includes runs:create.
2. Store the token in your CI system's secrets store (EVALIQA_TOKEN).
3. On PR open / push call the trigger endpoint from your CI workflow. See the Integrations reference for the exact payload shape and per-CI/CD-tool examples. The payload accepts "column_binding": {"type": "auto"}: the server resolves dataset columns by name (exact slot name first, then aliases such as question / prompt / query for input and answer / reference / ground_truth for expected output), so the snippet works unchanged across plan modes. Pass an explicit binding object instead when you need to override the match; an unresolvable required column fails the call with a BINDING_UNRESOLVED error that lists the available columns.
4. Poll or webhook for the run's completion. The trigger returns a run id; the CI job either polls until the run is completed or waits for a webhook callback with the pass rate.
5. Gate the merge on the pass-rate threshold.

The endpoint proxies to the eval engine via a service JWT internally, so from CI's perspective you're calling one endpoint and getting a run id back. Fast enough to sit inside a normal PR pipeline.

Which plan to trigger from CI. The one that runs in under a minute against the golden set. Full multi-turn or red-team plans belong on the nightly / weekly cron, not per-PR.

How to wire scheduled runs

Same trigger endpoint, called from a cron. Two ways:

• External cron. Your existing scheduler (GitHub Actions on cron, GitLab schedules, Jenkins, k8s cronjob, whatever) calls the trigger endpoint on the schedule. Full control, no lock-in.
• EvaliQA scheduled runs. If your workspace has this enabled, configure the schedule from the test plan's page. EvaliQA fires the run internally and stores the run history under the plan.

Both approaches surface identical run rows on the plan's Eval runs tab; the compare view works across scheduled and manual runs without distinction.

How to wire production tracing

The other big integration surface. Adding the SDK to your agent takes an hour to a day depending on the framework.

1. Install eval-ai-library in your agent's Python environment (or the equivalent SDK for other languages, listed in the SDK setup guide).
2. Get a project SDK key from the project's Tracing tab.
3. Wire the env vars the SDK expects.
4. Wrap your agent's entry point with the tracing helper.

Once traces are flowing, decide two policies:

• Sample rate. Sample 100% while you're validating; drop to 1-10% once you're confident in coverage vs cost.
• Online evaluation. Which metrics run automatically on the sampled traces. Start with a cheap subset (Toxicity, PII Leakage, Restricted Refusal) and add more once you're happy with the alert quality.

See Runtime tracing setup and Online evaluation.

Environments: dev, staging, prod

Where does each stage of the pipeline run against?

• PR / CI runs: usually against the staging connector (the agent as it will be deployed if this PR merges).
• Nightly / weekly runs: against the production connector, so drift from provider changes shows up.
• Red-team runs: against staging when you're iterating on defenses, against production for monthly release qualification.
• Trace ingestion: production only. Never trust trace data from dev environments as regression signal.

Each stage typically has its own EvaliQA project (or one project with clearly-tagged plans) so the KPIs stay separate.

Ownership: who does what

A pipeline needs owners or it decays.

• Test plan and metrics: usually owned by the engineer / team building the feature under test. They know what "good" should look like.
• Golden set: owned collectively but with a clear editor (often the QA / prompt engineer). Additions and prunings go through review.
• CI wiring: owned by whoever owns your CI in general.
• Scheduled runs and alerts: usually owned by whoever handles on-call for the AI product.
• Reports: written by the person who ran the eval, reviewed by the PM before sharing widely.
• Trace review: rotates among the team, weekly is a healthy cadence.

The failure mode is "everyone can add rows, no one prunes". Assign a single golden-set editor.

What "success" looks like at each rung

Sanity checks that you've actually reached the rung, not just built the plumbing.

• Rung 1. Anyone on the team can run the plan in under 2 minutes and read the pass rate.
• Rung 2. Merging a PR that breaks the golden set is blocked. Everyone has hit this at least once.
• Rung 3. Someone reads the nightly result within 24 hours, every day.
• Rung 4. Interesting production traces get promoted into the golden set weekly. Golden set grows by 3-15 rows a month organically.
• Rung 5. The quarterly report goes to leadership without needing follow-up questions. Product roadmap references concrete failure patterns from the pipeline.

If a rung's plumbing exists but the sanity check fails, you're running the pipeline without benefiting from it. Fix the human process before adding more automation.

Tips and pitfalls

• Wire one rung fully before adding the next. A half-wired CI gate that gets ignored is worse than no gate at all.
• Don't gate on aggregate pass rate too tightly. A 3 pp drop can be judge noise (see judge variance). Gate on a threshold that leaves noise room, and alert on larger drops.
• Keep CI runs small. A 30-row PR gate that finishes in 40 seconds is used every time. A 300-row PR gate that takes 6 minutes gets bypassed on urgent PRs.
• The golden set is the pipeline's spine. If it's stale, everything downstream is stale. Prioritise the weekly promotion review.
• Alerts on production metric drops need a human to check them. Automate signal, not decisions. Every alert should land in someone's inbox with the run link.
• Costs add up. Rung 5 with GPT-4-class judge and daily + weekly + monthly cadences can be hundreds of dollars a month on a mid-size product. Model this into your budget from the start.
• Reports close the loop. A pipeline that produces runs but no reports doesn't produce decisions. Make the quarterly report part of the cadence, not an aspiration.

Related

• Datasets, methodology hub
• Golden dataset: the set your release gate points at.
• Test plans hub: the recipes each stage runs.
• Metrics hub: the signals the pipeline surfaces.
• Evaluation results: reading the runs the pipeline produces.
• Preparing a report: turning results into communicable decisions.
• Runtime tracing SDK setup: wiring rung 4.
• Online evaluation: the auto-scoring half of rung 4.
• Integrations reference: CI / CD trigger endpoints and per-tool examples.