DocsReference

Integrations reference

The calls that start an EvaliQA run from CI and read its result: token header, requests, responses, errors and a release gate, plus how production traces reach EvaliQA.

Last updated 2026-10-14

EvaliQA has two public integrations: the CI integration, which starts an evaluation run and reads its result, and trace ingest for production traffic. Everything else happens in the app.

Start a run from CI

Create a token on Integrations, API tokens. A token belongs to the workspace and runs as the person who created it. Its value is shown only once, so put it in your CI secrets as EVALIX_TOKEN right away.

The quickest start is Integrations, CI/CD tools: pick GitHub Actions, GitLab CI/CD, Jenkins, CircleCI, Bitbucket Pipelines, Azure Pipelines or plain curl, choose the plan, dataset and connector, and copy the step it generates. The snippet already holds the full URL and the three ids. It starts the run, waits for it, and fails the job when the run fails or its pass rate is below the minimum set in the snippet (EVALIX_MIN_PASS_RATE, 0.8 by default).

Request

POST https://<your EvaliQA host>/api/test-plan/triggers/eval-run
X-Evalix-Token: <token>
Content-Type: application/json

{
  "test_plan_id": "…",
  "dataset_id": "…",
  "connector_id": "…"
}
FieldRequiredMeaning
test_plan_idYesThe plan to run.
dataset_idYesThe dataset to run it on.
connector_idYesThe AI system under test.
column_bindingNo{"type": "auto"} by default: dataset columns are matched by name, the exact slot name first, then aliases such as question, prompt or query for the input and answer, reference or ground_truth for the expected output. Send an explicit binding object to override the match.
nameNoRun name, up to 120 characters.
concurrencyNoRows processed at once, 1 to 20, default 1. At most 5 for red-team plans.

Response

202 Accepted with the queued run:

{
  "triggered_by_token_id": "…",
  "id": "…",
  "status": "queued",
  "url": "https://<your EvaliQA host>/eval-runs/…",
  "eval_run": { "id": "…", "status": "queued", … }
}

Read id, status and url. eval_run is kept for scripts written against it; its shape may change.

The call returns as soon as the run is queued. To gate a release on the result, read the run's status until it is final (below).

Errors

StatusWhen
401The header is missing, or the token is invalid or revoked.
400The run cannot start as configured: the dataset is not generated yet, the plan has no metrics or no judge, it is a voice plan, a red-team limit is exceeded (200 rows, concurrency 5), or BINDING_UNRESOLVED when a required column cannot be matched (the response lists the available columns).
402A plan limit is reached, red teaming is not on your plan, or credits ran out.
403The token's owner has no access to the plan's project.
404The plan, dataset or connector does not exist in this workspace.
409A run is already in progress on this dataset; the response holds its id.
422The request body is malformed.
5xxTemporary. Retry the call.

Run on a schedule

EvaliQA has no scheduler of its own. Call the same endpoint from the one you already use: a GitHub Actions schedule trigger, GitLab pipeline schedules, Jenkins, a Kubernetes CronJob.

Read a run's result

The same token reads the status and result of a run it can see, so a pipeline can wait for a run and fail the build on it. Only runs of the token's workspace; a user sign-in token is not accepted here.

Status request

GET https://<your EvaliQA host>/api/test-plan/triggers/eval-runs/<id>
X-Evalix-Token: <token>

Status response

200 OK:

{
  "id": "…",
  "status": "completed",
  "phase": null,
  "progress": { "current": 6, "total": 6 },
  "started_at": "2026-10-15T10:00:00Z",
  "finished_at": "2026-10-15T10:02:00Z",
  "pass_rate": 0.8333,
  "passed_rows": 5,
  "failed_rows": 1,
  "errored_rows": 0,
  "metrics": [
    { "name": "Answer Relevancy", "pass_rate": 0.8333, "mean_score": 0.86 }
  ],
  "url": "https://<your EvaliQA host>/eval-runs/…"
}
FieldMeaning
statusOne of the status values below.
phaseWhat a running run is doing, or null.
progressRows done and rows in total.
started_at, finished_atWhen the run started and finished, or null.
pass_ratepassed / (passed + failed) rows, 0 to 1, as on the run page. Errored rows are out of it and counted in errored_rows. Null until a row is scored.
passed_rows, failed_rows, errored_rowsRows that passed every metric, rows that failed one, and rows that got no verdict (a connector failure, an HTTP error from your system, a runner or metric error).
metricsEach metric's name, pass rate and mean score over the scored rows.
urlThe run's page in EvaliQA.

The response holds no row data, prompts, outputs or judge details.

Status values

queued, running, completed, failed and cancelled. The last three are final: stop reading once you see one.

Polling

Read every 10 seconds or so. Each token can read 60 times a minute; above that the call answers 429 with a Retry-After header.

Status errors

StatusWhen
401The header is missing, or the token is invalid or revoked.
404No run with this id in the token's workspace.
422The id is not a UUID.
429Too many reads for this token in a minute.
5xxTemporary. Retry the call.

Gate a release

Fail the job when the run fails or its pass rate is below your minimum. The snippets on Integrations, CI/CD tools do exactly this; in plain sh:

STATUS_URL="https://<your EvaliQA host>/api/test-plan/triggers/eval-runs/$RUN_ID"
while :; do
  RUN=$(curl -fsS "$STATUS_URL" -H "X-Evalix-Token: $EVALIX_TOKEN")
  STATUS=$(printf '%s' "$RUN" | grep -o '"status":"[a-z_]*"' | head -n 1 | cut -d '"' -f 4)
  case "$STATUS" in completed|failed|cancelled) break ;; esac
  sleep 10
done
PASS_RATE=$(printf '%s' "$RUN" | grep -o '"pass_rate":[^,}]*' | head -n 1 | cut -d ':' -f 2)
[ "$STATUS" = "completed" ] || exit 1
[ "$PASS_RATE" != "null" ] || exit 1
awk -v r="$PASS_RATE" -v t=0.8 'BEGIN { exit !(r >= t) }'

Send production traces

Production traffic reaches EvaliQA as traces, sent with the SDK or over OpenTelemetry (OTLP), with a tracing API key you create on the project's Tracing tab. Online evaluation then scores them with the metrics you choose.