Quick eval

Quick eval is the shortest path from "I have a bot" to "I have a score". The guided setup walks you through six pages: a credential, a project, a connector, a test plan, a dataset and a run. Quick eval asks the same questions on one page, /quick-eval, and creates all six things for you. Expect about two minutes from the first keystroke to the first scored row.

The screencast (1:29) shows the whole run, including how to copy the request your chat makes from the browser as a cURL command.

What you fill in

The page has three numbered steps that open one after another, and one button. A step collapses to a one-line summary once you continue past it; click Edit on the summary to reopen it.

1. What are we testing. The fastest way is a cURL command copied from your browser: open DevTools, the Network tab, right-click the request your chat makes and choose Copy as cURL, then paste it under Paste a cURL command instead and press Use this. EvaliQA finds the field that holds your question, replaces it with {{input}}, sends the request once with the very question you captured, and reads which field of the response holds the answer. The status line under the form says what it found, for example: Found your question in message and replaced it with . Tested: HTTP 200 in 210 ms, answer read from reply. When the endpoint answered, the page moves on to step 2 by itself. When the question could not be located, the body template opens so you can put {{input}} where the question goes; when the test failed, the status shows the HTTP code and the reason and the fields stay editable. A short screencast next to the URL field shows the whole thing. You can also enter the URL by hand, add an auth header (stored encrypted, never shown again), and press Test endpoint, which runs the same analysis on what you typed. If a project of yours already has an active connector, pick it instead.

2. Judge key (own credential only). On platform credits this step is not on the page: the platform model is the judge and the description is step 2. When the Platform AI agent runs on your own credential, the judge scores answers with your model. If you already stored a credential, the newest one is preselected together with its default model. Otherwise pick a provider and paste the key; the page asks for exactly the variables that provider needs. EvaliQA never stores its own provider keys.

3. What to test. One or two sentences, at least twenty characters: the scenarios, business processes or failure modes this run should probe. Every generated test case targets what you write here, so "check refunds of used gear — the 14-day window, item condition, and refusing hygiene-sensitive items" produces a far sharper dataset than "support bot for a store". Set how many test cases to generate (1 to 50, ten by default).

The text becomes the plan's description and goals, and is sent to the generator as the focus of the run. It is not a description of your system — that lives on the project, and the two are combined:

## Target system          ← the project: description, domain,
                            scenarios, capabilities, out-of-scope
## Focus of this evaluation ← what you typed here

If the selected project has no description, the page says so and links you to it. Without project context the generator only knows the system's name and type, and the questions come out generic no matter how good your focus is. Filling in the project's description, domain, business scenarios and out-of-scope topics is the single biggest lever on generation quality — and it pays off on every later run, not just this one. Quick eval runs fine without it; it just produces a blunter dataset.

When no project exists yet, quick eval creates one and seeds its description from this text. Enrich it afterwards on the project page.

Basic metrics only

Quick eval starts with four basic metrics, shown as pills you switch on and off: answer relevancy, bias, toxicity and task success. When an existing project is selected, the ones that fit are pre-selected and marked with a small spark. Nothing outside these four is ever added on your behalf: specialised metrics such as faithfulness or tool correctness need dataset columns a generated first run does not have. Add them on the plan page afterwards. At least one pill has to be on.

The card on the right sums up what will happen: the project name (derived from the first sentence when a new one is created), the endpoint, the mode, the metrics, the number of cases and the judge.

What it creates

Pressing Run evaluation creates, in order:

1. a credential from the judge key (skipped when a stored one was picked, or on platform credits),
2. a connector from the endpoint, after one live request to check it answers,
3. a project, unless an existing one was chosen,
4. a test plan in single-turn evaluation mode with the chosen metrics,
5. a dataset generated from the project context plus your focus text,
6. an eval run against that dataset.

The first four happen while you wait; the page shows each step turning green. Generating the dataset and starting the run take a little longer, so the page keeps polling and takes you to the run the moment it starts.

When a step fails

If a step fails, the page shows which one, the message, and for endpoint problems the HTTP status, the URL that was called and the raw body that came back. Fix and retry brings the form back and highlights the block to fix.

A retry never duplicates what already exists. The ids of the entities the first attempt created are sent back with the retry: a credential that was stored is reused rather than stored twice, an existing project is reused, and a connector that already passed its test is reused. A muted note under the form lists what is being reused.

If the run itself fails later, for example because the judge rejected the key while generating cases, the page shows the error with a link to the test plan so you can fix the setup there.

Editing afterwards

Nothing Quick eval creates is special. The connector is on the project page, the plan is under Test plans, the dataset under Datasets, the run under Eval runs. Add test cases, change thresholds, swap the judge or wire the plan into CI from those pages like you would for anything made by hand. The banner on the run page links to the most common next steps.

Prefer to see every option? The guided six-step setup is still on the Get started tab of the dashboard, and the full test plan wizard is one click away from the summary card.