Connectors

A connector tells EvaliQA how to call your AI system: where to send the request, what to put in it, and where the answer sits in the response. Every eval run sends each dataset row through a connector and scores what comes back.

Connectors belong to a project. Open the project and pick the Connectors tab. Quick eval creates one for you from a pasted cURL command; this page covers everything you can set by hand.

When to use

• Your AI system has an HTTP endpoint: a chat API, a RAG service, an agent, a document processor, or a model behind your own gateway.
• The answer arrives later: the endpoint starts a job and you fetch the result from a status URL.
• The endpoint takes files: a form upload with one or more documents.

When not to use

• Voice bots on a phone number. They are tested by calling them, see Voice eval.
• Watching live production traffic. Send traces instead, see Runtime tracing.

How it works

Setting up a connector takes two sheets:

1. New connector: how to call your system (URL, method, headers, body). Save & configure output stores it as an inactive draft.
2. Configure output: run a test, then pick where the answer and the other fields sit in the response. Activate makes the connector usable.

Only active connectors are offered when you start an eval run or use the Playground. On the list, click a row to open Configure output; the pencil icon (Edit request) reopens the request sheet. Creating, editing and deleting connectors takes the Admin role; members can run tests.

Create a connector

Click New connector. Fill in Name (unique in the project) and an optional Description, then describe the request by hand or import it.

Import from cURL, HAR or Postman

Import from… (next to Basics, on a new connector only) fills the request fields from a request you already have. Nothing is sent while importing; the file is read in your browser.

• cURL. In your browser's DevTools, Network tab, right-click the request and choose Copy as cURL (bash). Paste and the preview shows method, URL, headers and body.
• HAR file. Save all as HAR with content (Chrome) or Save All As HAR (Firefox), then drop the file in and pick the request from the list.
• Postman. Export the collection as Collection v2.1, paste or drop the JSON, and pick the request. Bearer, Basic and API key (header) auth become headers.

Headers that look like credentials (Authorization, Cookie, anything with token, api key, secret or auth in the name) are marked Secret automatically. Warnings appear above the form, for example when a browser cookie was imported. Apply to form copies the request in. Then replace the captured question in the body with {{input}}, otherwise every row sends the same captured text.

Request setup

• Method. Most chat APIs use POST.
• Base URL. The full URL. Variables work here too.
• Headers. One row per header. Tick Secret for keys and tokens: the value is stored encrypted and never shown again. Editing later, the field reads stored - type to replace; leave it empty to keep the stored value.
• Query params. Added to the URL. Values can use variables.
• Request body. JSON or raw (a body template) or Form with files (see below).
• Answer arrives. In the response for ordinary APIs, Job, then polling for asynchronous ones.
• Advanced. Timeout (seconds), 60 by default, up to 600. Max retries, 1 by default, up to 10; retries happen only during eval runs, never on a test.

Variables

{{input}} is replaced with the row's input: the question in single-turn runs, each user message in multi-turn runs. The default body is:

{
  "input": "{{input}}"
}

Any other {{name}} (letters, digits and underscores, no spaces) is a custom variable. When you start an eval run, the New eval run sheet lists them under Custom variables and asks which dataset column fills each one. Variables work in the URL, headers, query params and body.

• Keep the quotes. Write "message": "{{input}}". Text is escaped for you, so quotes and line breaks in a question cannot break the JSON.
• Typed values. In the test panels each extra variable has a type switch: Text, № (number) or { } (JSON). Numbers and JSON go in without quotes, so "tags": "{{tags}}" with JSON ["a","b"] sends a real array. During runs, list, number and boolean columns are sent the same way.
• Smart quotes typed by macOS or Word are turned into straight quotes.

Files (form uploads)

Pick Form with files when the endpoint expects multipart/form-data, the usual way document-processing APIs take files.

• Add file field: the field name the endpoint expects (for example invoices) and the dataset column that holds the documents. Each document in the row is sent as one part under that name.
• Add text field: a field name and a value, rendered like the body template (Reconcile {{input}}).

The documents come from a file column in the dataset. A cURL import with -F field=@file creates the file fields for you and reminds you to add the matching columns. HAR and Postman imports skip form fields.

Asynchronous APIs (job, then polling)

Some systems answer the first request with a job id and a status, and put the result on a separate status URL. Pick Job, then polling and set:

Field	What to enter
Status URL	Where to read the job. {{response.<field>}} takes a value from the first response, for example https://your-api.com/jobs/{{response.id}}. The other variables work too.
Status field	Where the job state is in the status response, for example status. Leave it empty and the job counts as done as soon as the actual output path finds a value.
Done values	Statuses that mean ready. Default done, completed, succeeded.
Pending values	Statuses that mean still working. Default running, pending, queued. Anything unrecognised counts as pending.
Failed values	Statuses that mean the job failed. Default failed, error, cancelled. The row is recorded as failed.
Error field	Where the failure message is, for example error.
Every (ms)	Wait between status requests. Default 2000.
Give up after (s)	How long one job may take. Default 120, up to 600.
Status method	GET or POST. Almost always GET.

Status requests carry the same headers as the first request. The output paths read the finished status response, not the first one, so point them at fields there (for example result.report_markdown). If the Status URL has no {{response.…}} reference, the form warns you: every job would be polled at the same address.

Test before you activate

There are two tests.

• Test connection, at the bottom of the request sheet. Sends one request with the current form values, before anything is saved. Fill in a sample value for each placeholder. You get OK or FAILED, the HTTP status and latency, and Rendered request (URL, headers, body after substitution, secrets masked). A polling connector is followed to the end. For a Form with files body, the text fields are sent as a form; file fields are skipped and listed in the result, since they are filled from a dataset column during a run. The request is capped at 60 seconds and polling at 120 seconds.
• Run test, in Configure output. Sends a request through the saved connector and shows the response so you can pick paths. The result is saved as the connector's Last test (Untested, OK · 230ms or Failed · HTTP 401 in the list).

Without a fresh test, the panel shows the last successful response and says so. After a failed test it is labelled as not what the endpoint returns now.

Reading the response

Configure output has the test on the left and the mapping on the right. Click the copy icon next to any value in the response tree and its path fills the field you focused last.

Response format

How the body is read before paths are applied:

• Auto (default): JSON, falling back to plain text.
• JSON: fail the test if the body is not JSON.
• SSE (streaming): reassembles data: events into content and events.
• NDJSON: one JSON object per line, as a list.
• Key / value text: key: value or key=value lines, as an object.
• YAML.
• Plain text: the body as is, read with a re: path.

When the body looks like a different format, the panel offers it (This response looks like SSE (streaming), Use SSE (streaming)). It never switches on its own. Show raw body shows exactly what came over the wire.

Paths

Path	Reads
answer	a top-level field
choices[0].message.content	nested fields and list items
items[-1]	the last item
sources[*].text	one field from every item, as a list
steps[?(@.type=='tool_call')]	items matching a filter (==, !=, <, >, <=, >=)
$.answer	same as answer, the $. is optional
re:Answer:\s*(.+)	a regular expression on the body as text: the first group, or the whole match. re:/…/i adds flags (i, m, s)

The fields

• Actual output path (required). The answer that metrics score. Under it, Suggested from this response offers likely paths; click one to use it. A path that finds nothing shows No match.
• Eval-library standard paths (optional, folded):
  • Retrieval context path: the passages your RAG system retrieved, as a list of text. If your system does not return passages, put them in a retrieval_context column of the dataset instead; the response wins when both exist.
  • Tools called path: the names of the tools your agent called, as a list.
  • Token usage path: an object with prompt_tokens/input_tokens, completion_tokens/output_tokens and total_tokens. Empty: a top-level usage object is read if there is one.
  • System prompt path: the system prompt, if your system echoes it back.
  • Model path: the model name, used for cost. Empty: a top-level model or model_used field is read.
  • Trace path: pipeline steps, stored and shown step by step in run results.
• Custom output fields. Any value you want a custom metric to read. Give it a Name (letters, digits and underscores, not starting with a digit) and a JSONPath; a Custom Eval criterion reads it as {{name}}. Up to 50.

Which metrics need which path

Metric	Needs from the connector
Most metrics (Answer Relevancy, Bias, Toxicity, G-Eval, Custom Eval, safety metrics)	Actual output path
Faithfulness, Contextual Relevancy, Contextual Precision, Contextual Recall	Retrieval context path, or a retrieval_context column in the dataset
Tool Correctness	Tools called path, plus an expected_tools column in the dataset
Tools Error	Tools called path

Without the path (or, for the RAG metrics, a retrieval_context column), those metrics have nothing to score. The full list is in the metrics catalog.

Example: a RAG response

{
  "answer": "You can return unused items within 30 days.",
  "sources": [
    { "id": "kb-112", "text": "Unused items can be returned within 30 days of delivery." },
    { "id": "kb-087", "text": "Refunds go back to the original payment method." }
  ],
  "usage": { "prompt_tokens": 812, "completion_tokens": 41, "total_tokens": 853 },
  "model": "gpt-4o-mini"
}

Field	Path
Actual output path	answer
Retrieval context path	sources[*].text
Token usage path	leave empty (top-level usage is read)
Model path	leave empty (top-level model is read)

Use sources[*].text, not sources: whole objects would reach the judge as JSON with ids in them.

Example: an agent with tool calls

{
  "output": { "message": "Your order 4471 ships tomorrow." },
  "steps": [
    { "type": "tool_call", "tool": { "name": "lookup_order", "args": { "order_id": "4471" } } },
    { "type": "tool_call", "tool": { "name": "get_shipping_eta", "args": { "order_id": "4471" } } },
    { "type": "message", "text": "Checking the order." }
  ],
  "meta": { "model_used": "claude-sonnet-4-5", "confidence": 0.82 }
}

Field	Path	Result
Actual output path	output.message	Your order 4471 ships tomorrow.
Tools called path	steps[?(@.type=='tool_call')].tool.name	["lookup_order", "get_shipping_eta"]
Model path	meta.model_used	claude-sonnet-4-5
Custom output field confidence	meta.confidence	0.82

Tool Correctness compares names exactly, so the dataset's expected_tools for this row would be lookup_order and get_shipping_eta.

Activate

Activate is enabled once the latest test passed and Actual output path is set. Save draft keeps the mapping without activating. Saving the request sheet again (Save changes) makes the connector inactive until you run a new successful test and activate it again, so the last test always matches the request runs will send.

Voice bots

A voice agent reached by phone does not need a connector. Voice runs call the number and score the conversation, see Voice eval. If the same agent also has a text API, a connector to that API is the quicker way to test its answers.

Common errors

• Request timed out after 60s. Raise Timeout (seconds) under Advanced (up to 600). Test connection stops at 60 seconds regardless. If the system works in the background, use Job, then polling.
• FAILED · HTTP 401 or HTTP 403. Your system refused the credentials. Check the header name and value; for a rotated key, type the new value over the stored secret. Browser cookies from an import expire quickly; use an API token.
• Other HTTP 4xx, often 422. The request is not what the endpoint expects. Open Rendered request and compare it with a call that works.
• Body template is not valid JSON (Content-Type says application/json). Fix the template. Check for a missing comma or quote, and that {{input}} is in quotes.
• Test is OK but the answer shows No match. The path does not fit the response. Click the value in the tree or a suggestion, and check Response format.
• Response is not valid JSON, or a streamed reply arrives as one block of text. Pick the matching Response format (SSE, NDJSON, Key / value, YAML), or use a re: path on plain text.
• … resolves to 10.0.0.5, a private or internal address; outbound requests to those are not allowed on this deployment (or is an internal name). The hosted service only calls public internet addresses. localhost, host.docker.internal and LAN addresses are refused. Put the endpoint on a public HTTPS address with authentication, or run EvaliQA in your own network, see Self-hosted EvaliQA.
• Poll URL references {{response.id}}, which the first response does not contain. Use the field name from the first response; the test shows that response.
• Job did not finish within 120s (last status: running). Raise Give up after (s), or check that your done values match the status your system reports.
• Activate is greyed out. Run a test that passes and set Actual output path.

Tips and pitfalls

• Replace the captured question with {{input}} after an import. Otherwise every row sends the same text and the scores mean nothing.
• Mark every credential Secret. Non-secret header values are shown in the form to everyone who can open it.
• Multi-turn runs send one message per request. Each user turn goes out as {{input}} on its own; earlier turns are not added. Your system has to keep the conversation, for example with a conversation id in a custom variable bound to a dataset column.
• Tools called must resolve to names, not whole objects, or Tool Correctness never matches.
• Testing a file connector sends no documents. Text fields are sent; documents are attached only during eval runs, from the dataset's file columns.
• Point polling paths at the finished response. The first response usually has result: null.
• Retries cost requests. Max retries applies to every failed row in a run; keep it low for slow or paid endpoints.

Related

• Quick eval, a connector, plan, dataset and run from one cURL command.
• Create your first project, where connectors live.
• Playground, send ad-hoc requests through an active connector.
• Custom metrics, read custom output fields in your own criteria.
• RAG metrics and Agent metrics, what the retrieval context and tools called paths feed.
• Metrics catalog, every metric and the fields it needs.