DocsProduction

Verify and troubleshoot

Smoke-test both doors and diagnose the usual failure modes.

Last updated 2026-09-07

Smoke test: SDK

Fire a synthetic trace and confirm it lands:

Export the four env vars in your shell (or paste them into a scratch .env).

Run:

python -c "from eval_lib.tracing import tracer; tracer.start_trace('smoke-test'); tracer.set_trace_metadata(model='smoke', input='ping', output='pong', session_id='smoke-session', user_id='smoke-user'); tracer.end_trace()"

Open Runtime eval → Traces. The trace appears within a couple of seconds. Because the test set a session_id and a user_id, it also shows on Sessions and Users.

Smoke test: OpenTelemetry

No SDK needed, one curl. The body is a single root span in OTLP/JSON:

curl -sS -X POST "https://<your-evaliqa-host>/api/traces/otlp/v1/traces" \
  -H "x-api-key: evx_..." \
  -H "Content-Type: application/json" \
  -d '{
    "resourceSpans": [{
      "resource": {"attributes": [{"key": "service.name", "value": {"stringValue": "smoke"}}]},
      "scopeSpans": [{
        "spans": [{
          "traceId": "5b8efff798038103d269b633813fc60c",
          "spanId": "eee19b7ec3c1b174",
          "name": "smoke-test",
          "kind": 1,
          "startTimeUnixNano": "1700000000000000000",
          "endTimeUnixNano": "1700000001000000000",
          "attributes": [
            {"key": "gen_ai.operation.name", "value": {"stringValue": "chat"}},
            {"key": "gen_ai.request.model", "value": {"stringValue": "smoke"}},
            {"key": "gen_ai.usage.input_tokens", "value": {"intValue": "3"}},
            {"key": "gen_ai.usage.output_tokens", "value": {"intValue": "2"}}
          ]
        }]
      }]
    }]
  }'

A 2xx with an empty partialSuccess means the batch was accepted. The span has no parent, so it's the root and the trace is stored at once: open Traces and it's there with one llm_call span and source: otlp in its metadata. Change the ids before you send it again; a repeated traceId is skipped as a duplicate.

Common failure modes

Nothing arrives

TRACING_ENABLED is unset or false. The default is off.
The URL isn't reachable from the agent's host: a VPN, a firewall rule, or an internal DNS name that doesn't resolve from production.
A corporate proxy strips the Authorization header. Try the OTLP door with x-api-key; proxies rarely touch custom headers.

401 Invalid API key

The key was revoked, or you pasted the prefix shown in the key list instead of the raw value. Only the raw evx_… value works, and it's shown once. Create a new one at Project → Tracing → New API key.

On the OTLP door, a 401 whose message names both x-api-key and Authorization means no credential reached EvaliQA at all. Check that OTEL_EXPORTER_OTLP_HEADERS is set where the exporter actually runs, and that the value has no quotes around it.

402 Payment required

The workspace is over its plan's runtime-trace allowance. Traces are refused, not queued. See Billing.

503 auth service unreachable

EvaliQA's auth service is restarting or unhealthy. The SDK retries with backoff (twice by default, TRACING_MAX_RETRIES) and then drops the trace, logging the failure, or raising it if TRACING_STRICT is on. OTel exporters retry on their own schedule; EvaliQA sends Retry-After: 5.

Traces land but Sessions is empty

The traces have no session_id. Sessions only exist for traces that carry one; the Traces tab shows everything. Set it in set_trace_metadata, or as a gen_ai.conversation.id / session.id span attribute. See Sessions and users.

Spans nest wrong

Use tracer.trace(...) as a context manager (with tracer.trace(...) as span:), not a bare call. Bare calls don't push onto the span stack, so nested spans end up as siblings of their parent.

OTLP traces show up 20 seconds late

The root span never arrived, so EvaliQA waited out the quiet window before storing what it had. Usually the root lives in another service that isn't exporting to EvaliQA, or head sampling dropped it. Export from the service that owns the request, or make sure the sampler keeps the root whenever it keeps the children.

OTLP spans are all `custom`

Your instrumentation isn't setting any attribute EvaliQA classifies on. Add gen_ai.operation.name (chat, execute_tool, invoke_agent) or switch to an instrumentation library that follows the GenAI semantic conventions. The classification order is on the OpenTelemetry page.

Cost is missing or tagged `estimated`

estimated is normal for OTLP: exporters ship tokens, not dollars, and EvaliQA priced them from the model catalogue. Missing cost means the model name didn't resolve; for Bedrock, Azure, Vertex and similar hosts set gen_ai.system so the right price table is used. On the SDK, pass cost_usd= and cost_source="reported" when your provider gives you a number.

400 unsupported Content-Encoding

The OTLP door accepts gzip, deflate, and identity. Brotli and zstd are refused; set OTEL_EXPORTER_OTLP_COMPRESSION=gzip (or none).

The SDK sends but EvaliQA answers 4xx on every span

TRACING_STREAM=true is on. EvaliQA's ingest takes whole traces only; per-span partial_span payloads fail validation and the SDK doesn't retry 4xx. Turn streaming off.