What is EvaliQA

EvaliQA is an evaluation platform for teams building with AI. If you ship a chatbot, a RAG assistant, a voice agent, a copilot, or any product that leans on an LLM to make decisions, EvaliQA is the place you go to check whether it actually works, before release, and while it runs in production.

It replaces the usual mix of ad-hoc scripts, spreadsheets, and "let's ship it and watch Slack" with a single workspace where prompts, datasets, metrics, runs, and live traces all live together and stay comparable over time.

Who this is for

EvaliQA fits three roles most naturally, and one workspace usually has all three sharing it:

• AI / ML engineers: you want fast feedback while iterating on prompts, models, and retrieval. You need to know whether a change is a win, a loss, or noise, and you want that answer in minutes, not days.
• QA and prompt engineers: you own the test bank. You curate the tricky inputs, define what "good" looks like, and want to run the whole battery against every candidate release without babysitting scripts.
• Product managers and stakeholders: you want a plain-English report: did quality go up, where does the agent still trip, what does that mean for users? EvaliQA generates human-readable reports on top of every run.

You do not need to be an ML researcher. If you can describe what a good response looks like in a sentence or two, EvaliQA can turn that into a metric.

The two loops

The whole product is organized around two workflows that run side by side and share the same datasets, metrics, and models:

Offline evaluation answers "is my next release better than the last one?" Live evaluation answers "what is happening right now, and is it still good?" Together they give you a continuous quality signal across the whole lifecycle of the product.

What you can do in EvaliQA

Here is a tour of the main things you'll spend time on:

Design a test bank

Create a project for each AI product you own, then build datasets of representative inputs, you can upload a CSV, paste rows by hand, generate synthetic ones from a description, or capture real traces from production and promote them into a dataset. Datasets are versioned, so you can grow the bank over time without breaking old runs.

Define what "good" means

Attach metrics to a test plan. EvaliQA ships with a library of ready metrics (correctness, faithfulness, tone, toxicity, latency, cost, and many more), and you can write custom metrics in natural language when the built-in ones don't fit. Custom metrics use an LLM as a judge, so you don't write code, you describe what should hold, and EvaliQA turns it into a score.

Run evals and compare

Pick a model (any provider, you plug in your own API keys), pick a dataset, pick metrics, hit Run. EvaliQA executes every row in parallel, scores it, and lands you on a results page where you can:

• see the aggregate score per metric,
• drill into any single row to see the input, the model's response, the judge's reasoning, and the pass/fail verdict,
• compare two runs side by side to see exactly which rows changed and how,
• promote a failing row into your dataset so it never regresses again.

Test conversational and voice agents

Beyond single-turn scoring, EvaliQA can drive multi-turn conversations: a simulated user persona talks to your agent for N turns while EvaliQA scores the whole exchange. It also supports voice evaluation end-to-end: audio in, audio out, transcription, and voice-specific metrics like ASR accuracy and TTS naturalness.

Watch production

Add the EvaliQA SDK to your agent, and every real conversation streams into EvaliQA as a trace. You can filter by user, session, latency, cost, or any custom tag, replay the exact prompt path that led to a bad answer, and turn on online evaluation so every live session gets scored against the same metrics your offline runs use.

Ship on a schedule

Wire EvaliQA into your CI / CD: every pull request or nightly build can kick off a test plan and block the merge if quality drops. Set up alerts so someone gets pinged when a live metric slides below its threshold.

Read the report

Every eval run generates a written report: what improved, what regressed, patterns across failing rows, suggested next fixes. You can share the URL with a PM who never has to look at raw JSON.

What you do not need to bring

• Model API keys are yours to bring. You add them once in Credentials and reuse them across runs. EvaliQA never charges you for tokens on top, you pay your providers directly.
• No custom code is required to get started. Everything through the first eval run can be done in the UI. The SDK is only needed for live tracing.
• No infra work for the basics. The hosted product is ready to use. Self-hosting is available if you need it, see the Administration section.

Where to go next

Pick the path that matches what you want to do today:

• I just signed up and want to run my first eval. Follow the Getting started walkthrough, signup to a green run in about ten minutes.
• I want to understand the vocabulary first (projects, datasets, test plans, runs, metrics). Read the Concepts section, one short page per idea.
• I already know the product and want to design a proper test suite. Jump into Testing for the deeper how-to.
• I want to hook production traffic into EvaliQA. Start with Runtime tracing setup and then Online evaluation.
• I need to plug EvaliQA into CI or a webhook trigger. The Reference section has the API, SDK, and integration catalog.

If you get stuck, every page in these docs has a "Was this helpful?" footer that goes straight to us, we read every note.