Preparing a report

An eval run is a spreadsheet full of scores. A report is a paragraph someone else can read. Both are useful; they serve different audiences. The row-level detail is for the person iterating on the product; the report is for the person deciding whether to ship.

EvaliQA can generate a first-draft report from any completed run. That's a starting point, not the finished artefact. This page is the methodology of getting from the auto-draft to something worth sending.

When to generate a report

Not every run needs one. Reasonable rules of thumb:

• Every release qualification run: generate a report and share it as part of the release notes.
• Every model swap or prompt change that ships: report the delta vs the previous baseline. Even a paragraph is enough.
• Every red-team run: brief report of the breach summary, regardless of whether anything broke.
• Every failed run (systemic failure): brief report of what broke and what's known so far, so the on-call person doesn't have to piece it together from the raw traces.
• Every quarterly review: aggregate several runs into one report for stakeholders who don't watch continuous quality.

Skip:

• Iteration runs during a coding session. Read the numbers, don't write about them.
• Runs on datasets that shifted. Report a run only if the results are meaningful.

How to generate one

From /eval-runs/{id}:

• If no report exists yet, the header shows a Generate report button. Click it. Takes 15-60 seconds depending on the run size and the report model. When it finishes, you land on /reports/{id}.
• If one report already exists, the header shows Open report linking to the newest one, and New report to draft a fresh one (useful when you've changed the underlying analysis or want a different tone).
• Report generation is disabled while the run is queued or running. Wait for completion.

The report is editable once generated. Export options: PDF, DOCX, HTML, Markdown.

What the AI-generated draft covers

Roughly, out of the box, the auto-report includes:

• Executive summary. What was tested, on what, what happened.
• Headline numbers. Rows scored, pass rate, cost, latency.
• Per-metric breakdown. Score distribution and pass rate for each metric, with a sentence of interpretation.
• Notable failure patterns. Categories and personas that underperformed.
• Red-team summary (for red-team runs). Per-vulnerability breach counts.
• Suggested next steps. Where to iterate next.

The tone is neutral and factual, useful for internal readers and for putting in front of a PM who wants an honest picture.

What you should always add or edit before sharing

The auto-report is a competent first draft. It doesn't know your product, your customers, or your priorities. What to add:

1. The "so what" line at the top

The generated summary says what happened. Add one sentence saying what it means for the reader's decision:

• "We're clear to ship the new prompt to production this week."
• "Ship blocked, PII leakage on 3 rows must be patched first."
• "Quality is fine but latency degradation is unacceptable, no ship this week."

This is the sentence a PM actually needs. Nothing else in the report matters if this sentence is missing.

2. Business context

Numbers without stakes are hard to prioritise. Add:

• What product feature this run tested (link to the ticket / PR / RFC driving the change).
• What the previous baseline was (from the prior run's report).
• What audiences / segments are affected.

3. What was tested, what wasn't

The auto-report says what was in the dataset. Add what wasn't:

• "This run covered English single-turn support scenarios. Voice and multi-turn are covered by [other plan]."
• "Adversarial coverage is limited to a smoke test; the full red-team pass runs next Wednesday."

Otherwise the reader assumes everything was tested and forms wrong confidence.

4. Recommended action

Auto-generated "suggested next steps" tend to be generic. Replace them with specifics:

• "Add 20 rows for the [X] scenario, currently underrepresented."
• "Raise Toxicity threshold to 0.85 before next release."
• "Rerun on Wednesday against the updated prompt (see PR #421)."

5. What to strip

The auto-report can be too long. Trim:

• Repeated numbers (the same pass rate cited in three sections).
• Filler ("This run demonstrates a comprehensive evaluation of...").
• Long lists of every metric when only two matter for this audience.

A report that runs to two pages sends the message "read this carefully". A report that runs to five pages sends the message "skim this".

Reports for different audiences

The same run needs different reports for different readers.

For engineers on the team:

• Full per-metric breakdown.
• Row-level examples of the top 3 failure patterns.
• Direct link to the run for anyone who wants to drill in.
• Skip the "so what" sentence, they already know the stakes.

For the product manager:

• One-paragraph executive summary.
• The "so what" sentence, front and center.
• Ship / hold recommendation.
• Delta vs previous release.
• Skip the per-row details.

For non-technical stakeholders (leadership, sales, customer success):

• The "so what" sentence.
• Business context (what feature, which customers, which risks).
• One or two illustrative examples of what "good" and "bad" look like in plain English.
• No per-metric jargon. Say "answer quality", not "Answer Precision".

For compliance / audit:

• Formal record: date, plan, dataset version, judge model, model under test.
• Full metric coverage including security metrics.
• Failure examples with row identifiers for traceability.
• Signed off by owner (add a signature line at the bottom).

The auto-report gives you the raw material. Which audience is reading it determines what stays.

Reports as documentation

A report is not just a one-time communication. It's the durable record of what the product looked like on a specific date. Store them accordingly:

• Version reports by release. "v2.1 release qualification" beats "eval run 42".
• Link the report from the release notes. Anyone auditing the release later can trace back to the evidence.
• Keep old reports. They're the baseline you compare next quarter's report against. Deleting them is throwing away history.

Common report mistakes

• Sharing the auto-draft as-is. It's a first draft. Without the "so what" line and business context, readers form their own interpretation, and it's often wrong.
• Padding the report to look thorough. A 5-page report reads less carefully than a 1-page one. Cut anything the audience won't act on.
• Reporting without a comparison. A number alone is meaningless. "78% pass rate" tells you nothing; "78% pass rate, up from 71% last release" tells you everything.
• Reporting on a run with known dataset drift. If the dataset changed between this run and the baseline, say so explicitly. Otherwise the reader assumes it's a fair comparison.
• Reporting a red-team run without a breach summary. The vulnerabilities that were tested and the breach counts are the entire point. Bury them in the middle of the report and no one will find them.
• Auto-generating without reading. The AI can hallucinate a summary that sounds right but doesn't match the numbers. Read the draft against the actual run before you share.

Tips

• Regenerate the report after you've iterated on the analysis. You'll see things in the second reading that the auto-report missed.
• Keep a template for common report shapes. Release qualification, red-team monthly, quarterly review. Fill in the numbers each time.
• Timestamp the report and the run. Both matter. Auditors care when the run happened; readers care when the report was written.
• Reports don't replace runs; runs don't replace reports. One is the evidence, the other is the interpretation. Both are the record.

Related

• Evaluation results: the hub.
• Finding bottlenecks: what feeds the report's "notable failure patterns" section.
• Comparing runs: the delta that gives numbers meaning.
• Reports page: UI reference for generating, editing, and exporting reports.