What's new in EvaliQA, October 28, 2026

Eval runs list shows how many rows errored. The Progress column on the Eval runs list now counts errored rows next to scored rows. Hover over it to see the full count, the same way the dashboard and project pages show it.
Mark whether the judge was right on each Custom Eval score. Under each saved Custom Eval score in run results, answer "Was the judge right?" with Right or Wrong, and add an optional reason. The Judge agreement column on the Custom metrics page shows how often the judge agrees with people, and Calibrate opens the metric so you can fix it.
Custom metric names and thresholds show correctly across the app. When a plan used more than one judge of the same metric type, some screens lost the metric's name and threshold. Runs, compare, trends, leaderboards, reports, online evaluation, tracing panels and alerts now show the right name, and judges from different plans no longer merge on leaderboards.
Calibrate a metric by hand and check it on your labeled rows. On the Calibration tab, Calibrate by hand lets you edit the criteria, strategy, pass threshold and few-shot examples, including rows the judge got wrong (up to 20). Check the draft on every labeled row first, then apply it as the next default version or discard it.
Improve with AI raises judge agreement without moving your threshold. On a metric's Calibration tab, Improve with AI learns from every row you checked, tries up to three rounds of changes and keeps your pass threshold. It shows the before and after agreement, what it changed and the rows still wrong. You can only apply the result when it agrees with people more often.
Metric score card stays in place when a time range has no data. In Online evaluation, picking a time range or scope with no scores now keeps the Metric score card and its controls, with "No scores in this time range.", so you can switch back. Judge agreement on Custom metrics also falls back to the newest labeled version when the default has no labels.
Run results show which metric version judged them. A saved Custom Eval metric now has a version tag next to its name in run results and on its chart in Evaluation Metrics. Older runs keep showing the version they used after the metric gets a new one.