How we evaluate AI systems, and what the metrics actually measure.
3 posts in Expertise tagged Online evaluation. Clear filters
Deterministic rules, semantic similarity and LLM judges each answer a different question about an AI system. Here is how to stack them so the cheap checks carry the weight and the judge does only what nothing else can.
Read the postA production AI failure you fixed but never wrote an AI test for comes back with the next prompt change or AI update. This is how to turn the trace behind it into a golden-set row, with an expected output and a metric that catch the failureof AI, that CI checks on every pull request.
Read the postA coding agent that edits a test to make it pass still produces an ordinary pull request. This post shows how to trace every Claude Code or Codex session into EvaliQA, score it as it arrives with detectors, metrics and the AI Analyst, and promote the bad sessions into your test dataset.
Read the post
