How to set eval release thresholds
Separate critical failures, acceptable quality changes, and missing evidence. A useful gate can return pass, fail, or inconclusive without hiding behind an average.
An eval threshold turns measurement into a release decision. It should express what the team is willing to accept for a particular task, rather than inherit an arbitrary score such as 0.8 from a tutorial.
Define three outcomes: pass, fail, and inconclusive. A completed run with unacceptable behavior fails. A run with missing results or inadequate evidence has not established that the candidate is acceptable. Both can block promotion, but they need different fixes.
Separate requirements before choosing numbers
Do not blend every score into one average. A concise answer cannot compensate for changing the wrong account, and a lower token bill cannot compensate for a critical task failure.
| Requirement | Example policy | Why it is separate |
|---|---|---|
| Critical behavior | No observed unauthorized writes in required cases | An aggregate quality gain cannot offset this failure |
| General quality | Meet an absolute floor and remain within an approved regression margin | A weak baseline should not define acceptable quality forever |
| Important task groups | Each required group meets its own rule | Common easy cases can hide a regression in a smaller group |
| Latency and cost | Stay within a separately specified budget | These are operational tradeoffs, not substitutes for correctness |
| Evidence completeness | Every required case has a valid result | Missing scores must not silently reduce the denominator |
Choose requirements with the people responsible for the workflow. Critical cases need direct assertions where possible; subjective quality may need a calibrated judge.
Compare a candidate with a named baseline
Pin the dataset, scorer, application, prompt, and model configuration. Run the accepted version and candidate on the same cases, keeping the evaluation conditions comparable. Record provider changes or other conditions you cannot control.
Review the transitions: cases where both pass, both fail, the candidate improves, and the candidate regresses. In an illustrative 100-case suite, a baseline passes 90 cases. The candidate fixes five failures and breaks three previously passing cases, finishing at 92. The two-point gain does not tell you whether the three regressions are acceptable.
Decide whether your policy requires every critical case to pass, permits known baseline failures, or allows an aggregate regression margin. An accepted baseline with known failures is an explicit product decision, not permission to treat those failures as successes.
Use uncertainty when making a rate claim
A result of 19 passes out of 20 cases is an observed 95% pass rate. It is not strong evidence that the true success rate exceeds 95%. For independent, representative binary observations, a Wilson interval or an exact binomial interval is more informative than the point estimate alone. NIST guidance on intervals for proportions.
Repeated runs of one fixture are not the same as new independent tasks. If cases share a conversation, customer, or template, preserve that grouping in uncertainty estimates. A hand-picked regression suite measures coverage of those examples; it does not become representative production data by adding a confidence interval.
For a baseline comparison, define a practically acceptable margin before seeing the result. Suppose higher scores are better and the allowed decrease is two percentage points. A paired difference interval from minus one to plus three points supports that margin; an interval from minus four to plus three leaves it unresolved. These are illustrative values, not recommended universal thresholds.
Failure to detect a statistically significant regression does not establish that a meaningful regression is absent. If the decision is inconclusive, gather the additional evidence specified in the policy or route it to review.
Account for model and judge variability
Fixed inputs do not make a model-based eval deterministic. Repeat selected cases to estimate variation, especially near the decision boundary. Choose the repetition count and retry rules before evaluating the candidate; repeatedly rerunning until a green result appears biases the decision.
Keep application failures, judge failures, timeouts, and missing traces visible. Retry infrastructure errors under a bounded policy and retain the attempts. If a required result is still unavailable, report the run as inconclusive rather than dropping that case.
Use a holdout when tuning the prompt or scorer. Repeatedly optimizing against the same release suite can improve the measured score without improving behavior on new inputs.
Make the policy enforceable
An illustrative release policy might require all critical cases to pass, valid results for every required case, a minimum quality score per task group, and a paired comparison within the approved margin. Store the policy version with the experiment and record any exception with its owner, reason, and expiry.
The CI command must return a failing status when the policy rejects promotion. A result table or pull-request comment alone does not enforce that decision. On GitHub, configure the relevant status check as required and verify the bypass policy. GitHub branch protection documentation.
Before relying on the gate, submit a deliberately failing candidate and an incomplete run. Confirm that neither can pass through the ordinary release path. Then connect the accepted version to deployment and rollback, and use production incidents to improve the next dataset.
Continue with the evals implementation guide for the full workflow.