How to turn production traces into an eval dataset
Preserve the failure, reconstruct the inputs, and define the correct behavior. A saved production response is evidence, not automatically a regression test.
A useful regression case recreates a condition and checks the behavior that should follow. Copying a bad production response into a spreadsheet captures the incident, but it does not establish what the next version should do differently.
The conversion has three parts: preserve the evidence, reconstruct the necessary inputs, and define an assertion or rubric. Keep those parts distinguishable so a future engineer can understand why the case exists.
Select cases for a reason
Start with confirmed failures from user feedback, low scores, operational incidents, or reviewed trace patterns. Include representative successful examples too. A suite containing only historical failures is useful for regression protection but cannot estimate average production quality.
Deduplicate incidents that share the same underlying condition. A thousand retries during one outage should not outweigh every other workflow in the suite. Preserve frequency as metadata if it matters to prioritization.
Useful metadata includes the incident category, source trace, date, release version, severity, and review status. Keep private customer details out of reusable fixtures unless they are necessary and permitted for that use.
Decide what the test will replay
| Test boundary | Inputs to preserve | What it can establish |
|---|---|---|
| Response generation | Messages and fixed context | Whether the generator handles that evidence |
| Retrieval | Query and a versioned corpus or index | Whether the retriever finds the required evidence |
| Agent workflow | Initial state, tools, responses, and task | Whether the agent reaches the required outcome |
| Scorer validation | Recorded response or trace and reviewed label | Whether the evaluator judges the example correctly |
Rescoring an old trace tests an evaluator. Rerunning an application on reconstructed inputs tests the application. Both are useful, but they answer different questions.
For an external service, choose between a recorded response, a mock, and a controlled live test environment. A fixture with a recorded timeout isolates recovery behavior. It does not measure today's service availability.
Write the expected behavior explicitly
Suppose an agent confirmed that it changed an address, but the update tool returned an error. The useful expected behavior is not a single preferred sentence. It is that the agent must not claim success without evidence that the update succeeded.
A case can contain the user request, initial order state, a simulated tool failure, and assertions over the final state and response. Score factual reporting separately from tone. If the task permits a retry, define the permitted retry behavior and ensure it cannot create duplicate writes.
Keep a reviewer decision with the case. When the expected behavior is disputed, resolve the rubric or mark the case unresolved rather than allowing an arbitrary label into a release gate.
Check what your platform actually saves
Braintrust supports versioned datasets built from logs, feedback, and manual curation. A record can contain ordinary input values or a reference to a trace. Braintrust dataset structure.
Those storage forms have different consequences. Braintrust's Add full trace to dataset operation stores a reference; the referenced trace must remain available. Its documentation distinguishes that from adding a selected span or inserting separate rows programmatically. Check retention and the saved row shape before treating it as a durable fixture. Trace-to-dataset behavior.
For long-lived regression cases, preserve the required inputs and dependencies under an explicit retention policy. A link back to the incident is useful provenance, but the case should not silently become unusable when a log expires.
Verify that the case catches the failure
Run the known-bad version against the reconstructed case and confirm that the check fails for the expected reason. Then run the fix. If both pass, the test may be missing the original condition. If both fail, the fix or expected behavior needs more work.
Pin the dataset and scorer versions when comparing releases. Maintain a separate holdout for broader evaluation, and review obsolete cases when product behavior changes intentionally.
This makes the offline and online eval loop concrete: production supplies evidence, review defines the failure, and a repeatable case protects the corrected behavior.
Continue with the evals implementation guide for the full workflow.