Prompt versioning and rollback in production
Save immutable prompt versions, evaluate a specific release, and record what each request actually used. Rollback depends on more than restoring the text.
A prompt edit changes application behavior even when no application code changes. Treat it as a release: preserve the previous version, evaluate the candidate, deploy it deliberately, and retain a way to restore the last acceptable configuration.
Version history makes the edit recoverable. Request-level version logging makes its effects diagnosable. You need both.
Version the configuration that shapes the answer
The prompt text is only one input. Record the system and user templates, model identifier, generation parameters, tool schemas, output schema, retrieval configuration, and application revision.
These can live in different systems, but each release should resolve to a specific combination. A release manifest could contain immutable identifiers for each component. Avoid storing credentials in that manifest.
For example, a hypothetical support release might use prompt version 18, tool schema version 4, a pinned application revision, and retrieval index snapshot 12. If the previous prompt expects tool schema version 3, restoring only the prompt may create a new failure.
Distinguish a version from a deployment label
A version identifies a saved object. A label such as production points to the version selected for an environment. Moving that label changes what future requests fetch; it does not alter the historical version.
Langfuse documents immutable versions and movable labels for prompt management. This supports promoting a tested version and moving the production label back when necessary. Its SDK also caches prompts, which makes cache behavior part of deployment planning. Langfuse prompt concepts.
Log the resolved version on each request, not just the label. “Production” may refer to version 17 in one trace and version 18 in another. The label alone cannot explain which instructions produced the answer.
Evaluate the exact candidate
Run the baseline and candidate on the same dataset snapshot with the same scorers. Include cases covering the reason for the edit and cases protecting established behavior. Inspect per-case changes and critical failure categories before relying on an overall average.
If the edit changes a tool schema or response format, check downstream compatibility explicitly. A better natural-language response can still break a consumer expecting a particular field.
Braintrust provides a prompt-versioning workflow that combines prompt iterations, evaluations, and switching between versions. The release decision still depends on your dataset and acceptance criteria. Braintrust prompt versioning cookbook.
Define the rollback procedure before promotion
| Question | What the release record should specify |
|---|---|
| What are we restoring? | Previous prompt and compatible configuration versions |
| What triggers rollback? | Named failure categories and operational thresholds |
| Who can act? | An owner and the access needed to move the release pointer |
| When does it take effect? | Cache refresh, worker reload, and request-boundary behavior |
| How do we confirm it? | Traces showing the restored version and outcome checks |
For a staged rollout, assign the version consistently for the unit you care about. A multi-turn workflow may need to retain one release across its conversation rather than switch instructions halfway through. Decide that behavior explicitly.
A rollback prevents future requests from using the candidate once the change propagates. It does not undo emails, database writes, or other actions already completed. Those need their own recovery procedure.
Monitor the release by version
Compare quality, error rate, latency, and cost for the baseline and candidate while accounting for differences in traffic. A candidate routed only to simpler requests will look better even if it has no quality advantage.
Record scorer versions too. Changing the prompt and the judge simultaneously makes a score change harder to interpret. Use judge calibration when the grading logic changes.
If a release fails, preserve the incident and turn it into a regression case. The next prompt edit should be evaluated against the behavior that made this rollback necessary.
Continue with the evals implementation guide for the full workflow.