Self-hosted LLM evaluation: deployment options and tradeoffs
A local eval runner, a self-hosted platform, and a private data plane solve different problems. Compare the boundary you control before comparing features.
“Can we self-host it?” is an incomplete buying question. You may need local execution, private trace storage, control over the dashboard and authentication, or a deployment with no external dependencies. Those requirements lead to different architectures.
Write down which components must run in your environment before selecting an evaluation platform. Running the eval process locally does not tell you where its outputs, judge requests, or account metadata go.
Compare the deployment boundary
| Deployment shape | What you operate | What to verify |
|---|---|---|
| Local runner | The process executing cases | Model endpoints, result export, telemetry, shared review |
| Self-hosted application | Application services and their storage | Authentication, backups, upgrades, external services |
| Private data plane | Services holding evaluation and trace data | Managed control-plane dependencies and metadata flows |
| Managed cloud | Instrumentation and configuration | Data location, retention, access, export |
An air-gapped requirement is stricter than any generic self-hosting label. Verify model access, authentication, licensing, updates, and required outbound connections against the actual deployment design.
Langfuse: operating the application yourself
Langfuse documents self-hosting through Docker Compose and Kubernetes/Helm, along with cloud-specific deployment options. Its architecture and enterprise features need to be considered alongside the open-source deployment. Langfuse self-hosting documentation.
This is a candidate to examine when your team wants to operate the observability and evaluation application. Evaluate the complete setup your workload needs, including worker services, databases, storage, authentication, and recovery. A container starting successfully is a much smaller milestone than a production deployment surviving a restore.
Do not infer that every managed-cloud or enterprise control is included in every self-hosted edition. Check the edition and version you intend to deploy.
Braintrust: private data plane with a managed control plane
Braintrust documents Enterprise self-hosting in which the customer operates the data plane while Braintrust hosts the UI, authentication, and control plane. The documentation distinguishes stored trace and evaluation data from control-plane information such as project names and account settings. Braintrust self-hosting architecture.
That makes it a different option from operating the entire application stack. Assess it when private storage and infrastructure control are central requirements and a managed control plane is acceptable. If all components must be independent of the vendor's hosted services, evaluate that requirement explicitly with the vendor.
Your team still needs ownership of data-plane capacity, upgrades, backups, and incident response. A managed UI does not remove the operational work associated with the infrastructure you deploy.
Promptfoo: distinguish the runner from the shared server
Promptfoo documents a basic self-hosted server for storing and sharing evals. Its current documentation warns that this server is intended for individual or experimental use and lacks built-in authentication, multi-team controls, and horizontal scaling. It directs production deployments with those needs to the Enterprise platform. Promptfoo self-hosting documentation.
This distinction matters if your immediate requirement is a local or CI evaluation workflow. That workflow can be narrower than a production service hosting results for many teams. Decide which one you need before treating a simple deployment as an enterprise workspace.
Test operations with your own workload
Run a pilot that ingests representative traces, executes your scorers, restores a backup, and upgrades a staging deployment. Measure ingestion lag, query responsiveness, storage growth, and the work needed to investigate a failed run.
Estimate total cost from infrastructure, storage, model and judge calls, engineering ownership, and any required license or support agreement. A zero software-license fee does not establish a lower total cost; a managed subscription does not establish lower operational risk for your requirements.
Use the Braintrust and Langfuse comparison for the broader product discussion, then validate the deployment boundary against the current documentation. The decisive requirement is which services and data your team must control.
Continue with the evals implementation guide for the full workflow.