Two of the most-searched eval questions are also the most misunderstood: what is an offline eval? and what is an online eval? The short version is that they run at different times, against different data, and answer different questions — and most teams that struggle with evals are running one when they needed the other.
Here's the distinction that actually matters.
Offline evals: before you ship, against cases you chose
An offline eval runs against a fixed dataset. You assemble a set of inputs you care about — golden examples, past failures, tricky edge cases — attach expected behavior or a scoring rubric, and run your prompt, model, or agent against all of them at once. Scores come from code assertions, LLM judges, or human graders.
The defining trait is that you picked the inputs. Nothing is live. You can run the same suite a hundred times, diff two model versions on identical cases, and get a clean before/after. Offline evals answer one question:
Did this change make things better or worse on the cases I already know I care about?
That's a regression question. It's exactly what you want to answer before code reaches a user.
Online evals: after you ship, against traffic you didn't choose
An online eval runs against live production traffic. Real users send inputs you never anticipated, your system responds, and you score those responses — either every one or a sample. There's usually no ground-truth label, because nobody wrote these cases; the user did. So online scoring leans on reference-free LLM judges, heuristics, and user signals (thumbs, retries, escalations, abandonment).
Online evals answer a different question:
Is the thing actually good in the wild, on inputs I never imagined?
That's a discovery question. It's the only way to find the failure modes your curated dataset didn't have — because if you'd imagined them, they'd already be in the offline suite.
CI evals vs. continuous evals is the same split, wearing work clothes
Once you see the offline/online distinction, the CI-vs-continuous question answers itself:
- CI evaluation is offline evals wired into your pipeline. Every pull request runs the curated suite; a score below threshold blocks the merge. It's a gate. It's deterministic enough to gate on because you control the inputs.
- Continuous evaluation is online evals running perpetually against production — feeding dashboards, firing alerts on quality drift, and (the part people skip) harvesting new failure cases back into the offline set.
CI evals prevent regressions. Continuous evals detect the ones you couldn't have written a test for. You need both, and they are not interchangeable.
The two mistakes that follow from confusing them
Mistake one: gating CI on online-style scores. Reference-free judge scores on live traffic have a noisy distribution — the input mix shifts hour to hour, so the aggregate score wobbles for reasons that have nothing to do with your code. Wire that into a merge gate and you get a flaky build that blocks good PRs and trains your team to ignore the eval. Gates belong on fixed datasets.
Mistake two: no path from production back to the suite. This is the more common and more expensive one. Teams stand up online monitoring, watch the dashboard, and stop there. The failures scroll by and nothing captures them. The whole point of running online is to discover cases worth defending — so every interesting production failure should become a row in the offline dataset that CI now guards forever. If that loop isn't wired, you're paying for observability and throwing away the only thing that made it valuable.
They're a loop, not a choice
The reason "offline vs. online" is the wrong framing is that neither works alone:
- Offline without online means you optimize against your own guesses. Your suite gets greener while the product quietly fails on the inputs you never thought to add. This is the same trap as treating the eval as the whole spec — a green suite proves the output matches your rubric, not that the rubric matches reality.
- Online without offline means you notice regressions but can't prevent them. You find out something broke from a dashboard spike or a user complaint, fix it, and have no gate to stop it from breaking again next week.
The working shape is a cycle: curate an offline set → gate CI on it → ship → run online evals on production → mine the failures → feed them back into the offline set. Each loop makes the offline suite a better model of the real world, which makes CI a stronger gate, which lets you ship faster.
How to set this up without overbuilding
If you're standing up evals for an AI product from scratch:
- Start offline, small. Twenty to fifty hand-picked cases with a clear rubric beats a thousand scraped ones. This is your CI gate on day one.
- Wire it into CI immediately. Run the suite on every PR that touches prompts, models, or agent logic. Set a threshold you can actually hold.
- Turn on production logging before you turn on production scoring. You can't run online evals on traffic you didn't capture. Get tracing and observability in place first.
- Add online scoring on a sample, then close the loop. Score a slice of production, review the low scorers, and promote the real failures into your offline set.
Offline evals defend a definition of good. Online evals keep discovering that the definition was incomplete. The teams that ship reliable AI aren't the ones that picked the right one — they're the ones that kept both running and let each feed the other.