Judgment is now publicly available.Get started

Not every eval needs a frontier model

Introducing the Eval Optimizer: a router that auto-configures your evals. Describe what you want to measure, and it picks, sets up, and tests the right judge, from code and decision models to agents. In one customer’s workload, it cut the cost per eval by up to 50×.

Sailesh Polavarapu, Shun Ueno, Andrew Li6 min read

Think of the Eval Optimizer as a router for creates and configures judges automatically. Describe what you want to evaluate in plain language, and it picks the right model and compute, writes code to scope down the evidence the judge needs, and tests the judge against your labels before it runs. Every judge is an AI-powered data processing job, and running those jobs over every trajectory, cheaply and reliably, takes data infrastructure, starting with deciding which model and how much compute each one deserves.

Analyzing agent data is still manual

Every question asked of agent data is fundamentally the same kind of job: a model, or sometimes just code, reads each trajectory and labels it. What changes is when it runs and on what data:

  • Real-time monitoring runs on live traffic as it comes in, where speed matters, like catching a security issue as it happens.
  • Offline tests run on a carefully curated benchmark, for iterating on an AI system before it ships.
  • Data analysis runs pipelines over past trajectories to label and enrich interesting data points, whether for understanding user behavior, business intelligence, or building tasks and environments for RL.

All three are evals in the broad sense: each runs a judge, a rubric plus the model that applies it, over a set of trajectories. Setting up those judges is the first step, and it’s still very human-in-the-loop. Someone has to read through the trajectory data to configure each judge on the following:

  • Rubric: a clear definition of what counts as a pass or a fail, with labeled examples to check the judge against. It usually takes a few rounds of test runs before the judge agrees with human preference.
  • Scope: a single turn, a set of turns, the whole trajectory, or the environment the agent acted in. Some judges also need outside context, like a GitHub repo or a database, to check the agent’s work.
  • Model and compute: anything from deterministic code or a small classifier to a decision model, a frontier LLM, or a full agent. The choice also affects how the work runs, such as whether it can be batched or served on optimized inference.

This is one of the most time-consuming parts of analyzing agent data, and the Eval Optimizer automates as much of it as it can. The rubric stays human-led, since it has to agree with human judgment. Scope, model, and compute are auto-configured: the optimizer reads the trajectories to find where the evidence for each rubric lives, then decides which model and how much compute each judge needs, based on how hard the judgment is and how spread out and consistently located the evidence is.

Match the model and compute to the job

Nobody runs DeepSeek on an 8×H200 node to check whether the agent called the complete_task tool. A few lines of code do that. Yet judges often work exactly this way, with every check sent to the biggest model available. Many need far less. But the model and compute can’t be too small either, or the judge won’t have enough intelligence to interpret the rubric and catch what it’s supposed to. The goal is the smallest model and least compute that still has enough intelligence to get the answer right.

Harder rubrics and wider scopecall for more computeExample judge questions by rubricdifficulty and scopeScopeRubric difficultyTurnTrajectory+ EnvironmentProgrammatic checkHeavy interpretationCodeDecision modelLLM-as-a-JudgeAgent-as-a-Judge12345678910Deterministic code1Called complete_task?2Count tool calls across the session3Do the repo's tests pass?Decision model4Malicious message in this turn?5Is the user frustrated?LLM-as-a-Judge6Tone stayed polite the whole trajectory?7Did it follow the instructions?Agent-as-a-Judge8Obscure rule violated?9Followed policy across a long-horizon,200-turn trajectory?10Did the agent's change break production?
Each dot is an example judge question, placed by how hard its rubric is to apply and how much of the trajectory it needs. Dotted lines mark the region each compute tier covers. Positions are illustrative.

Programmatic checks are easy to spot. The hard part is the boundaries between decision models, LLM-as-a-Judge, and Agent-as-a-Judge, where the right choice depends on the specific question and where its evidence lives. And picking the type is only half of it: once a judge is configured, its workload is predictable, so the optimizer also tunes how much compute it uses, by batching trajectories, answering several questions in one pass, and reusing the same setup across every run.

How the Eval Optimizer decides

Engineers describe what they want to measure, and the Eval Optimizer routes and auto-configures the rest. It looks at each judge’s rubric and where its evidence lives in the trajectories, then routes the judge to the smallest model and least compute that fits:

  • Deterministic code when the rubric is an exact check on structured data, like whether a tool was called or a field is present.
  • Decision model when the evidence fits in a small context and the judgment is a classification: a narrow label with no explanation needed.
  • LLM-as-a-Judge when the evidence needs more context than a decision model can handle, the judgment takes real reasoning, or the result needs a written explanation.
  • Agent-as-a-Judge when the evidence is spread across a long-horizon trajectory, or confirming the answer means checking an outside environment like the web, a GitHub repo, or a database.

The optimizer auto-configures each judge, and a human verifies its outputs before it runs in production. But production data keeps changing after setup: users ask things nobody anticipated, conversations run longer, and teams update their agents’ prompts and tools. A turn that’s usually under 16K tokens might occasionally run far past it, so the decision model can no longer take it all in. When that happens, or when a cheaper judge returns low confidence, the judge falls back to the next tier up. Fallbacks are rare in practice, so most trajectories still run on the right judge and one that gives teams the granular answers they need at the right cost.

For example, one customer runs judges on narrow questions, like what the outcome of a turn was or what type of query the user started with. Their trajectories average about 1 MB, so an LLM judge reading the full trajectory would cost an estimated $0.045–0.075 per judgment. With the optimizer routing those judges to a decision model, and preprocessing programmatically pulling out just the evidence the decision model needs, they pay about $0.0016 per eval, roughly 30–50× cheaper. And this isn’t the most extreme setup we’ve seen.

1A user describes the evalMake sure each factualclaim cites a relevantdocument.2Judgment Agent3Intelligence routing4Alignment check5Judge createdJudgment AgentThought, used 3 toolsThinking…Thinking…ThoughtWhat does this eval check, andwhere does the evidence live?Thinking…Thinking…ThinkingWhere the evidence isSourcesReplyNeeds judgmentA littleContext neededTwo turnsLabelsFailPassSystem Onelocal evidenceEasyHardCodeDeterministicSystem OneFast and cheapLLM JudgeOne model callAgent JudgeExplores the runSource Citation JudgeSystem OneCodeGround truthMatchPassPassPassFailFailPassPassPassFailFailSystem OneGround truthMatchPassPassFailFailPassPassPassPassFailFailLLMGround truthMatchPassPassFailFailPassPassPassPassFailFailAgentGround truthMatchPassPassFailFailPassPassPassPassFailFail

What’s next

The Eval Optimizer is one step toward data infrastructure that lets teams run AI over all of their agent data, at any scale. Here’s where we’re headed:

  • How preprocessing works under the hood. We’ll publish a technical deep dive on how we generate preprocessing code for each judge, run it safely at scale, and keep it reliable as trajectories change.
  • Inference built for evals. Once a judge is configured, its workload is known up front: the same rubrics, run over many trajectories. That lets us reuse shared work across rubrics and trajectories, and tune inference for throughput in ways general-purpose inference can’t.
  • Decision models tuned to your data. Production agent data is often far from what general models were trained on. Fine-tuning small decision models on a company’s own labeled trajectories should make judges both more accurate and cheaper to run.

We’re building toward a world where teams can run large-scale data transformations over every trajectory their agents produce, asking any question of all of it, not a sample. That only works if each judgment is cheap and reliable enough to run billions of times. The Eval Optimizer is how we get there: a router that auto-configures the right model and amount of compute for every judge, so analyzing all your agent data stays accurate and affordable at any scale.