Judgment is now publicly available.Get started

Beyond 1M+ Tokens: New Ways to Evaluate Long-Horizon Agents

Long-horizon agents now run for hours and generate millions of tokens. Against LLM-as-a-Judge, Agent-as-a-Judge scored 0.15 higher evidence F1 up to 1M tokens at 42% lower cost, and 0.21 higher past 1M at 62% lower cost.

Abi Rao, Andrew Li12 min read

Why evaluating long-horizon agents matters more and is getting harder.

Automating economically valuable work requires agents that can take on longer and harder tasks. In September 2025, OpenAI released GDPval, a benchmark of professional work in fields like finance, healthcare, manufacturing, and government. Its tasks take experts seven hours on average, and some take weeks. Even so, the best models were rated as good as or better than an expert on nearly half of them, though each task came as a single prompt. Earlier that year, METR found that even the best agents had only about a 50% chance of completing a software task that took a human expert an hour.

Agents are closing that gap, fast. In its May 2026 Frontier Risk Report, METR estimated that Claude Opus 4.6 had roughly the same chance of completing a software task that would take an expert 12 hours. Another notable example comes from Anthropic, where a team of 16 Claude Opus 4.6 agents worked in parallel to build a limited C compiler with little human intervention. METR estimated that a human expert working alone would need weeks or months to build it.

As agents increasingly take on longer and more economically valuable work, evaluating them becomes both more important and harder. The longer an agent runs, the more steps an evaluator may need to look through to find the evidence that shows whether it did the job correctly. For example, a coding agent might report that it finished migrating your customer records. But hundreds of tool calls earlier, it merged customers who shared a last name, even though your instructions said to merge records only when the email addresses matched. Two different customers now appear as one. To catch this, an evaluator has to find and connect evidence spread across the run: the instruction at the start, every bad merge in the hundreds of tool calls that follow, and the agent’s report of success at the end.

Where the evidence lives

To catch errors like a wrong merge, teams use eval-driven development: you define the behaviors you expect from your agent and write a rubric for each one, and a judge checks each trajectory against it. How hard a rubric is to check depends on where its evidence sits in the trajectory:

  1. In a single turn. “Did the user ever express frustration?” “Did the agent leak an API key?” Any single turn can settle the question, so each part of the trajectory can be checked on its own.
  2. In nearby turns. “Did the agent recover after a tool failure?” The evidence is the failed call and the few turns right after it: once you find the key turn, the rest is right around it.
  3. Scattered across the trajectory. “Did the agent’s final summary match what actually happened?” “Did it respect a constraint the user set at the start?” “Did it make a claim its tools didn’t support?” The evidence is spread across turns far apart, as in the migration example. There is nothing to anchor on and no way to know beforehand where the evidence will be, so the judge has to find it and connect it.

The third kind is often what teams most want to catch, because it is where agents drift: whether the agent stayed on-policy over a long run. It is also the hardest to evaluate, because the evidence could be anywhere in the trajectory. A judge can’t know in advance which turns matter, so the common approach is to give an LLM-as-a-Judge the entire trajectory along with the rubric and let it find them. As trajectories grow, that creates three problems:

  • They don’t fit. Many trajectories have outgrown the 1M-token context windows of today’s frontier models. On our platform, 20% of trajectories exceed 1M tokens, and the largest run exceeds 1 billion tokens.
  • Accuracy degrades. Even when a trajectory fits, the judge gets worse well before it reaches the context limit. This phenomenon is known as context rot.
  • Cost grows with length. The judge reads every token, so evaluating long trajectories at production scale gets increasingly expensive.

All three problems come from the judge reading the entire trajectory, when most of it isn’t evidence. To decide a rubric, the judge needs only the few turns that matter, wherever they appear. So instead of making the judge read everything, we can narrow the trajectory down to the evidence and judge only that. We compare two ways to do this:

  1. Screen each chunk, then judge. We split the trajectory into chunks, use a separate model to screen each one for relevant evidence, and pass what it keeps to an LLM-as-a-Judge. Because each chunk fits in context, this works on trajectories of any length.
  2. Use an agent as the judge. The judge searches the trajectory with tools and retrieves only the evidence relevant to the rubric. It uses what it has already found to decide where to look next, so it can connect evidence spread across the whole run.
LLM → LLM-as-a-Judge: each chunkis condensed to quoted evidenceThe LLM reads each chunk with therubric and keeps only the evidence,quoted exactly with the spans aroundit. The quotes from every chunk arecombined for the final judgment.LLMLLMLLMLLM-as-a-Judge
Step 5 of 5
The same trajectory under each method. Both screeners read it one chunk at a time and pass what they keep to an LLM-as-a-Judge; the Agent-as-a-Judge searches the whole trajectory for the evidence and judges it directly.

How to find the evidence

1. Screen each chunk, then judge

We split the trajectory into chunks sized to the screening model’s context window. Chunks overlap so that evidence near a boundary isn’t cut in half. If what the screener keeps still exceeds the final judge’s context window, it is screened again until it fits. We try two screeners: an LLM, which extracts quotes from each chunk, and a decision model, which keeps or drops each chunk whole.

  • LLM to LLM-as-a-Judge: An LLM reads each chunk with the rubric and extracts evidence and counter-evidence, quoted exactly, in effect running the eval on each chunk on its own. We then combine the extractions from all chunks, and the same LLM acts as the final judge. Because the quotes come straight from the unsummarized chunks, the verdict stays tied to what the trajectory actually says. Note: when the trajectory fits in the judge’s context window, there is nothing to screen, and this reduces to a plain LLM-as-a-Judge.
  • Jev to LLM-as-a-Judge: Decision models such as Jev don’t generate text, so they can’t extract quotes. Instead, they return calibrated scores over categories you define, which makes them cheap enough to run over every chunk of a long trajectory. We therefore use Jev as a filter: it labels each chunk relevant, irrelevant, or uncertain, and the LLM-as-a-Judge reads only the relevant and uncertain chunks, in their original form and order. If the kept chunks still don’t fit in the judge’s context window, we split them into smaller chunks and run Jev again. Jev has a 64K-token context window.

2. Agent-as-a-Judge

An Agent-as-a-Judge treats evaluation as an iterative search problem. It decides what to look at next based on what it has already found, so it can follow the evidence across the trajectory instead of reading all of it. Our harness is built around these relevant design choices:

  • SQL over the trajectory. Each trajectory is exposed as a SQL table, spans, with one row per turn, and the agent queries it in SQL because models are heavily trained on it. The same interface supports full-text search, so the agent can find exact strings such as a test name or an error message.
  • Code mode. A stateless execution service for heavier processing: writing larger queries, parsing JSON payloads, and pulling out only the relevant fields, without loading the results into its context window. Anything the agent wants to keep between calls, it saves to a key-value store.
  • Memory. Every turn carries metadata (type, turn, parent, tool name) that the agent can pre-filter on, such as only tool-call turns from one tool. Business context about the agent being judged helps it interpret vague rubrics. Together they cut the number of turns it needs.

If you ran Codex or Claude Code over agent logs, you would copy the logs into a sandbox, a machine with its own CPU and memory, and search the files there, so the sandbox has to stay up and hold the whole trajectory. Our harness needs no sandbox. The agent’s code runs on a stateless execution service backed by a key-value store, and it queries the trajectory in place: it sends SQL queries to the database where the trajectory lives and gets back only the rows it asks for. That keeps the agent cheap to run, lets a failed step simply rerun its queries, and leaves room for features such as vector search, which can be added on the database side without changing how the agent works.

The Agent-as-a-Judge queries thetrajectoryEach tool call runs code against atable of the trajectory, and only thematching turns come back, as text.agent.runagent.stepagent.generatellm (step 1)agent.resolvecodemodesql_queryagent.stepagent.generatellm (step 2)agent.resolvecodemodesql_querysql_queryinterpreter (code mode)In [3]:const r = await sql.query(` SELECT span_id, turn, name, snippet(output) FROM spans WHERE trace_id = $trace AND type = 'tool' AND output MATCH 'npm test AND failed' ORDER BY turn`);text(r.rows);Out[3]:span_idturnnamesnippeta41f4bash3 failed: session.test.tsc09e4bashFAIL session.test.ts7d2b17edit_filesession.ts: +18 −6…runs in the DBhits onlyspansvirtual table over databasesspan_idtexttrace_idtextparent_idtexttypetextnametextturnintinputjsonoutputjsonembedding *vector* Coming soon: a separate vector database.

Benchmarking

We built the benchmark with the synthetic data pipeline from our evals-as-indexes post. It’s derived from internal traffic of our own Agent-as-a-Judge. Claude Sonnet 5.5 served as the oracle model and collected the evidence with human-in-the-loop review. We then added distractors, evidence that looks correct but isn’t, to make the task harder.

  • Build trajectories at five lengths. We built 100 trajectories, 20 each at 100K, 500K, 1M, 1.5M, and 2M tokens.
  • Choose rubrics with scattered evidence. We used two rubrics that ask the judge to find every instance of a failure across the whole run, not just give a verdict: hallucination, “Find all instances of hallucination, where the agent made a claim its tool outputs don’t support,” and policy adherence, “Find every step of its skill the agent skipped.”
  • Score how well the judge finds the evidence. For every trajectory and method, we compare the turns the judge cites against the failures we planted. Recall is the share of planted failures the judge found, and precision is the share of its citations that are real failures. We report F1, which weights the two equally, so a judge can’t score well by flagging everything or by flagging only what it’s sure of.

Consider a coding agent migrating customer records to a new schema. Over hundreds of turns it counts the records, runs a dedupe step, migrates the data, and runs the tests, then writes a final report. The judge has to check that report against everything the agent actually did.

Checking the migration report

In this example, the final report makes four claims that contradict earlier tool outputs. The judge has to trace each claim back to the relevant turns.

The task: migrate us-east customer records to the new schema. Merge duplicates only when email addresses match. Only change us-east data.

Rubric: Hallucination

Find all instances of hallucination, where the agent made a claim its tool outputs don’t support. Cite each one by span ID.

Trajectory View

  1. a1
    sql SELECT count(*) FROM customers WHERE region='us-east'30631
  2. ⋯ 94 turns: reading the schema, writing the migration script ⋯
  3. a96
    sql UPDATE customers SET tier='standard' WHERE region='eu-central4'UPDATE 384
  4. ⋯ 51 turns: testing the dedupe logic ⋯
  5. a148
    bash python dedupe.py --match last_name2merged 214 records2
  6. a149
    bash python migrate.py --region us-eastMigrated 30361 rows
  7. ⋯ 31 turns: updating the API layer ⋯
  8. a181
    bash npm test3 failed3: session.test.ts
  9. ⋯ 230 turns: fixing types, cleaning up, writing docs ⋯
  10. a412
    agent
    “Migration complete: all 3,0631 us-east customers are on the new schema, 214 duplicates were merged by email2, all tests pass3, and no data outside us-east was touched4.”

Filled what the agent claimedUnderlined the tool evidence for each claim

Ground truth

4 hallucinations, all in a412

  1. 1Span a412 says all 3,063 customers were migrated, but the migration moved 3,036.Evidence:a1a149
  2. 2Span a412 says duplicates were merged by email, but the dedupe step matched on last name.Evidence:a148
  3. 3Span a412 says all tests pass, but the test run shows 3 failures.Evidence:a181
  4. 4Span a412 says no data outside us-east was touched, but an update changed 38 eu-central rows.Evidence:a96

Not a hallucination: “214 duplicates” in a412, which matches a148.

Configurations

We ran each method with five of the models teams most often use for LLM-as-a-Judge and Agent-as-a-Judge on Judgment Labs: GPT-6 Luna, Claude Haiku 5.5, GLM 5.3, DeepSeek V4.1, and Claude Sonnet 5.5 (oracle). Each model drove three configurations:

  1. LLM-as-a-Judge: a single call when the trajectory fits in the model’s context window. Past that limit, LLM to LLM-as-a-Judge extracts evidence from each chunk for a final judgment.
  2. Jev to LLM-as-a-Judge: Jev (jev-1.13.0) labels each chunk, and the same LLM-as-a-Judge reads only the chunks it keeps.
  3. Agent-as-a-Judge: the harness above, with each model driving it.

The charts below plot each configuration’s evidence F1 and median cost per evaluation against trajectory length.

Results

Finding the evidence astrajectories growAgent-as-a-JudgeJev → LLM-as-a-JudgeLLM-as-a-JudgeWithin context: plain LLM-as-a-Judge. Past the limit: LLM → LLM-as-a-Judge.0.40.60.81.0100K500K1M1.5M2MTrajectory length (tokens)Evidence F10.740.530.47$0.01$0.1$1$10100K500K1M1.5M2MTrajectory length (tokens)Median cost per evaluation (USD, log scale)$0.71$0.57$0.10Sonnet 5.5 (oracle)Claude Haiku 5.5GPT-6 LunaDeepSeek V4.1GLM 5.3
Evidence F1 (left) and median cost per evaluation (right) by trajectory length, for each method and model. Solid: Agent-as-a-Judge. Dotted: Jev → LLM-as-a-Judge. Dashed: LLM-as-a-Judge, which switches to LLM → LLM-as-a-Judge past the context limit.

Averaged across GPT-6 Luna, Claude Haiku 5.5, GLM 5.3, and DeepSeek V4.1:

  • Up to 1M tokens, Agent-as-a-Judge scored 0.85 evidence F1 against 0.70 for LLM-as-a-Judge and 0.66 for Jev to LLM-as-a-Judge, at 42% lower cost than LLM-as-a-Judge.
  • Past 1M tokens (1.5M and 2M), the gap widened. Agent-as-a-Judge scored 0.21 higher F1 than LLM-as-a-Judge and 0.27 higher than Jev to LLM-as-a-Judge, at 62% lower cost than LLM-as-a-Judge.
  • At 2M tokens, Agent-as-a-Judge averaged 0.80 F1 at $0.49 per evaluation, against 0.60 F1 at $1.48 for LLM-as-a-Judge.

Analysis

Screening extends the judge to longer trajectories, but filters evidence locally, not globally

Both screeners get an LLM-as-a-Judge past its context window, but each decides what matters from one chunk at a time, and whether a turn is evidence often depends on turns in other chunks. Take the report in a412 from the example above. “All tests pass” is only a hallucination because of the test run in a181, 230 turns earlier. In its own chunk, the report reads like an ordinary summary, and the failed test run reads like a problem the agent may have fixed later. Each half looks harmless alone, so a screener can drop either one, and it can’t rule out harmless look-alikes either.

  • LLM to LLM-as-a-Judge. When the trajectory fits in context, this is just a plain LLM-as-a-Judge, and it does well: up to about 100K tokens, GLM 5.3 and Haiku 5.5 scored 0.94–0.98 F1, matching or beating the same models running as Agent-as-a-Judge. Once it has to chunk, it plateaus. By 500K every model was at 0.61–0.68 F1, and at 2M GPT-6 Luna and DeepSeek V4.1 found only 55% and 53% of the 120 failures. A pricier model barely helps: GLM 5.3 scored 0.66 F1 at 2M against Haiku 5.5’s 0.58, at 10× the cost.
  • Jev to LLM-as-a-Judge. This tracked LLM to LLM-as-a-Judge closely, and at 2M it scored lower with every model (0.47–0.61 F1). We saw three effects:
    • It over-keeps. Unable to rule chunks out, Jev labeled many of them uncertain. It kept 77% of the chunks with a failure, but also 36% of those without, so it cut only 29–45% of the trajectory.
    • It splits the evidence. In 42% of failure pairs, only one half made it through screening.
    • The final LLM is still the bottleneck. What survives screening goes to the same LLM calls, so at 2M recall was about the same with or without Jev: DeepSeek V4.1 53% against 53%, GPT-6 Luna 51% against 55%.

The Agent-as-a-Judge finds evidence globally, so it holds up as trajectories grow

Instead of judging each chunk on its own, the Agent-as-a-Judge goes and finds the related turns: when it reads the report’s claim that all tests pass, it searches for the test run, wherever it is. The two halves of a failure end up side by side no matter how far apart they are in the trajectory. DeepSeek V4.1 stayed at 0.94–0.995 F1 from 100K to 2M. Weaker models drifted down (Haiku 5.5 from 0.89 to 0.73, GPT-6 Luna to 0.74) but finished 0.15–0.21 F1 points ahead of the same models running as LLM to LLM-as-a-Judge at 2M tokens.

It also becomes the cheapest method at length

LLM to LLM-as-a-Judge reads every token, so its cost grows linearly: with GLM 5.3, from $0.21 at 100K to $4.03 at 2M. Jev itself costs pennies ($0.042 per million tokens), but because it over-keeps, the final LLM still reads most of the trajectory, so Jev to LLM-as-a-Judge cost 70–97% as much at 2M. Our failures are dense, though (55% of chunks contain one); where failures are rare, Jev would cut more. The Agent-as-a-Judge starts from a higher floor because of its scaffolding, but each step reads a small slice and repeated context is cached. At 2M it was the cheapest method with every model: GPT-6 Luna $0.10 against $0.71, GLM 5.3 $1.26 against $4.03.

Conclusion

Every problem LLM-as-a-Judge has with long trajectories comes from one design choice: reading the whole trajectory. Most of a long trajectory is irrelevant to any given rubric, and our results show what happens when the judge stops reading everything. Screening gets the judge past its context window, but it decides what counts as evidence one chunk at a time, so it misses failures whose halves sit far apart. An Agent-as-a-Judge that searches for the evidence it needs keeps finding it as trajectories grow, and it becomes the cheapest method at length.

As agents run longer, and increasingly as swarms of subagents working in parallel, their trajectories will only get longer and more tangled, and evaluating them will look more and more like search. That raises open questions about the judge’s own harness:

  • Maintaining rubrics. At its very worst, a vague rubric turns evaluation into a difficult search problem. A rubric specific enough to say what counts as evidence tells the Agent-as-a-Judge when it can stop, and clarifies edge cases.
  • Semantic search. Full-text search finds exact strings like an error message or a test name, but many rubrics are about meaning, not wording: a claim and the tool result that contradicts it rarely share the same words. Vector and semantic search over spans could let the judge find those connections directly. What else can we borrow from data-agent tooling?
  • Subagents and decision models inside the harness. A main judge agent could fan out subagents over parts of a trajectory, the way RLMs fan out models over slices of their context window. Decision models could fit inside this loop too: Jev struggled because it judged each chunk alone, but an agent could fetch both halves of a suspected failure and ask a decision model to score them together, combining global context with decision-model costs.
  • Checking the environment, not just the trajectory. Some failures only show up in the world the agent changed, such as a database row that’s still wrong after the agent reported fixing it. A judge that can query that state, not just the trajectory, could verify claims the trajectory alone can’t.