Searching a billion tokens of agent data

Drag the slider to see how evals turn raw traces into searchable indexes.

Searching through agent traces is a new and hard problem

Most agent failures don’t come with a big flashing error. You usually find them by accident, while digging into how users interact with your agent, when you notice it did something unexpected. Then comes the harder question: how often has this happened? Answering it means searching through billions of tokens of production data.

Large-scale data systems have long turned vast amounts of raw data into useful signals. Meta’s ads platform, for example, used tens of thousands of engineered features (e.g. recently clicked ads and page-visit counts) to personalize recommendations. At its core, the recommendation system turns vast amounts of user activity into better insights about people's preferences.

Long-horizon agents now produce data at a much larger scale. Their traces do contain structured events, such as searches, page visits and tool calls, but much of what we care about lives in free-form text across those events. A single run can produce thousands of model calls, tool invocations and intermediate decisions, most of them unstructured text. Whether a run actually succeeded often depends on context spread across many spans: a request the agent quietly dropped, a correction from the user, or a tool call that “succeeded” but did the wrong thing. However, the infrastructure and best practices for making sense of this data are still developing. Reading traces by hand helps you understand your agent’s behavior and discover what to look for. But finding every instance of that behavior across billions of tokens requires a more scalable approach.

The natural next step is a data agent: an agent that answers questions about your production traffic by writing SQL over your traces, reading the relevant spans, and citing the evidence it finds. But pointed at raw traces alone, it hits the same wall we do. The behaviors we care about are often latent, so keyword search can’t reliably find them, and the data agent has to find and evaluate them from scratch on every question.

Luckily, LLMs have made it much easier to semantically parse the traces we’re storing. Compared to earlier BERT-style models, LLMs are more capable of handling arbitrary, complex rubrics without fine-tuning. That lets us break each run down along multiple semantic dimensions:

• Was the user frustrated?
• Did the agent follow the specified instructions?
• What was the action’s outcome?

Each of these is an eval, and many teams already run evals like these as online monitoring, to track quality and catch regressions in production. But the same results can also serve as an index, a column you can filter and search on. That makes your data much easier to understand, and makes it far more likely you’ll catch what’s going wrong.

Why search needs these evals

Without evals, a data agent typically falls back on one of two approaches:

• Read traces one by one. With no index to narrow the search, it has to inspect trace after trace. It’s slow, and for a data agent it fills the context window with irrelevant results. As the context grows, the agent suffers from context rot, and the evidence it found earlier gets truncated or compacted away, so the search gets worse the longer it runs.
• Search by keyword. SQL LIKE or grep only finds the words it thinks to try, and some questions have no words to search for. Take “did the agent follow the policy it was given?” It may not know what each agent’s policy says, and breaking it doesn’t leave any particular word behind. To answer, it have to read the policy, read what the agent did, and compare the two.
  • What about vector search? Vector search can find relevant passages even when their wording differs from the query, and a reranker can help prioritize the results. But answering questions like “did the agent repeat a mistake after being corrected?” still requires evaluating the evidence, sometimes across multiple spans. Evals-as-indexes do that work ahead of time for behaviors you already know you want to track. Subsequent searches can filter the stored results and inspect the supporting evidence, rather than evaluate those behaviors from scratch.

Say you run a sales agent and want to find requests it did not complete. Consider the query: “Find email requests that were not completed.”

For this example, we use three evals, some run as standard LLM-as-a-judge and some with Jev:

• requested_action: what did the user ask the agent to do? For a sales agent: send_email, update_pipeline, research_account, schedule_meeting.
• action_outcome: what actually happened to it? completed, partially_completed, pending_approval, not_done or failed.
• failure_recovery: if something failed, did a later attempt fix it? recovered, partially_recovered or unresolved.

Evals can see an entire trace or an individual span. Their results are stored in telemetry.scores, alongside the traces and spans they describe, with the eval name (judge), its result (label) and an explanation (reason). Each result links to the relevant trace and supporting span. In this example, the evals assess whether the email request was completed at the trace level and cite supporting spans within that trace. We assume at most one result per eval per trace.

-- Find evidence via grep-style text matching
SELECT
  trace_id,
  span_id,
  span_name,
  substringUTF8(input, 1, 500) AS input_preview,
  substringUTF8(output, 1, 700) AS output_preview
FROM telemetry.spans
WHERE
  span_name IN ('send_email', 'save_email_draft')
  OR (
    span_name = 'chat_completions'
    AND (
      output ILIKE '%not been sent%'
      OR output ILIKE '%haven''t sent%'
      OR output ILIKE '%did not send%'
      OR output ILIKE '%nothing was sent%'
      OR output ILIKE '%saved a draft%'
      OR output ILIKE '%not yet%'
    )
  )
ORDER BY trace_id, started_at;

-- Find evidence by filtering across three evals
SELECT
  trace_id,
  groupUniqArray(span_id) AS evidence_span_ids,
  anyIf(label, judge = 'requested_action') AS action,
  anyIf(label, judge = 'action_outcome') AS outcome,
  anyIf(reason, judge = 'action_outcome') AS reason,
  anyIf(label, judge = 'failure_recovery') AS recovery
FROM telemetry.scores
WHERE judge IN (
  'requested_action', 'action_outcome', 'failure_recovery'
)
GROUP BY trace_id
HAVING action = 'send_email'
  AND outcome IN (
    'not_done', 'failed',
    'pending_approval', 'partially_completed'
  )
  AND coalesce(recovery, '') != 'recovered';

The raw query can find the right trace even when the sales agent tells the user “Done, the follow-up is ready.” A save_email_draft receipt with sent_at: null tells us the message was still a draft at that point. The search can miss the evidence that tells us whether the email was ever sent. A later delivery confirmation, for example, might appear in a span that matches neither the tool names nor the phrases in the query. With evals-as-indexes, it can filter for requests labeled not_done, exclude those labeled recovered, and inspect the supporting spans.

Using evals-as-indexes helps the data agent find relevant evidence more accurately, finish faster, and consume fewer input tokens. The search costs below exclude the upfront cost of generating the evals, which can be reused across subsequent searches.

The walkthrough below uses a simplified SQL schema to make the Sonnet 5.5 run easier to follow.