Judgment is now publicly available.Get started

A Missing Primitive for AI Agent Improvement

How labels make production traces useful for agent improvement.

Enyu Rao4 min read
Fixes Without EvidenceDetects coding sessions where edits and final claims are not grounded in observed repository/test evidence.Past 7 DaysFilter traces.../3224168006/30 8pm7/1 2pm7/2 8am7/3 2am7/3 8pm7/4 2pm7/5 8am7/6 2am7/6 8pmTimestampAa NameInputOutput2026-07-07 20:08:23Claude_AgentNow look for any adjacent edge cases ...{"content":[{"text":"##2026-07-07 20:08:23Claude_AgentAdd coverage for the edge case you ...{"content":[{"text":"Per2026-07-07 20:08:22Claude_AgentThis todo_list module is used by other...{"content":[{"text":"Und2026-07-07 20:08:21Claude_AgentRun the full test suite and report ...{"content":[{"text":"Fai2026-07-07 20:08:21Claude_AgentCheck for a regression around empty ...{"content":[{"text":"**

Overview

Labels record a judge’s assessment of a trace or session. They can capture what a user asked for, how the agent handled the task, or what went wrong in the environment. Teams can track these patterns across production traffic, retrieve labeled traces, and reuse those examples to test agent changes.

Why Continuous Improvement Breaks in Production

Agents are increasingly growing in capabilities to autonomously take on longer horizon tasks. However, take a peek into what’s been driving these capabilities, and you’ll find teams passively waiting on frontier model releases or mimicking the newest harness engineering trend.

The industry has long known the mechanism to achieve continuous self-improvement: have an intelligent system (agent) define its own hill to climb, design its own feedback loop then let it recursively learn from its own past experiences. Projects like Auto Research show the loop working in miniature setups and on verifiable domains: give an agent a small training setup, let it edit the code, run a bounder experiment, check whether the metric improved then repeat. But most production long-horizon agents are never a single training file with a single hill to climb.

In production, the improvement signal is buried across millions of sessions. Teams take their best jab at continuous improvement by hand-rolling eval CI/CD pipelines on top of tracing and observability tools. In practice, that usually means handing exploratory data analysis off to a brutal raw-log query to coding agents.

A typical query from an engineer might look like: "For USWest1 and USEast1 service areas, what are the top 5 most frequently asked user requests, out of those requests, which ones are product-feature related and where the user expressed frustration, and how is this agent failure trending over the last month?"

Imagine what you’re actually asking Claude/Codex to do. The agent has to first find the right slice of latest production traces across service areas, users, timestamps, tools and sessions. Then it has to infer which requests are actually product-feature related (keyword-match “feature” or “product” is never enough). Then it has to attribute user frustrations to an agent failure, a missing product capability, or a incomplete environment response. Then it has to aggregate the pattern over time without losing the trace examples that prove the claim.

It’s easy to see how quickly this pipeline breaks down. Once a long-horizon agent grows in complexity and starts producing GB-scale session trajectories across scaled production interactions, the same queries keeps getting rerun. Each investigation forces the agent to re-read raw traces, re-infer the same concepts: cost exponentially climbs, and accuracy becomes impossible to trust.

The missing layer is a way to make the petabytes of data reusable before the question is even asked.

In Judgment, judges apply labels to production traces and sessions. This post focuses on behavior labels: labels that capture patterns in users, agents, and their environments.

What are Labels in Judgment?

Traces show what happened in a run or session. Labels become the semantic tags for recurring patterns inside production agent sessions.

A binary judge applies a label when its criterion is met. A classification judge applies one of its defined choices. Each label belongs to the judge that produced it and points back to the evaluated trace or session. Numeric judge outputs remain scores. See how judge labels work.

At Judgment Labs, we see three types of behaviors worth labeling:

  • User behaviors: what the user is asking for or implying in the session. Repeated asks after a missed answer, explicit complaints, escalation or handoff requests.
  • Agent behaviors: what the agent decides or does. Stopping after one empty search result, saying it cannot find a record that exists, claiming a failed action succeeded, or leaking/editing data it should not.
  • Environment behaviors: what the agent is given by its environment. Empty tool results, stale knowledge, schema mismatches, and timeouts or rate limits.
Labels identify recurring patterns in tracesHighlights show evidence for labels across the traces in a session.One agent sessionRepeated AskComplaintStale RetrievalEmpty ResultFalse SuccessHandoffComplaintTimeoutPremature StopSkipped ToolEmpty ResultEscalationBad RetrievalUnsafe EditSchema ErrorRepeated AskFalse SuccessRefusalUserEnvironmentAgent
Highlights show user, agent, and environment events that support a judge’s assessment. Labels apply to the evaluated trace or session, with the underlying evidence available for inspection.

These behaviors highlights the important windows of a full session trajectory (e.g user frustration on unresolved support tickets, agent stopping after one empty search result, stale retrieval that sends the agent down the wrong path.)

The important part is that the Label stays attached to the trace. Once applied, it becomes a durable signal that teams can monitor, search, collect into a dataset, or use to test whether an agent change climbed the right hill.

How Labels Close the Improvement Loop

Once a Label exists, it does two jobs.

For online monitoring, teams define judges for patterns such as PII leakage, unauthorized edits, or false refusals. With long-horizon agents, the relevant evidence can sit deep inside a trajectory. We instrument these with Agent Judge so the signal is cost-effective to product and accurate enough to act on.

To track these patterns on new production traffic, enable Online monitoring on the judge and configure its evaluation scope, sampling, and triggers. Creating a judge alone does not enable monitoring. Teams can then track the labels it returns and inspect the evidence behind each result.

Labels make production traces searchableSearch traces by their labels01 / TRACE CORPUS WITH LABELS000000100000020000003116553102 / SEARCH BY LABEL
user frustration · premature stop · refund-policy retrieval
03 / NARROWED SET OF MATCHING TRACES
Trace
Output
Labels
001
sent final reply
user frustrationpremature stop
002
returned refund policy
refund-policy retrievaluser frustration
⋮
173
closed conversation
premature stoprefund-policy retrieval
Labels help teams find traces with recurring patterns, then inspect the evidence behind each match.

The second is behavior-indexed search. With Labels, teams can semantically query over traces that already have rich precomputed meaning. "Support tickets are not resolving" becomes searchable indexes like user frustration, agent premature stops, irrelevant refund-policy retrieval, and repeated escalation. Once Labels give you the ability to get down to the right slice of data, you can turn that slice into a dataset to curate specific benchmarks, use automations to chain downstream actions like alerts or triage root-cause agents, and run tests to compare whether the proposed change actually affects new rollouts.

Bring us a sample of your production traces. We’ll show you how to label recurring user, agent, and environment patterns, then use those labels to monitor failures and find examples for testing.