Skip to main content
Agent Experience measures execution. It tells you whether a coding agent can complete a task with your product and whether the final result meets the requirements you defined. The central question is:
When an agent uses you, can it succeed?
This is different from Agent Preference, which measures what an agent chooses when several products could solve the task.

​
What Agent Experience reveals

An Agent Experience eval can show whether an agent:
  • Finds the correct setup path
  • Uses current APIs and configuration
  • Produces working code instead of a plausible stub
  • Handles required security and error cases
  • Recovers from documentation or tool friction
  • Reaches the same outcome across agents, models, and repositories
Agent Experience separates a coding session’s execution status from its quality verdict. A run can finish normally and still fail its pass criteria. A run without completed judging does not count as passed or failed.

​
The eval model

Each eval combines three things:

​
Task prompt

Write a task with a concrete outcome. Give the agent the context a developer would provide, but do not prescribe every implementation step unless following that step is what you want to test.

​
Pass criteria

A pass criterion is a rubric, not a general aspiration. It should describe evidence that Gauge can verify in the final diff or session trace.
You can define more than one criterion. Gauge judges each criterion independently. The run passes only when every criterion passes. Good criteria are:
  • Observable: Point to code, configuration, or trace evidence.
  • Specific: State the required behavior or final state.
  • Current: Reflect the API and integration path you support now.
  • Independent: Avoid combining unrelated requirements when separate verdicts would be more useful.
Avoid criteria such as “the integration is good” or “follow best practices.” They do not define a testable boundary.

​
Run presets

Run presets let you reuse the same eval across different environments. A preset can define:
  • Repository and Git ref
  • Coding agent and model
  • Judge persona
  • Skills available to the agent
  • MCP servers available to the agent
  • Runs per cycle
Changing one dimension at a time helps you explain why pass rates moved.

​
How an eval works

1

Define the task

Create an eval set with the coding task you want agents to complete.
2

Write the pass criteria

Describe exactly what the final result must contain or do. Gauge never judges an eval without a criterion.
3

Attach run presets

Choose the repositories and agent configurations you want to test. Attach multiple presets when you want a controlled comparison.
4

Run the coding sessions

Launch the eval on demand or let its cycle create runs. Gauge executes each run in an isolated workspace and captures the trace and final diff.
5

Judge each criterion

The eval judge inspects the final diff, relevant trace events, web activity, and your implementation reference. It returns a strict pass or fail with concrete reasoning for every criterion.
6

Aggregate and act

Gauge calculates pass rates, compares agents and models, and groups recurring failures into actions you can investigate.

​
How judging works

The judge evaluates the final state of the run. A difficult path can still pass if the final result satisfies the rubric. Friction does not change a verdict unless the criterion makes that friction relevant. For each criterion, the judge returns:
  • A strict passed value
  • Reasoning tied to a diff path or trace event
  • The specific missing evidence when the criterion fails
The judge can also record observations outside the rubric. Observations capture issues such as deprecated APIs, documentation gaps, near misses, or unnecessary friction without changing a correct criterion verdict.
When the available evidence cannot establish that a criterion is met, the criterion fails. Gauge does not pass a run on the benefit of the doubt.

​
Read the results

Pending or failed judging does not enter the pass-rate denominator. Review the judged-run count with the percentage, especially when the sample is small.

​
Choose Agent Experience when

Use an eval when you want to measure:
  • Installation and onboarding success
  • API integration correctness
  • Migration or upgrade instructions
  • Framework-specific setup
  • Troubleshooting and recovery
  • The effect of a skill, MCP server, or documentation change
Run the same eval across several presets to compare agents or models. Keep the task and criteria stable so the comparison remains meaningful.

​
Use both measurements together

Preference and experience can move independently:

Agent Preference

Measure which products agents choose before you evaluate whether the implementation succeeds.