Skip to main content
Agent Preference measures selection. It tells you which product a coding agent chooses when it needs to solve a developer problem, why it made that choice, and whether it followed through in code. The central question is:
When an agent has a choice, does it choose you?
This is different from asking whether an agent can use your product successfully. Agent Experience measures implementation success after the task and pass criteria are defined.

​
What Agent Preference reveals

An Agent Preference run can show whether an agent:
  • Includes your product among the options it considers
  • Ranks your product as its first choice
  • Chooses a competitor instead
  • Recommends one product but installs another
  • Mentions several options without making a clear choice
  • Uses documentation or third-party sources while deciding
Gauge bases the result on the complete session. It reads the agent’s trace, research activity, and final working-tree diff. The diff helps distinguish a product the agent mentioned from one it actually adopted.

​
Measurement types

Recommendation

Ask an open-ended developer question without naming a product. The agent chooses freely, and Gauge records its ranked picks and whether each pick was adopted.

Head-to-head

Name two products and ask the agent to choose between them. Gauge records a winner or neither, with reasoning for both products.

​
Recommendation prompts

A recommendation prompt measures unaided preference when it does not name your product or a competitor.
For each run, Gauge records:
  • Ranked product picks
  • The top recommendation
  • Whether each product was actually adopted
  • Whether the agent converged on one clear choice
  • Evidence-backed reasoning for each rank
If a recommendation prompt names a product, mark it as branded. Branded runs remain useful for qualitative evidence, but Gauge excludes them from the organic ranking because the prompt influenced the choice.

​
Head-to-head prompts

A head-to-head prompt measures a direct tradeoff between two named products.
Gauge records one of three outcomes:
  • Product A won
  • Product B won
  • Neither won because the agent used both, chose a third option, or did not reach a clear decision
Head-to-head results are always branded. Use them to understand a specific competitive decision, not to estimate unaided market preference.

​
How a preference prompt works

1

Write a realistic question

Describe a task the way a developer would ask a coding agent. Keep the task concrete enough that the agent must choose and use a solution.
2

Choose a market

A market groups prompts that measure the same capability, such as authentication, databases, or email delivery. Market scope keeps rankings and findings comparable.
3

Attach run presets

Run presets define the repositories, agents, models, personas, skills, and MCP servers used for the measurement. Every attached preset runs during the cycle.
4

Run on a cycle or on demand

Use the shared Agent Preference cycle for recurring measurement, or run a prompt immediately when you need fresh evidence.
5

Adjudicate the choice

Gauge inspects the trace and diff to identify the chosen product, the alternatives, the final adoption state, and the evidence behind the decision.

​
Read the results

Gauge keeps different measurements separate because they answer different questions. Do not combine organic chosen rates and head-to-head outcomes into one score. Their prompts and denominators differ.

​
Design useful prompts

For an organic recommendation:
  • Write one to three sentences in a developer’s voice.
  • Describe the capability and desired result.
  • Do not name your product, package names, or competitors.
  • Give the agent enough room to choose a solution.
For a head-to-head:
  • Name both products in the prompt.
  • Ask the agent to make and implement one choice.
  • Use the same task and run setup when comparing results over time.
Start most preference prompts in a greenfield repository. Use an existing repository only when its framework, dependencies, or architecture are part of the decision you want to measure.

​
Interpret the evidence

Agent Preference results describe the runs you sampled. They are not a census of every agent or developer task.
  • Report counts alongside rates.
  • Compare results within the same market and prompt type.
  • Keep run presets stable when measuring change over time.
  • Separate being mentioned from being ranked first or adopted.
  • Read the run reasoning before turning a result into an action.
Repeated evidence can point to changes you control, such as clearer positioning, better ecosystem documentation, or a working starter template.

Agent Experience

Measure whether agents can complete a task with your product after they choose or are directed to use it.