When an agent has a choice, does it choose you?This is different from asking whether an agent can use your product successfully. Agent Experience measures implementation success after the task and pass criteria are defined.
What Agent Preference reveals
An Agent Preference run can show whether an agent:- Includes your product among the options it considers
- Ranks your product as its first choice
- Chooses a competitor instead
- Recommends one product but installs another
- Mentions several options without making a clear choice
- Uses documentation or third-party sources while deciding
Measurement types
Open-ended
Ask an open-ended developer question without naming a product. The agent chooses freely, and Gauge records its ranked picks and whether each pick was adopted.
Head-to-head
Name two products and ask the agent to choose between them. Gauge records a winner or
neither, with reasoning for both products.Open-ended prompts
An open-ended prompt measures unaided preference when it does not name your product or a competitor.- Ranked product picks
- The top recommendation
- Whether each product was actually adopted
- Whether the agent converged on one clear choice
- Evidence-backed reasoning for each rank
Head-to-head prompts
A head-to-head prompt measures a direct tradeoff between two named products.- Product A won
- Product B won
- Neither won because the agent used both, chose a third option, or did not reach a clear decision
How a preference prompt works
1
Write a realistic question
Describe a task the way a developer would ask a coding agent. Keep the task concrete enough that the agent must choose and use a solution.
2
Choose a market
A market groups prompts that measure the same capability, such as authentication, databases, or email delivery. Market scope keeps rankings and findings comparable.
3
Configure settings and scenarios
Choose the prompt’s agents, models, skills, MCP servers, credentials, and samples. Add scenarios for the repository and persona combinations you want to measure. Every scenario uses those shared settings.
4
Choose a cadence or run on demand
Set the prompt to daily, weekly, monthly, or manual cadence. Run now launches all scenarios, or a selected subset through the CLI, without changing the schedule. A prompt with no scenarios does not run on a schedule.
5
Adjudicate the choice
Gauge inspects the trace and diff to identify the chosen product, the alternatives, the final adoption state, and the evidence behind the decision.
Measure preference in a pull request
Keep preference prompts in code with a Markdown case undergauge-cases/. Use
preference.kind: open-ended or head-to-head in place of experience criteria.
The task and scenario settings stay in the same file, and optional file filters
run the case only for relevant PR changes.
Git-authored preference results are advisory: selecting a competitor is a
measurement, not a failed check. These cases use the app’s preference analysis
without creating saved prompts or entering their market rankings. See
GitHub PR Checks for
complete examples, CLI requirements, and mixed-suite behavior.
Read the results
Gauge keeps different measurements separate because they answer different questions.
Do not combine organic chosen rates and head-to-head outcomes into one score. One measures a free choice; the other asks the agent to choose between two named products.
Design useful prompts
For an unbranded open-ended prompt:- Write one to three sentences in a developer’s voice.
- Describe the capability and desired result.
- Do not name your product, package names, or competitors.
- Give the agent enough room to choose a solution.
- Name both products in the prompt.
- Ask the agent to make and implement one choice.
- Use the same task and run setup when comparing results over time.
Interpret the evidence
Agent Preference results describe the runs you sampled. They are not a census of every agent or developer task.- Report counts alongside rates.
- Compare results within the same market and prompt type.
- Keep scenarios stable when measuring change over time.
- Separate being mentioned from being ranked first or adopted.
- Read the run reasoning before turning a result into an action.
Agent Experience
Measure whether agents can complete a task with your product after they choose or are directed to use it.