Skip to main content
Set up an eval directly on the eval. Choose its repository, models, tools, and schedule. For Agent Preference, choose models, tools, and a schedule on the prompt. Then add scenarios for the repositories and personas you want to test. Every scenario uses the prompt’s settings.

​
Choose the inputs

For private repositories, connect the GitHub App and grant repository access in the organization that owns the eval. A private URL alone does not grant access. Model-provider keys are separate from service credentials; check billing and workload eligibility before launching.

​
Configure an eval

Set the environment on gauge evals create or edit an existing eval:
Replace the example repository with one you can access. Use --agent <harness:model> to pin a model, and repeat --agent to select more than one target. run uses the saved configuration and asks for confirmation before launching. Add resources with --skill <name@version>, --mcp <name@version>, and --connection <credential-id>. Use gauge skills, gauge mcp, and gauge connections list to discover them. Settings you leave out of an edit stay the same. Use the relevant --clear-* flag to remove a setting; see gauge evals edit --help for the available fields. To test the same prompt in a different repository or with different tools, duplicate the eval in the app and edit the copy. Keep the task and criteria constant when comparing configurations.

​
Configure preference scenarios

A preference prompt shares its agents, models, skills, MCP servers, credentials, cadence, and samples across all its scenarios. Each scenario supplies only the repository, Git ref, and persona.
Omit all flags on scenarios add for an empty project with the default persona. Add at least one scenario before expecting the prompt to run on a schedule. To launch only selected scenarios, repeat --scenario:

​
Choose when and how much to run

--cadence none makes a measurement manual. Use daily, weekly, or monthly for recurring runs. There is no separate pause switch for measurements. The CLI displays none cadence as manual when reading a measurement. Run now uses the saved settings. On a recurring cadence, it also moves the next scheduled run to one cadence period after the launch. Running only selected preference scenarios leaves the schedule unchanged.
  • Eval: sessions per launch = agent/model targets × samples.
  • Preference: sessions per launch = selected scenarios × agent/model targets × samples.
Start with one sample and check the logs before adding models or running more often. Review the resulting run count and credit usage before scaling up. If setup fails, use Run troubleshooting.