Skip to main content
An Agent Experience eval combines a task, pass criteria, and its own run configuration. This guide starts with a small file-writing task so you can verify the full workflow before testing your product’s integration.

​
Create the eval

Open Agent Experience and select New eval.
1

Write a realistic task

Name the eval Create a greeting file. Use this prompt:
For your own product, describe an outcome a developer would request. Include the context the developer would have, without giving away the implementation you want to test.
2

Say what must be correct

Add this criterion:
Gauge uses an LLM judge to check each criterion. Keep each check specific and focused on one outcome. A run passes only when every criterion passes.Pass criteria should match what the prompt asks the agent to do. If one outcome depends on another—such as choosing your tool before implementing it—use separate evals to test them. See Writing pass criteria for an example.
3

Configure this eval

Choose the agent, model, persona, and how many times to run the task. Leave the repository empty for this example. For a product task, select the repository and Git ref, skills, MCP servers, and credentials a developer would have.These settings belong to this eval. Select multiple model targets to compare them, or duplicate the eval and edit the copy to compare a different repository or tool setup. See Configure runs.
4

Create and run

Choose manual cadence so the eval runs only when you ask. Start with one sample—one run per selected model. Enable Run now to launch when creating the eval, or save it and launch later. Review billing eligibility before running.

​
Run the same workflow from the CLI

These CLI commands use hosted Gauge. The CLI uses the local backend by default, so set hosted once after you sign in, or add --hosted to each command. See Choose a backend.
Copy the returned eval ID, then launch its saved configuration:
The CLI asks for confirmation before launching. Use --yes only when your automation has already established the configuration and intended spend.

​
Inspect the result

Open the session from the eval, or use its run ID:
Review the final response, messages and tool activity, changed files, pages accessed, and criterion verdicts. For this example, verify the file in the diff and read the judge’s evidence. A research task may have no diff; inspect its final response and web activity instead. The agent may finish before Gauge has checked its work. Until judging finishes, the run is left out of the pass rate. Check how many runs were judged before drawing conclusions from the percentage.

​
Generate evals from a repository

For broader coverage, use Generate evals from repo in Agent Experience:
  1. Connect the Gauge GitHub App and grant access to the repository. If an installed connection cannot see it, use Add repositories.
  2. Choose the connected repository and the surface to evaluate: CLI, MCP server, or developer documentation.
  3. Add repository context, such as the package directory to inspect or areas to ignore, and start generation.
  4. After the scan, review the proposed tasks and criteria. Edit the proposal to reflect real developer goals and keep the evals you want.
  5. Review the run settings, choose whether to run immediately, and select Create eval suite.
If the scan fails or reports an unsupported repository, read its explanation before retrying with a different surface or clearer context. Check that the suggested tasks matter to your users and that each pass criterion describes something Gauge can verify.

​
Use evidence to improve

For a product eval, identify the specific point where the agent struggled: an unclear setup instruction, missing example, outdated API, or ineffective skill. Keep the task and pass criteria the same while testing your change. Click New optimization on the eval and follow Improve agent results. After reviewing and publishing a change through your normal process, run the eval again against the live product. Repeat important tests with more runs and different models. One successful session does not tell you how every agent will behave.