Test whether coding agents can use your latest product changes before you merge a pull request.
Gauge Agents tests whether coding agents can complete a task with your latest CLI, package, docs, or Skills before you merge. The GitHub App reads the configuration and eval cases from your PR, runs coding sessions with the candidate inputs, and reports the results in a Gauge evaluations check and PR comment.
Start with your coding agent. We recommend the gauge-agents Skill and Gauge CLI. The Skill’s dedicated PR setup instructions help your agent inspect the repository, reuse existing CI or previews, and verify the configuration. Follow the copy-paste setup below.
Use Gauge CLI 0.15.0 or later for local setup verification. It requires Node.js 22.12 or later; install it with npm install -g @withgauge/cli, or update an existing installation with gauge update.Automatic PR evaluations launch real coding sessions. Start with one case, one agent, and one sample while getting set up. Installing the App or verifying configuration alone does not launch sessions.The examples below use placeholder names and paths. Replace them with yours. Screenshots use example data.
Then start a new coding-agent session in the repository so it can discover the installed Skill, and paste:
Use the gauge-agents Skill and its GitHub PR setup instructions toset up Gauge PR checks in this repository. Run gauge instructions first.Inspect the repo and reuse its existing CI artifacts or public previews.Help me choose the product surface to test, a realistic task, and successcriteria. Use my actual Gauge organization; ask if there is a choice.Create or update gauge.json and one eval case with one agent and one sample,preserving existing configuration and cases. Consult the published configand case schemas for supported fields and enum values.Run gauge evals verify -o json, fix definite failures, and explain anyremaining unknowns. Show me the changes and the remaining GitHub Appinstallation or target-branch settings needed before I open a PR.
The agent can prepare the repository files; complete the GitHub App installation and repository settings below before opening the PR. The install, sign-in, and verification commands do not launch coding sessions.
Open Settings → GitHub and select Install GitHub App.
Connect GitHub from your organization's settings.
Screenshot text: The disconnected GitHub settings page shows Connect your repositories and an Install GitHub App button. Select that button to begin installation.On GitHub, choose your account and select the repositories whose CLI, docs, or Skills you want to test. Return to Gauge to see the connected account and its repositories.If GitHub requests an account owner’s approval, finish that approval before continuing. See repository access and troubleshooting.
2
Configure pull request checks for each repository
On the same Settings → GitHub page, find Pull request checks. Enable checks for each repository you want to test.Use Edit beside Target branches to choose which branches should receive checks. For example, select main to test PRs that merge into main. These are the PR’s destination branches.Repositories without custom settings default to checks enabled for main. Branch names are exact names, such as main or release/next, rather than glob patterns. Changing these settings does not itself launch an evaluation; use a new PR update or rerun an existing check after setup.
Enable checks and choose target branches independently for each repository.
Screenshot text: The Connected panel has an Add or remove repositories button. Under Pull request checks, each repository has an Enabled toggle and an Edit action beside Target branches. In this example, acme/cli and acme/skills target main; acme/docs targets both main and staging.
3
Create gauge.json
Create gauge.json at the repository root. Copy an input configuration below and set org to your Gauge organization slug from the app URL. That organization must own the App installation connected to this repository.Use the configuration schema for accepted fields, input types, and enum values. See schemas and enums for validation guidance.This file tells Gauge what to make available to the agent. Packages and native executables come from your existing CI, docs pages come from a public preview, and Skills or files can come directly from the PR’s commit. Gauge consumes these inputs; it does not trigger builds. Custom prepare/install scripts are not supported in gauge.json.gauge-evals/**/*.md is the default case location. The examples declare it explicitly. No github.enabled field is needed; repository access and target branches are controlled in Settings → GitHub.
4
Write your first eval case
Create gauge-evals/onboarding.md. Start with this template and replace the task and success criterion before opening the PR.
gauge-evals/onboarding.md
---criteria: - name: Replace with your success criterion rubric: >- Describe the observable result that shows the agent completed the task successfully.config: agents: - agent: CLAUDE_CODE sampleCount: 1---Write what an agent should be able to do with your product here,like onboarding. Include any context the agent needs and how itshould verify that the task worked.
The Markdown body is the agent’s task. The criterion tells Gauge how to judge the result and is not shown to the agent. For example, an onboarding task might ask the agent to set up your product and run its first command; the criterion could require a successful response from that command.The eval-case schema describes the YAML frontmatter, including the exact agent enum values. Use CLAUDE_CODE, CODEX_CLI, PI, or OPENCODE; lowercase CLI aliases such as claude-code are not valid here.By default, the case uses every input in gauge.json. To choose specific inputs, add inputs: [cli], inputs: [docs], or inputs: [product-skill] under config, using the names from your configuration.If the task needs an existing application, add repoUrl and repoRef under config, such as https://github.com/your-org/example-app and main. Otherwise, the agent starts in an empty workspace. Each case owns its agent and sample settings. Start with one agent and one sample.The consumer application is separate from the repository supplying the PR’s inputs. Use a pinned commit in repoRef when you need a repeatable starting point. Omitting models uses the harness default; run gauge models list -o json after signing in to choose and pin available model IDs under each agent’s models array.
5
Verify the setup locally
From the repository, check your working files before committing:
gauge evals verifygauge evals verify -o json
Verification works offline without login or a PR. It inspects configuration, case input references, Git input paths in the working tree, and recognizable workflow declarations. yes means a local check passed, no means a definite problem, and unknown means that declaration could not be resolved statically. Exit code 1 indicates a definite problem; exit code 0 can still include unknowns.Fix the no results and inspect any unknown results. Verification does not execute CI, inspect built artifacts, prove installation, check App activation, or start sessions. A repository with no cases is reported as information, so also confirm that it found the case you just wrote.After committing, optionally sign in and preview the committed setup:
gauge auth logingauge evals plan
Plan shows the session count and input readiness without launching sessions. Unlike verify, it reads committed HEAD and ignores uncommitted edits.
6
Open a pull request and review the result
Commit gauge.json, your eval case, and the changes you want to test. Push a branch in the connected repository and open a PR targeting one of its configured branches.Gauge adds a Gauge evaluations check to the latest commit. It reads the committed evals, waits for any required package or docs preview, then starts the sessions. You can include the configuration in your first PR; it does not need to be merged first. Pull requests from forks do not launch evals.The PR comment updates through input preparation, execution, and judging. When judging finishes, the check shows whether the evals passed. Open its session links to review messages, commands, code changes, and criterion explanations. Use that evidence to improve your product, then push a new commit to test again.
The YAML between the case’s --- delimiters: criteria, agents, models, samples, and input selection. The Markdown task body is separate.
Enum values are case-sensitive. Two common examples:
Field
Accepted values
Case config.agents[].agent
CLAUDE_CODE, CODEX_CLI, PI, OPENCODE
npm input install
environment (CLI commands on PATH), workspace (a dependency in the consumer application)
Model IDs are not a fixed enum in the schema. Use gauge models list -o json after signing in to discover available models for each agent.Associate the schemas through your editor’s schema settings or use an external validator that supports YAML frontmatter. Do not add $schema to gauge.json or case frontmatter; both reject unknown fields. Run gauge evals verify -o json to check the files and their references together.
Choose the configuration that matches what you’re changing. Each example is a complete gauge.json. You can combine the named entries under inputs to test more than one together.
Skills
Docs pages
npm package
Native CLI
Other Files
Gauge takes the Skill from the PR’s commit and installs it where the coding agent discovers Skills. The agent attempts your task with the revised instructions, and Gauge judges whether it completes the outcome you defined.
Set path to the directory containing your SKILL.md. Use "." if the Skill is at the repository root. This configuration reads the committed files directly and does not require a build workflow.Skills can also come from an Actions artifact. Select a directory, or a ZIP, .tar.gz, or .tgz file whose root contains SKILL.md.
Gauge waits for the PR’s docs preview to deploy. When the agent visits your normal docs URL during its task, Gauge serves the preview’s pages so you can see whether the revised instructions help it succeed.
Set url to your normal HTTPS docs origin, without a page path. Set environment to the GitHub deployment environment used for your PR previews. Gauge requires a successful deployment for the exact candidate commit with a public HTTPS preview URL. Mintlify, Cloudflare, Vercel, and other providers can work when they expose this evidence.Include your canonical docs URL in the task. Gauge routes requests at that origin to the candidate preview; a URL printed in the task alone does not configure routing. Private previews, authentication gates, and preview URLs under a path prefix are unsupported. Preview pages remain live and can change or expire.If your provider posts an existing PR comment instead of a GitHub Deployment, use the comment-based source below.
Gauge waits for your PR’s build workflow to succeed, downloads the packed npm package, and puts its CLI commands on the agent’s PATH. The agent then attempts your task using that version of the CLI.
Set workflow to your existing packaging workflow and artifact to its upload name. Set path to the packed .tgz inside that artifact. Your workflow should build the PR’s version and upload the package. Gauge consumes the output; it does not build your CLI.For a library or SDK, use "install": "workspace" to add the package as a dependency in the consumer application instead. "environment" is for packages with declared CLI commands. Each input can name a different workflow.Ensure CI checks out the candidate PR head and packages that revision. Paths are relative to the uploaded artifact root, not necessarily the repository root. When using actions/upload-artifact@v7, set archive: true so Gauge receives the expected artifact archive. Reuse an existing suitable build/upload step.See GitHub’s guides to workflows and uploading artifacts.
Build a Linux x86-64 executable in CI and upload its unpacked files, including any companion files. Gauge validates the selected executables, restores executable permissions, and exposes the declared commands on PATH.
commands maps the command an agent will invoke to an actual selected file inside the artifact. Match paths and command paths to the uploaded layout. For actions/upload-artifact@v7, use archive: true.Native inputs support Linux x86-64 ELF executables compatible with the session runtime. They do not unpack nested release archives, install system dependencies, or run custom installers. Shell scripts, macOS, Windows, and ARM binaries are unsupported. Two inputs selected by the same case cannot expose the same command name.
Supply committed docs or other supporting files. Gauge stages them outside the consumer application and tells the agent where to find them.
File inputs can also select files from an Actions artifact. They do not automatically install Python wheels or other package formats. Use an npm package or supported native executable for automatic installation, or make any required installation work explicit in the task.
Use github-pr-comment when the preview provider already posts a comment that identifies both a ready URL and the candidate commit. Match the exact author login, an identifying text marker, and a readiness pattern with named url and sha captures.For example, this complete configuration expects a comment from github-actions[bot] in this format:
Adapt these fields to your provider’s existing comment. The pattern uses JavaScript regular expressions with multiline/global matching. The captured SHA must identify the candidate commit; comments with only a preview URL or timestamp are insufficient. Gauge uses the newest matching author/marker comment, so a newer pending or stale comment cannot silently reuse an older ready URL. This source needs a unique open PR in the same repository at the candidate head, including when launched from the CLI.
The App handles PRs opened, updated with new commits, reopened, or retargeted to another base branch. The PR must be open, come from the same repository, target an enabled branch, and contain a valid gauge.json with the connected organization’s org. Fork PRs do not launch evaluations.To skip unrelated file changes, add a top-level checks entry to gauge.json:
Use repository-relative, case-sensitive globs; matching any pattern is enough. Changes to gauge.json or configured case paths always count. Without checks.paths, every eligible update can run. After a passing evaluation, Gauge compares subsequent changes against that evaluated commit. A failed or unfinished evaluation, changed target branch, or unavailable comparison causes another evaluation rather than hiding the result behind a skip.
Use Gauge CLI 0.16.0 or later to validate, plan, or run cases with per-eval path filters.To give an individual eval its own file filter, add checks.paths under config in its Markdown frontmatter. For example, this case runs when documentation or its preview workflow changes:
gauge-evals/docs-onboarding.md
---criteria: - name: Completes onboarding with the candidate docs rubric: The agent follows the docs and successfully runs its first example.config: agents: - agent: CLAUDE_CODE sampleCount: 1 inputs: [docs] checks: paths: - "docs/**" - ".github/workflows/docs-preview.yml"---Use https://docs.example.com to set up the product and run its first example.
Use the input name, task, and repository paths that match your project. Case patterns are relative to the repository root, just like the top-level filter. A match on any pattern selects the case. Cases without config.checks.paths run whenever the suite runs.If you also set top-level checks.paths in gauge.json, that remains the gate for the whole suite. Include every path that should trigger any of its cases, or omit the top-level filter and let each case decide. A case’s config.inputs selects the inputs it receives; it does not determine when it runs.Changes to gauge.json select every case. Editing a case selects that case even if its watched paths did not change. Failed or unfinished evaluations, target-branch changes, and unavailable comparisons run conservatively instead of skipping cases. Explicit GitHub reruns bypass both levels of path filtering.If no cases match, the check is skipped without launching sessions or reserving credits.These conditions apply to automatic GitHub PR checks. Local gauge evals verify validates all discovered cases; manual gauge evals plan and gauge evals run use your case selection without filtering by changed files.A repository with no cases gets No evals configured, a skipped check with no sessions or credit reservation. This lets you commit input configuration before adding your first case.Use GitHub’s rerun action on the current Gauge evaluations check to evaluate the same commit again. Explicit reruns bypass both path filters and launch new sessions, which can consume credits. A rerun reuses a complete captured input set when its inputs are unchanged. If previously skipped cases need additional inputs, Gauge resolves a new set from existing CI or previews; it does not trigger builds. Push a new commit to evaluate a new candidate.
Gauge maintains a summary comment for the current PR head. During preparation it shows planned cases and input waits; during execution it shows queued, running, and judging states. Pending judgments are not failures.The table links each session and shows the case, agent/model, outcome, criteria passed, duration, and token usage. For large suites it summarizes cases and limits the individual session table.Once results are settled, each failed criterion gets a separate follow-up comment with the expected behavior, the judge’s explanation, the case, agent/model, commit, and a session link. Execution and judging errors receive their own callouts so you can distinguish a failed product outcome from an incomplete evaluation. Open the linked session to inspect the agent’s work.
Criteria failed: the session completed, but at least one judged outcome failed.
Execution failed: the session failed or timed out; inspect its failure reason and trace.
Judging failed: Gauge could not produce a complete judgment; this is separate from a failed product outcome.
Input preparation failed: an artifact, preview, or configuration prevented sessions from starting.
The check completes after sessions and required judging finish. Each new head or explicit rerun updates the summary. Older sessions remain available from their links.
Confirm App repository access, the organization in gauge.json, enabled target branches, and a same-repository PR. After changing settings, trigger a new PR update.
No evals configured
Add a Markdown case under gauge-evals/, or correct the cases globs. Commit it before updating the PR.
Waiting for CI
Confirm the named workflow runs for the candidate commit. Approve it in GitHub if approval is required. Gauge discovers existing runs; it does not trigger them.
Missing package or executable
Check the artifact name and its internal file layout. Match path/paths to uploaded files. Native files must already be unpacked; npm input must select the packed package.
Waiting for a preview
Check the candidate SHA, deployment environment and successful status, or the exact comment author/marker/readiness pattern. Make the preview publicly accessible.
Local edits have no effect on plan
plan and run use committed HEAD. Use verify for working files, then commit and try again.
Discovery waits up to five minutes for a missing build or preview. A known queued or running build/deployment can continue waiting, with abandoned requests expiring after seven days. Waiting for inputs creates no coding sessions or session debit. A failed workflow or absent/expired artifact produces an input error.
