GitHub App

The Gauge GitHub App connects your GitHub repositories to your Gauge Agents organization. With it, Gauge can:

Give coding agents access to private repositories during runs.
Generate evals from a repository.
Run evals on each pull request and report the result as a GitHub check and a comment.

You choose which repositories the App can use.

Install the App

Open GitHub settings in Gauge

In Gauge Agents, open Settings → GitHub for the organization that should own the runs. Select Install GitHub App.

Choose repositories on GitHub

GitHub asks which account to install on and which repositories to share. Select only the repositories Gauge needs.

Confirm the connection

GitHub returns you to Gauge, which shows GitHub App connected.

If you are not an owner of the GitHub account, GitHub sends an install request to an owner instead. Gauge shows Install requested until the owner approves it. To change access later, select Add or remove repositories.

Evaluate pull requests

Pull request evals check a change before you merge it. For example, a pull request that changes your CLI can run the tasks you expect agents to complete with that CLI. If agents can no longer finish a task, the pull request shows a failing check. To turn them on, add a gauge.json file at the repository root. It names your Gauge organization and lists the eval files to run:

gauge.json

{
  "org": "your-org-slug",
  "github": {
    "enabled": true,
    "files": ["evals/healthcheck.md"]
  }
}

Each eval file describes one task and the criteria Gauge judges:

evals/healthcheck.md

---
criteria:
  - name: Healthcheck responds
    rubric: The endpoint returns HTTP 200 without authentication.
config:
  agents:
    - agent: CLAUDE_CODE
---
Add a /health endpoint and run the relevant app checks.

config.agents lists the coding agents to run. Write criteria that Gauge can verify from the session. See Agent Experience.

What happens on a pull request

When a pull request is opened or receives new commits, Gauge:

Adds a Gauge evaluations check to the latest commit.
Runs every listed eval file.
Marks the check as passed or failed after the runs are judged.
Posts a comment with the results and links to the runs in Gauge. Gauge updates the same comment on later pushes.

To evaluate the same commit again, select Re-run on the check in GitHub. Pull requests from forks do not start runs. Pull request evals use your organization’s Gauge Agents credits, like any other run.

Troubleshooting

What you see	What to check
No Gauge evaluations check appears	Check that `gauge.json` is at the repository root, is valid JSON, and sets `"enabled": true`. Check that `org` matches the organization where you installed the App and that the App can access the repository.
The check fails	Open the comment or the check’s Details link to see which criteria failed. See Run troubleshooting.
Settings shows Install requested	An owner of the GitHub account must approve the App.

Get started

Concepts

Guides

API reference

Endpoints

Install the App

Evaluate pull requests

What happens on a pull request

Troubleshooting

​Install the App

​Evaluate pull requests

​What happens on a pull request

​Troubleshooting

Install the App

Evaluate pull requests

What happens on a pull request

Troubleshooting