Test proposed CLI commands

Use gauge run --hosted --cli-rules rules.json to test a CLI change before you ship it. Rules can return a fake response, run a replacement command, or change an existing command’s arguments and output. You do not need an optimization, an issue, or a previous session. The file stays on your machine; Gauge sends its validated contents with the launch and freezes them for those sessions. It does not change the saved eval, preference, or fixture repository. Pass the flag again on each launch that should use the rules.

Stub a proposed subcommand

Suppose your CLI is published as @example/cli with the executable example, and you want to test a proposed example token create command. Make the existing CLI available in the session as usual. A rule replaces a matching invocation; it does not install a package or make an absent executable discoverable. Save this as rules.json, replacing the package and executable names with yours:

{
  "schemaVersion": 4,
  "target": {
    "kind": "npmBin",
    "packageName": "@example/cli",
    "program": "example"
  },
  "shippingIntent": "interaction",
  "rules": [
    {
      "id": "create-token",
      "selector": { "kind": "exact", "argv": ["token", "create"] },
      "operations": [
        {
          "id": "fake-token",
          "kind": "stub",
          "stdout": "test-token-not-a-real-credential\n",
          "stderr": "",
          "exitCode": 0
        }
      ]
    }
  ]
}

A stub skips the original command and supplies stdout, stderr, and an exit code. Use an obvious test value, and keep the task small enough to complete with it. A fake token will not authenticate against a real API. Launch a saved eval or preference:

gauge run --eval <eval-id> --hosted --cli-rules rules.json
gauge run --preference <prompt-id> --hosted --cli-rules rules.json

For an immediate exploratory session, pass a task directly:

gauge run "Use example token create and report the token it returns." \
  --hosted --cli-rules rules.json

The direct prompt creates an open-ended Agent Preference session. Use an eval with criteria when you want a task-success verdict. The same flag works with case files and JSON case specs:

gauge run --hosted --cli-rules rules.json --dry-run
gauge run --hosted --cli-rules rules.json
gauge run --apply cases.json --hosted --cli-rules rules.json

Previews for prompts, case files, and case specs report the rule count and a digest of the normalized rules. A preview launches no sessions, but may upload configured inputs. Saved hosted evals/preferences do not support --dry-run.

Test discovery separately

Naming the proposed command in the task tests whether the agent can use it. To test discovery, give the agent only the goal and put the command in the help or docs it is likely to consult. You can add a rule for example --help that advertises token create, alongside the rule that stubs it. Rules do not teach the agent about a command automatically. Unmatched invocations run normally, so a command the agent invents may still fail. Check whether the agent read the proposed help, chose the command, and used its result before crediting the change.

Redirect an invocation

To run a replacement program instead of returning a fixed response, replace the stub operation with:

{
  "id": "replacement",
  "kind": "exec",
  "argv": ["/bin/echo", "test-token-not-a-real-credential"]
}

exec.argv is the complete replacement argument list. Its first entry must be an absolute executable path in the session. Arguments are literal; this is not a shell command string. Each stub or exec must be the only operation in its rule.

Matching and limits

A file targets one npm package/executable pair or one sandbox path. For a path target, use {"kind":"path","path":"/absolute/path/to/program"}.
An exact selector matches the argument list after the executable, including argument order. The example matches example token create, not example token create --json.
A command selector matches parsed positional tokens and declared options. Declare option arity when using it; unknown or ambiguous options can prevent a match. Start with exact selectors for focused tests.
Schema v4 supports stub and exec. Schema v3 remains supported for argument, environment, stdin, and output changes. Do not supply a bundle digest or runtime limits in the authoring file; Gauge seals the rules.
The flag takes a JSON file path, not stdin. Files are limited to 2 MiB and 50 rules. It is hosted-only and does not apply to legacy battery specs, --verify, --schema, or --example.

Review what happened

Open the session viewer and select View CLI rules to see its frozen rules. Applied changes also appear beside nearby commands in the trace; open an annotation to see what ran and what changed. Placement uses the recorded event time, so confirm the command shown in the details. From the CLI, use gauge run commands <run-id> --hosted and gauge run trace <run-id> --hosted to inspect the agent’s commands and subsequent behavior. Configured rules alone do not prove that the agent used them. For a coding agent authoring a test, load gauge instructions cli-rules --hosted. To edit and rerun existing sessions, see Improve agent experience.

Get started

Concepts

Guides

API reference

Resources

Test proposed CLI commands

Stub a proposed subcommand

Test discovery separately

Redirect an invocation

Matching and limits

Review what happened

​Stub a proposed subcommand

​Test discovery separately

​Redirect an invocation

​Matching and limits

​Review what happened

Stub a proposed subcommand

Test discovery separately

Redirect an invocation

Matching and limits

Review what happened