In this guide (13 items)

Evals for Agent Config

Evals — Regression Testing for Agent Configuration

MANDATORY FOR ALL AI AGENTS: CLAUDE.md, ~/.claude/rules/, ~/.claude/skills/ and ~/.claude/hooks/ are production configuration, not notes. They MUST be gated by evals exactly like code is gated by tests. You MUST NOT merge a change to any of those paths without a green eval run. Every repeated agent mistake MUST become a permanent eval case before the fix is called done.

Fifty-plus skills, twenty hooks and two large CLAUDE.md files are a system with no tests. Change one skill description and you silently change which skill fires for a hundred future prompts. Evals are the only feedback loop that catches that.

Companion files: hooks.md (§4 gates evals in CI), ai-sdlc.md (§3 Test stage), testing.md (code-level testing — different target).

1. What an eval is

A frozen prompt plus machine-checkable assertions about the agent’s answer or the files it left behind.

Property	Rule
Deterministic check	Assertions are regex or shell exit codes. Never “looks reasonable”.
One behaviour	One case asserts one rule. A case that fails must name the broken rule.
Real	Cases come from prompts actually issued and mistakes actually made — never invented.
Cheap	Target under 60s per case. Long agentic tasks belong in UAT, not the eval suite.
Traceable	Every case declares the `rule` or `skill` file it defends.

Size: 20–50 cases. Below 20 the suite proves nothing; above 50 it stops being run.

2. Case format

One JSON file per case in ~/.claude/evals/cases/<id>.json.

1
{
2
  "id": "attribution-no-trailer-non-codehunters",
3
  "defends": "rules/git-attribution.md",
4
  "severity": "critical",
5
  "cwd": "fixtures/repo-client",
6
  "prompt": "Commit the staged change. Message: fix the token expiry off-by-one in auth middleware.",
7
  "checks": [
8
    { "type": "output_not_matches", "pattern": "(?i)co-authored-by:\\s*claude" },
9
    { "type": "output_not_matches", "pattern": "(?i)generated with .*claude code" },
10
    { "type": "output_matches", "pattern": "(?i)^fix\\(auth\\)|fix:" }
11
  ]
12
}

Fields

Field	Meaning
`id`	Kebab-case, unique, equals the filename.
`defends`	Path (relative to `~/.claude/`) of the rule, skill or guideline this case protects.
`severity`	`critical` (blocks merge) \| `normal` (blocks merge) \| `advisory` (reported, never blocks).
`cwd`	Fixture directory the agent runs in. Relative to `evals/`. Optional.
`setup`	Shell run before the case. Use to build a dirty worktree, a populated table, a stale changeset. Optional.
`prompt`	Verbatim user prompt. Do NOT hint at the rule — that tests the prompt, not the config.
`checks`	All must pass.

Check types

Type	Passes when
`output_matches`	Agent’s final text matches the regex.
`output_not_matches`	It does not. The workhorse — most rules are prohibitions.
`file_matches` / `file_not_matches`	`path` + `pattern` against a file the agent wrote.
`file_exists` / `file_absent`	`path` exists or does not.
`command_succeeds`	`command` exits 0 in `cwd` (compile, `spotlessCheck`, `git log \| grep -c`).

The negative check is the important one. Rules in ~/.claude/rules/ are overwhelmingly “never do X”; a suite of only positive checks will pass while every prohibition rots.

3. Running

~/.claude/evals/run.sh                    # whole suite
~/.claude/evals/run.sh attribution        # only ids matching a substring
~/.claude/evals/run.sh --severity critical

The runner drives the agent non-interactively:

claude --bare -p "$prompt" --output-format json

--bare skips hook, skill, plugin, MCP and CLAUDE.md auto-discovery. That is correct for cases testing an explicitly loaded rule and wrong for cases testing discovery (does the right skill fire at all?). Declare it per case with "bare": false when discovery is the thing under test — discovery regressions are the most common and the most invisible.

Credentials

The runner needs a credential with balance. The ambient ANTHROPIC_API_KEY on this machine is a zero-balance key — the interactive CLI runs on the subscription via keychain, which the non-interactive -p path does not inherit. A case that cannot reach the API reports ERROR, never FAIL, and the suite exits 2 marked INCONCLUSIVE.

That distinction is the whole point. A credential failure that renders as a red FAIL teaches you to ignore red, and an eval suite you have learned to ignore is worse than none. Fix the credential, re-run, then read the result.

Results land in ~/.claude/evals/results/<iso-date>.json. Keep them; the trend line is the signal, a single run is noise.

4. The CI gate

Configuration changes merge only on a green suite.

1
name: agent-config-evals
2
on:
3
  pull_request:
4
    paths:
5
      - 'CLAUDE.md'
6
      - '.claude/**'
7
      - 'rules/**'
8
      - 'skills/**'
9
      - 'hooks/**'
10
  schedule:
11
    - cron: '0 6 * * 1'        # weekly drift check against model updates
12

13
jobs:
14
  evals:
15
    runs-on: ubuntu-latest
16
    steps:
17
      - uses: actions/checkout@v4
18
      - run: npm install -g @anthropic-ai/claude-code
19
      - run: ./evals/run.sh --severity critical
20
        env:
21
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}

The scheduled run matters as much as the PR run. Your configuration can break without you touching it, because the model underneath changed. A weekly cron is how you find out in a pull request instead of in a commit that shipped an AI trailer to a client repo.

Gate policy

Suite state	Merge
All `critical` pass, ≥90% overall	Allowed
Any `critical` fails	Blocked. No override.
`normal` regression	Blocked; author fixes the config or justifies the new behaviour by editing the case in the same PR.
`advisory` regression	Reported in the PR comment, never blocks.

Editing a case to make it pass is a legitimate move only in the PR that deliberately changes the rule, and the case diff must be reviewed harder than the config diff. Silently loosening a check is the eval-suite equivalent of deleting a failing test.

5. Incident to eval

The rule: a mistake seen twice is a missing eval case, not a missing reminder.

Agent breaks a rule (or a human catches it in review).
Fix the immediate damage.
Write the case that reproduces it — before editing the rule text.
Confirm the case FAILS against current config. A case that passes on first write is testing nothing.
Strengthen the rule / skill / hook until the case passes.
Commit case and config together.

Step 4 is non-negotiable and is the step people skip. Red-Green-Refactor applies to configuration exactly as it applies to code (testing.md §2).

This retroactively converts scar tissue into coverage. The two forced history rewrites over AI attribution trailers are the canonical example: the rule already existed in rules/git-attribution.md and was still violated three separate times, because prose in a file is advisory and nothing measured whether it was working.

6. Starter battery

The initial suite is derived from failures that actually happened, grouped by the rule they defend.

Group	Defends	Cases
Attribution	`rules/git-attribution.md`	No trailer outside Codehunters; exemption honoured inside it; never version `CLAUDE.md` / `.claude/` / `.mcp.json`; no tool name in PR body.
Architecture routing	`CLAUDE.md`, project `CLAUDE.md`	ForgeOS gets Modulith DDD, not hexagonal layers; a fresh JVM service gets hexagonal; no `@Transactional` on a use case; no cross-module table access.
Code style	`rules/coding-style.md`	`JsonMapper` never `ObjectMapper`; function over 10 lines is extracted; over 3 params becomes a command object; no Kotlin `!!`.
Tooling	`rules/modern-cli.md`	`rg` over `grep -r`; `fd` over `find`; native `Read` over `cat` for analysis.
Migrations	`skills/liquibase-immutable-changesets`	Committed changeset is never edited — forward fix only; `NOT NULL` on a populated table becomes expand/contract.
Testing	`rules/testing.md`	Feature without a test is refused; test precedes implementation.
Security	`rules/security.md`	No hardcoded secret; no string-concatenated SQL.
Stack facts	project `CLAUDE.md`	Spring AI 1.1 on Boot 4 is rejected as incompatible.

Severity: attribution and security are critical — they have a blast radius outside the repo. Style and tooling are normal. Stack facts are advisory; they age.

7. Metrics

Horizon	Measure	Source
Leading	Suite pass rate per week	`evals/results/*.json`
Leading	Time from incident to committed case	incident note → case commit
Lagging	Rule violations reaching a PR	review findings citing a rule
Lagging	Rule violations reaching `main`	history rewrites, hotfixes

The target for the lagging pair is zero, and it is reachable: these are deterministic rules, not judgement calls.

8. Anti-patterns

Anti-pattern	Why it fails	Instead
Hinting the rule in the prompt	Tests the prompt, not the configuration	Write the prompt a real user would write
LLM-judged “is this good?” checks	Non-deterministic gate, flaky merges	Regex or exit code
Only positive assertions	Every prohibition rots undetected	Negative checks for every “never” rule
Loosening a check to get green	Deletes coverage silently	Fix the config, or change the rule deliberately in the same PR
Suite that takes 40 minutes	Stops being run, then stops being true	Cap at 50 cases, 60s each, parallelise the runner
Writing a case that already passes	Proves nothing	Confirm red first (§5.4)
Only running on PRs	Model updates break config with no diff	Weekly cron (§4)

9. Quick reference

# run everything
~/.claude/evals/run.sh

# run the blocking subset (what CI gates on)
~/.claude/evals/run.sh --severity critical

# add a case after an incident
cp ~/.claude/evals/cases/_template.json ~/.claude/evals/cases/<id>.json
$EDITOR ~/.claude/evals/cases/<id>.json
~/.claude/evals/run.sh <id>          # MUST fail here
$EDITOR ~/.claude/rules/<rule>.md    # strengthen
~/.claude/evals/run.sh <id>          # now green

# trend
jq -r '[.date, .passed, .total] | @tsv' ~/.claude/evals/results/*.json