Testing Claude skills

Claude skills are packaged instructions an agent loads before doing a task — and like any code path, they fail silently unless exercised. This page is the canonical reference for how I test them: paired runs with and without the skill, graded deterministically against written assertions on what the agent actually did, with the untested parts listed honestly.

The evidence base is deliberately concrete: one production skill — a release-notes generator I built for client work, which turns a release pull request into paste-ready, plain-text notes for a ticket tracker — carried through a full eval cycle, an optimisation round, and one recorded production run. Where this page generalises, it generalises from that.

The shape of the test: paired runs

Each skill carries an evals.json: a list of cases, each with a verbatim user prompt, a prose description of the expected output, and a list of natural-language assertions. The release-notes skill has three cases — the user pastes the PR URL; the user asks casually with just a PR number; the user asks with no PR at all — totalling sixteen assertions. The third case is the interesting one: its two assertions are that the agent asks which PR and produces no file. A guard, not an output.

Every case runs twice: once with the skill (the agent is pointed at SKILL.md and told to read it first) and once as a baseline, explicitly fenced off from the skills repo and told to approach the task however it naturally would. Both configurations get the same rule: don’t ask the user anything mid-run — if you would ask a clarifying question, write it as your final message and stop. That one instruction is what makes ask-versus-guess behaviour gradeable. Each run writes its artifact plus a transcript.md capturing the exact final user-facing message.

Two practical details that cost me real debugging time: runs that write the same output path or touch the clipboard will race, so the six runs went out in two waves; and these runs hit real infrastructure — the live GitHub PR through an authenticated CLI, the real ticket tracker through its connector — because a skill whose whole job is joining two external systems tells you nothing when tested against fixtures.

Grading is deterministic, and behavioural

Grading is a Python script, not a model: string and regex checks against the transcript and the filesystem, one verdict per assertion, with the matched evidence recorded. Four of the seven happy-path assertions are behavioural — they check what the agent did, not what the notes say:

• The transcript must show the clipboard was used and state the output path and give the command to open it — the hand-off ritual, three conjunctive checks.
• The discrepancy report (tickets referenced in the PR body with no matching commit, or vice versa) must appear in the chat transcript and must not appear in the notes file. A channel assertion: the file has to stay clean paste material.
• In the no-PR case: the transcript must ask for a PR number or URL, and the output directory must contain no notes file. An assertion about a refusal, and an assertion about the absence of a side effect.

One honest caveat: these assertions inspect the final transcript and the filesystem, not a structured tool-call log. “The clipboard was used” is proxied by the agent saying so plus the artifact existing.

And one honest incident: deterministic graders have false negatives. In the verification round, the ask-for-PR check failed a transcript that plainly asked for the PR — the regex didn’t cover the phrasing — and I patched the grading file by hand with the evidence string “manual review (regex false-negative)”. The headline result of that round, nine of nine, includes exactly one human override, and this page would be lying by omission if it didn’t say so. The upstream methodology I follow puts it well: a passing grade on a weak assertion is worse than useless, because it manufactures confidence.

What the numbers said

First iteration, six runs, all on the same model, same day:

• Pass rate: 100% with the skill, 14% without — 16/16 assertions versus 3/16 raw.
• Time: 40.6 seconds faster on average with the skill (79.7s vs 120.2s).
• Tokens: about 11,000 fewer with the skill (54,107 vs 65,327 mean per run).

The single most instructive pair is the no-PR case. With the skill, the agent asked which PR in 19.4 seconds and 26,165 tokens. Without it, the agent spent 172 seconds and 82,091 tokens auto-detecting a release and drafting confident notes for a version nobody had asked about — roughly nine times slower and three times more expensive than asking. Both baseline happy-path runs also produced markdown headings and bold into a field that renders markdown literally as punctuation. That is the general lesson I now design assertions around: an unguided agent’s failure mode isn’t refusal, it’s plausible wrongness — output that reads well and wastes a release cycle.

The with-skill time variance (±53s) looked alarming until I read it: the guard case finishes in 19 seconds because asking a question is fast. The variance was structural, not flakiness — worth checking before “fixing” a suite.

What the harness itself got wrong

The test apparatus failed more often than the skill did, and always silently:

• The aggregation script printed an all-zeros benchmark three consecutive times — wrong directory naming, then a missing nesting level, then a null timing key that short-circuited the token counts. Never an error, always a plausible table of zeros. Every check I now write asserts non-emptiness before asserting properties (the vacuous-green lesson from spec-driven development, relearned in miniature).
• The committed expectations went stale against reality: the eval file asserted a specific set of tickets as “referenced in the body only”, and the runs proved a different set. The runs were right; the fixture still says otherwise.
• Timing and token counts were captured by hand from run metadata into a JSON file — there is no instrumentation — and the generated benchmark recorded the model as a literal <model-name> placeholder. I have precise numbers and a self-documented gap in where they came from.

What I haven’t tested

The honest register, which earlier drafts of this page optimistically listed as sections:

Trigger reliability — closed 2026-08-09. Every with-skill run was told to read the skill file, so what this page’s method measured is compliance given the skill, not whether the skill fires from a cold prompt. That gap now has its own harness and its own numbers: 36 cold prompts across four skills — six paraphrases plus three decoys each — run five times over, 177/180, with both imperfect cases named and rated rather than averaged away. Method, results, and the description rewrite that attributably fixed one silent miss are in Testing whether Claude skills trigger at all.

Cross-model regression. Both eval iterations ran about an hour apart on the same model. No suite has yet been re-run because the model underneath changed. The section will get written when it has data behind it.

The second skill. The repo’s CI runs a structural validator — kebab-case directory names, frontmatter present, name matching the directory, a warning under 40 description characters or over 500 lines, green in 9–15 seconds — and that’s all it gates. A second skill was merged with no evals at all, and CI stayed green. Structure is checked by a machine; the eval discipline is currently a practice, not a gate. That gap is the roadmap.