Testing Claude skills

Claude skills are packaged instructions an agent loads before doing a task — and like any code path, they fail silently unless exercised. This page is the canonical reference for how I test them: paired runs with and without the skill, graded deterministically against written assertions on what the agent actually did, with the untested parts listed honestly.

The evidence base is deliberately concrete: one production skill — a release-notes generator I built for client work, which turns a release pull request into paste-ready, plain-text notes for a ticket tracker — carried through a full eval cycle, an optimisation round, and one recorded production run. Where this page generalises, it generalises from that.

The shape of the test: paired runs

Each skill carries an evals.json: a list of cases, each with a verbatim user prompt, a prose description of the expected output, and a list of natural-language assertions. The release-notes skill has three cases — the user pastes the PR URL; the user asks casually with just a PR number; the user asks with no PR at all — totalling sixteen assertions. The third case is the interesting one: its two assertions are that the agent asks which PR and produces no file. A guard, not an output.

Every case runs twice: once with the skill (the agent is pointed at SKILL.md and told to read it first) and once as a baseline, explicitly fenced off from the skills repo and told to approach the task however it naturally would. Both configurations get the same rule: don’t ask the user anything mid-run — if you would ask a clarifying question, write it as your final message and stop. That one instruction is what makes ask-versus-guess behaviour gradeable. Each run writes its artifact plus a transcript.md capturing the exact final user-facing message.

Two practical details that cost me real debugging time: runs that write the same output path or touch the clipboard will race, so the six runs went out in two waves; and these runs hit real infrastructure — the live GitHub PR through an authenticated CLI, the real ticket tracker through its connector — because a skill whose whole job is joining two external systems tells you nothing when tested against fixtures.

Grading is deterministic, and behavioural

Grading is a Python script, not a model: string and regex checks against the transcript and the filesystem, one verdict per assertion, with the matched evidence recorded. Four of the seven happy-path assertions are behavioural — they check what the agent did, not what the notes say:

• The transcript must show the clipboard was used and state the output path and give the command to open it — the hand-off ritual, three conjunctive checks.
• The discrepancy report (tickets referenced in the PR body with no matching commit, or vice versa) must appear in the chat transcript and must not appear in the notes file. A channel assertion: the file has to stay clean paste material.
• In the no-PR case: the transcript must ask for a PR number or URL, and the output directory must contain no notes file. An assertion about a refusal, and an assertion about the absence of a side effect.

One honest caveat: these assertions inspect the final transcript and the filesystem, not a structured tool-call log. “The clipboard was used” is proxied by the agent saying so plus the artifact existing.

And one honest incident: deterministic graders have false negatives. In the verification round, the ask-for-PR check failed a transcript that plainly asked for the PR — the regex didn’t cover the phrasing — and I patched the grading file by hand with the evidence string “manual review (regex false-negative)”. The headline result of that round, nine of nine, includes exactly one human override, and this page would be lying by omission if it didn’t say so. The upstream methodology I follow puts it well: a passing grade on a weak assertion is worse than useless, because it manufactures confidence.

What the numbers said

First iteration, six runs, all on the same model, same day:

• Pass rate: 100% with the skill, 14% without — 16/16 assertions versus 3/16 raw.
• Time: 40.6 seconds faster on average with the skill (79.7s vs 120.2s).
• Tokens: about 11,000 fewer with the skill (54,107 vs 65,327 mean per run).

The single most instructive pair is the no-PR case. With the skill, the agent asked which PR in 19.4 seconds and 26,165 tokens. Without it, the agent spent 172 seconds and 82,091 tokens auto-detecting a release and drafting confident notes for a version nobody had asked about — roughly nine times slower and three times more expensive than asking. Both baseline happy-path runs also produced markdown headings and bold into a field that renders markdown literally as punctuation. That is the general lesson I now design assertions around: an unguided agent’s failure mode isn’t refusal, it’s plausible wrongness — output that reads well and wastes a release cycle.

The with-skill time variance (±53s) looked alarming until I read it: the guard case finishes in 19 seconds because asking a question is fast. The variance was structural, not flakiness — worth checking before “fixing” a suite.

What the harness itself got wrong

The test apparatus failed more often than the skill did, and always silently:

• The aggregation script printed an all-zeros benchmark three consecutive times — wrong directory naming, then a missing nesting level, then a null timing key that short-circuited the token counts. Never an error, always a plausible table of zeros. Every check I now write asserts non-emptiness before asserting properties (the vacuous-green lesson from spec-driven development, relearned in miniature).
• The committed expectations went stale against reality: the eval file asserted a specific set of tickets as “referenced in the body only”, and the runs proved a different set. The runs were right; the fixture still says otherwise.
• Timing and token counts were captured by hand from run metadata into a JSON file — there is no instrumentation — and the generated benchmark recorded the model as a literal <model-name> placeholder. I have precise numbers and a self-documented gap in where they came from.

What I haven’t tested

The honest register, which earlier drafts of this page optimistically listed as sections:

Trigger reliability. Every with-skill run was told to read the skill file, so what got measured is compliance given the skill, not whether the skill fires from a cold prompt. The one recorded production use invoked it explicitly as a slash command. There is no should-trigger/should-not-trigger prompt set and no hit-rate across paraphrases — the only trigger engineering so far is a doctrine line in the repo’s contributing guide: Claude under-triggers skills, so make the description a little pushy.

Cross-model regression. Both eval iterations ran about an hour apart on the same model. No suite has yet been re-run because the model underneath changed. The section will get written when it has data behind it.

The second skill. The repo’s CI runs a structural validator — kebab-case directory names, frontmatter present, name matching the directory, a warning under 40 description characters or over 500 lines, green in 9–15 seconds — and that’s all it gates. A second skill was merged with no evals at all, and CI stayed green. Structure is checked by a machine; the eval discipline is currently a practice, not a gate. That gap is the roadmap.