Testing whether Claude skills trigger at all

My skill evals had a blind spot I’d documented but not closed: they measured whether a skill improves behaviour once loaded, never whether the agent loads it unprompted. So I built trigger evals — 36 cold prompts, four skills, deterministic grading on which skill fired. First run: 34/36. After a rewrite and five repetitions: 177/180, with the flakiness measured.

The untested register, item one

The testing-claude-skills reference keeps an honest register of what its methodology hasn’t covered, and the first entry was trigger reliability: every with-skill eval run was told to read the skill file, and the one recorded production use invoked the skill explicitly as a slash command. What had never been measured was the cold path — a user phrases a task naturally, and the agent either recognises that a skill applies or it doesn’t. A skill that never fires has the same production value as no skill, delivered at the same maintenance cost.

Designing trigger evals

The harness lives in this repo (evals/trigger/) and tests this site’s own four project skills — scaffolding, publishing, revising, and reviewing content — because they’re public, so every prompt, description, and rate can be shown here unredacted.

Each case spawns a fresh headless session from the repo root, so the skills load exactly as they would in real work, and the prompt never names or hints at a skill. Two constraints make the runs safe and cheap: the session may only invoke the Skill tool — every other tool is denied — and it gets two turns before the run ends. A prompt like “publish the draft” can reveal which skill the agent would reach for without being able to publish anything. Grading is a script, not a model: it scans the machine-readable transcript for skill invocations, and an empty transcript fails loudly rather than counting as “no skill fired” — a vacuous pass being the failure mode I spent three days filing bugs about elsewhere.

Paraphrase sets and pass criteria

Each skill gets six should-trigger prompts, ordered from direct (“the draft is ready — publish it”) to oblique (“the ag-grid post’s canonical link changed — sort it out”), and three should-not decoys chosen to be adjacent: publishing an npm package is not publishing a post; deploying the site is not publishing a post; editing a draft is not revising a published post. A should-trigger case passes if the target skill fires. A decoy passes if the target doesn’t — another skill firing legitimately is fine, because the question is boundary precision, not silence.

Results

First run, 36 cases on claude-fable-5, roughly 15 seconds per case:

The two misses failed differently, and the difference matters. “I want to write a new essay for the site” didn’t under-trigger — it lost a competition: a general-purpose brainstorming skill from an installed plugin, whose instructions aggressively claim all creative work, fired instead of the project’s scaffolding skill. The canonical-link prompt lost to silence: the one genuinely oblique phrasing fired nothing at all, which is exactly the under-triggering the register predicted.

The decoys produced the result I’m most confident in: all twelve stayed clean, and they didn’t just stay quiet — they routed to the correct other skill. The typo-fix decoy fired the revising skill, the deploy decoy fired a deployment skill, the PR decoy fired a code-review skill. Boundary precision was better than raw sensitivity.

What changed, and the catch

The silent miss had a diagnosable cause: the prompt describes a fact that changed (“the canonical link changed — sort it out”) while the description only claimed edits. I rewrote it to claim any change to a published post however phrased — facts that changed since publish, frontmatter, links — and re-ran all 36 cases. The targeted fix landed: the canonical-link prompt now fires the revising skill in 13 seconds.

The catch: the competition miss also passed on the re-run, and I hadn’t touched anything related to it. The brainstorming skill simply didn’t win the race this time. One flipped result with no causal change means trigger competition is stochastic, and a single run of any suite — including my 36/36 — overstates certainty. The honest reading of the two runs together: the description rewrite fixed one miss attributably, the other fix is unearned, and the only honest unit is a rate. So the harness grew a --runs flag.

Rates, not runs: the whole suite, five times

The suite ran five more times — 180 case-runs, about twelve minutes at concurrency four — and every earlier claim now has a rate attached. The rewrite’s fix held: the canonical-link prompt fired the revising skill 5/5, so that flip was earned. The competition case is confirmed stochastic: “I want to write a new essay” fired the scaffolding skill 3/5, and both losses went to the same brainstorming plugin — never to silence. The prompt is a coin-flip between two legitimate claimants, which no description rewrite on my side fixes; that’s a prioritisation question for the harness around skills, not the skills themselves.

Repetition also surfaced the one thing two single runs had agreed to hide: a decoy went unclean. “How many visits are we getting from AI crawlers?” — a data question, deliberately adjacent to the reviewing skill’s territory — pulled that skill in one run of five, 4/5, after showing clean in both single runs. Decoy precision is a rate too; it just needed more samples to show its tail. The five-run totals: should-fire prompts 118/120, decoys 59/60, 177/180 overall — and the two imperfect numbers are now named cases with measured frequencies instead of anecdotes about flips. All raw results — the two single runs and the five-run set — are committed alongside the harness; the register entry this post closes gets its revision when this publishes.