# Testing whether Claude skills trigger at all Claude skills fail silently in two ways: doing the task wrong once loaded, and never loading at all. My evals covered only the first. This log covers the trigger harness that closes the gap: 36 cold prompts across four skills — six paraphrases plus three decoys each — graded deterministically on whether the right skill fired, and what one description rewrite changed — verified across five repetitions of the whole suite. Published: 2026-08-09 Canonical: https://umar.codes/testing-skill-triggers ## Revisions - 2026-07-28 — Created. - 2026-07-30 — Wrote the full body from the first two harness runs. - 2026-07-30 — Added the five-repetition variance run (--runs flag); rewrote the closing from its results. - 2026-08-09 — Published. Pulled forward from the planned 2026-08-24 date — Search Console shows live query demand for the topic. --- My skill evals had a blind spot I'd documented but not closed: they measured whether a skill improves behaviour once loaded, never whether the agent loads it unprompted. So I built trigger evals — 36 cold prompts, four skills, deterministic grading on which skill fired. First run: 34/36. After a rewrite and five repetitions: 177/180, with the flakiness measured. ## The untested register, item one The [testing-claude-skills](/testing-claude-skills) reference keeps an honest register of what its methodology hasn't covered, and the first entry was trigger reliability: every with-skill eval run was *told* to read the skill file, and the one recorded production use invoked the skill explicitly as a slash command. What had never been measured was the cold path — a user phrases a task naturally, and the agent either recognises that a skill applies or it doesn't. A skill that never fires has the same production value as no skill, delivered at the same maintenance cost. ## Designing trigger evals The harness lives in this repo (`evals/trigger/`) and tests this site's own four project skills — scaffolding, publishing, revising, and reviewing content — because they're public, so every prompt, description, and rate can be shown here unredacted. Each case spawns a fresh headless session from the repo root, so the skills load exactly as they would in real work, and the prompt never names or hints at a skill. Two constraints make the runs safe and cheap: the session may only invoke the `Skill` tool — every other tool is denied — and it gets two turns before the run ends. A prompt like "publish the draft" can *reveal* which skill the agent would reach for without being able to publish anything. Grading is a script, not a model: it scans the machine-readable transcript for skill invocations, and an empty transcript fails loudly rather than counting as "no skill fired" — a vacuous pass being the failure mode I spent [three days filing bugs about](/dogfooding-specdx) elsewhere. ## Paraphrase sets and pass criteria Each skill gets six should-trigger prompts, ordered from direct ("the draft is ready — publish it") to oblique ("the ag-grid post's canonical link changed — sort it out"), and three should-not decoys chosen to be adjacent: publishing an npm package is not publishing a post; deploying the site is not publishing a post; editing a draft is not revising a published post. A should-trigger case passes if the target skill fires. A decoy passes if the target *doesn't* — another skill firing legitimately is fine, because the question is boundary precision, not silence. ## Results First run, 36 cases on claude-fable-5, roughly 15 seconds per case: | skill | trigger rate | decoys clean | | --- | --- | --- | | new-post | 5/6 | 3/3 | | publish-post | 6/6 | 3/3 | | revise-post | 5/6 | 3/3 | | content-review | 6/6 | 3/3 | The two misses failed differently, and the difference matters. "I want to write a new essay for the site" didn't under-trigger — it lost a *competition*: a general-purpose brainstorming skill from an installed plugin, whose instructions aggressively claim all creative work, fired instead of the project's scaffolding skill. The canonical-link prompt lost to *silence*: the one genuinely oblique phrasing fired nothing at all, which is exactly the under-triggering the register predicted. The decoys produced the result I'm most confident in: all twelve stayed clean, and they didn't just stay quiet — they routed to the *correct other* skill. The typo-fix decoy fired the revising skill, the deploy decoy fired a deployment skill, the PR decoy fired a code-review skill. Boundary precision was better than raw sensitivity. ## What changed, and the catch The silent miss had a diagnosable cause: the prompt describes a *fact that changed* ("the canonical link changed — sort it out") while the description only claimed *edits*. I rewrote it to claim any change to a published post however phrased — facts that changed since publish, frontmatter, links — and re-ran all 36 cases. The targeted fix landed: the canonical-link prompt now fires the revising skill in 13 seconds. The catch: the competition miss *also* passed on the re-run, and I hadn't touched anything related to it. The brainstorming skill simply didn't win the race this time. One flipped result with no causal change means trigger competition is stochastic, and a single run of any suite — including my 36/36 — overstates certainty. The honest reading of the two runs together: the description rewrite fixed one miss attributably, the other fix is unearned, and the only honest unit is a rate. So the harness grew a `--runs` flag. ## Rates, not runs: the whole suite, five times The suite ran five more times — 180 case-runs, about twelve minutes at concurrency four — and every earlier claim now has a rate attached. The rewrite's fix held: the canonical-link prompt fired the revising skill 5/5, so that flip was earned. The competition case is confirmed stochastic: "I want to write a new essay" fired the scaffolding skill 3/5, and both losses went to the same brainstorming plugin — never to silence. The prompt is a coin-flip between two legitimate claimants, which no description rewrite on my side fixes; that's a prioritisation question for the harness *around* skills, not the skills themselves. Repetition also surfaced the one thing two single runs had agreed to hide: a decoy went unclean. "How many visits are we getting from AI crawlers?" — a data question, deliberately adjacent to the reviewing skill's territory — pulled that skill in one run of five, 4/5, after showing clean in both single runs. Decoy precision is a rate too; it just needed more samples to show its tail. The five-run totals: should-fire prompts 118/120, decoys 59/60, 177/180 overall — and the two imperfect numbers are now named cases with measured frequencies instead of anecdotes about flips. All raw results — the two single runs and the five-run set — are committed alongside the harness; the register entry this post closes gets its revision when this publishes.