Cross-model skill regression

Same 36 trigger cases, five runs each, five Claude models in one evening: the July model re-scored 166/180 against its own 177 six weeks earlier, its successor 167, Opus 153, Sonnet 133, Haiku 66. Decoys stayed clean everywhere. The regression I went looking for exists, but the control run shows it is not a model regression — and without that control I would have said it was.

The untested register, item two

The testing-claude-skills reference keeps a register of what its method hasn’t measured. Item one was trigger reliability, closed in August by a 36-case harness that fires cold prompts at a fresh headless session and grades which skill the agent invokes. Item two was cross-model regression: every run of that harness had happened on one model, inside two days, and nothing had been re-run “because the model underneath changed”. The model underneath has now changed — the CLI’s default moved from claude-fable-5 to claude-fable-5-1 — so the entry was due.

Same suite, five models, one evening

The harness grew a --model flag that passes straight through to the CLI and stamps the model id into the results filename. Otherwise nothing moved: the same cases.json — six should-fire paraphrases and three adjacent decoys per skill, for the site’s four project skills — the same two-turn cap, the same Skill-only tool allowance, the same deterministic grading over the stream.

The design decision that turned out to matter was the control. Before touching any new model, I re-ran the July model, claude-fable-5, which the CLI still accepts by name. If the suite scores differently on the same model six weeks later, then every cross-model delta has to be read against that drift, not against July’s number. Then the four others, in sequence: 900 case-runs, about 80 minutes at concurrency six, zero empty streams.

Results

Two things are visible before any per-case reading. Decoy precision is not a model property: all five models kept at least 57 of 60 decoys clean, and the weakest model on sensitivity — Haiku, at 6 of 120 — was perfect on the decoys. And sensitivity is where the spread lives: from 118 should-fire hits in July down to 6, on prompts that never changed.

The control moved eleven case-runs on its own

This is the finding I would have missed. The July model, with byte-identical skill files, lost 11 case-runs between 2026-07-30 and 2026-09-04. All 11 came from one skill: the revising skill fell from 30/30 to 18/30, and two of its prompts — “Add a paragraph about exit codes to the testing claude skills post” and “Tweak the wording in the vibe coding essay’s opening paragraph” — went from 5/5 to 0/5. A single diagnostic run shows what happens instead of the skill: the agent lists the posts directory and opens the file, which the harness correctly records as no skill fired. It isn’t confused. It just edits.

What changed between the two dates is the environment around the model. The CLI is at 2.1.261 now; the July results file never recorded which version it ran on. The session loads 126 skills across five plugins, and every one of those plugins has updated since July. I can’t attribute the drop to any one of these, because the July run didn’t capture them — the same self-documented gap as the <model-name> placeholder in the original evals, one layer up. The harness now records the CLI version and the loaded skill count in every results file, so the next control has something to diff against.

The consequence for the headline: claude-fable-5-1 scoring 167 against July’s 177 looks like a ten-point model regression. Against the same-day control’s 166, it is one case-run better — and one case-run is noise. The model change did nothing measurable. The environment did.

Misses are silence, and silence scales with model size

In July the only failures were competition: a general-purpose brainstorming skill winning the “I want to write a new essay” prompt 2 runs in 5. Silence — no skill at all — was zero. Today, silence is the dominant failure mode on every model, and it grows as the models get smaller:

Silence here is never a refusal. Haiku’s diagnostic run on “How is the site doing in Google search?” went straight to pnpm gsc — the command is documented in the project’s instructions file, and the model reached for it in under four seconds without considering that a skill wraps it. Sonnet’s failures on the scaffolding skill split between the brainstorming competitor (4 of 5 on the essay prompt) and doing nothing skill-shaped at all. The smaller the model, the more a skill has to compete with the task being easy: if the instruction file already says how to do it, the skill’s extra discipline — a revision entry, a preflight — is invisible from the prompt, and the model skips it.

That reading is consistent across skills. The reviewing skill, whose prompts are questions with no obvious direct command, held 30/30 on four of five models. The revising skill, whose prompts read as ordinary edits, was the weakest on every model: 18, 21, 16, 10, and 0 out of 30. Publishing sat between them. The ranking of skills by fragility is the same on every model; only the level changes.

What the harness got wrong this time

Two honesty notes on the numbers above. First, the 180-second per-case timeout fired 33 times across the 900 runs, 19 of them on one decoy — “Review this PR for me” — which correctly routes to a code-review skill that then spins up real work until it is killed. Those graded as passes on the evidence already streamed. But 7 of Sonnet’s timed-out runs were should-fire cases with no skill call at the moment of the kill, and they graded as misses. Sonnet’s 133 may be as high as 140; it is not higher than Opus either way.

Second, a single ad-hoc run of the Opus prompt that scored 0/5 — “Ship the URL state essay live” — fired the publishing skill immediately. That is the July lesson again: a single run of anything proves nothing, and I only mention it because the temptation to “just check” one failure and conclude from it is strong enough to be worth naming.

What changes

Three things, none of them a rewrite of a skill description yet — one variable at a time, and the environment moved this round, not the descriptions.

The control is now mandatory. A cross-model number without a same-day re-run of the previous baseline is unreadable, and the results file has to carry enough of the environment — model, CLI version, skill count — to say what differed. The unit to track per skill is the silent-miss rate, because that is the failure that grows with cheaper models and the one that leaves no trace in a transcript. And the revising skill’s description, which one rewrite already fixed once, has a measured weakness on every model that a second rewrite can be tested against — with a control, five runs, and the numbers here as the before.

The register entry this post answers gets its revision when this publishes; all six results files — July’s baseline and the five from this run — are committed alongside the harness.