Running spec-drift detection on a real repo

Spec-drift detection checks that an implementation still matches its specs without anyone re-reading both. This site is the test subject: three specs, an Astro codebase, and sdx check now wired into the verification gate. Before declared artifacts landed in specdx, the honest answer here was exit code 3 — nothing checkable. Here’s what changed, verified.

Exit code 3 is an honest zero

specdx’s check command finds drift by extracting checkable surfaces from the implementation — routes, types, tests — and those extractors were framework-specific. Pointed at this Astro repo, the scan came back framework=none detected, routes and tests not scanned, and exited 3: “coverage not assessed”. That exit code is a design decision worth copying: zero errors because nothing was checked is not a pass, and the tool refuses to let the two look alike. Under this site’s dogfooding contract that friction became an upstream issue rather than a workaround, and the fix that closed it went framework-agnostic instead of adding an Astro extractor: any spec can now declare the artifacts: it claims to produce — file paths, plus named exports per file — and check verifies them with no framework knowledge at all. The design went in as the specdx suite’s first ADR; per-framework extraction stayed deferred.

Declaring artifacts in three specs

The declarations took twelve entries across the site’s three specs. The crawler-logger design names its four files — the middleware, the export and serve scripts, and the bot-list module with its two exports. The project-context spec declares the modules its rules govern: the identity constants (ENTITY_STRING and friends), the content schema with its reserved-slug list, the structured-data builders, the OG renderer, and tokens.css — the file that’s the only place colour may be defined. The content-calendar epic declares the plan file, the posts directory (directories verify as paths), and the trigger-eval harness entry point. Twelve paths plus thirteen named exports makes 25 checks. A declaration asserts existence — this path is real, this export is present — and nothing more; that limit is the price of working on any stack.

The first run scored 100%, which mostly proves freshness

sdx check on the declared suite: exit 0, artifacts: 25 verified, 100% implementation coverage. The honest reading is that this proves freshness more than correctness — the crawler logger shipped four days before this run, and its spec was corrected during implementation, so there’d been no time to drift. A checker that has never failed is an unverified checker, so I broke it twice on purpose. First a plausible export rename: the upstream issue’s illustrative example had guessed the bot list was called BOT_SIGNATURES; the code’s real export is BOT_USER_AGENTS. Declaring the guessed name — exactly what pasting the example blind would have done — dropped coverage to 96% and exited 1 with a finding that names both legal fixes: “Export BOT_SIGNATURES from src/lib/bots.ts or update the artifacts list in crawler-logger.” Then a missing file: declaring a renamed script produced the mirror finding (“Create scripts/serve-static.mjs or update the artifacts list”). Both failure modes work, and both findings treat drift as symmetric — the code may be wrong or the spec may be, and the tool doesn’t presume which.

With that verified, specdx check joined the site’s verification gate: pnpm check-specs now runs validate, lint, and check on every commit and in CI. The interesting findings arrive later, the first time a refactor moves a declared file and the gate — not a reader — notices.

Drift the checker still can’t see

Three classes stay invisible, deliberately. Config-key assertions were deferred with the ADR, so the redirect rules in vercel.json and the spec suite’s own wiring can drift silently. Route extraction for Astro was deferred too — the framework override list still reads express, hono, nextjs — so no spec can claim “this page exists” and have it checked. And existence-checking says nothing about content: check proves ENTITY_STRING is exported, while the rule that its bytes stay identical everywhere is enforced by a different tool in the same gate (check-content). Layered checkers, each honest about scope, beat one checker pretending to cover everything. The reopen criterion from the ADR is recorded: if declarations prove insufficient on a real write-up, the deferred extractors get revisited. This write-up is the first data point, and declarations were sufficient.

Update, 2026-08-09. The suite has grown since the run logged above: a fourth spec — a scheduled drain moving crawler logs to durable storage — entered as a draft declaring artifacts it plans to produce, and the gate now reports four specs and 29 declarations. The two unbuilt ones (a cron endpoint file, one middleware export) show as pending, not failing: planned artifacts are excluded from the coverage score until the spec’s status flips to approved, at which point they become enforceable. A checker that treats “not built yet” as different from “missing” is the same honesty as exit code 3, applied to the other end of a spec’s life.