# Spec Kit, Kiro, OpenSpec, BMAD, specdx: what each one actually checks Of five spec-driven development tools, two run their own checks on a spec without an agent. I seeded one password-reset spec with eight defects: OpenSpec 1.14.3 caught three and specdx 0.5.0 caught five. Spec Kit 1.2.3 and BMAD 6.12.4 leave spec review to the agent and caught none without one. Kiro pairs an LLM with an SMT solver; I did not run it. No validator caught "load fast". Published: 2026-10-23 Canonical: https://umar.codes/spec-driven-development-tools ## Revisions - 2026-10-23 — Created; one password-reset spec with eight seeded defects, run that day through Spec Kit 1.2.3, OpenSpec 1.14.3, BMAD 6.12.4 and specdx 0.5.0. Kiro described from its documentation, not run. - 2026-10-23 — Published, on its slot; auto-drafted by the publishing run because no draft existed for the row. --- Two of the five tools check a spec with code. I wrote one password-reset spec with eight seeded defects and ran each tool's own command on it. OpenSpec 1.14.3 caught three and specdx 0.5.0 caught five. Spec Kit 1.2.3 and BMAD 6.12.4 give spec review to the agent, so they caught none without one. I did not run Kiro. ## What each spec-driven development tool checks without an agent Every one of these tools gives an agent a workflow: write requirements, then a design, then tasks, then code. They differ in what happens to a spec when no model reads it. Some ship a validator that runs in CI. Others ship prompts that ask the agent to review the spec, and those checks run only when an agent runs them, in a different way each time. | Tool | Version | Check that runs as code | Exit code on the seeded spec | Seeded defects caught | | --- | --- | --- | --- | --- | | Spec Kit | 1.2.3 | `check-prerequisites.sh`: the files exist and `plan.md` is not empty | 0 | 0 of 8 | | Kiro | — | Requirements analysis: an LLM writes logic, an SMT solver checks it | not run | not measured | | OpenSpec | 1.14.3 | `openspec validate` | 1 | 3 of 8 | | BMAD | 6.12.4 | None for the PRD; `lint_spine.py` for the architecture document | — | 0 of 8 | | specdx | 0.5.0 | `specdx lint` and `specdx check` | 1 | 5 of 8 | I maintain specdx, and I chose the eight defects. Both facts lean the test toward the checks I think matter, so the next section gives the result for each defect and you can weigh them yourself. All four installs and runs are from 2026-10-23. ## Eight seeded defects and which tool caught each one The feature is a password reset by email: six requirements, written once in each tool's own format. Each copy carries the same eight defects. | Defect | Spec Kit | OpenSpec | BMAD (PRD) | specdx | | --- | --- | --- | --- | --- | | D1. A requirement with no acceptance scenario | — | error | — | warning | | D2. "The reset page must load fast and be user-friendly" | — | — | — | — | | D3. A requirement with no MUST or SHALL | — | warning | — | — | | D4. "The token must expire after TBD minutes" | — | — | — | warning | | D5. The same requirement written twice | — | error | — | — | | D6. A reference to a spec that does not exist | — | — | — | error | | D7. A source file named in the spec that does not exist | — | — | — | error, once approved | | D8. A section heading with nothing under it | — | — | — | warning | OpenSpec and specdx overlap on one defect, D1. Between them they catch seven of the eight. One defect, D2, passed every validator I ran. Read D6 and D7 with care. specdx has frontmatter fields for a reference to another spec and for the files a spec promises, so in the specdx copy those two defects sit in fields a program reads. The other formats have no such fields, so there the same defects are a sentence of prose. Two of the five specdx catches come from that difference in format, not from a stricter check of the same text. ## Spec Kit leaves every content check to the agent `specify init` for Claude Code wrote 10 skills, about 2,500 lines of instructions, and 6 shell scripts. The scripts create feature folders and resolve paths. The only check on a spec is in `check-prerequisites.sh`: it fails when the feature folder or `plan.md` is missing, when `plan.md` is empty, and, with flags, when `spec.md` or `tasks.md` is missing. On the seeded spec it printed the feature path and exited 0. The real review is the `/speckit-analyze` skill. It tells the agent to run six detection passes: duplication, ambiguity, underspecification, alignment with the project constitution, coverage gaps, and inconsistency between the spec, plan and tasks. The ambiguity pass names the exact words in D2: "Flag vague adjectives (fast, scalable, secure, intuitive, robust) lacking measurable criteria". It also names placeholders such as TODO, and the duplication pass covers D5. But no code runs them. The skill even asks for "Deterministic results: Rerunning without changes should produce consistent IDs and counts", which is an instruction to a model, not a property of a program. ## Kiro checks requirements with an LLM and an SMT solver Kiro is an IDE and a CLI, not an npm package, so I did not run it; this section comes from its [specs documentation](https://kiro.dev/docs/specs/) and its [12 May 2026 post on requirements analysis](https://kiro.dev/blog/deep-spec-analysis/). A Kiro spec is three files: `requirements.md`, `design.md` and `tasks.md`. Requirements analysis is an optional step before design. An LLM first rewrites vague criteria into testable ones. A second LLM step translates each criterion into SMT-LIB logic, samples several translations, and groups them by logical equivalence. Where the samples disagree, Kiro asks a question. An SMT solver then checks the model for conflicts and gaps. Each finding becomes a question with two options: keep the requirement, or change it. That design aims at D2, a requirement at the wrong level of detail, which no validator in my test caught. The solver step is deterministic, but its input is the output of an LLM, so the full pipeline is not. I have no measurement of how often it is right. ## OpenSpec validates requirement structure `openspec validate` is a 953-line validator. It requires a Purpose and a Requirements section, at least one `#### Scenario:` for each requirement, and unique requirement names. It warns when a requirement has no SHALL or MUST, when a Purpose is under 50 characters, and when a requirement is over 500 characters. For change proposals it checks that a "Why" section is 50 to 1,000 characters and that no requirement is both added and removed in the same change. On the seeded spec it found D1 and D5 as errors and exited 1. D3 is only a warning: alone, it passes with exit 0, and fails only with `--strict`. OpenSpec has no field for the files a spec promises, and its references point at other OpenSpec stores, not from one spec to another, so D6 and D7 were prose that it does not read. It also passed the inline "TBD" in D4: its placeholder check reads only the Purpose section. ## BMAD lints the architecture document, not the PRD The `bmad-method` CLI has three commands: install, status, uninstall. There is no validate. The install for Claude Code wrote 29 skills in 235 files. PRD validation is a rubric for a "validator subagent" with seven dimensions, and it says so directly: it wants judgement, "not box-ticking". The seeded PRD had no code to run against it, so BMAD caught 0 of 8 without an agent. My walkthrough of the method is linked from [the vibe coding post](/vibe-coding). BMAD does ship one linter, `lint_spine.py`, for the architecture spine. Its docstring is the clearest sentence any of these tools has on the subject: "LLMs miscount IDs and miss literal placeholders; a grep does not." It checks four things: placeholder text, duplicate or out-of-order decision IDs, decisions missing their Binds, Prevents or Rule field, and stack rows with no version. On a seeded spine it found the TBD, the reused ID, the missing Rule and the unpinned version, and it passed a Rule that said the page "must load fast and be user-friendly". It also exits 0 every time, by design: the findings go back to the agent, and the agent decides. ## specdx lints references, placeholders and declared files specdx 0.5.0 has 15 lint rules: 6 for structure and 9 for content. One of them, `clarity/ambiguity-score-ai`, calls no model; it prints a pointer to an AI mode when an API key is set. The other 14 run as code. `specdx check` then compares the files and exports a spec declares with the files on disk. It caught D6 as an error and D4 and D8 as warnings. It caught D1 as an empty Acceptance Criteria section, because in specdx a story's scenarios are a section. It caught D7 only after I set the design spec to `approved`. While the spec is a draft, `specdx check` treats its files as planned, not missing, and exits 3: nothing checkable. It missed D3 and D5. Its vague-language list has 13 phrases, among them "TBD", "as needed" and "simply", and "fast" is not on it. It also printed three warnings about missing edge cases that I had not seeded. This site pins specdx 0.4.0-alpha.24 in CI; the [sdx reference page](/sdx) covers the tool, and [what dogfooding changed in specdx](/what-dogfooding-changed-in-specdx) covers the fixes this site caused. ## No validator caught "load fast" D2 is the easiest of the eight to write by accident, and it is the one every validator passed. "Fast" and "user-friendly" are well-formed, have a MUST, and have an acceptance scenario. Nothing in the text is wrong in a way a parser can see. The defect is that nobody can write a test for it. That is the line between the two kinds of tool. Code catches what is missing or broken in the structure: a section, a scenario, a reference, a file. Judging whether a requirement means something takes a reader, and every tool here that tries it, whether Spec Kit's analyze skill, BMAD's rubric or Kiro's solver, puts a model in that seat. I wrote about where the structural checks stop in [spec drift detection](/spec-drift-detection), and about the practice around them in [spec-driven development](/spec-driven-development). So choose by which half you want code to own. If you want CI to fail on a broken spec, the choice is OpenSpec or specdx, and the per-defect table says which defects each one covers. If you want a review of what the requirements mean, all three of the others give that to the agent, and only Kiro adds a solver. ## How to rerun the comparison The versions were Spec Kit 1.2.3 (`uvx --from git+https://github.com/github/spec-kit.git@v1.2.3 specify init --integration claude`), OpenSpec 1.14.3, BMAD 6.12.4 with the `bmm` module, and specdx 0.5.0, on Node.js 24.13.0. The commands were: ```sh bash .specify/scripts/bash/check-prerequisites.sh --json --require-tasks --include-tasks openspec validate --specs --strict --no-interactive specdx lint && specdx check python3 .claude/skills/bmad-architecture/scripts/lint_spine.py --workspace ``` Write the eight defects into each tool's own format, not a shared one: an OpenSpec requirement needs a `#### Scenario:` heading, a specdx PRD feature needs a `**F1**:` prefix and a user story to cover it, and a Spec Kit requirement needs an `FR-001` ID. A defect written in the wrong format tests the parser, not the check. A new release can change any row, and this page's revision log records each rerun.