The user-agent header is a claim, not evidence

A user-agent header is a string the client chooses. A Web Bot Auth signature is a proof the client cannot fake: Ed25519 over the request, verified against a key the claimed agent publishes at its own domain. fastify-web-bot-auth is the Fastify plugin I published to check that at the origin.

What 882 crawler records showed

When I started logging AI crawlers on this site, I matched them the ordinary way — a list of user-agent substrings, checked at the edge. The first 882 records made the flaw obvious. Around 45% of them requested credential paths that no crawler has any reason to want. They were not crawlers. They were scanners wearing a crawler’s name, because the name costs nothing to wear.

The partial fix was to check the client IP against the range files operators publish. That works, and it now covers ten operators. It has two limits worth naming. It only works for operators who publish a file at all, and it makes me responsible for keeping ten lists fresh. Every new agent vendor is another list I do not have.

Both approaches share a shape: I am inferring identity from things the request happens to carry. What I actually want is for the caller to prove who it is.

What a Web Bot Auth signature proves

Web Bot Auth is an IETF draft built on RFC 9421, HTTP Message Signatures. The agent signs a defined set of request components with an Ed25519 key, sends the signature in Signature and Signature-Input headers, and names itself in a Signature-Agent header — an https origin such as https://chatgpt.com.

The origin is the interesting part. The key is not distributed by me, or by a registry I have to trust. It is fetched from the agent’s own domain, at /.well-known/http-message-signatures-directory, and selected by JWK thumbprint. So a request claiming to be ChatGPT must carry a signature made with a private key whose public half chatgpt.com is publishing right now. Claiming the name is still free. Producing the signature is not.

It is worth being precise about what this does not establish. It proves key possession for a domain. It says nothing about whether the agent behaves well, respects robots.txt, or is doing something you want. It replaces a guess about identity with a fact about identity, and leaves policy entirely to you.

Why the check belongs at the origin too

OpenAI, Google, and Amazon already sign agent requests in production. Cloudflare, Vercel, and Akamai already verify them at the edge. If you sit behind one of those, much of this is handled for you.

I did not want the capability to be a property of my hosting choice. If verification only exists at the CDN, then it disappears when you move hosts, it is absent in local development, and it is unavailable to anyone serving directly. Cloudflare publishes a low-level web-bot-auth library for exactly this reason, but it is framework-agnostic: you still write the hook, and you still own the key directory logic, which is where the failure modes live. Express has a community verifier. Fastify had none, which is the gap this fills.

Observe mode is the default because enforcement breaks traffic you cannot see

The plugin never blocks anything until you ask it to. Registered with no options, it adds one onRequest hook that decorates every request with a verdict and gets out of the way:

interface WebBotAuthResult {
  verified: boolean;   // signature verified against the agent's own directory
  agent?: string;      // https origin from Signature-Agent
  keyid?: string;      // base64url JWK SHA-256 thumbprint
  trusted?: boolean;   // your trust policy's answer; only set when verified
  reason?: string;     // why it failed; one of six values
  elapsedMs: number;
}

Verification never throws into the request lifecycle. Every failure becomes one of six reason values — unsigned, expired, bad-signature, unknown-key, directory-unreachable, malformed — so a broken signer degrades into a label, not a 500.

The default matters more than it looks. Ordinary human browser traffic is unsigned, so it arrives as reason: 'unsigned', which is indistinguishable at the header level from a scraper. Any origin that switches on enforcement globally without looking first will reject its own readers. Observe mode exists so that the first thing you do is watch: log the verdicts, learn which agents call you and how they fail, add a trust policy while still not blocking, then enforce on individual routes via route config before you consider enforcing globally. Enforcement is the last step, not the first.

The hard part is the key directory, not the crypto

I did not write any cryptography. Ed25519 verification is WebCrypto underneath Cloudflare’s library, and reimplementing it would be the least defensible line of code in the project. Nearly all of the work went somewhere less glamorous: fetching, caching, and rotating other people’s keys without turning that into a way to hurt the server.

Directories are cached per origin, with the TTL taken from Cache-Control and clamped to a range of 60 seconds to 24 hours, defaulting to one hour. Stale entries are served while a refresh runs in the background, and concurrent refreshes for the same origin are single-flighted. That stale window is itself capped at 24 hours past the last successful fetch — after that the keys are treated as unreachable rather than served forever, because “the signer’s directory has been down for a week” should not silently keep authenticating requests.

The adversarial cases shaped the rest. An unknown keyid triggers one forced refresh, which is how key rotation is meant to work — but a fabricated keyid would otherwise turn every request into an outbound fetch, so forced refreshes are throttled to one per origin per 30 seconds. Failing origins are negative-cached for 30 seconds so a dead directory cannot slow every request down. The number of tracked origins is capped at 1000 with oldest-first eviction, so spraying invented Signature-Agent values cannot grow memory without bound. Directory responses are capped at 64 KB, redirects are followed same-origin only for at most three hops, and the fetch timeout covers the body read rather than just the headers.

The suite that holds this together is 93 tests at 97.8% line and 92.4% branch coverage, signing with the RFC 9421 Appendix B.1.4 Ed25519 test key. Two further tests are network-gated and run weekly rather than on the pull request path, verifying real signatures against Cloudflare Research’s live test endpoint, which publishes that same key. CI runs lint, typecheck, tests on Node 20, 22 and 24, and a smoke test of the built output from both ESM and CJS consumers.

What it does not do

Version 0.1.2 keeps no replay store. Nonces are format-checked, but a captured signature can be replayed until it expires. This is deliberate for a first version and it is a real limit: treat Web Bot Auth as identity, not as a transaction-level anti-replay mechanism, and keep signature windows short.

The SSRF protections reject loopback, private, link-local and unspecified addresses, in both IPv4 and IPv6 forms including IPv4-mapped ones. Those checks cover literals. A hostname that resolves to an internal address will pass them, so DNS rebinding remains a network-level problem that wants an egress proxy or firewall rules, not a string check.

Two things will cost someone an afternoon if they are not written down. The clockSkew option cannot widen the accepted time window, because the underlying library enforces created <= now <= expires strictly before the plugin’s own checks run — a signature one second past expires is rejected regardless of what you set. And behind a TLS-terminating proxy, Fastify sees http and possibly a rewritten Host, so the signature base no longer matches what the agent signed and every signature fails as bad-signature until Fastify is configured to trust the forwarded headers. Both are in the README’s threat-model notes for the same reason they are here: the failure is silent and the cause is not where you would look.

Outbound signing, rate limiting, and bot-detection heuristics are all out of scope. The plugin establishes who is calling. What to do about it is a policy question, and policy belongs in the application.

The source is on GitHub and the package is on npm, MIT licensed, published with npm provenance.