# The user-agent header is a claim, not evidence

Nearly half of the first 882 crawler hits logged on this site were forged, because a user-agent string is something any client can assert. Web Bot Auth answers it with Ed25519 signatures over RFC 9421, checked against keys the agent publishes at its own domain. fastify-web-bot-auth is the Fastify plugin Umar Gora built to verify those signatures at the origin server.

Published: 2026-08-22
Canonical: https://umar.codes/fastify-web-bot-auth

## Revisions

- 2026-08-22 — Created.
- 2026-08-22 — Published. Corrected one claim before it went live: the suite signs with the RFC 9421 Appendix B.1.4 Ed25519 test key, it does not run the full B.1.4 vector set.

---

A user-agent header is a string the client chooses. A Web Bot Auth signature is
a proof the client cannot fake: Ed25519 over the request, verified against a key
the claimed agent publishes at its own domain. `fastify-web-bot-auth` is the
Fastify plugin I published to check that at the origin.

## What 882 crawler records showed

When I [started logging AI crawlers on this site](/logging-ai-crawlers), I
matched them the ordinary way — a list of user-agent substrings, checked at the
edge. The first 882 records made the flaw obvious. Around 45% of them requested
credential paths that no crawler has any reason to want. They were not crawlers.
They were scanners wearing a crawler's name, because the name costs nothing to
wear.

The partial fix was to check the client IP against the range files operators
publish. That works, and it now covers ten operators. It has two limits worth
naming. It only works for operators who publish a file at all, and it makes me
responsible for keeping ten lists fresh. Every new agent vendor is another
list I do not have.

Both approaches share a shape: I am inferring identity from things the request
happens to carry. What I actually want is for the caller to prove who it is.

## What a Web Bot Auth signature proves

Web Bot Auth is an IETF draft built on RFC 9421, HTTP Message Signatures. The
agent signs a defined set of request components with an Ed25519 key, sends the
signature in `Signature` and `Signature-Input` headers, and names itself in a
`Signature-Agent` header — an https origin such as `https://chatgpt.com`.

The origin is the interesting part. The key is not distributed by me, or by a
registry I have to trust. It is fetched from the agent's own domain, at
`/.well-known/http-message-signatures-directory`, and selected by JWK
thumbprint. So a request claiming to be ChatGPT must carry a signature made
with a private key whose public half `chatgpt.com` is publishing right now.
Claiming the name is still free. Producing the signature is not.

It is worth being precise about what this does *not* establish. It proves key
possession for a domain. It says nothing about whether the agent behaves well,
respects `robots.txt`, or is doing something you want. It replaces a guess about
identity with a fact about identity, and leaves policy entirely to you.

## Why the check belongs at the origin too

OpenAI, Google, and Amazon already sign agent requests in production. Cloudflare,
Vercel, and Akamai already verify them at the edge. If you sit behind one of
those, much of this is handled for you.

I did not want the capability to be a property of my hosting choice. If
verification only exists at the CDN, then it disappears when you move hosts, it
is absent in local development, and it is unavailable to anyone serving directly.
Cloudflare publishes a low-level `web-bot-auth` library for exactly this reason,
but it is framework-agnostic: you still write the hook, and you still own the key
directory logic, which is where the failure modes live. Express has a community
verifier. Fastify had none, which is the gap this fills.

## Observe mode is the default because enforcement breaks traffic you cannot see

The plugin never blocks anything until you ask it to. Registered with no options,
it adds one `onRequest` hook that decorates every request with a verdict and gets
out of the way:

```ts
interface WebBotAuthResult {
  verified: boolean;   // signature verified against the agent's own directory
  agent?: string;      // https origin from Signature-Agent
  keyid?: string;      // base64url JWK SHA-256 thumbprint
  trusted?: boolean;   // your trust policy's answer; only set when verified
  reason?: string;     // why it failed; one of six values
  elapsedMs: number;
}
```

Verification never throws into the request lifecycle. Every failure becomes one
of six `reason` values — `unsigned`, `expired`, `bad-signature`, `unknown-key`,
`directory-unreachable`, `malformed` — so a broken signer degrades into a label,
not a 500.

The default matters more than it looks. Ordinary human browser traffic is
unsigned, so it arrives as `reason: 'unsigned'`, which is indistinguishable at
the header level from a scraper. Any origin that switches on enforcement
globally without looking first will reject its own readers. Observe mode exists
so that the first thing you do is watch: log the verdicts, learn which agents
call you and how they fail, add a trust policy while still not blocking, then
enforce on individual routes via route config before you consider enforcing
globally. Enforcement is the last step, not the first.

## The hard part is the key directory, not the crypto

I did not write any cryptography. Ed25519 verification is WebCrypto underneath
Cloudflare's library, and reimplementing it would be the least defensible line
of code in the project. Nearly all of the work went somewhere less glamorous:
fetching, caching, and rotating other people's keys without turning that into
a way to hurt the server.

Directories are cached per origin, with the TTL taken from `Cache-Control` and
clamped to a range of 60 seconds to 24 hours, defaulting to one hour. Stale
entries are served while a refresh runs in the background, and concurrent
refreshes for the same origin are single-flighted. That stale window is itself
capped at 24 hours past the last successful fetch — after that the keys are
treated as unreachable rather than served forever, because "the signer's
directory has been down for a week" should not silently keep authenticating
requests.

The adversarial cases shaped the rest. An unknown `keyid` triggers one forced
refresh, which is how key rotation is meant to work — but a fabricated `keyid`
would otherwise turn every request into an outbound fetch, so forced refreshes
are throttled to one per origin per 30 seconds. Failing origins are
negative-cached for 30 seconds so a dead directory cannot slow every request
down. The number of tracked origins is capped at 1000 with oldest-first
eviction, so spraying invented `Signature-Agent` values cannot grow memory
without bound. Directory responses are capped at 64 KB, redirects are followed
same-origin only for at most three hops, and the fetch timeout covers the body
read rather than just the headers.

The suite that holds this together is 93 tests at 97.8% line and 92.4% branch
coverage, signing with the RFC 9421 Appendix B.1.4 Ed25519 test key. Two further
tests are network-gated and run weekly rather than on the pull request path,
verifying real signatures against Cloudflare Research's live test endpoint,
which publishes that same key. CI runs lint,
typecheck, tests on Node 20, 22 and 24, and a smoke test of the built output
from both ESM and CJS consumers.

## What it does not do

Version 0.1.2 keeps no replay store. Nonces are format-checked, but a captured
signature can be replayed until it expires. This is deliberate for a first
version and it is a real limit: treat Web Bot Auth as identity, not as a
transaction-level anti-replay mechanism, and keep signature windows short.

The SSRF protections reject loopback, private, link-local and unspecified
addresses, in both IPv4 and IPv6 forms including IPv4-mapped ones. Those checks
cover literals. A hostname that resolves to an internal address will pass them,
so DNS rebinding remains a network-level problem that wants an egress proxy or
firewall rules, not a string check.

Two things will cost someone an afternoon if they are not written down. The
`clockSkew` option cannot widen the accepted time window, because the underlying
library enforces `created <= now <= expires` strictly before the plugin's own
checks run — a signature one second past `expires` is rejected regardless of
what you set. And behind a TLS-terminating proxy, Fastify sees `http` and
possibly a rewritten `Host`, so the signature base no longer matches what the
agent signed and *every* signature fails as `bad-signature` until Fastify is
configured to trust the forwarded headers. Both are in the README's threat-model
notes for the same reason they are here: the failure is silent and the cause is
not where you would look.

Outbound signing, rate limiting, and bot-detection heuristics are all out of
scope. The plugin establishes who is calling. What to do about it is a policy
question, and policy belongs in the application.

The source is on [GitHub](https://github.com/umxr/fastify-web-bot-auth) and the
package is on [npm](https://www.npmjs.com/package/fastify-web-bot-auth), MIT
licensed, published with npm provenance.