Logging AI crawlers on a static site

This site is fully static, so I had no record of which AI crawlers actually read it. Edge middleware now matches crawler user agents and writes each hit to storage, while human requests pass through untouched. The first 882 records then taught me something I had not planned for: the user-agent header is a claim, not evidence.

Why a static site has no visibility

Building fully static was a deliberate choice, and its trade-off is that there’s no request log you own. The host has access logs; you can’t query them the way this needs. For a site whose stated purpose is being read by AI systems, that’s flying blind on the one metric that matters. The local dev server had logged bot hits to .crawler-log/crawler-log.jsonl since the rebuild — production had been a blocker entry for two days short of the site’s whole life.

The design: match at the edge, write async

The production answer is routing middleware in front of the static output, built to three rules. First, humans pay nothing: a request whose user-agent matches no crawler short-circuits after one list scan — no storage call, no await. Second, bots never wait either: a matched hit is written fire-and-forget (the response returns while the write completes), one LPUSH onto a Redis list capped at 50,000 entries so a stampede can’t grow storage unboundedly. Third, the logger fails open: any thrown error, missing configuration, or slow dependency and the site just serves — an observability feature must never become an availability bug.

A daily cron drains the list oldest-first in batches of 500, each batch removed only after its contents are safely stored, into the same JSONL shape the local server writes — one analysis toolset for both sources. The record started as four fields: timestamp, raw user-agent, matched bot name, path. It is now six. The two I added later are the subject of most of this post.

Which user agents count as AI crawlers

The shared list started at twelve entries and now holds thirty, but the families matter more than the count: training crawlers (GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended) collect for model training; live retrieval fetchers (ChatGPT-User, Claude-User, Perplexity-User) fetch a page because a user asked something right now; and AI search indexers (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Bingbot) build the indexes AI search answers from. A training hit is a bet on future recall; a retrieval hit is a citation happening in real time.

I closed that section, when I first wrote it, with a caveat: everything here is user-agent-declared and unverified, anything can claim to be GPTBot, so treat the counts as indicative rather than audited. I wrote that as a disclaimer. It turned out to be the finding.

Nearly half the log was a scanner in a costume

Thirteen days in, the log held 882 records and I finally looked at the paths rather than the bot names. The largest single “crawler” in the dataset was Perplexity-User, at 347 hits. Of those, 321 asked for /.env, /.ssh/id_rsa, /credentials.json, /@fs/proc/self/environ, and eighty-odd variations on the same theme. Perplexity does not want your SSH key. That is a vulnerability scanner wearing Perplexity’s name.

Sorted by how many of each bot’s requests went to paths only a credential scanner asks for:

Across every bot, 394 of 882 records — 45% — were this. My fetch-side instrument, the one built specifically to answer “is any of this working?”, was 45% noise, and I had been reading its totals for two weeks.

Note what that table is and is not. It sorts by path, which is a heuristic: a strong one, since nothing legitimate requests /.ssh/id_rsa, but still an inference rather than proof. It is the best that can be done retrospectively, because those 882 records store no client address. That is the honest reading and I would rather state it than round it up.

Verifying the claim instead of trusting it

The fix is not clever. Major crawler operators publish the IP ranges they crawl from — OpenAI, Perplexity, Microsoft and Google all serve a JSON file of prefixes. Snapshot those into the build, and verifying a user-agent claim becomes a CIDR containment test against the client address: no DNS lookup, no network call, nothing added to the response path. Ten bots and 1,491 prefixes at the time of writing.

Each record now carries the client IP and a verdict. verified means the address sits inside a range the claimed operator publishes. forged means the operator publishes ranges and the address is outside all of them — a positive finding, not a shrug. Everything else is unknown, which means absence of evidence and nothing more.

Two decisions I would defend. Forged records are labelled, not dropped. A false claim is a fact about the world, and deleting the rows would make the forgery rate itself unmeasurable — the exact mistake that let this sit undetected for two weeks. The verdict never guesses upward. A missing address, an unparseable one, or an operator with no published ranges all read unknown, never verified.

Which leads to a mistake worth recording, since this post is otherwise about not trusting claims. I first wrote this section stating that Anthropic published no range file, on the strength of four plausible URLs that all returned 404. Anthropic publishes one. It is at claude.com/crawling/bots.json, linked from their crawler documentation — which I had not read. Guessing hostnames is not a search, and I made that error on the single operator where it mattered most: ClaudeBot is the largest genuinely-behaving crawler in this log at 260 hits, with a 2% scanner share. It verifies now, along with Claude-User and Claude-SearchBot.

The wider point survives the correction, though. Ten of the thirty agents on the list publish ranges; Meta, Apple, Amazon, ByteDance and the research crawlers do not, and read unknown by construction. A single “verified share” figure for this site would therefore measure which operators publish their IP ranges, not how much of the traffic is real. The numbers only mean anything per-bot.

The first fourteen hours

The numbers in this section predate verification, so read them as user-agent-declared. Their paths are clean — reconnaissance and content, no credential probing — which is weak evidence for them being genuine, not strong evidence.

The logger went live on 29 July, and the first 14.5 hours logged 31 genuine crawler hits (my own verification requests excluded): ClaudeBot 17, GPTBot 10, Bingbot 2, OAI-SearchBot 2. The shape of the traffic is the interesting part. Nineteen hits — over 60% — were reconnaissance: robots.txt eleven times, sitemap.xml eight. Six content pages got actual reads, the spec-driven development essay first among them. One crawler requested a blog URL from the 2021 version of this site, retired a year of redesigns ago and absent from even the redirect map — crawler memories are long, and by August eleven such URLs were still being requested often enough that I gave them redirects. And llms.txt, the file this site maintains specifically for AI consumption: zero fetches so far. That last number gets its own post when there’s enough data to be fair about it.

What it costs

Per human request: one substring scan over thirty entries, no I/O. Per bot request: the same scan, a CIDR test against the compiled-in prefix snapshot, and an async write the response doesn’t wait for. Verification is pure arithmetic on two integers, which is why it could go in the request path at all — the moment it needed a DNS lookup it would have belonged somewhere else. At the cap, storage tops out around 12 MB — comfortably inside a free Redis tier — and the cap plus fail-open design means the worst realistic failure is lost log lines, never a slow page. The whole system is one middleware file, one shared bot list, and one export script.

What I’ll do with the data

Three posts consume this pipeline: a data report after roughly ten weeks of accumulation, the llms.txt fetch-rate question, and the year-end retrospective. The experiment shipped nineteen days ahead of its calendar slot precisely so those posts would have something worth reporting — data compounds, and the cheapest way to have ten weeks of it in October is to start logging in July.

All three now carry a caveat I did not expect to write. Verification landed on 10 August, so every record before that date is unverified and unverifiable, and any comparison spanning that date has to be segmented on it or the change in the numbers is just the instrument changing. That is the second lesson here, and it is the more general one: an instrument is not trustworthy because you built it carefully. It is trustworthy once you have tried to find out how it lies.