What AI crawlers actually read: five verified weeks

AI crawlers mostly read my sitemap. In 36 days of verified traffic, ClaudeBot made 797 requests and 670 of them were robots.txt or sitemap.xml; it fetched a post 51 times. GPTBot fetched posts 43 times in 116 requests. Both picked up every new post within a day. A person asked an assistant something that fetched a page here 14 times.

Why the window is five weeks and not seven

The crawler logger went live on 29 July, so the calendar says seven weeks of data. I am using 36 days, from 10 August to 15 September, because that is when the middleware started recording the client IP and checking it against each operator’s published ranges. The 882 records before that date have no IP and can never be verified, and the first pass over them found 45% were a vulnerability scanner wearing a crawler’s user agent. Reporting seven weeks would mean adding two weeks I know to be polluted to five I can stand behind.

The verified segment holds 2,681 records with a verdict. 1,231 are verified: the IP sits inside a range the operator publishes. 67 are forged: the operator publishes ranges and the IP is outside all of them. The remaining 1,383 are unknown, which means the operator publishes nothing and the label says only that I could not check. Everything below is per bot, because an aggregate “verified share” would measure which companies publish IP lists, not how much traffic is real.

ClaudeBot reads the sitemap nine times a day

ClaudeBot is the largest verified source at 797 requests, 22 a day. The shape of those requests is the finding. 335 were robots.txt and 335 were sitemap.xml, usually as a pair a few seconds apart: nine sitemap checks a day, each with its robots check. The remaining 127 requests split into 41 for the homepage, 35 for section and tag pages, 32 for post HTML and 19 for the markdown twin of a post.

So the crawler is polling for change far more than it is reading. That is rational behaviour for a site that published twice a week through this window: the sitemap is 3.6 KB, a post page is 17 to 25 KB, and the sitemap tells it whether anything moved. Over the window ClaudeBot touched 19 of the 20 posts live at the time at least once, so the polling does convert into reads. It just converts at a ratio of 13 polling requests per post fetch.

GPTBot spends its requests on content

GPTBot made 116 verified requests, 3.2 a day, and the mix is inverted. 29 were post HTML, 14 were markdown twins, 21 were section pages, 13 the homepage and 39 the sitemap. It reached 14 distinct posts. Per request, GPTBot is the crawler most likely to be reading something I wrote rather than checking whether I wrote anything.

OAI-SearchBot, OpenAI’s search index crawler, behaves differently again: 150 requests, of which 123 were robots.txt. It fetched eight posts. A crawler that re-reads the robots file three times a day and content 14 times in five weeks is checking permission, not building an index, and that matches its role as the gate in front of ChatGPT search rather than the model’s training corpus.

Bingbot, the only conventional search engine on the list, made 140 verified requests and read 16 different posts. It is on the list because Bing’s index feeds Copilot, and it is the one crawler here whose reads I can tie to a click: Search Console reports 3 clicks in the same window.

New posts are fetched on publish day

Seven posts were published inside the window. For every one of them, the first verified AI fetch was ClaudeBot, and for six of the seven it came on the day of publication. The seventh was fetched the day after. GPTBot followed within the same week in all seven cases.

Bingbot reached two of the seven. The sitemap polling that looks wasteful in the previous section is what makes the zero-day lag possible. A site that wants AI systems to have its newest work has that already, for free, from the two operators that verify. The conventional search engine is the slow one.

The markdown twins get read, llms.txt does not

Every post here has a markdown twin at the same path with .md appended, published for agents that would rather not parse HTML. In the window, 39 verified requests were for a markdown twin: 19 from ClaudeBot, 14 from GPTBot and 6 from OAI-SearchBot. Twelve different posts were fetched that way. It is a small number, but it is not zero, and the two crawlers that read posts most are the two that use it.

llms.txt and llms-full.txt were fetched zero times by any verified bot, and zero times by any unverified one. I am holding a fuller write-up until the file has been live long enough for zero to be a finding rather than an early reading, so I will only note here that the markdown twins earned 39 reads in the same period the index file earned none.

Fifteen times a person was behind the request

Two user agents mean a human asked a question and the assistant fetched a page to answer it: ChatGPT-User and Claude-User. These are the requests that a citation strategy is for. In 36 days there were 15 verified ones, 14 of them for a page. ChatGPT-User fetched 12 pages and Claude-User 2. The pages were the ones I would hope for: the sdx page three times, the crawler-logging post twice, and single reads of the two skills-testing posts (testing Claude skills and testing skill triggers), the specdx dogfooding post and the spec-driven development essay. PerplexityBot, which is an index crawler and not a user fetcher, verified 13 times and forged 7. Perplexity-User, the largest single “bot” in the polluted pre-verification log at 347 records, has not appeared once since IP checks began.

The same two agents were forged 34 times. ChatGPT-User was forged more often than it was real, 20 to 12, and Claude-User 14 to 3. The forged requests went to /@fs/etc/passwd, /.env and AWS credential paths. Before verification shipped, those 34 hits would have tripled the count of “people reading via assistants”. The Web Bot Auth write-up covers why an IP list is the floor of verification and not the ceiling.

The biggest crawler cannot be checked

The single largest user agent in the raw log is meta-externalagent, at 823 requests, more than ClaudeBot. Meta publishes no IP ranges, so every one of those records is unknown. The same is true of PetalBot at 249, Amazonbot at 192 and Bytespider at 95. Together they are 1,383 records, over half the segment, and I can say nothing about how many are real. What I can say is where the unverified traffic went: 371 of the 419 requests for tag pages came from unknown-status bots, and meta-externalagent alone touched 44 distinct post and section URLs.

That is the limit of this kind of measurement. Four of the six largest sources are checkable and behave in ways I can describe. The other two outrank them and are opaque. Any post that reports AI crawler traffic without saying which operators it could verify is reporting the size of its own blind spot.

What five weeks changes about the plan

Three things. First, the sitemap is the interface, and its lastmod values are the signal ClaudeBot polls nine times a day, so a revision that does not update lastmod is invisible to the crawler most likely to notice it. Second, the markdown twins earn their build time and stay. Third, the citation metric I care about is the 14 user-triggered page reads, not the 1,231 verified requests, and that number is small enough that a single post being cited in one answer would move it. That is the number I will report next time, and the only one worth optimising for.