How to verify GPTBot and ClaudeBot by IP, not user agent

Test the request’s IP against the ranges the operator publishes, not the user agent. OpenAI publishes one JSON file per crawler, and Anthropic publishes one for all three of its agents. An address inside a range proves the claim. An address outside one is weaker evidence than it looks, because the lists change and do not cover everything.

A user agent is a string the client writes, and anything can write it. Before this site checked addresses, 436 of its first 977 logged “crawler” requests, 44.6%, asked for paths that only a credential scanner asks for: /.env, /.ssh/id_rsa, /@fs/proc/self/environ. The log’s busiest “AI agent” was a scanner.

Where GPTBot, ClaudeBot and the other crawlers publish their IP ranges

Ten of the 30 agents this site logs come from operators that publish their egress ranges as JSON. Each file has a prefixes array of ipv4Prefix or ipv6Prefix entries in CIDR form.

Anthropic’s file is easy to miss. On 10 August I recorded that Anthropic published no ranges, after four guessed hostnames returned 404. The file was linked from Anthropic’s crawler support article the whole time, and I corrected the record the same day. Read the operator’s crawler documentation before you decide a list does not exist.

Two cases have no list of their own. Google-Extended is a robots.txt token, not a crawler, so a request whose user agent says Google-Extended is forged by definition; this site’s logger checks it against Google’s crawler ranges anyway, and all 8 such requests failed. This site has no range list for Meta, Apple, Amazon, ByteDance or the research crawlers, so their requests stay unknown. I have not checked every one of those operators for a list, and after the Anthropic mistake I do not assume there is none.

The lookup: a CIDR test with no network call

The check runs in this site’s middleware on every logged request, so it must cost nothing. The ranges are fetched by hand into a committed snapshot, and the request path does integer arithmetic only:

type Verdict = "verified" | "forged" | "unknown";

function verifyClaim(ip: string | null, bot: string, ranges: Record<string, string[]>): Verdict {
  const cidrs = ranges[bot];
  if (!cidrs?.length || !ip) return "unknown"; // no list, or no address: no evidence
  return cidrs.some((cidr) => ipInCidr(ip, cidr)) ? "verified" : "forged";
}

ipInCidr parses both sides to a bigint and compares the top bits bits. Two details matter. An IPv4-mapped IPv6 address such as ::ffff:203.0.113.9 must be unwrapped to its IPv4 form, or a proxied client never matches the IPv4 prefixes operators publish. And an IPv4 address must never be compared with an IPv6 prefix: as raw integers, 0.0.0.1 equals ::1.

There are three verdicts, not two. unknown means no evidence either way, and it is never counted as a pass or a failure.

What 78 days of verified requests show, per agent

IP logging went live on 10 August, and what the verified crawlers read is its own post. From then to 28 October, 78 days, the log holds 6,573 requests with an address. For the agents with published ranges, this is what the logger recorded:

The last row is the clearest result in the log. Before addresses were checked, Perplexity-User was the log’s largest agent: 347 requests, 321 of them for scanner paths. Its last request came on 9 August. In 78 days since, not one request has claimed to be Perplexity-User, real or forged. I report these per agent and not as one total, because a total measures which operators publish lists, not how much traffic is real.

The 81 forged requests came from 12 addresses. Two Google Cloud addresses sent 53 of them, in two bursts on 12 and 13 August, each wearing six crawler names in turn: ClaudeBot, GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot and Google-Extended. They asked for /.aws/credentials, /etc/passwd and /proc/self/environ. A scanner that rotates user agents does not care which one you trust; it is looking for the one you exempt from rate limits.

Why 7 of the 81 “forged” requests were not forgeries

I re-checked all 81 against the range files as they stood on 28 October, then looked up the owner of every address. Two groups do not fit the label, and a third fits it only narrowly.

A stale snapshot: 4 requests. This site’s snapshot was taken on 10 August and was 78 days old. Since then OpenAI’s ChatGPT-User file gained 33 prefixes and lost 54, and Anthropic’s file replaced 16 of its 20 prefixes with 10 new ones, leaving 14. Four ChatGPT-User requests on 19 and 27 October came from addresses in ranges added after the snapshot. They were real, and the logger called them forged.

The operator’s own unlisted network: 3 requests. Three Claude-User requests on 21 October, with Anthropic’s full user agent string, came from 160.79.106.0/24. Whois assigns 160.79.104.0/21 to Anthropic, PBC, and bots.json does not list it. Anthropic’s support article says an address on the list means the crawler is Anthropic’s. It does not say the reverse, and these three show why that matters. I cannot prove they were Claude-User, but “forged” claims more than I know.

A user agent no range list can cover: 1 request. One request today said Claude-User (claude-code/2.1.312) and came from a hosting provider’s address. Claude Code runs on its user’s own machine, so a fetch it makes leaves from that machine. I count it as forged, because it is not Anthropic’s infrastructure, but it is not an attack either.

The other 74 hold up. Most are the two scanner bursts. The rest include 14 requests in one second from an AWS address with the bare user agent Claude-User, asking only for MCP discovery paths such as /.well-known/mcp.json, and four /.env probes wearing Claude-User’s full string from a small hosting provider.

Three rules for verifying crawlers by IP

• A match is proof; a miss is a question. Treat verified as fact. Treat forged as “outside the list on the day of the snapshot”, and look up the owner of the address before you call it hostile. For agents that fetch on a user’s behalf, an outside address is common.
• Refresh the snapshot on a schedule. Four of the 81 errors came from a 78-day-old copy, and OpenAI’s ChatGPT-User file was regenerated the day before I checked. This site still refreshes by hand, which is how it got to 78 days. A rise in forged for an agent that was clean usually means a stale list, not a new attacker.
• Label, do not drop. Every row keeps its address and verdict. That is the only reason I could re-check 81 verdicts against today’s lists and find the 7 that were wrong. If I had blocked or deleted them on the day, the error would have stayed invisible.