Infrastructure
The plumbing behind this site's crawler data: logging bots on a static site, verifying who they really are, and what they fetch, including whether anything reads llms.txt.
- Web Bot Auth from the verifier's side: what a signature covers
- Three months of Search Console on an essay-led site
- How to verify GPTBot and ClaudeBot by IP, not user agent
- Does anyone read llms.txt?
- What AI crawlers actually read: five verified weeks
- The user-agent header is a claim, not evidence
- Logging AI crawlers on a static site●