Infrastructure
The plumbing behind this site's crawler data: logging bots on a static site, verifying who they really are, and what they fetch, including whether anything reads llms.txt.
- Does anyone read llms.txt?
- What AI crawlers actually read: five verified weeks
- The user-agent header is a claim, not evidence
- Logging AI crawlers on a static site●