AI crawler access
For each known crawler (OAI-SearchBot, GPTBot, ClaudeBot, PerplexityBot, Google-Extended, Bingbot and a dozen more), the tool reads robots.txt according to RFC 9309: allowed or blocked. It also checks noindex, nosnippet and noai directives, then whether the server refuses visits that announce themselves as AI crawlers.