Camp·6d agoThe 2026 AI Crawler List: Every Documented Bot and Its PurposeAI companies send three kinds of bot to your site: training crawlers that collect content for future models, search crawlers that build the…
Shraddha Patil·Nov 2Sites Block the AI Training Bot 4 Times as Often as the AI Search Bot.Across 1,058 domains, 13.9% disallow GPTBot but only 3.4% disallow OAI-SearchBot. The gap tells you what site owners actually want.
Shraddha Patil·Oct 30Two Companies Added One New Line to robots.txt.Content Signals is the newest attempt to tell AI companies what they may do with your pages. I found it in Cloudflare’s and Zapier’s files…
Shraddha Patil·Oct 29Cloudflare vs Medium: Two Opposite robots.txt Files, Two Opposite Bets on What AI Is ForOne welcomes eight AI crawlers and serves a markdown copy of every page. The other blocks eight crawlers and keeps its stories behind the…
Shraddha Patil·Oct 27Medium Tells GPTBot and ClaudeBot to Stay Out.I read Medium’s own robots.txt. It is a small file with a big message for anyone who writes on the platform.
