Camp·6d agoThe 2026 AI Crawler List: Every Documented Bot and Its PurposeAI companies send three kinds of bot to your site: training crawlers that collect content for future models, search crawlers that build the…
Shraddha Patil·Nov 2Sites Block the AI Training Bot 4 Times as Often as the AI Search Bot.Across 1,058 domains, 13.9% disallow GPTBot but only 3.4% disallow OAI-SearchBot. The gap tells you what site owners actually want.
Shraddha Patil·Oct 30Two Companies Added One New Line to robots.txt.Content Signals is the newest attempt to tell AI companies what they may do with your pages. I found it in Cloudflare’s and Zapier’s files…
Shraddha Patil·Oct 29Cloudflare vs Medium: Two Opposite robots.txt Files, Two Opposite Bets on What AI Is ForOne welcomes eight AI crawlers and serves a markdown copy of every page. The other blocks eight crawlers and keeps its stories behind the…
Shraddha Patil·Oct 27Medium Tells GPTBot and ClaudeBot to Stay Out.I read Medium’s own robots.txt. It is a small file with a big message for anyone who writes on the platform.
Shraddha Patil·Oct 26I Read the robots.txt Files of 14 Well-Known Tech Sites. Only 2 Block AI Crawlers by Name.Nine of them say nothing about AI at all. Here is what each company’s file actually says, and what the silence means.
Beste·Oct 25Should You Block AI Crawlers? robots.txt for Business SitesFor most small business sites, the answer is: allow the search bots and the user-triggered fetchers, and decide separately about the…
Camp·Oct 24noindex vs robots.txt: Which to Use and the Mistake to Avoidnoindex is for keeping a page out of results; a robots.txt Disallow is for stopping crawlers from requesting URLs. The two aren’t…
InQueryRev ResearchbyQueryRev·Oct 23State of AI crawler access, October 2026: what 9,973 top sites tell the 11 crawlers we measuredThe robots.txt and llms.txt of a fixed panel, on three denominators. Open data, CC BY 4.0.
Yudha Pratama·Oct 22Web Security Day 2 Part 1: PortSwigger Writeup — Unprotected Admin FunctionalityHello everyone! Continuing my Web Security journey on day one, I’m sharing my writeup for the second PortSwigger lab: Unprotected Admin…
