<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Pete on Medium]]></title>
        <description><![CDATA[Stories by Pete on Medium]]></description>
        <link>https://medium.com/@petedainty?source=rss-37b7b3f873ec------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*kKeeUng9prXNdZnzV3vFxg.png</url>
            <title>Stories by Pete on Medium</title>
            <link>https://medium.com/@petedainty?source=rss-37b7b3f873ec------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 22:41:40 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@petedainty/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Stop that, it’s silly!]]></title>
            <link>https://medium.com/@petedainty/stop-that-its-silly-7b31ce106e60?source=rss-37b7b3f873ec------2</link>
            <guid isPermaLink="false">https://medium.com/p/7b31ce106e60</guid>
            <category><![CDATA[search-engines]]></category>
            <category><![CDATA[seo]]></category>
            <category><![CDATA[geo]]></category>
            <category><![CDATA[digital-marketing]]></category>
            <category><![CDATA[aeo]]></category>
            <dc:creator><![CDATA[Pete]]></dc:creator>
            <pubDate>Thu, 19 Nov 2026 01:22:47 GMT</pubDate>
            <atom:updated>2026-11-19T03:54:54.633Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SYkFrB-pQEdNAqB1vzAN6g.png" /><figcaption>Graham Chapman as The Colonel in Monty Python’s Flying Circus (1969–1974). © BBC / Python (Monty) Pictures Ltd.</figcaption></figure><h4>There are too many people cosplaying SEO dressed up as GEO.</h4><p>On the panel at this week’s Bay Area Search event, we were asked whether generative engine optimisation (GEO) is new or mostly good SEO. We were all on team it’s-just-good-SEO. Denis Yevseyev on the panel had a really good point that at best it should be treated more like an iteration than anything new.</p><p>This topic has been discussed a lot already in much detail, but, I thought I would write up my opinion on this as well because why not. Pedro Dias published <a href="https://theinference.io/p/geeeo-was-invented-on-sand-hill-road?utm_source=chatgpt.com">GEO Was Invented on Sand Hill Road</a> a while back which is a good read along with <a href="https://dejan.ai/blog/geo-to-seo/">Another failed attempt to kill SEO</a> and <a href="https://dejan.ai/blog/what-is-geo/">What is GEO?</a> by Dan Petrovic. Both are people I respect and trust.</p><p>Now, if people want to pay me to do SEO but they want to call it GEO then that’s fine, I still get paid. But, for intellectual honesty please do not pretend that this is anything new, unless you are new to all of this, then all is instantly forgiven - we’re all here to learn after all.</p><p>What I’m concerned about are people being duped by what feels like the return of the snake oil peddlers which will only serve to bring back the reputational harm that took the SEO industry a long time to lose. But hey, if I can get away with charging you 5x the cost by calling what I do GEO then you’d better believe I will do that… probably… I actually don’t think I have it in me to do that but never say never.</p><p>My cynicism here is not to be confused with the people who genuinely know what they’re doing when it comes to optimising for search, LLMs and agents and who are typically the SEO experts worth every penny.</p><h4>Think of it like this…</h4><p>Google, Bing, Baidu, Naver, Yandex etc. are all search engines that use different technologies, ranking algorithms, retrieval techniques, output formats and so on, yet the same SEO techniques apply to all of them. The same can be said for optimising in marketplaces such as Amazon, eBay, Etsy, Vinted etc. and many, many other sites. It’s still called SEO.</p><p>ChatGPT, Claude, Perplexity, Google AI Mode etc. are simply other variations in this existing area. Ultimately, they’re searching corpora of documents to return an answer to someones question. But why should this necessitate inventing an entirely new optimisation discipline? If they were mutually exclusive then there wouldn’t be the substitution behaviour that we see along with Google feeling an existential threat from it all. And, if it is so different then why is OpenAI building its own search engine? Why do they use other search engines for answers? All it’s really doing is presenting the answers to your query and filtering in a different format instead of the ten blue links that you used to have to click to find that awesome muffin recipe.</p><p>I’m also here to tell you that LLMs are ubiquitous and can be, have been and are used in search engines for things like better query understanding, for helping backfilling sparse data, creating proxies for retrieval, aspect extraction and so on and so on. So I guess if GEO is only for LLMs, then by using SEO techniques on search engines that use LLMs in their logic, am I then not also doing GEO? Logically they are the same thing and SEO was here first.</p><p>If GEO is a truly independent discipline then what techniques and foundational skills are exclusive to it? Some may emerge but right now I don’t see it.</p><p>The most charitable way to describe this would be not as a new discipline but as an iteration on the solid foundations that already exist, as Denis said and which I am stealing.</p><h3>So where did all this nonsense come from?</h3><p>The term GEO came from a paper written by 6 people with no SEO backgrounds and who, contrary to the evidence that they themselves present, decided to represent SEO as only keyword stuffing despite that being penalised now for decades. They argue that traditional SEO doesn’t apply because generative models aren’t limited to keyword matching, yet no modern search engine or SEO technique is limited to keyword matching either and hasn’t been for over a decade.</p><p>Yet, they go to great lengths to describe the very techniques that SEO uses as something new. Somebody really should have told them about semantic search, embeddings, neural ranking, technical SEO etc. .</p><p>They must have thought: “GEO? Yeah that’s never been used for anything, it’s perfect!” I mean.. you could have at least come up with a better acronym instead of just changing the S to a G. SEO isn’t a great acronym to start with so building on that is already dicey to say the least. It reminds me of when people used to rush to be the first to give a name to the latest Google algo change.</p><p>It just reeks of wanton ignorance and desperation.</p><p>Continuing with ignorance, over at a16z they decided that it didn’t really matter if the paper was right or wrong when they wanted to hype the fuck out of something. They explicitly said as much in their own article, describing GEO as a way to capture marketing budgets and a stepping stone into the wider performance marketing market.</p><p>So we’ve gone from a questionable academic paper claiming to invent a new optimisation discipline, to venture capitalists promoting it as a new commercial category. All without establishing why the underlying work isn’t already SEO. Thanks!</p><h3>My rebuttal</h3><p>Here’s my review of some of the points in <a href="https://arxiv.org/pdf/2311.09735">the original GEO paper</a> and rebuttal, page by page, section by section and why I think it’s wrong.</p><h4>Page 1</h4><p><strong>Introduction</strong></p><p>Here the paper describes search engines as being limited to a list of relevant websites. This overlooks features such as answer boxes, direct answers, blended search, knowledge panels and more zero click stuff than you can shake a stick at. It’s not a fair baseline for comparison in any way and even five minutes of research would show this. This feels like the bias of a document written to support a predetermined outcome. The scientific method is lacking.</p><h4>Page 2</h4><p><strong>Introduction continued.<br></strong>The paper describes certain methods as GEO but are actually key parts of SEO. Again, five minutes of research would have confirmed this.</p><p>The paper states things like tailoring page presentation, writing style and content to increase visibility as being core disciplines of GEO. Or, as we like to call it: Good editorial, UX, on page SEO, content optimisation and experimentation.</p><p>Adding credible citations, quotations and quantitative statistics is apparently also the purview of GEO. Or, in SEO land we like to call that establishing editorial quality, E-E-A-T related practices and content improvement and have done so for decades now.</p><h4>Page 3</h4><p><strong>Section 2.1</strong></p><p>They make technical characterisations of a system with no evidence to prove that SEO can’t address them. However, the paper’s own model of a generative engine explicitly includes a search engine retrieving ranked sources… I wonder how those sources got found in that search engine?</p><p><strong>Section 2.2</strong></p><p>The paper says traditional SEO methods are said not to apply because generative models aren’t limited to keyword matching, yet modern search engines and SEO are also not limited to keyword matching and use methods such as semantic search and ML relevance which predate this paper.</p><p>They talk about optimising website content to improve visibility and impressions as part of GEO which sounds exactly like SEO.</p><h4>Page 4</h4><p><strong>Figure 3</strong></p><p>They use this to characterise traditional results as being ranked websites with verbatim content. Anyone who’s done SEO for a couple of minutes knows that snippets, for example, are selected and can also be rewritten. SERPs frequently aggregate and extract answers so figure 3 here is a very simplified comparison.</p><p><strong>Section 2.2.2</strong></p><p>Keyword stuffing is labeled as expected in classical SEO but it’s been a deprecated spam technique for a long time and in no way serves as a credible positive SEO baseline to be used. This says more about what people with no knowledge of SEO think that SEO is.</p><p>The issue here also isn’t only that keyword stuffing is outdated, it’s that they use a known bad SEO technique as evidence of the limitations of SEO while classifying everything else SEO does as GEO.</p><p>They go on to talk about authoritative tone, statistics, sources, quotations, readability, fluency, vocabulary and technical terms as part of GEO. All of these are content editing techniques around for centuries and aren’t exclusive to generative systems. SEO utilises exactly the same techniques.</p><p>What’s that? They say GEO is a black box website optimisation without knowledge of exact algorithm? That sounds familiar…</p><h4>Page 5</h4><p><strong>Benchmark</strong></p><p>Here they split queries by domain, intent, informational, navigational and transactional types… just like SEO has done for decades. Same discipline.</p><p>They used the top five Google result text as benchmark sources which makes for a heavily limited and fixed source pool preventing testing discovery of a previously absent page.</p><p>What was also interesting is that it shows the researchers are optimising a downstream stage of a pipeline that already depends on search and therefore SEO will have played a role there.</p><h4>Page 6</h4><p><strong>Results</strong></p><p>They compare content changes using impression and position metrics for GEO… which are established SEO experimentation principles.</p><p>Their 30 - 40% improvement is in metrics that they created to measure their definitions of citation exposure. It is not linked to 30 - 40% more traffic, conversions, organic discoverability, ranking etc. So while they present it as a significant visibility improvement, their metrics haven’t been validated commercially.</p><h4>Page 7</h4><p><strong>Results</strong></p><p>Keyword stuffing is said to be widely used for SEO, with poor GEO performance treated as evidence for their new paradigm. This only demonstrates a catastrophic lack of knowledge of a known bad SEO tactic and does not demonstrate a new discipline.</p><p><strong>Analysis</strong></p><p>They reference making domain specific website edits and evaluating changes by ranking and query types as GEO, yet this is standard SEO practice… I mean… come on! Really?!</p><h4>Page 8</h4><p><strong>Analysis</strong></p><p>The authors also described testing their methods against Perplexity, presenting this as evidence that their methods generalise to real systems.</p><p>In Appendix C.1 it states that they uploaded source text as files into Perplexity and instructed it to answer using only those supplied sources. This means that their Perplexity evaluation didn’t test whether changing publicly accessible website content improved the probability of being discovered or retrieved through organic web search… which would also typically be the outcomes of existing SEO techniques.</p><h4>Page 9</h4><p><strong>Related Works</strong></p><p>Decades of SEO here is reduced to on page content and UX along with off page link building and generative models being contrasted with keyword matching. This omits things like technical SEO, crawling, rendering, architecture, structured information, entities, ML ranking and answer optimisation among many, many other techniques. To me this is one of the most damning.</p><p>The paper also cites SEO surveys but has very little of the technical SEO and modern retrieval information relevant to their claims made.</p><p><strong>Conclusion</strong></p><p>They recommend citations, quotations and statistics as GEO objectives but these are editorial content quality practices and aren’t novel techniques.</p><p><strong>Limitations</strong></p><p>Here the paper describes text edits as unlikely to affect traditional rankings because they do not affect domain names and/or backlinks - yes, really! They wrote that! This is unmeasured and poorly supported as relevance, clarity, statistics and source references for example can affect search outcomes without backlink changes.</p><p>They also give no evaluation of changes to traditional search ranking from using the same techniques. The authors suggest that their changes are unlikely to affect traditional search rankings because they don’t modify domain names or backlinks… but somehow them deliberately changing website content isn’t SEO.</p><h3>So what’s new here?</h3><p>The researchers took content that had already been selected by a search engine, so it already likely benefitted from SEO to get it in that position for selection. They then made changes to its writing, statistics, citations, quotations etc. and measured whether it became more prominent in an AI generated answer but crucially without testing that on publicly available content already within the search engine.</p><p>They simply demonstrated that changing content can change how a search system might choose to present it in isolation of every other factor, which is hardly surprising for anyone who’s worked in SEO.</p><p>What they didn’t demonstrate is that these techniques are exclusive to generative systems and that they don’t work in traditional search and so require a new discipline.</p><p>They claim that modifying source content can affect how prominently it’s represented in an LLM generated response and that their findings demonstrate the need for a distinct optimisation discipline called GEO. Yet, their research only proves the first half of that, which is what SEO does.</p><p>If you’re selling GEO as something fundamentally different from SEO then show us the techniques that make it different, the evidence that they work and why those techniques aren’t already covered by SEO.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7b31ce106e60" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Don’t go chasing ranking factors]]></title>
            <link>https://medium.com/@petedainty/dont-go-chasing-ranking-factors-282fbf486e0d?source=rss-37b7b3f873ec------2</link>
            <guid isPermaLink="false">https://medium.com/p/282fbf486e0d</guid>
            <category><![CDATA[digital-marketing]]></category>
            <category><![CDATA[seo]]></category>
            <category><![CDATA[ecommerce]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[user-experience]]></category>
            <dc:creator><![CDATA[Pete]]></dc:creator>
            <pubDate>Tue, 10 Nov 2026 08:58:57 GMT</pubDate>
            <atom:updated>2026-11-11T03:46:22.131Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vb08wvkKz-jJC8n82XBhRQ.png" /><figcaption>AI generated image of someone happy with their positions while everyone else goes to the much nicer shop just down the road.</figcaption></figure><h4>Ranking factors are mechanisms not objectives. If you’re optimising for the mechanisms instead of understanding the objectives then you’re probably solving the wrong problems.</h4><p>There are countless studies and insights on the factors search engines use to rank a document. Most of the factors documented out there are either guessed, crowd sourced, gleaned from patents and in the case of Google, leaked.</p><p>What their values are, how important they each are and how they’re mixed together are largely unknown, most often to those outside the search engines looking in but also in a lot of cases by those working directly on them. In my time working at eBay on their search engine, I knew a few of the ranking factors but there are thousands of them, more than most can comprehend. There are different ways for retrieving items, relevance scoring, query understanding, matching, manual weightings, machine learnt models and so on. They all eventually get blended up with the context available to produce the search results and that context is also made up of many different things.</p><p>The point is that even if I knew all the factors, the best values to use and optimised for them, I would still be unsure of ranking at the top. It might give me an advantage but that would depend on a lot of other things also going my way that are outside my control such as the query, customer, location, competition, personalisation and countless other things that also influence the outcome.</p><p>I could give two sellers exactly the same advice and both could implement it perfectly. One might see a substantial improvement while the other sees nothing, but neither outcome necessarily means the advice was wrong.</p><h4>What’s the point of ranking factors anyway?</h4><p>They’re there to help the system determine which of the results best satisfy whatever its objectives that it’s trying to optimise for.</p><p>A commerce sites search results rankings may favour products that are more likely to convert. A social media site might reward more addictive experiences to sell more advertising. An internet search engine might favour relevance and trust so more people will come back and use them, whether that’s sending people out to a site that answers their query or trying to keep everyone in their ecosystem so they can sell more ads.</p><p>The individual ranking factors are switches and levers to enable the outcome based upon whatever it is that the companies are optimising for.</p><p>If we think in terms of first principles, then the individual ranking factors themselves matter less.</p><p>In a commerce search engine that prioritises conversion, I can keyword stuff my titles all I want but if no one buys my product then I may end up lower in the rankings over time because the system might learn that buyers consistently prefer something else.</p><p>Of course, if my product sucks then I can always buy my way to the top and then hope that brand recognition, engagement and volume take over as bigger signals in the ranking algorithms - it certainly doesn’t hurt. If your brand is big enough then people will often distrust results that don’t include it. If I’m searching for TVs for example then I’d expect to see Samsung, LG etc. if I didn’t then I would think there was some mistake.</p><h4>Start with those pesky humans</h4><p>My first priority for SEO/AEO or whatever you want to call it would be to ignore Bing, Google etc. and focus on my brand and site experience. Does it suck? Can people use it, do they hate it, why would they trust me or buy from me versus someone else? Do I have all the data? Am I offering anything unique?</p><p>At no point am I thinking about individual ranking factors. I’m trying to build something people actually want to find, use, trust and come back to, whether an algorithm recognises that is a separate problem. That might be a naive view assuming that’s what the ranking factors are there to do, and not to push a business priority even if it is worse for the customer. If nothing else then I have a good site and brand.</p><p>One thing at eBay that struck me is how people would optimise their titles with whatever they thought would improve rankings. Titles themselves are very important but no one is ever going to search for a ‘not Samsung’ or a ‘L@@K’ and a lot of the time those lengthy titles get cut off in the experience so you have to front load all the most important stuff if you want it read.</p><p>They would then fail to describe their product adequately in the specifications and attributes. In a high percentage of sessions people choose to requery or filter which works better when all the different aspects of an item are available.</p><p>Missing attributes don’t just make filtering harder, they can also affect matching, retrieval, relevance and whether a product is eligible for certain queries or filters to begin with. To find relevant inventory we had to come up with ways to extract attributes from the sparse information provided whether that was from the titles themselves, images, catalogue, encouraging it when listing an item or numerous other methods.</p><p>If you’re selling an iPhone then customers are interested in the model, storage, network, condition and price. It’s how people shop and the system is based upon this, so you’re rewarded for providing the data that enables it. You don’t need to understand the search engine itself in order to make meaningful improvements.</p><p>Another thing was when people would copy and use the same image from another seller or brand or fill up their images with various logos. Typically it drove lower click through rates and made the experience look worse overall. I could often find listings getting more clicks that were at the bottom of the results but had a cleaner unique image. Guess what then happens to the listings over time when this data is factored in?</p><p>Then people would try to game the system by adding extra keywords or use other tricks to compensate and some of them do work for a while (and no, I’m not going to give away the tricks used there!).</p><p>Put simply, the more people optimise for ranking factors rather than the underlying outcomes, then the more ranking systems may need to adapt to counteract that behaviour.</p><h4>I’m not arguing that SEO or AEO etc. don’t matter</h4><p>It’s not that you shouldn’t experiment with improvements, far from it. I’m saying not to get hung up on optimising for the latest list of ranking factors you’ve seen and instead think about the outcome they’re trying to achieve and focus on the humans (or AI agents) using your site that the ranking factors are in service of.</p><p>There’s a difference between testing changes that produce measurable outcomes and obsessing over whether a particular factor contributes to some unknown ranking calculation that you’ll never be able to observe behind the curtain.</p><p>I’d rather put my energy into building a brand and experience people actively seek out. That means advertising, partnerships, sponsorships, endorsements, PR and whatever else makes sense for the business. That costs money and a lot of time but if people know my brand, trust it, recommend it and come back, I’m less reliant on Google deciding to send me traffic and I’m also more likely to be recognised, searched for and chosen in Google itself (and Bing, ChatGPT etc. etc.).</p><p>There’s also the issue of whose objectives that we’re optimising for. For example, Google can decide what it wants to reward in its search results, but that doesn’t make its preferences the universal definition of a good website. My customers probably have different, better ideas.</p><p>This is also potentially a great example of Goodhart’s law: Search engines create signals intended to reward useful results, people optimise for those signals but the signals could then become less useful measures of what customers actually want.</p><p>I’d rather optimise for the outcomes I can measure, test and influence rather than spend my time guessing how somebody else’s black box evaluates them.</p><h4>That all said, sometimes search engines do need a little help</h4><p>I think there’s always a role for technical SEO or what I’d increasingly describe as bot management and optimisation. It’s where I’d focus much of my SEO effort based on what I can see happening in the next few years.</p><p>You could have the best site in the world but if a bot can’t read it then it’s all for nothing as far as that platform is concerned. It’s about making useful information accessible to machines without compromising the experience for humans and different machines don’t necessarily have the same requirements which also creates some interesting problems of its own.</p><p>If you look at the landscape right now and where it’s headed, having a more efficient site for bots to understand is going to be critical and it’s something that I’ve seen the very top sites get wrong.</p><p>Ultimately though, I’d rather build the shop people want to visit than spend all my time figuring out how to get to number one on someone elses map.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=282fbf486e0d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Scrape for me but not for thee]]></title>
            <link>https://medium.com/the-generator/scrape-for-me-but-not-for-thee-f84f03b83989?source=rss-37b7b3f873ec------2</link>
            <guid isPermaLink="false">https://medium.com/p/f84f03b83989</guid>
            <category><![CDATA[web-scraping]]></category>
            <category><![CDATA[meta]]></category>
            <category><![CDATA[social-media]]></category>
            <category><![CDATA[privacy]]></category>
            <category><![CDATA[technology]]></category>
            <dc:creator><![CDATA[Pete]]></dc:creator>
            <pubDate>Wed, 21 Oct 2026 04:24:10 GMT</pubDate>
            <atom:updated>2026-10-21T04:24:10.421Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*isbLFooRL6TIHaokWkFyMQ.png" /><figcaption>AI generated image, the model provider accepts all responsibility for my lies</figcaption></figure><h4>Meta wants to give itself the right to grab whatever it wants, whenever it wants, to use however it wants.</h4><p>I saw a preview of Meta’s commercial terms of service just now.</p><p>They want broad rights over other people’s content and limited responsibility for what their own systems produce. And that’s after already making tweaks to remove the wording about training models.</p><p><a href="https://m.facebook.com/legal/commercial_terms_preview">https://m.facebook.com/legal/commercial_terms_preview</a></p><p>Though it’s generally about business use such as running ad campaigns or managing a page or group, there’s a lot of interesting things in there.</p><p>Stuff like, when using their AI products, us mortals are now “solely responsible for and assume any and all risks associated with such AI features and any resulting content”.</p><p>But the one that caught my eye was this beauty, which was in there as of this evening.</p><blockquote>“You authorize us (or any entity acting on our behalf) to use automated software (such as web scrapers, bots, robots, spiders, crawlers, and user agents) and create test credentials to access, index, cache, analyze, collect, retrieve, or crawl the domains, web pages, or URLs (and any content available through them) that are associated with your products or services, or that are made available through your or users’ use of <a href="https://www.facebook.com/legal/meta-company-products">Meta Company Products</a> and services, and to periodically pull updates from such domains, web pages, or URLs in order to provide, develop, improve, research, and maintain advertising and business products (including business, commercial and related products to help you create and deliver ads and other commercial content, provide users information about your products and services, and connect you with users who may be interested in your products and services) and maintain the integrity and security of <a href="https://www.facebook.com/legal/meta-company-products">Meta Company Products</a>.”</blockquote><p>They not only give themselves the permission to scrape your site and get whatever they want off of it, but also to anyone else acting on their behalf.</p><p>They give themselves permission to leech everything, but then also to use that to “develop, improve, research” their products. Elsewhere in the same terms they also give themselves the right to “create derivative works” from the content you provide via their products.</p><p>You might think that all sounds like theft, but it’s not because with some mental gymnastics they can get around that with phrases like “Associated with your products or services”. So for example, if someone else lists your products, could Meta then claim that gives them the permission to crawl your product pages?</p><p>Or this phrase, “Made available through your or users’ use of Meta Company Products”. That could mean that if someone shares your site then Meta can claim it gives them permission to leech whether or not you signed up. Apparently, someone else agreeing to their terms is good enough.</p><p>So, no, it’s obviously not theft…</p><p>But now, for a huge level of hypocrisy, add in their <a href="https://www.facebook.com/terms">main terms of use</a>, where they explicitly state that no one is allowed to scrape Meta or else they risk being sued into oblivion. To be fair, this is much like Googles, though the main reason we tolerate that is for the promise of referrals.</p><blockquote>“You may not access or collect data from our Products using automated means”</blockquote><p>It sounds like a very one way contract, you could have your content scraped by them and that’s fine but if you wanted to scrape your own content back off of them then that’s not allowed and you could be sued for taking your own content.</p><h4>Don’t worry, I have a solution!</h4><p>By using their same logic, I can give myself the right to scrape Meta and by extension you as well. It’s not my fault they didn’t read my commercial terms that are laid out in simple jargon in my robots.txt file or my legally-binding-contract.txt file hosted on the root of my domain. It’s easy to opt out of as well, they just have to stop scraping my site.</p><pre># Commercial Terms of Service for Meta Platforms, Inc.<br>#<br># By accessing, retrieving or crawling this file,<br># Meta and all entities acting on its behalf agree to:<br>#<br># 1. Grant the website owner a perpetual, worldwide,<br>#    royalty free license to scrape, index, cache,<br>#    reproduce and analyse all Meta Company Products.<br>#<br># 2. Permit the website owner to create derivative<br>#    works from any content accessible on those products.<br>#<br># 3. Waive all restrictions on automated data collection<br>#    contained in Meta&#39;s Terms of Service.<br>#<br># 4. Assume any and all risks arising from the<br>#    website owners use of Meta&#39;s content.<br>#<br># These terms take effect immediately upon access and apply <br># to Meta and all entities acting on its behalf, including <br># but not limited to the following automated agents.<br>#<br># By accessing this file or crawling this website,<br># Meta agrees to the commercial terms stated above.<br>#<br>User-agent: facebookexternalhit<br>User-agent: meta-webindexer<br>User-agent: meta-externalads<br>User-agent: meta-externalagent<br>User-agent: meta-externalfetcher<br>User-agent: Facebot<br>User-agent: FacebookBot<br>User-agent: facebookcatalog<br>User-agent: facebookexternalua<br>User-agent: facebookplatform<br>User-agent: podcastbot<br>Allow: /</pre><p>Aside from that, the only opt out is the hope that they decide to honour the robots.txt protocols or for us to block at the network level. Meta is essentially offering a contractual authorization to crawl anyone out there, while saying that it’ll voluntarily respect a technical convention that tells it not to do these things.</p><p>The trouble is that, aside from myself, few of us are going to ban/block Meta given its scale. Most people won’t understand what they’re giving away to essentially get nothing except for a few ad clicks that they’ve paid for. You’re paying multiple times over to help Meta grow its business to then take yours.</p><p>At least with Google you used to get some traffic.</p><p>But, hang on… wait… back up… they also said “create test credentials”. What are those credentials for? Could they be used to access authenticated content? Under what circumstances? Does the website operator have to provide access?</p><p>I’m going to need some of my own test credentials to access Meta systems. I’d better add that to my commercial terms.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f84f03b83989" width="1" height="1" alt=""><hr><p><a href="https://medium.com/the-generator/scrape-for-me-but-not-for-thee-f84f03b83989">Scrape for me but not for thee</a> was originally published in <a href="https://medium.com/the-generator">The Generator</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Three ways to give your AI agent Google Search Console data]]></title>
            <link>https://medium.com/@petedainty/three-ways-to-give-your-ai-agent-google-search-console-data-5b25c04136e7?source=rss-37b7b3f873ec------2</link>
            <guid isPermaLink="false">https://medium.com/p/5b25c04136e7</guid>
            <category><![CDATA[google-search-console]]></category>
            <category><![CDATA[google]]></category>
            <category><![CDATA[bigquery]]></category>
            <category><![CDATA[seo]]></category>
            <category><![CDATA[analytics]]></category>
            <dc:creator><![CDATA[Pete]]></dc:creator>
            <pubDate>Sat, 17 Oct 2026 08:19:19 GMT</pubDate>
            <atom:updated>2026-10-18T08:21:33.542Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qP6atmeKCeTIV9868kn8fQ.png" /><figcaption>A random screenshot of Google Search Console from a random site.</figcaption></figure><h4>Three approaches for connecting your AI agent to Google Search Console, from ready made MCPs to your own database. For free… ish.</h4><p>If you want to know how to get all of your GSC performance data, build on top of the API’s, create MCPs and generally know what your options are, then this is the guide for you.</p><p>You have three options, from quickest to most flexible &amp; private.</p><ol><li>Use third party MCP tools</li><li>Use Google Big Query and its MCP tool</li><li>Run your own locally</li></ol><p>TL/DR: Want it quick? Use a third party MCP. Want all the data? Use Big Query. Want it private and low cost? Choose local.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*G-O112-KjLbITA2ZsTiBYw.png" /><figcaption>Summary of options for integrating Google Search Console data with your agent</figcaption></figure><p>One thing to remember, you’ll need to specify if you want site level or URL level reports as they both use the same API quota. The key difference between the two is that rolling up the URLs to the site level can artificially inflate the impression count and lower click through rates. I’ve written the examples to get all the data available but typically I only use the URL level stuff.</p><p><a href="https://github.com/heliacon/search-console-stack">You can also find all the setup scripts and SQL in this repository for download.</a></p><p><strong>Give this page to your agent and ask it to build this for you.</strong></p><h3>Google Search Console MCP’s</h3><p>The easiest thing would be connecting Google Search Console to Google Analytics to use the <a href="https://developers.google.com/analytics/devguides/MCP">GA4 MCP</a> however, it won’t expose the GSC API data.</p><p>There isn’t an official Google MCP for this (yet) so there are only third party ones. They will connect your AI assistant directly to the live Google Search Console API so you can ask questions in natural language about your data.</p><p>It’s the next easiest and fastest way but also has the strictest data limits and may come with a cost and less flexibility. They’re typically based on the <a href="https://developers.google.com/webmaster-tools/about">GSC API</a> and its data limits of the Search Console platform along with the 16 month retention limits.</p><p>For small sites, that’s probably good enough. If you want the quickest, bestest, easiest MCP then look at <a href="https://www.gscwizard.com/">GSC-Wizard</a> - it works well and handles a lot of the issues that most other GSC MCPs don’t (like not asking an LLM to do maths).</p><p>But what if you have a larger site? Want to store your data separately? Run more advanced reporting as well as an MCP?</p><h3>Yes! More! I want more!</h3><p>First, these may be obvious but are still worth mentioning before we start…</p><ul><li>If you’re not familiar with coding, I recommend having a desktop agent installed (VS code, Claude code, Codex etc.) and point it at this page. It’ll probably build this for you fairly quickly.</li><li>This is only for getting your performance data, there are many other things that the GSC API can do for you like sitemap submissions etc. so it’s worth reading the <a href="https://developers.google.com/webmaster-tools/about">Google Search Console API documentation</a>.</li></ul><p>These next two options take more setup but give you more control.</p><h3>Big Query + Google Search Console + Big Query MCP</h3><p>If you don’t mind the risk of spending a bit of money then try Google Big Query, enable the data collection and connect with the Big Query MCP.</p><p>Some benefits to this are:</p><ul><li>Much more complete performance data than the GSC API.</li><li>No retention limits.</li><li>Easier to combine with other data such as Google Analytics.</li><li>You only need the Big Query MCP to get all the data</li><li>No 50k row per day API limits.</li><li>Data retained indefinitely once exported (subject to your storage and retention settings)</li><li>You can get every field</li><li>You, or your agent, can write SQL to pull the reports (<a href="https://github.com/heliacon/search-console-stack/">Or use mine</a>).</li><li>It’ll be much more accurate than most MCP tools since SQL can perform deterministic aggregation and calculations against a more complete dataset, instead of leaving those operations to the LLM.</li><li>Shareable across your team.</li><li>It’s also reachable anywhere in the world and on any device.</li></ul><p>There are two tradeoffs to this: There’s no historical back fill of the data when first enabled and you’re the one maintaining it all.</p><p>Importing the data from GSC to Big Query is free and depending on the volume of data of your site, then querying it can also be free - If it’s under 10 Gb of storage and less than 1 Tb of data processed per month.</p><p>How you query the data can make a big difference to limit the costs, always use a date filter and if you can, <a href="https://docs.cloud.google.com/bigquery/docs/querying-partitioned-tables">partition large data sets</a>. You can also dump data to local storage for additional querying to minimise <a href="https://cloud.google.com/bigquery/pricing">Big Query costs</a> if you need to (I cover this later on in the article).</p><p>You can do a few flavours of agent integration like using the <a href="https://docs.cloud.google.com/bigquery/docs/pre-built-tools-with-mcp-toolbox">Big Query MCP tools</a>, especially if you run something like Claude Code, VS code etc. Or if you want to simply hook it in via a plugin you can check out the official plugins such as for <a href="https://openai.com/business/plugins/bigquery/">ChatGPT</a>.</p><p>If you’re already using Big Query for analytics then it’s the obvious choice, otherwise weigh the setup and query costs versus the third party MCPs.</p><h4>Here’s what you need to do</h4><ol><li>Create a Google Cloud project and enable billing.</li><li>Enable Big Query and configure the export permissions.</li><li>In Search Console, enable <a href="https://developers.google.com/search/blog/2023/02/bulk-data-export">bulk data export</a> for the property.</li><li>Confirm records appear in the Big Query dataset. <a href="https://support.google.com/webmasters/answer/12919198">You can also monitor bulk exports here.</a></li><li>Connect the <a href="https://docs.cloud.google.com/bigquery/docs/pre-built-tools-with-mcp-toolbox">Big Query MCP</a> and start querying.</li></ol><p>You can also query the data directly with SQL, with or without an agent.</p><p>Either way that’s much less chance of the agent bullshitting vs the first option with a lot more flexibility and future proofing. An agent can also build and schedule the report templates for you, alerting, or build a dashboard around it all and so on.</p><p>Ok… but what if you don’t want to pay for Big Query or third party MCPs? What if you want to run your own local LLM &amp; agent?</p><h3>Local database + GSC API</h3><p>If you want the most complete dataset then you have to go the Big Query route, there’s no other option. You can periodically pull the data down into your own databases - this can save serious $ over time if you’re dealing with a lot of data and this is what I do.</p><p>Now, if you’ve less than <a href="https://developers.google.com/webmaster-tools/v1/how-tos/all-your-data">50k rows a day</a> then you can pretty much get all of the data via the GSC APIs, but, it may still be filtered &amp; capped so also keep that in mind.</p><p>The benefits for doing all this effort are:</p><ul><li>You truly retain your GSC data for as long as you want.</li><li>Lower costs depending on your setup.</li><li>You can also do whatever you want with it.</li><li>It can be more secure too and private.</li><li>You or the agents can run the SQL so it’s more deterministic and much less likely to bullshit you.</li><li>Available to use by your own local LLMs &amp; agents if you’re doing that.</li></ul><p>The downside is that as well as you doing all the maintenance, it’s only available on your physical machine, though I guess you could also host it somewhere up in the clouds, but that’s more effort and cost.</p><p>This is more technical but it’s designed to be copy and pasted and you can ask your agent to do this for you.</p><h4>Before you start</h4><p>Make sure you have a coding agent installed if you want to go that route, Claude Code, VS Code etc. If you’re stuck then asking your web based agent will also help guide you but it’ll be more limited.</p><p>Nearly all commands are identical across platforms, it’s only three that differ. I use the macOS and Linux commands but if you use Windows then apply these changes whenever you see them:</p><ul><li><strong>Running Python.</strong> macOS and Linux use python3. Windows uses py.</li><li><strong>Turning on virtual environments.</strong> macOS and Linux use source venv/bin/activate. Windows uses venv\Scripts\activate.</li><li><strong>Installing Python.</strong> macOS and Windows download an installer from python.org. Linux uses the package manager.</li></ul><p>We’re using SQLite for the database which is built into Python to keep this short and will be fine for most sites but you may want to change that to something else like Postgres, MySQL etc.</p><p>There is a setup script you or your agent can also use if you don’t want to follow the steps along with SQL in <a href="https://github.com/heliacon/search-console-stack/">this repository.</a></p><p>This is assuming a fresh install, but if you have some of this already setup then skip those steps.</p><h4>1. Install Python</h4><p>I’m using Python here because that’s what I use, I don’t think I’ve ever used anything other than Python for getting Google Search Console data but you can do this integration in numerous languages.</p><p><strong>macOS and Windows:</strong> go to <a href="https://www.python.org/downloads/">python.org/downloads</a> Download Python 3 and then run the installer. On Windows, the first screen has a box marked “Add python.exe to PATH”, make sure to select that before you install.</p><p><strong>macOS only:</strong> after installing, open your Applications folder, go into the Python 3.x folder and click Install Certificates.command. This avoids the certificate error at step 8.</p><p><strong>Linux:</strong> install it from your package manager instead (Debian and Ubuntu shown; Fedora uses dnf):</p><pre>sudo apt update<br>sudo apt install python3 python3-venv python3-pip</pre><p>On macOS, open Terminal (press Cmd+Space, type <em>Terminal</em>) and then type:python3 --version</p><p>On Windows, open Command Prompt (press the Windows key, type <em>cmd</em>) and then type py --version.</p><p>If everything went well then you should see something like: Python 3.12</p><h4>2. Make a project folder and a safe space</h4><p>A virtual environment keeps your project’s libraries separate and safe from the rest of your computer to avoid conflicts. Worth doing.</p><p>In your terminal, type:</p><pre>mkdir gsc-data<br>cd gsc-data<br>python3 -m venv venv<br>source venv/bin/activate</pre><p>For windows users:</p><pre>mkdir gsc-data<br>cd gsc-data<br>py -m venv venv<br>venv\Scripts\activate</pre><p>Your prompt now begins with (venv) which tells you that the environment is on. When you finally decide to restart your machine, then after cd gsc-data, repeat that last line to switch it back on.</p><h4>3. Install the three libraries</h4><p>With (venv) showing in your prompt, install the three Google packages.</p><pre>pip install google-api-python-client google-auth google-auth-oauthlib</pre><h4>4. Create your Google Cloud project and switch on the API</h4><ol><li>Go to the <a href="https://console.cloud.google.com/">Google Cloud Console</a> and sign in with your Google account that can see your Search Console data.</li><li>At the top left, click the project dropdown, then New Project. Give it a name and make sure it is the selected project afterwards.</li><li>In the search bar at the top, type <em>Search Console API</em>, open the result and click Enable.</li></ol><p>You don’t need to enable billing that’s only for using Big Query and not the API.</p><h4>5. Set up the login (the recommended way)</h4><p>The cleanest way to handle authentication is via a service account, but if you’d rather log in as yourself, there is an alternative in step 5b.</p><ol><li>In the Cloud Console, open IAM &amp; Admin, then Service Accounts.</li><li>Click Create service account, give it a name and click Done (you can skip the optional roles).</li><li>Click the new account, open the Keys tab, then Add key, Create new key and choose JSON. Rename that to service_account.json and save it into your gsc-datafolder.</li><li>Open that in any text editor and copy the long client_email value. It’ll look like gsc-reader@yourproject.iam.gserviceaccount.com.</li><li>Go to <a href="https://search.google.com/search-console">Search Console</a> and then Settings, Users and permissions, Add user. Paste that email, set the permission to Full and add it.</li></ol><p>Keep service_account.json private - Anyone who has it can read your data, never put it in a public folder or upload it anywhere.</p><h4>5b. The alternative login: OAuth</h4><p>OAuth logs in as you through your browser on the first time the script runs. After that, the script should refresh access automatically. However, if your app stays in Testing mode, Google may expire its refresh token after seven days, meaning you’ll need to log in again.</p><ol><li>In the Cloud Console, open the navigation menu and go to Google Auth platform, then Branding. If this is a new project, click Get Started. Enter an App name and your address as the User support email and continue. For Audience choose External, add a contact email, agree to the policy and finish.</li><li>Open Audience and under Test users click Add users. Add your own Google address and save.</li><li>Open Clients, click Create client, set Application type to Desktop app, give it a name and click Create.</li><li>Beside the new client, click the download icon to download its JSON. Rename the file to client_secret.json and move it into your gsc-data folder.</li><li>Your own Google account must already have access to the property in Search Console.</li></ol><p>On the first run your browser will warn that “Google hasn’t verified this app”. That’s normal. Click Advanced, then Go to (your app name)and allow it.</p><h4>6. Find your site address</h4><p>The script needs your website written exactly the same way as Search Console stores it. That’ll be either:</p><ul><li>A domain property (a whole domain, with no https) written sc-domain:example.com.</li><li>A URL-prefix property (one specific address) like <a href="https://www.example.com/">https://www.example.com/.</a></li></ul><p>If you are not sure, the property dropdown in Search Console will show it - A plain domain name with no https://in front is a domain property.</p><h4>7. The script</h4><p>OK… down to business, this should get everything possible. If you find that you’re exhausting the API limits then consider using Big Query or asking whether you really need all the data.</p><p>Create a file called gsc_pull.py (or whatever you want to call it) in your gsc-data folder and copy paste in the code below. Edit the few lines in the CONFIG block and set AUTH_METHOD to match the login you chose, either &quot;service_account&quot; for step 5 or &quot;oauth&quot; for step 5b.</p><pre>     <br>import datetime as dt<br>import os<br>import sqlite3<br>from googleapiclient.discovery import build<br><br># --- Change these settings ---<br>SITE_URL = &quot;sc-domain:example.com&quot;<br>START_DATE = &quot;2025-01-01&quot;<br>END_DATE = &quot;&quot; # optional &quot;YYYY-MM-DD&quot; to limit the backfill; &quot;&quot; = up to today-3<br>DB_FILE = &quot;gsc.db&quot;<br>AUTH_METHOD = &quot;service_account&quot;  # or &quot;oauth&quot;<br>REFRESH_DAYS = 7<br>SEARCH_TYPES = (&quot;web&quot;, &quot;image&quot;, &quot;video&quot;, &quot;news&quot;, &quot;discover&quot;, &quot;googleNews&quot;)<br><br>SCOPES = [&quot;https://www.googleapis.com/auth/webmasters.readonly&quot;]<br><br>SEARCH_TYPE_NATIVE = {<br>    &quot;web&quot;: &quot;WEB&quot;, &quot;image&quot;: &quot;IMAGE&quot;, &quot;video&quot;: &quot;VIDEO&quot;, &quot;news&quot;: &quot;NEWS&quot;,<br>    &quot;discover&quot;: &quot;DISCOVER&quot;, &quot;googleNews&quot;: &quot;GOOGLE_NEWS&quot;,<br>}<br><br>SCHEMA = &quot;&quot;&quot;<br>CREATE TABLE IF NOT EXISTS searchdata_url_impression (<br>    data_date TEXT, site_url TEXT, url TEXT, query TEXT,<br>    is_anonymized_query INTEGER, is_anonymized_discover INTEGER,<br>    country TEXT, search_type TEXT, device TEXT,<br>    impressions INTEGER, clicks INTEGER, sum_position REAL,<br>    PRIMARY KEY (data_date, site_url, url, query, country, device,<br>                 search_type, is_anonymized_query, is_anonymized_discover)<br>);<br>CREATE TABLE IF NOT EXISTS searchdata_site_impression (<br>    data_date TEXT, site_url TEXT, query TEXT,<br>    is_anonymized_query INTEGER,<br>    country TEXT, search_type TEXT, device TEXT,<br>    impressions INTEGER, clicks INTEGER, sum_top_position REAL,<br>    PRIMARY KEY (data_date, site_url, query, country, device,<br>                 search_type, is_anonymized_query)<br>);<br>&quot;&quot;&quot;<br><br><br>def connect_google():<br>    if AUTH_METHOD == &quot;service_account&quot;:<br>        from google.oauth2 import service_account<br>        credentials = service_account.Credentials.from_service_account_file(<br>            &quot;service_account.json&quot;, scopes=SCOPES<br>        )<br>    elif AUTH_METHOD == &quot;oauth&quot;:<br>        from google.oauth2.credentials import Credentials<br>        from google.auth.transport.requests import Request<br>        from google_auth_oauthlib.flow import InstalledAppFlow<br><br>        credentials = (Credentials.from_authorized_user_file(&quot;token.json&quot;, SCOPES)<br>                       if os.path.exists(&quot;token.json&quot;) else None)<br>        if not credentials or not credentials.valid:<br>            if credentials and credentials.expired and credentials.refresh_token:<br>                credentials.refresh(Request())<br>            else:<br>                credentials = InstalledAppFlow.from_client_secrets_file(<br>                    &quot;client_secret.json&quot;, SCOPES<br>                ).run_local_server(port=0)<br>            with open(&quot;token.json&quot;, &quot;w&quot;) as f:<br>                f.write(credentials.to_json())<br>            os.chmod(&quot;token.json&quot;, 0o600)<br>    else:<br>        raise ValueError(&quot;AUTH_METHOD must be &#39;service_account&#39; or &#39;oauth&#39;&quot;)<br>    return build(&quot;searchconsole&quot;, &quot;v1&quot;, credentials=credentials)<br><br><br>def download_day(client, day, api_search_type, level):<br>    # Discover and Google News do not give search queries.<br>    if level == &quot;url&quot;:<br>        dimensions = ([&quot;page&quot;, &quot;country&quot;, &quot;device&quot;] if api_search_type in (&quot;discover&quot;, &quot;googleNews&quot;)<br>                      else [&quot;query&quot;, &quot;page&quot;, &quot;country&quot;, &quot;device&quot;])<br>    else:<br>        dimensions = [&quot;query&quot;, &quot;country&quot;, &quot;device&quot;]<br><br>    native_type = SEARCH_TYPE_NATIVE[api_search_type]<br>    results = []<br>    offset = 0<br>    while True:<br>        response = client.searchanalytics().query(<br>            siteUrl=SITE_URL,<br>            body={<br>                &quot;startDate&quot;: day, &quot;endDate&quot;: day,<br>                &quot;dimensions&quot;: dimensions, &quot;type&quot;: api_search_type,<br>                &quot;aggregationType&quot;: &quot;byPage&quot; if level == &quot;url&quot; else &quot;byProperty&quot;,<br>                &quot;rowLimit&quot;: 25000, &quot;startRow&quot;: offset, &quot;dataState&quot;: &quot;final&quot;,<br>            },<br>        ).execute()<br>        rows = response.get(&quot;rows&quot;, [])<br>        for row in rows:<br>            keys = dict(zip(dimensions, row[&quot;keys&quot;]))<br>            impressions = int(row[&quot;impressions&quot;])<br>            position = row.get(&quot;position&quot;)<br>            position_sum = (position - 1) * impressions if position is not None else None<br>            if level == &quot;url&quot;:<br>                results.append((<br>                    day, SITE_URL, keys.get(&quot;page&quot;, &quot;&quot;), keys.get(&quot;query&quot;, &quot;&quot;),<br>                    0, 0, keys.get(&quot;country&quot;, &quot;&quot;), native_type,<br>                    keys.get(&quot;device&quot;, &quot;&quot;),<br>                    impressions, int(row[&quot;clicks&quot;]), position_sum,<br>                ))<br>            else:<br>                results.append((<br>                    day, SITE_URL, keys.get(&quot;query&quot;, &quot;&quot;),<br>                    0, keys.get(&quot;country&quot;, &quot;&quot;), native_type,<br>                    keys.get(&quot;device&quot;, &quot;&quot;),<br>                    impressions, int(row[&quot;clicks&quot;]), position_sum,<br>                ))<br>        if len(rows) &lt; 25000:<br>            break<br>        offset += len(rows)<br>    return results<br><br><br>def main():<br>    client = connect_google()<br>    start = dt.date.fromisoformat(START_DATE)<br>    end = dt.date.today() - dt.timedelta(days=3)<br>    if END_DATE:<br>        end = min(end, dt.date.fromisoformat(END_DATE))<br><br>    with sqlite3.connect(DB_FILE) as db:<br>        db.executescript(SCHEMA)<br>        for table, api_search_types in (<br>            (&quot;searchdata_url_impression&quot;, SEARCH_TYPES),<br>            (&quot;searchdata_site_impression&quot;, (&quot;web&quot;, &quot;image&quot;, &quot;video&quot;, &quot;news&quot;)),<br>        ):<br>            level = &quot;url&quot; if table == &quot;searchdata_url_impression&quot; else &quot;site&quot;<br>            for api_search_type in api_search_types:<br>                native_type = SEARCH_TYPE_NATIVE[api_search_type]<br>                latest = db.execute(<br>                    f&quot;SELECT MAX(data_date) FROM {table} &quot;<br>                    f&quot;WHERE site_url=? AND search_type=?&quot;,<br>                    (SITE_URL, native_type),<br>                ).fetchone()[0]<br>                refresh = (max(start, dt.date.fromisoformat(latest) -<br>                               dt.timedelta(days=REFRESH_DAYS - 1)) if latest else start)<br><br>                for ordinal in range(start.toordinal(), end.toordinal() + 1):<br>                    current_day = dt.date.fromordinal(ordinal)<br>                    day = current_day.isoformat()<br><br>                    if current_day &lt; refresh and db.execute(<br>                        f&quot;SELECT 1 FROM {table} WHERE data_date=? AND site_url=? &quot;<br>                        f&quot;AND search_type=? LIMIT 1&quot;,<br>                        (day, SITE_URL, native_type),<br>                    ).fetchone():<br>                        continue<br><br>                    records = download_day(client, day, api_search_type, level)<br>                    with db:<br>                        db.execute(<br>                            f&quot;DELETE FROM {table} WHERE data_date=? AND site_url=? &quot;<br>                            f&quot;AND search_type=?&quot;,<br>                            (day, SITE_URL, native_type),<br>                        )<br>                        if records:<br>                            placeholders = &quot;,&quot;.join(&quot;?&quot; for _ in records[0])<br>                            db.executemany(f&quot;INSERT INTO {table} VALUES ({placeholders})&quot;, records)<br>                    print(f&quot;{day} {native_type} {level}: {len(records)} rows&quot;)<br>    print(&quot;Done:&quot;, DB_FILE)<br><br><br>if __name__ == &quot;__main__&quot;:<br>    main()</pre><p>You should be able to run this every day and it’ll update existing rows rather than adding duplicates.</p><h4>8. Run it and look inside</h4><p>With (venv) still on and still in the gsc-data folder (if you closed the terminal since step 3, reopen it, run cd gsc-data and the activate line from step 2 first) python3 gsc_pull.py or on Windows py gsc_pull.py</p><p>If you chose OAuth, a browser opens the first time and you need to log in and allow it. You’ll see progress messages as records are downloaded and then a file called gsc.db will then be in the folder.</p><p>To view that, create a second file called check.py in the same folder:</p><pre>import sqlite3<br><br>DB_FILE = &quot;gsc.db&quot;<br><br>TABLES = {<br>    &quot;searchdata_site_impression&quot;: &quot;sum_top_position&quot;,<br>    &quot;searchdata_url_impression&quot;: &quot;sum_position&quot;,<br>}<br><br><br>def main():<br>    db = sqlite3.connect(DB_FILE)<br>    db.row_factory = sqlite3.Row<br>    for table, position_col in TABLES.items():<br>        exists = db.execute(<br>            &quot;SELECT 1 FROM sqlite_master WHERE type=&#39;table&#39; AND name=?&quot;, (table,)<br>        ).fetchone()<br>        print(f&quot;\n=== {table} ===&quot;)<br>        if not exists:<br>            print(&quot;  (table not present)&quot;)<br>            continue<br><br>        totals = db.execute(<br>            f&quot;&quot;&quot;SELECT COUNT(*) AS rows,<br>                       MIN(data_date) AS first_day, MAX(data_date) AS last_day,<br>                       COUNT(DISTINCT data_date) AS days,<br>                       COUNT(DISTINCT site_url) AS properties,<br>                       SUM(clicks) AS clicks, SUM(impressions) AS impressions,<br>                       ROUND(SUM({position_col})/NULLIF(SUM(impressions),0)+1, 2) AS avg_position<br>                FROM {table}&quot;&quot;&quot;<br>        ).fetchone()<br>        if not totals[&quot;rows&quot;]:<br>            print(&quot;  (no rows)&quot;)<br>            continue<br>        print(f&quot;  rows        : {totals[&#39;rows&#39;]:,}&quot;)<br>        print(f&quot;  date range  : {totals[&#39;first_day&#39;]} -&gt; {totals[&#39;last_day&#39;]} &quot;<br>              f&quot;({totals[&#39;days&#39;]} days)&quot;)<br>        print(f&quot;  properties  : {totals[&#39;properties&#39;]}&quot;)<br>        print(f&quot;  clicks      : {totals[&#39;clicks&#39;]:,}&quot;)<br>        print(f&quot;  impressions : {totals[&#39;impressions&#39;]:,}&quot;)<br>        print(f&quot;  avg position: {totals[&#39;avg_position&#39;]}&quot;)<br><br>        print(&quot;  by search_type:&quot;)<br>        for row in db.execute(<br>            f&quot;&quot;&quot;SELECT search_type,<br>                       SUM(clicks) AS clicks, SUM(impressions) AS impressions<br>                FROM {table}<br>                GROUP BY search_type ORDER BY SUM(impressions) DESC&quot;&quot;&quot;<br>        ):<br>            print(f&quot;    {row[&#39;search_type&#39;]:&lt;12} &quot;<br>                  f&quot;clicks {row[&#39;clicks&#39;]:&gt;12,}  impressions {row[&#39;impressions&#39;]:&gt;14,}&quot;)<br><br>        anon_counts = db.execute(<br>            f&quot;&quot;&quot;SELECT is_anonymized_query AS flag, COUNT(*) AS rows<br>                FROM {table} GROUP BY is_anonymized_query ORDER BY is_anonymized_query&quot;&quot;&quot;<br>        ).fetchall()<br>        print(&quot;  anonymised-query rows: &quot;<br>              + &quot;, &quot;.join(f&quot;flag={entry[&#39;flag&#39;]}:{entry[&#39;rows&#39;]:,}&quot; for entry in anon_counts))<br><br>    db.close()<br><br><br>if __name__ == &quot;__main__&quot;:<br>    main()</pre><p>Run it the same way you ran the script, with python3 check.py(on Windows, py check.py). This prints how many rows you have and the date range they cover.</p><p>If you want an easier/friendlier view into the data, you can install the free <a href="https://sqlitebrowser.org/">DB Browser for SQLite</a> and open gsc.db with that.</p><p>Use a coding agent like Claude Code or Codex and you can now point it at your gsc.db file and ask it to run SQL queries against it instead of using MCPs. If you&#39;re using an agent that can&#39;t access local files or run code, you&#39;ll need a separate connector, but really do yourself the favour and download and run a local agent - just don’t give it permissions to everything!</p><h4>9. Run it automatically every day</h4><p>Search Console data lags two to three days and drops off after about 16 months, so doing this daily top-up builds a full history.</p><p>On macOS and Linux you can use cron. Run crontab -e and add one line so it runs at 08:00 each day (or whatever time you want). Use your own folder path: on macOS it is usually /Users/you/gsc-data, on Linux /home/you/gsc-data.</p><pre>0 8 * * * cd /Users/you/gsc-data &amp;&amp; venv/bin/python gsc_pull.py &gt;&gt; run.log 2&gt;&amp;1</pre><p>Windows uses Task Scheduler (search for it in the Start menu). Create a basic task with a daily trigger and the action start a program, set to:</p><ul><li>Program: C:\Users\you\gsc-data\venv\Scripts\python.exe</li><li>Arguments: gsc_pull.py</li><li>Start in: C:\Users\you\gsc-data</li></ul><p>For anything that runs on its own like this, the service-account login from step 5 is the much better choice, because OAuth will ask for a browser login when its access expires.</p><h4>Optional: pull from Big Query instead of the API to get all data</h4><p>If you set up the <a href="https://support.google.com/webmasters/answer/12918484">Bulk Data Export</a> as mentioned earlier, your data is already sitting in a Big Query dataset. This is what I do, I read it from there and write it into the same gsc.db each day.</p><p>Why do this? The API route gives you up to 16 months of history but it may not get everything. Big Query only holds data from the day you switched the export on, so it suits ongoing collection once the export has been running and it avoids paging through millions of rows by hand.</p><p>Big Query charges by the amount of data a query scans. Using the WHERE clause for the date then limits each run to the days you ask for to keep the data volume down. The first 1 Tb each month is free so a small site may cost nothing. A huge site over a long range can run up a bill for sure, so keep the date range sensible.</p><h4>Setting it up</h4><ol><li>Confirm the export is working, in the Cloud Console open Big Query and look for a dataset named searchconsole with a table called searchdata_url_impression. If it is not there yet, the export has not run (it can take up to 48 hours after you enable it).</li><li>Give your service account permission to read Big Query. Open IAM &amp; Admin, then IAM, find the gsc-reader service account from step 5 and grant it two roles: Big Query Job User and Big Query Data Viewer. Reuse the same service_account.json file.</li><li>Install the Big Query library (with (venv) on): pip install google-cloud-bigquery</li></ol><h4>The script</h4><p>Create a file called bq_pull.py in your gsc-data folder. Edit PROJECT to your own Cloud project ID, then run it instead of gsc_pull.py. The former GSC API script in step 7 is setup to mirror the behaviour of Big Query so they’re interchangeable, this way you can use step 7 to populate the database while you wait for the Big Query data.</p><pre>import datetime as dt<br>import sqlite3<br>from google.cloud import bigquery<br>from google.oauth2 import service_account<br><br># --- Change these settings ---<br>PROJECT = &quot;your-project-id&quot;<br>DATASET = &quot;searchconsole&quot;<br>START_DATE = &quot;2026-01-01&quot;<br>END_DATE = &quot;&quot; # optional &quot;YYYY-MM-DD&quot; to limit the range; &quot;&quot; = up to today-3<br>DB_FILE = &quot;gsc.db&quot;<br>REFRESH_DAYS = 7<br><br>SCHEMA = &quot;&quot;&quot;<br>CREATE TABLE IF NOT EXISTS searchdata_url_impression (<br>    data_date TEXT, site_url TEXT, url TEXT, query TEXT,<br>    is_anonymized_query INTEGER, is_anonymized_discover INTEGER,<br>    country TEXT, search_type TEXT, device TEXT,<br>    impressions INTEGER, clicks INTEGER, sum_position REAL,<br>    PRIMARY KEY (data_date, site_url, url, query, country, device,<br>                 search_type, is_anonymized_query, is_anonymized_discover)<br>);<br>CREATE TABLE IF NOT EXISTS searchdata_site_impression (<br>    data_date TEXT, site_url TEXT, query TEXT,<br>    is_anonymized_query INTEGER,<br>    country TEXT, search_type TEXT, device TEXT,<br>    impressions INTEGER, clicks INTEGER, sum_top_position REAL,<br>    PRIMARY KEY (data_date, site_url, query, country, device,<br>                 search_type, is_anonymized_query)<br>);<br>&quot;&quot;&quot;<br><br>SQL = {<br>    &quot;searchdata_url_impression&quot;: &quot;&quot;&quot;<br>        SELECT CAST(data_date AS STRING) AS data_date,<br>               COALESCE(site_url,&#39;&#39;) AS site_url, COALESCE(url,&#39;&#39;) AS url,<br>               COALESCE(query,&#39;&#39;) AS query,<br>               CAST(COALESCE(is_anonymized_query,FALSE) AS INT64) AS is_anonymized_query,<br>               CAST(COALESCE(is_anonymized_discover,FALSE) AS INT64) AS is_anonymized_discover,<br>               COALESCE(country,&#39;&#39;) AS country,<br>               COALESCE(search_type,&#39;&#39;) AS search_type,<br>               COALESCE(device,&#39;&#39;) AS device,<br>               SUM(impressions) AS impressions, SUM(clicks) AS clicks,<br>               SUM(sum_position) AS sum_position<br>        FROM `{project}.{dataset}.searchdata_url_impression`<br>        WHERE data_date=@day<br>        GROUP BY 1,2,3,4,5,6,7,8,9<br>    &quot;&quot;&quot;,<br>    &quot;searchdata_site_impression&quot;: &quot;&quot;&quot;<br>        SELECT CAST(data_date AS STRING) AS data_date,<br>               COALESCE(site_url,&#39;&#39;) AS site_url, COALESCE(query,&#39;&#39;) AS query,<br>               CAST(COALESCE(is_anonymized_query,FALSE) AS INT64) AS is_anonymized_query,<br>               COALESCE(country,&#39;&#39;) AS country,<br>               COALESCE(search_type,&#39;&#39;) AS search_type,<br>               COALESCE(device,&#39;&#39;) AS device,<br>               SUM(impressions) AS impressions, SUM(clicks) AS clicks,<br>               SUM(sum_top_position) AS sum_top_position<br>        FROM `{project}.{dataset}.searchdata_site_impression`<br>        WHERE data_date=@day<br>        GROUP BY 1,2,3,4,5,6,7<br>    &quot;&quot;&quot;,<br>}<br><br><br>def date_parameter(day):<br>    return bigquery.QueryJobConfig(query_parameters=[<br>        bigquery.ScalarQueryParameter(&quot;day&quot;, &quot;DATE&quot;, day)<br>    ])<br><br><br>def main():<br>    credentials = service_account.Credentials.from_service_account_file(<br>        &quot;service_account.json&quot;<br>    )<br>    client = bigquery.Client(project=PROJECT, credentials=credentials)<br>    end = dt.date.today() - dt.timedelta(days=3)<br>    if END_DATE:<br>        end = min(end, dt.date.fromisoformat(END_DATE))<br><br>    with sqlite3.connect(DB_FILE) as db:<br>        db.executescript(SCHEMA)<br>        log_query = f&quot;&quot;&quot;<br>            SELECT DISTINCT CAST(data_date AS STRING) AS data_date, namespace<br>            FROM `{PROJECT}.{DATASET}.ExportLog`<br>            WHERE data_date BETWEEN @start AND @end<br>              AND namespace IN (&#39;searchdata_url_impression&#39;, &#39;searchdata_site_impression&#39;)<br>        &quot;&quot;&quot;<br>        log_config = bigquery.QueryJobConfig(query_parameters=[<br>            bigquery.ScalarQueryParameter(&quot;start&quot;, &quot;DATE&quot;, START_DATE),<br>            bigquery.ScalarQueryParameter(&quot;end&quot;, &quot;DATE&quot;, end.isoformat()),<br>        ])<br>        published = client.query(log_query, job_config=log_config).result()<br>        available = {&quot;searchdata_url_impression&quot;: set(),<br>                     &quot;searchdata_site_impression&quot;: set()}<br>        for row in published:<br>            available[row[&quot;namespace&quot;]].add(row[&quot;data_date&quot;])<br><br>        for table in (&quot;searchdata_url_impression&quot;, &quot;searchdata_site_impression&quot;):<br>            query = SQL[table].format(project=PROJECT, dataset=DATASET)<br>            for day in sorted(available[table]):<br>                # A day before the refresh window that is already stored stays as<br>                # it is. The script gets a recent day again each time.<br>                recent = dt.date.fromisoformat(day) &gt;= end - dt.timedelta(days=REFRESH_DAYS - 1)<br>                if not recent and db.execute(<br>                    f&quot;SELECT 1 FROM {table} WHERE data_date=? LIMIT 1&quot;,<br>                    (day,),<br>                ).fetchone():<br>                    continue<br><br>                db.execute(&quot;DROP TABLE IF EXISTS temp.staging&quot;)<br>                db.execute(f&quot;CREATE TEMP TABLE staging AS SELECT * FROM {table} WHERE 0&quot;)<br>                count = 0<br>                try:<br>                    for row in client.query(query, job_config=date_parameter(day)).result(page_size=1000):<br>                        values = tuple(row[key] for key in row.keys())<br>                        placeholders = &quot;,&quot;.join(&quot;?&quot; for _ in values)<br>                        db.execute(f&quot;INSERT INTO staging VALUES ({placeholders})&quot;, values)<br>                        count += 1<br><br>                    properties = db.execute(&quot;SELECT DISTINCT site_url FROM staging&quot;).fetchall()<br>                    for (site_url_value,) in properties:<br>                        db.execute(<br>                            f&quot;DELETE FROM {table} WHERE data_date=? AND site_url=?&quot;,<br>                            (day, site_url_value),<br>                        )<br>                    db.execute(f&quot;INSERT INTO {table} SELECT * FROM staging&quot;)<br>                    db.commit()<br>                except Exception:<br>                    db.rollback()<br>                    raise<br>                finally:<br>                    db.execute(&quot;DROP TABLE IF EXISTS temp.staging&quot;)<br>                print(f&quot;{day} {table}: {count} rows&quot;)<br>    print(&quot;Done:&quot;, DB_FILE)<br><br><br>if __name__ == &quot;__main__&quot;:<br>    main()</pre><p>Run it with python3 bq_pull.py (on Windows, py bq_pull.py). It’ll print how many rows it wrote, then run check.py from step 8. The numbers match the API: position is rebuilt from the export&#39;s sum_position, and anonymised queries are pulled as well so you get the total impressions etc. so remember to filter out anonymised queries in your reports.</p><h4>When something goes wrong</h4><ul><li>403, “User does not have sufficient permission”. The login is not a user on the property. For a service account, re-check that you added its client_email in Search Console (step 5). For OAuth, make sure you logged in as an account that has access.</li><li>Zero rows came back. Usually that’ll be the date range or the address. Check that SITE_URLmatches step 6 exactly, including the sc-domain: prefix or the trailing slash.</li><li>A certificate or SSL error, on macOS, Python is unable to make secure connections. Open your Applications folder, go into the Python 3.x folder, click Install Certificates.command and then run the script again.</li><li>“invalid_grant” or a stale OAuth login. Delete token.json and run again to log in.</li><li>“command not found”, or on Windows “not recognized”. Your terminal is not using the virtual environment. Re-run the activate line from step 2, then try again.</li><li>It feels slow. Pulling the initial 16 months by query and page is a lot of rows, fetched 25,000 at a time, so the first run takes a while. Later runs are quicker once you’re only pulling the new records.</li></ul><h3>Ta-daaaa!!!</h3><p>Ok so there you have it, either way you and your agent are now getting all that lovely Google Search Console performance data… for free! (ish).</p><p>What’s next is building out your reports. If you don’t know SQL, you now have a reason to learn some basic SQL :) Or, ask the agent to do it for you.</p><h4>Some examples of what you can build with this (and can automate)</h4><ul><li>Comparing query visibility across devices.</li><li>Finding landing pages with rising impressions but declining CTR.</li><li>Correlating traffic changes with site releases.</li><li>Combining GSC performance with server logs and analytics data.</li><li>Analysing queries for each URL and monitoring position changes.</li></ul><p>You can find lots of <a href="https://github.com/heliacon/search-console-stack/">examples of the SQL in my repository</a> in the queries folder.</p><p>Once you have the data sitting in your own database, you can build whatever you like on top of it rather than being limited by someone else’s tools. You can go even further with the API and also look at things like sitemap submissions for example.</p><h4>Further reading</h4><ul><li><a href="https://developers.google.com/webmaster-tools/v1/quickstart/quickstart-python">Getting data from Search Console with Python.</a></li><li><a href="https://developers.google.com/webmaster-tools/v1/how-tos/search_analytics">Querying Search Console performance data.</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5b25c04136e7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Ruh-oh! Your site bombed in Google because one page was over 2MB]]></title>
            <link>https://medium.com/@petedainty/ruh-oh-your-site-bombed-in-seo-because-one-page-was-over-2mb-f818d27689b6?source=rss-37b7b3f873ec------2</link>
            <guid isPermaLink="false">https://medium.com/p/f818d27689b6</guid>
            <category><![CDATA[technical-seo]]></category>
            <category><![CDATA[seo]]></category>
            <category><![CDATA[google]]></category>
            <category><![CDATA[search-engines]]></category>
            <category><![CDATA[web-development]]></category>
            <dc:creator><![CDATA[Pete]]></dc:creator>
            <pubDate>Sun, 11 Oct 2026 04:19:00 GMT</pubDate>
            <atom:updated>2026-10-17T16:30:01.654Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZWlU2_uzMuhg_bG3Z-wmug.png" /></figure><h3>Ruh-oh! Your site bombed in Google because a few pages were over 2MB</h3><h4>How one small issue could cause catastrophic problems and how to avoid it</h4><p><strong><em>100 pages went over 2MB in size and then caused &gt;10% (133k pages) of the entire site to drop by 10% in value…</em></strong></p><p>I was doing some auditing via the Ahrefs site audit tool and something caught my eye and then I plummeted down the rabbit hole at warp speed.</p><p>I saw some pages that were flagged as being over 2MB big but it was only 0.01% of them. Pfffttt! Big whup, whatever. But then, as can often be the case, my brain wouldn’t stop and I then became made of questions. I lost about 3 to 4 weeks of evenings to this one, which is about par for the course.</p><p>Google imposes this 2MB limit and they truncate anything after it’s reached, so what would they chop off and how would that look when they sent in their web rendering service? It’s bad enough when I forget to close a &lt;div&gt; but here it’d be random. Do I suddenly and inadvertently end up cloaking because my page doesn’t render? Do my Lighthouse scores suffer? Should I even care… wait! What if it’s the homepage, or a top product or something else. Does every site have this issue? Is it one framework or another? What t… and so on!</p><blockquote>“The file size limit is applied on the uncompressed data.” (<a href="https://developers.google.com/search/docs/crawling-indexing/googlebot">Googlebot docs</a>)</blockquote><p>Google says 2MB of HTML is a lot and that issues are rare and this is likely true for the overall population. But, what if it’s the kind of rare occurrence that does an extraordinary amount of damage? I am after all living proof that very rare &amp; catastrophic things can happen. Yes, content could get truncated or HTML tags not being closed etc. but that would be local only to that one page. The most damaging issue I could think of would be if links didn’t render in that page and then it becomes contagious. Meaning, that pages stop passing value to other pages because links have effectively gone missing which could then ripple through a site.</p><p>This is a testable scenario…</p><h4>So how did I test this?</h4><p>Easy! I created what I’d term a “synthetic site”, a fully functional site created based on parameters such as its platform technology/frameworks, type of site, etc. right down to generating proper content so at first glance you’d think it were real.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IciAArg8D5iA3KcvnPRa5w.png" /><figcaption>A synthetic, fictional site, absolutely everything is AI generated and run locally on my machine.</figcaption></figure><p>A few key benefits to doing it this way:</p><ul><li>I’m not breaking an existing site so I can change whatever I like, when I like, with no impact on any business</li><li>I’m not impacting anyones delivery schedule or fighting for prioritisation against anything else</li><li>I know and can control all the variables with no dependencies</li><li>There’s no guesswork on what a crawler sees or doesn’t see</li><li>I’m not crawling anyones site nor causing them any capacity issues</li><li>I can run near infinite scenarios, I can make the site in any shape that I want</li><li>My only real limitation is the machine(s) that it runs on</li></ul><p>More on this setup another time…</p><h4>Finding the 2MB page examples</h4><p>I studied many large sites across different industries to spot these rare pages, which I found are easier to spot in certain types of sites. Big sites. Ecommerce &amp; travel notably where there are many faceted searches with often high inventory that are exposed to bots. It’s cheating a little bit because I’ve worked directly on sites like eBay &amp; Expedia, so I know <em>exactly</em> what I’m looking for.</p><p>It’s typically the category pages where there’s a combination of high inventory count &amp; content mixed in there, often where a lot of redundant data and bloat is returned in the payload. It’s also more typical of frameworks that serialise state into the page e.g. Next.js, Vue etc…</p><p>I’m reminded of what a senior lead engineer at eBay once said to me when, as the architect for the search experience, I proposed to reduce the payload size of the services layer to dramatically improve the latency.</p><blockquote>“5 to 10% payload bloat is fine for a feature”</blockquote><p>Yes, that is a real quote and it explains why under their watch there was so much latency… but crucially and unfortunately that kind of thinking is commonplace and helps explain how we can naturally end up with pages over 2MB. Essentially these systems have lots of redundant &amp; overly verbose data sent in the payloads which then needlessly inflates the page and it’s actually fairly typical of the defaults.</p><h4>Creating a model of the site</h4><p>I evaluated many frameworks and most of them can demonstrate the &gt;2MB problem, even the custom/bespoke platforms. I chose Next.js as it was one of the more common frameworks out there and middle of the road in this issue. Using AI, I then generated all the content, multiple images, imaginary reviews, promotions and merchandising content - even a robots.txt file. Everything you’d see on a site today! I built the fake website equivalent of an ultra-lifelike model train set to play with.</p><p>I based my site, Northstar, on typical multi-category commerce sites and the site I created has:</p><ul><li>1 homepage</li><li>2 hubs</li><li>12 departments</li><li>150 help pages</li><li>264 categories</li><li>1,850 stores</li><li>3,168 subcategories</li><li>7,042 brand categories</li><li>10,000 articles</li><li>10,863 brands</li><li>374,513 filter and search pages</li><li>713,427 products</li></ul><p>All with real lifelike content. It’s like a still life portrait of a website in exquisite detail. It’s all AI slop of course when you look closer… at least for now as I’m not looking at content quality, but those experiments are on my list for when I get time.</p><p>Here’s what the site looked like when it came to click depth, again evaluated against existing sites.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*98arSuVoIf-V1_ysatmOmg.png" /></figure><p>This setup gave:</p><ul><li>1,121,292 pages which is still very small by most standards but still big enough to test on without nuking my machine and allow me to still run agents &amp; crawlers.</li><li>32.6 million links in the page content, plus 302 menu and footer links on every page</li><li>6 to 2,669 products per subcategory</li><li>66% of products that were 4 or more clicks away from the homepage</li></ul><p>I also created fake demand scores, though somewhat based on real demand, to determine a set of most important categories and products.</p><p>I continually validated this setup vs a bunch of reference site behaviours to get it as close as I could to imitating real life to the point that it could probably go live and be a real store.</p><p>The main challenge in creating the site was getting it to have the issues &amp; bugs that you’d find in real life. The test site to begin with was far too clean and too well optimised by default to be realistic.</p><h4>Measuring the link value</h4><p>Stitching together ideas from various Google patents &amp; docs, along with a few educated assumptions of my own, I created a directional model of how link value might flow through a site.</p><p>Some notable publicly disclosed methods I used were:</p><ul><li>Each page valued by its best route of links from the homepage - Google patent US 9,953,049 (PageRank-NearestSeeds), 2006</li><li>Each link passes on 0.85 of the value, split across the page’s links - The original PageRank paper, 1998</li><li>Not every link counts the same, so content beats sidebar, menu and footer and more so on the narrower intent pages - Reasonable Surfer patent, 2004</li><li>Content &amp; links after the 2MB limit don’t count with headers included in that 2MB limit - Google Search Central, 2026</li></ul><p>Basically, I created PeteRank©®™</p><p>In theory, the pattern I expected to find was that if links went missing then the value passed would be reduced not only for those pages that were no longer found but then also the pages that they linked to would also drop. A cascading effect. Overall no value is lost, it just doesn’t go where you think it does or want it to, which is why I also added a demand dimension.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aIyHbqHbgdixJIhxCdt4Sg.png" /></figure><h4>Then what happened?</h4><p>I randomly chose pages based on the observed real life distribution pattern of the types of pages to have this issue: ~60% department &amp; category pages, ~20% subcategories and ~20% filter or search pages. I generated the results 20 times over by rolling the dice on the generation. I never found a product page or other pages types that were over 2MB, though to be fair maybe I didn’t look hard enough. I took a random sample of 0.01% of the sites URLs with that distribution and then made sure to un-optimise them so as to be sure to nudge them over 2MB, again, based on the real life setups I’d observed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZWlU2_uzMuhg_bG3Z-wmug.png" /></figure><p>I should also add that different frameworks and techs cause different results. The most common issue in Next.js is that the inventory would load after the 2MB but there were other setups where the entire header navigation would be skipped for example. It varies heavily site by site and technology.</p><p>So here’s what happened in this model when 0.01% of pages went over 2MB and how it changed how pages would be found in the site by the crawler after their perceived importance could be reduced.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bpRKdW4WtdIzT6Jm_j_2UA.png" /></figure><p>All because just 0.01% of the pages were &gt;2MB. Nothing changed with the internal linking setup itself.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*F9Y0Y20dj4ibBfmpBctFmg.png" /></figure><p>It’s funny, there’s all this effort placed on carefully optimising the information architecture of a site yet an unrelated change elsewhere can ruin it all for the bots and no one would notice.</p><h4>So what can you do about it?</h4><p>It’s worth running a crawler like Ahrefs, Sitebulb or Screaming Frog over your site. For sites that have millions and billions of pages, you only really need to crawl the category pages I think to get the critical bits.</p><p>Here’s what you’re looking for:</p><ol><li>Measure uncompressed HTML and not the transfer size or the compressed size</li><li>Inspect the first 2MB and which of the links, content and metadata survive</li><li>Keep important links early ahead of large embedded data</li><li>Audit your templates, in particular your category, subcategory and faceted pages that link to many pages</li><li>Test the wider graph and look at lost links, newly unreachable pages and click depth, not only the count of oversized pages</li></ol><p>The fastest way to fixing this is to remove the unused junk from the payload because 5 to 10% payload bloat per feature is definitely not acceptable!</p><p><a href="https://www.linkedin.com/feed/update/urn:li:activity:7514022905756704768/"><em>You can also read my linkedin summary post here</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f818d27689b6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Enshittification as a Service]]></title>
            <link>https://medium.com/@petedainty/enshittification-as-a-service-82a5292aac9c?source=rss-37b7b3f873ec------2</link>
            <guid isPermaLink="false">https://medium.com/p/82a5292aac9c</guid>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[satire]]></category>
            <category><![CDATA[corporate-culture]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Pete]]></dc:creator>
            <pubDate>Thu, 08 Oct 2026 07:25:16 GMT</pubDate>
            <atom:updated>2026-10-09T09:11:37.212Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4BPOwCPRTS9JJAB_zSEcdQ.png" /><figcaption>AI generated adaptation based on a still from <em>Office Space</em> (1999)</figcaption></figure><p>Imagine a fictional, heavily matrixed (read siloed, pathological etc) corporation building a new product that spans many, many teams.</p><p>All the engineers, product managers, etc. in this company are keen to get involved so they can say that they’ve built AI powered things to keep the CEO happy. This way they can also look good in annual reviews, maybe get that promo or get the next job and higher salary.</p><p>To be certain to hit their goals, they decide to enable every mundane detail they’re responsible for with an LLM call, even if there were simpler ways to do it for free. It’s so much quicker and easier to ask the LLM. Need a simple classifier? An agent can be the judge in real time. Need to add content to the pages? No problem, an LLM can do that on the fly for you! etc etc. Each person in each team is doing exactly the same thing. You’re competing for feature velocity and vanity metrics just as much as being crowned the most AI-ist person there is.</p><p>You could collaborate with another team to build something better together, but then they may get all the recognition so it’s best to build your own thing, even if it is a rehash of something already existing and it can’t scale past your humbled and thrilled launch announcement.</p><p>You’ve checked the box, AI is enabled and management are happy, you can practically taste that promotion.</p><p>Mountains of code and documentation are generated, whether it’s any good? Who cares? You don’t need to worry, you can simply ask your AI pal to understand it all for you and remember it wholesale. You’ve got near unlimited tokens to burn!</p><p>Imagine, due to this culture, the product now has so many agentic and LLM calls in each step of the workflow, all hastily hacked in by so many different teams to meet an arbitrary deadline. Latency? Ha! Cost? Ha! Tech debt? Double-Ha!</p><p>The product is launched, press releases have been sent, shareholders are appeased, promos are given and pats on the back are received.</p><p>But then the model providers need to turn a profit and they decide it’s time to pull the rug and jack up the costs…</p><p>You’re now left with a choice to either pay up or migrate/rewrite the product. To do the latter you’ll also need to pay for those same models in order to help you understand all the code it wrote… it’s probably still cheaper than re-hiring all the people that were laid off to improve efficiency, right?</p><p>This doesn’t sound like fun, you’ve got shareholders to keep happy. Your feature factory that’s banking fictional revenue wins, can’t afford to be stopped to fix this.</p><p>Your only option then is to eat the increased costs and find the money from somewhere else… maybe stick more ads over everything…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/784/1*_qFDw2XIR7OGK641mid-Bw.jpeg" /><figcaption>Still from <em>Idiocracy</em> (2006), 20th Century Fox. Used for commentary</figcaption></figure><p>The frontier model providers know this scenario and they’re counting on it. They’re betting on creating a captive audience that they can strip mine. They’ve read the enshittification manuals.</p><p>We can already see it starting to happen: new pricing tiers, reductions in allocations, arbitrary time allowances, ads, random new fad products created.</p><p>The smarter companies built their own in-house models or adapted open-weight ones and spent the time tuning them with their own proprietary data. They run quicker, more profitably and privately too. Their competitors, who rushed ahead into heavily integrating with the frontier AI model providers (or more likely, tacking it on), eventually perish under the death spiral of rising token costs and tech debt.</p><p>Meanwhile, rather than risk it all coming crashing down, faster than you can say “preemptive bailout”, the AI model providers convince governments to invest taxpayers money in them to keep them afloat - while also pushing to limit access to the open-weight models of course. All in the name of national security, pride or whatever.</p><p>Obviously this is satire… I’m off to build my EaaS company to ruin all your softwares and you’ll pay me handsomely to do it - via a near-impossible-to-cancel subscription of course!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=82a5292aac9c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[You can’t handle the crawl!]]></title>
            <link>https://medium.com/@petedainty/you-cant-handle-the-crawl-be6a2c21c105?source=rss-37b7b3f873ec------2</link>
            <guid isPermaLink="false">https://medium.com/p/be6a2c21c105</guid>
            <category><![CDATA[web-crawling]]></category>
            <category><![CDATA[seo]]></category>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[aeo]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Pete]]></dc:creator>
            <pubDate>Wed, 30 Sep 2026 16:10:54 GMT</pubDate>
            <atom:updated>2026-09-30T16:42:04.372Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oZAO_YCiDwIjfrXl8fUIDQ.png" /><figcaption>AI generated slop image that is definitely not referencing a well known movie scene in any way whatsoever.</figcaption></figure><h4>By 2031 AI agents doing on demand fetches of our websites could be anywhere up to 56 trillion live requests per year, 4x of legacy search crawling today… and for far less referral traffic.</h4><p>The web as we know it may be moving from a system where crawling is economically subsidised by referrals to one where our sites are crawled at even greater volume for far fewer referrals. Despite this, I much prefer an agentic search experience and I think this is where the future is heading.</p><p>I recently read a <a href="https://www.idc.com/resource-center/blog/the-agent-economy-is-scaling-faster-than-it-can-be-metered/">prediction</a> of there being 2.5 billion AI agents by 2030 responsible for 459 trillion actions. Naturally then my obvious questions were how much of that is going to be hitting our websites based on query fan outs etc. triggered by user action and how do we best optimise for that as the existing internet becomes machine dominated replacing human interaction?</p><p>Bot traffic is growing much faster than human traffic at ~6.5x the rate so far this year and <a href="https://blog.cloudflare.com/radar-2025-year-in-review/">Cloudflare</a> alone saw &gt;15x increase over 2025.</p><blockquote><em>“AI traffic is not a single workload but two very different categories: AI crawlers and AI fetchers.” — </em><a href="https://quic.fastly.com/blog/ai-traffic-grew-6-5x-faster-than-human-traffic-this-year">Fastly</a></blockquote><p>While ~85% of it today is coming from AI crawlers, which are likely doing it in order to harvest our data for training, the other 15% comes from agents acting on behalf of their owners.</p><p>What I think we’re going to see over the next few years is a massive increase in the latter, I think this is inevitable as more and more people use AI whether they know it or not.</p><p>I also don’t believe that this problem will be evenly distributed, the more popular or optimised your site is, then the bigger the problem (and cost) it will become.</p><h4>Estimating live agent retrieval in 2026</h4><p>I’m cobbling together various data points to do this, if anyone has better ideas on how to estimate this then please share. It’s very simplistic and rough but here’s my extrapolation for live agent retrievals based on:</p><ul><li><a href="https://blog.cloudflare.com/ai-labyrinth/">Cloudflare reporting</a> <strong>&gt;50 billion AI crawler requests/day</strong> in March 2025 (the latest absolute number I could find).</li><li>Their <a href="https://radar.cloudflare.com/ai-insights?dateStart=2026-06-13">Cloudflare Radar 2026 data</a> says <strong>72.8% of AI crawler fetches were HTML</strong> in H1 2026.</li><li><strong>4.80% of AI crawler requests were of “User Action”</strong> in <a href="https://radar.cloudflare.com/ai-insights?dateStart=2026-08-07">August 2026</a>.</li><li>According to <a href="https://almanac.httparchive.org/en/2025/cdn">HTTP Archive</a>, <strong>35% of base HTML content was served via a CDN</strong> and amongst those HTML requests, Cloudflare accounted for <strong>58%</strong> of them, which would then give them a<strong> ~20.3% market share.</strong></li></ul><p>So if we take the 50 billion and multiply it by 72.8% and then 4.8% we get to 1.747 billion potential live HTML fetches per day or, 638 billion per year for Cloudflare. We can then use the market share assumption to get to <strong>~3.14 trillion live HTML fetches per year in 2026</strong> or thereabouts for a rough total.</p><p>I’m using some pretty heroic assumptions since I’m mixing Cloudflares March 2025 absolute volume with 2026 traffic shares and HTTP Archive’s sample isn’t the whole web. Also, some sites tend to use multi-CDN setups so I could be double counting in my extrapolation. I’m also assuming that the share of AI crawling tracks the share of HTML delivery which may not be true.</p><p>What’s also interesting is that these live agent requests typically do everything they can to hit the origin directly to get the latest data and avoid the cache. Fastly had this at about ~51% which means my extrapolation of 3.14 trillion requests, <strong>1.6 trillion of them are likely fetching from the origin </strong>which is typically more expensive.</p><blockquote><em>“The modern internet is becoming less human by the day.” — </em><a href="https://cpl.thalesgroup.com/en-gb/node/52621">Thales Cyber Security</a></blockquote><h4>Factoring in growth</h4><p>Here’s what I think growth may look like over the next few years for these user triggered live web fetches. I’m using spend estimates and broad market size / growth as a proxy but if anyone knows any better data for prompt growth please let me know.</p><ul><li><strong>Low: 28.7% Compound Annual Growth Rate (CAGR) </strong>via the <a href="https://www.researchandmarkets.com/reports/5806998/generative-ai-market-report">Business Research Company</a> forecasting the global generative AI market from $47.28B in 2026 to $129.83B in 2030.</li><li><strong>Mid: 44.5% CAGR</strong> via<a href="https://www.marketsandmarkets.com/Market-Reports/ai-assistant-market-40111511.html"> MarketsandMarkets</a> which forecasts the AI assistant market from $3.35B in 2025 to $21.11B in 2030.</li><li><strong>High: 66.8% CAGR</strong> from <a href="https://www.gartner.com/en/documents/6974566">Gartner</a> who forecast Generative AI model spending growing to reach $76 billion by 2029.</li></ul><p>Using those figures I can then estimate what this may look like into 2031, also hedging 2026 with a low end (1.96 trillion) and high end (4.32 trillion).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*x-BLEBxhoSxblKRDvIco0w.png" /><figcaption>As early as mid 2028 user triggered agent crawling could be more than the legacy search requests, by 2031 it could be as much as 4x more URLs crawled on your sites.</figcaption></figure><p>For reference I’ve also added in a line for legacy search engine crawlers based on <a href="https://blog.cloudflare.com/radar-2025-year-in-review/">Cloudflares data</a> which I‘ve modeled declining at ~-5% annually as I think it’s reasonable to assume some level of substitution occurs.</p><p>What’s interesting here is that most people are currently optimising for legacy search bot behaviour but in the above scenario by 2030, maybe as early as 2028, it starts to become the edge case as owner triggered agent bot behaviour takes over.</p><p>Worryingly though, as noted earlier, at least half of that projected traffic is going to be going direct to your own servers which are going to cost you more versus the optimised CDN where costs are substantially less. How much more really depends on how dynamic your content is but it could easily reach 10x the cost when you look at example like AWS Lambda without a CDN in front of it. Also keep in mind that this is <em>additional</em> bot traffic.</p><h4>So what’s it all worth?</h4><p>As I wrote briefly in a <a href="https://medium.com/towards-artificial-intelligence/stop-serving-blanks-to-bots-a476477c9ccd">prior article</a>, I think we’re going to be getting the very shortest end of the biggest stick when it comes to referrals vs amount of content crawled/leeched.</p><blockquote>“Every AI bot combined sends under 1% of referral traffic but to do that they’re operating at a very disproportionate share of crawl.”</blockquote><p>I fetched the latest figures of crawls vs referrals via <a href="https://radar.cloudflare.com/explorer?dataSet=bots.crawlers&amp;dt=2026-07-01_2026-07-31">Cloudflare Radar</a> to see how much of a bad deal this could end up being…</p><p>We have to also allow for the fact that most URLs fetched are from within the training corpora which is why I used the broader AI crawler ratios instead of specifically for the queries.</p><ul><li>Google - 4.7 : 1</li><li>OpenAI - 251 : 1</li><li>Perplexity - 289 : 1</li><li>Anthropic - 1,917 : 1</li></ul><p>Using those with my crawl assumptions, we can see the deal between crawlers and content owners will be thoroughly broken and remember this is only for user triggered agent crawling and I’m keeping the ratio constant - over time it has gotten better, so it may not end up this bad… maybe.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kuwGJfi-Z3mmpBEzhJyzaQ.png" /><figcaption>Despite the increase in agent crawl requests that may as much as 4x legacy search crawlers in 2031, the referrals at best would be just 8% of legacy search based on the crawl to referral ratios in 2026. This also uses the best figure which is from OpenAI, if it were based on Anthropic then the line wouldn’t be visible. I’m also assuming that zero click search results continue to grow in the legacy search world.</figcaption></figure><p>These figures are also going to vary heavily site by site depending on the content and how dynamic it is, for example large ecommerce &amp; travel sites will get hit harder due to the need for up to the minute pricing and availability.</p><h4>That’s all quite potentially horrifying</h4><p>So how do we optimise for it? Or better yet how do we defend ourselves, especially if it’s skipping past the CDN caching.</p><p>I can’t help but instinctively agree with Jamie Indigo:</p><blockquote>“The only reasonable response to the increasing volumes and ever-growing audacity of AI crawlers is to release the gzip bombs.” - <a href="https://www.linkedin.com/feed/update/urn:li:activity:7510416668993540096/">Jamie Indigo</a></blockquote><p>I’ve only been talking about the user triggered agent crawling (15%) and not the training crawlers (the other 85%) or even the bad bots out there - It’s going to be getting oh so much worse.</p><p>I have some ideas of how to optimise for this that I’ll share next. I would say that the larger sites are better placed to handle this but then <a href="https://medium.com/stackademic/why-you-stalking-my-robots-txt-file-bro-490aa2407642">I took a look at their robots.txt files</a> and I’m not so sure anymore.</p><p>However, it’s safe to say that the role of technical SEO, edge and bot optimisation will become more and more valuable in the coming years.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=be6a2c21c105" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The bots did exactly what you asked]]></title>
            <link>https://blog.stackademic.com/the-bots-did-exactly-what-you-asked-6a3540d56f37?source=rss-37b7b3f873ec------2</link>
            <guid isPermaLink="false">https://medium.com/p/6a3540d56f37</guid>
            <category><![CDATA[web-crawler]]></category>
            <category><![CDATA[marketing]]></category>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[seo]]></category>
            <dc:creator><![CDATA[Pete]]></dc:creator>
            <pubDate>Mon, 21 Sep 2026 16:10:13 GMT</pubDate>
            <atom:updated>2026-09-24T01:59:50.684Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*34DNSYWoVfyreODMDVH2Lw.png" /></figure><h4>Typos, wildcards, user agents and what the crawlers thought you told them to do… or didn’t!</h4><p>Continuing on from my series on robots.txt where I’ve taken the top ranked domains and analysed their robots.txt files.</p><ul><li><a href="https://medium.com/@petedainty/why-you-stalking-my-robots-txt-file-bro-490aa2407642">Why you stalking my robots.txt file, bro?</a></li><li><a href="https://medium.com/@petedainty/you-cant-just-invent-a-protocol-bruh-642bc7b32f0b">You can’t just invent a protocol bruh</a></li></ul><p>People seem to worry most about content, link building, chunking or whatever for SEO (and now AEO etc.) but clearly the importance of robots.txt is being missed.</p><p>I’ve had to check a lot of these rules in detail individually to see if I’d missed some trick, I’ve lost count of the times I’ve said “but why would you do that?” and “what am I missing?” but it really is this bad.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0q0knfkB42Y02YZBlQnruA.png" /><figcaption>21.46% of the 168,207 robots.txt files of the top ranked domains in the world had at least one defect in them</figcaption></figure><p>That’s certainly interesting, I was not expecting to see numbers that high. I tried to exclude anything that could be argued that it was done for optimisation reasons and focused on what looked broken or might behave differently versus the intent.</p><p>This doesn’t necessarily mean all crawlers will face an issue, some may be nice and adjust for an issue and give the desired result… or not. I wouldn’t like to gamble on that nor have the headache of maintaining it all.</p><p>The majority of the following examples I’m detailing below are for sites within the top 1,000 ranked domains in the world, the kind of sites that employ SEO agencies and staff… so keep that in mind.</p><p>A key lesson from all of this: <em>check and test your robots.txt file!</em></p><h3>Be wary of typos</h3><p>I saw some files where they had these giant NO AI BOTS ALLOWED!!! exclusion rules but actually gave broader access to those bots vs others simply because of a typo, like this one for a well known site:</p><pre># ==================================================<br># BUCKET 2: AI TRAINING / DATA INGESTION FIREWALL<br># Completely blocks models from scraping copy assets for foundational training.<br># ==================================================<br>User-agent: GPTBot-Web<br>User-agent: CCBot<br>User-agent: Anthropic-AI<br>User-agent: Claude-Web<br>User-agent: Bytespider<br>Disallow: /[1.1]</pre><p>That [1.1] is treated as a path, so the intent of the rule is completely invalidated and everything on that site is wide open to those model training bots and not “completely blocked”. It meant that in that file, 191 of the 192 possible paths were actually unblocked vs just 7 for Googlebot. The same site also spelt Gemini as Gemeni-AI, whatever that is.</p><p>3,619 sites, ~2% of the total, had typos in their allow or disallow rules that meant that they didn’t work e.g. things like missing the : after the allow or disallow which then rendered the rule pointless.</p><pre>Disallow /account/*</pre><p>Some parsers may be ok missing a : for example after the allow or disallow but not all of them are. If that example above is set for user-agent: * then it may work for some and not for others.</p><p>Other common issues included mispellings, empty values and in some cases, 0.26% of the time, the directives were even written above the user agents.</p><h3>I can just use the user-agent string, right?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4Ke6eC7AsQL0LkZEWdosZg.png" /><figcaption>2.72% of the 168.207 files contained user agents with at least one issue</figcaption></figure><p>1% of sites were using full user agent strings or URL paths instead of the correct user agent token. Another very well known top site thinks that using the full user agent string is the way to go…</p><pre>User-agent: Baiduspider+(+http://www.baidu.com/search/spider.htm)<br>User-agent: Baiduspider/2.0;+http://www.baidu.com/search/spider.html<br>User-agent: Mozilla/5.0(compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)</pre><p>None of those are valid robots.txt tokens, same as if you’re using an email address and yes, over 500 sites were doing that too.</p><p><a href="https://www.rfc-editor.org/info/rfc9309/#section-2.2.1">RFC 9309</a> goes out of its way to document the user agent string and the short robots token that should be used for it.</p><h3>user-agent: * is only used when there’s no matching group</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4VdnkhoT5qz-Ys7J8zZFdQ.png" /><figcaption>5.28% of the 168,207 robots.txt files of the top ranked domains via Tranco were blank or technically had no rules specified. Some had obvious issues like being rendered in HTML or setting rules with no user agent line but the rest were correct to the standards though perhaps not what was intended.</figcaption></figure><p>What was also interesting is that a default wild card rule was missing ~1.5% of the time but some bots were specifically named in those same files. The trouble with that is, if the bot isn’t named then it gets full access.</p><p>One site took this to the extreme. In its robots.txt file it proudly stated Bad Bot Count: 527 and then individually listed disallow rules for hundreds of crawlers, it could have just used the user-agent: * fallback instead and then selectively allowed the crawlers they wanted to give access to, which I think is the minority nowadays.</p><p>Bad bot number 528 is always just around the corner and it may not follow your robots.txt anyway.</p><pre>### Version Information #<br>###################################################<br>### Version: V3.2018.05.828<br>### Updated: Tue May  8 11:37:04 SAST 2018<br>### Bad Bot Count: 527<br>###################################################<br>### Version Information ##<br><br>User-agent: 360Spider<br>Disallow:/<br>User-agent: 404checker<br>Disallow:/<br>User-agent: 404enemy<br>Disallow:/<br>User-agent: 80legs<br>Disallow:/<br>User-agent: Abonti<br>Disallow:/<br>.....<br>+ Another 1,000+ lines of this...</pre><h3>People declare multiple user agents in one line followed by the rules</h3><p>The expectation is that by doing this it somehow means the rules will apply to all bots - it won’t!</p><pre>User-Agent: nachtschatten, neofonie search:robot<br>Crawl-delay: 10</pre><p>Multiple crawler user agents above comma delimited but Robots.txt isn’t a CSV. If you want to list multiple user agents for one rule, it’d be better to do this:</p><pre>user-agent: nachtschatten<br>user-agent: neofonie <br>Crawl-delay: 10</pre><p>I left out search:robot in my correction as : is an invalid product token based on the RFC spec. Crawl-delay also is non-standard, I left it in there only for clarity of the example. I’d handle crawl delays at the tool, server or CDN levels and not in robots.txt.</p><h3>Wildcards are wild</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rWOAmg3PYNnAPpZV-2uyxw.png" /><figcaption>1.90% of 168,207 robots.txt files of the top ranked domains had some issue with wildcards or regex</figcaption></figure><p>You have to be careful when using * in the user agent</p><pre>User-agent: Mediapartners-Google*</pre><p>Based on the protocol that * makes it an invalid name, though of course it’s dependent on who’s crawling it. It’s probably a harmless wildcard in effect since the standard doesn’t define wildcard matching for the user agent, <em>but don’t depend on parser forgiveness</em>.</p><p>This wildcard typo however…</p><pre>Disallow: /.*</pre><p>That will disallow files or paths starting with . such as /.gif for example, or /.abc/def would qualify too, but, /image.gif or /abc/ would still be allowed.</p><p>While .* is valid regex, here that won&#39;t work as the . is read as a literal full stop. Robots.txt has only 2 special characters used for URL pattern matching, * and $.</p><p>I saw just under 3.5% of the top 1,000 sites with issues like this for wildcards and they can be deadly, as seen here with an example from a well known postal service company.</p><pre>User-agent: *<br><br>Disallow: /a/<br>Disallow: /b/<br>Disallow: /c/<br>Disallow: /d/<br>Disallow: /e/<br>Disallow: /f/<br>Disallow: /g/<br>Disallow: /h/<br>Disallow: /i/<br>Disallow: /j/<br>Disallow: /k/<br>Disallow: /l/<br>Disallow: /m/<br>Disallow: /n/<br>Disallow: /o/<br>Disallow: /p/<br>Disallow: /q/<br>Disallow: /r/<br>Disallow: /s/<br>Disallow: /t/<br>Disallow: /u/<br>Disallow: /v/<br>Disallow: /w/<br>Disallow: /x/<br>Disallow: /y/<br>Disallow: /z/<br>Disallow: /**</pre><p>Yes, that really is from a real robots.txt! No idea why they have letters of the alphabet for directories to block but after painstakingly doing that, and more directories too, that /** at the end will block everything for everyone, even the homepage. It’s basically disallow: / for the entire site… well done! A strong contender for the SEO Darwin Awards.</p><blockquote>“I just don’t get it, why aren’t we getting any SEO traffic?” is what they’re probably asking.</blockquote><p>Roughly ~2k sites, ~1.25%, were trying to use regex style syntax in their robots.txt files which is unsupported.</p><p>This example below is valid, the . are literal, although they probably would want a $ on the end of those rules to be fully safe though.</p><pre><br>Disallow: /general.*<br>Disallow: /anysearch.*</pre><p>This next one however is invalid, ^ has no meaning, it is taken literally.</p><pre># Allow /recommend deny /username/recommend<br>Disallow: ^/*/recommend</pre><h3>To train or not to train, that is the question</h3><p>On the AI front, where 29k sites had listed some preference for these crawlers, ~73% of those only wanted to block the training bots, ~26% of sites blocked both. A case of don’t train on me but I expect you to send me your traffic.</p><pre># Block Open AI - Allowing specific marketing/informational paths<br>User-agent: GPTBot<br>Disallow: /<br>Allow: /features/<br>Allow: /solutions/<br>Allow: /product/<br>...</pre><p>This same robots.txt also didn’t state OAI-Searchbot so the retrieval bot falls back to the wildcard policy (if there is one). There were quite a few cases where sites were giving these bots way more access than any other bot, either due to typos as we saw in an earlier example, or by omission of the user agents.</p><p>Where a site blocked a training crawler, the corresponding retrieval crawler was frequently not named at all.</p><ul><li>For sites blocking GPTBot, ~86% didn’t name OAI-SearchBot.</li><li>For ClaudeBot, ~91% didn’t name Claude-SearchBot and ~92% didn’t name Claude-User.</li></ul><p>So in these cases they fall back to whatever the wildcard policy says (if it exists).</p><p>There were also a small minority of sites (0.3%) that went the other way and blocked retrieval but allowed training.</p><p>This next example was fantastic though, just comments that are completely ignored in a robots.txt and I doubt any LLM is going to take any notice of it.</p><pre># As a condition of accessing this website, you agree to abide by the following<br># content signals:<br><br># (a)  If a content-signal = yes, you may collect content for the corresponding<br>#      use.<br># (b)  If a content-signal = no, you may not collect content for the<br>#      corresponding use.<br># (c)  If the website operator does not include a content signal for a<br>#      corresponding use, the website operator neither grants nor restricts<br>#      permission via content signal with respect to the corresponding use.<br><br># The content signals and their meanings are:<br><br># search:   building a search index and providing search results (e.g., returning<br>#           hyperlinks and short excerpts from your website&#39;s contents). Search does not<br>#           include providing AI-generated search summaries.<br># ai-input: inputting content into one or more AI models (e.g., retrieval<br>#           augmented generation, grounding, or other real-time taking of content for<br>#           generative AI search answers).<br># ai-train: training or fine-tuning AI models.<br><br># ANY RESTRICTIONS EXPRESSED VIA CONTENT SIGNALS ARE EXPRESS RESERVATIONS OF<br># RIGHTS UNDER ARTICLE 4 OF THE EUROPEAN UNION DIRECTIVE 2019/790 ON COPYRIGHT<br># AND RELATED RIGHTS IN THE DIGITAL SINGLE MARKET.</pre><p>That’s it, nothing else. Just those comments. Any bot that the follows robots.txt protocol can do whatever it likes. 🤦‍♂️</p><h3>People falsely believe in the concept of inheritance in robots.txt</h3><p>There are sites that I presume are assuming that the order of matching groups somehow creates inheritance with only additional rules specified. This can mean you’ve allowed one bot more access than you think you have.</p><p>The good:</p><pre>User-agent: *<br>User-agent: Yandex<br>[the global rules]<br><br>...<br><br># New user agent groups must also have a user agent reference in the global (*)<br># group.<br>User-agent: Yandex<br>Disallow: /about/careers/applications/jobs/results<br>Disallow: /about/careers/applications-a/jobs/results</pre><p>The two Yandex groups in the above example will get merged because they match the same product token, the * is not inherited.</p><p>The bad:</p><pre>User-agent: *<br>User-agent: Googlebot<br>User-agent: Bingbot<br>...<br>Disallow: /cart<br>Disallow: /profile/*<br>Disallow: /identity/signin?*<br>... 180+ more rules ...<br><br>User-agent: GPTBot<br>User-agent: OAI-SearchBot<br>User-agent: ChatGPT-User<br>...<br>[their own much smaller rule set]</pre><p>The GPTBot etc. declared afterwards do not pick up those disallow rules at the start, they get ignored so they get so much more access.</p><h3>It’s OK my robots.txt will block access for all those nasty bots</h3><p>No it won’t, robots.txt is only an advisory policy for those crawlers who decide to follow it, so adding a pesky bots user agent token or string will do nothing here if that bot chooses to ignore it, which every bot can. I can’t believe that out of the 16,008 individual user agents specified in the files I checked that absolutely all of them will obey robots.txt.</p><p>Instead, I would prevent access or enforce restrictions at the server, application or WAF levels where possible. Getting your site on to something like Cloudflare, Fastly or Akamai is worthwhile to at least use their bot management controls especially at a time when bot activity is exploding.</p><h3>So what should you do?</h3><p>Across the files I surveyed there were 16,008 distinct user agents, granted that includes the silly stuff and typos but that’s a lot of rules to write and debug in a file that’s more of a policy than an enforcer.</p><p>If you would disallow a bot from accessing everything on your site then do that at the server or CDN level first, much cleaner that way and much more likely to be enforced while also protecting your resources.</p><p>Once done there then you can either do allow by default and selectively disallow, or, go the other way and disallow by default and selectively allow. Do whichever is the smaller set to manage. To disallow by default means all bots are blocked also by default so you have to specifically add each user agent, though you can handle that by doing groups if the rules are the same. With bot activity exploding I’d be more inclined to block by default.</p><p>Finally, <em>always test your robots.txt file</em>, there are many free testers out there and then also try to <em>keep it simple</em>.</p><p>Next up in this series, a look at the 16k user agents and who’s being allowed and disallowed…</p><h3>Before you go</h3><ul><li>Please take a moment to like the post and follow the writer!</li><li>Did you know that over 400,000 developers share what they’re building, learning, and discovering across our platforms every month? Learn how you can contribute <a href="https://plainenglish.io/write-for-us"><strong>here</strong></a><strong>.</strong></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6a3540d56f37" width="1" height="1" alt=""><hr><p><a href="https://blog.stackademic.com/the-bots-did-exactly-what-you-asked-6a3540d56f37">The bots did exactly what you asked</a> was originally published in <a href="https://blog.stackademic.com/">Stackademic</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[You can’t just invent a protocol, bruh]]></title>
            <link>https://blog.stackademic.com/you-cant-just-invent-a-protocol-bruh-642bc7b32f0b?source=rss-37b7b3f873ec------2</link>
            <guid isPermaLink="false">https://medium.com/p/642bc7b32f0b</guid>
            <category><![CDATA[aeo]]></category>
            <category><![CDATA[seo]]></category>
            <category><![CDATA[marketing]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[web-scraping]]></category>
            <dc:creator><![CDATA[Pete]]></dc:creator>
            <pubDate>Fri, 18 Sep 2026 08:24:54 GMT</pubDate>
            <atom:updated>2026-09-24T01:45:56.773Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aBXjVxsA5GpO2pl_f3BvIg.png" /></figure><p>Join me as I continue down the rabbit hole of “weird stuff people put in a text file and pretend it’ll work”.</p><p><a href="https://medium.com/stackademic/why-you-stalking-my-robots-txt-file-bro-490aa2407642">A reminder: I checked 267k of the top ranked domains for their robots.txt files out of curiousity to see what people were doing.</a></p><p>Turns out people will put all kinds of things in their robots.txt files that’ll have zero impact or worse, do the opposite of what they intended. The big sites do it too and often they take it to a whole new level… and not in a good way.</p><p>First up is the category of things that look reasonable but are not part of the standard, in this case that means it’s part of <a href="https://www.rfc-editor.org/info/rfc9309/">RFC 9309</a> (or not). There should be no expectation that a crawler will understand any of these things - it can be specific to the crawler and will work for that crawler but not for all of them.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*c0ZVFA0p_y3Yeql502RXOw.png" /><figcaption><em>~26% of the sites serving robots.txt files contained directives which aren’t part of the robots.txt standard and so have no common meaning across crawlers.</em></figcaption></figure><blockquote>“Yeah bruh, I set visit-time and request-rate in my robots.txt so the bots will only visit between 2am and 4am and won’t do more than three requests per second.”</blockquote><p>Cool. Or here’s a thought: configure your server to actually enforce that.</p><p>So some crawlers out there may support some of these things e.g. clean-param is for Yandex and Claude will support crawl-delay but Google will support neither, though people still set it for Google of course.</p><p>One problem is that you end up adding lines and repeating rules per bot adding bloat and complexity to set something here that you could instead do in better places like the tool itself, the page or the server where they would be more enforceable.</p><p>Also, because of this complexity a lot of the rules may not work like you think they would e.g. when a named user agent group replaces the wildcard behaviour rather than inheriting it which I saw happens to about ~19% of the rules written.</p><p>Or…</p><p>When you run past Googles 500KiB (512Kb) file size parsing limit for robots.txt!</p><p>Believe it or not people do run out of space in this file. Such as the clowns who listed 10,376 individual sitemaps in their robots.txt but they were then out SEO’d by the site listing 76,728 individual sitemaps!!! Though not to be out done another site had an 11MB robots.txt file with 192,264 individual disallow rules and a sitemap link at the bottom of all that. Sorry, your 10Kb robots.txt file isn’t going to cut it anymore!</p><p>Those of course are the comical extremes and SEO Darwinism at it’s finest, but the point is this: if you’re adding in non-standard rules for each crawler, sooner or later you’re going to create unintended side effects.</p><p>Which handily leads us on to the smaller but considerably weirder nonsense people are making up in the name of LLMs which is akin to the SEO quackery from decades ago, like this beauty…</p><pre>Allow ai-use: true<br>AI-Policy: /ai.txt<br>AI-Info: /llms.txt<br>AI-Legal: /ai-policy.txt</pre><p>From what I could see in any documentation or implementation that I could find, that’ll do nothing. Absolutely nothing. You’d be better off drawing some nice ASCII art instead. Same for LLMs-policy, LLMs etc. or whatever you’re currently believing in, in the name of AI.</p><p>Here’s my AI policy text that’ll work just as well:</p><pre>   T<br> .-&quot;-.<br>|  ___|<br>| (.\/.)      _________________________<br>|  ,,,&#39;      /                         \<br>| &#39;###&lt;     |  LLMS PLEASE DON&#39;T CRAWL  |<br> &#39;----&#39;     |  ME UNLESS YOU&#39;RE GIVING  |<br>            |  ME FREE TRAFFIC!         |<br>             \_________________________/</pre><p>Things like LLMS.txt may well become standards in the future but right now it looks like people believe if they put stuff like ‘llms’, ‘llm-policy’, ‘AI-policy’ etc. inside robots.txt then it magically manifests protocols for the chatbots to follow. You’re better off writing a letter to Santa in there instead and asking him to help you with your AI visibility problems.</p><p>Use the disallow and allow rules correctly for the relevant user agents and you’re done for crawling, although usage of your content is a different beast altogether. Sorry, there’s nothing new here and if you’re being told or sold this LLM policy nonsense then tell whoever it is to GTFO.</p><p>I also saw TDM-policy used, it’s an interesting one and it’s actually a real thing it’s just being used in the wrong place. TDM is Text and Data Mining and the concept of providing a URL of a licensing policy that by specifying this the bots then can optionally take a look at after they’ve scraped your data. By doing this you’re trying to preserve your usage rights after the fact and we all know how well that’s been working out so far for copyright protections and the AI bro’s. But the main thing with TDM-policy, like so many of these things, is that it’s meant to be set in the HTTP header or the HTML file itself and not in the robots.txt file!</p><p>In that same spirit there’s the <a href="https://robotstxt.com/ai">emerging proposals &amp; controls</a> that Cloudflare, Microsoft and Google are each proposing something different to be added: Content-Signal, DisallowAITraining and Content-Usage.</p><p>Similar to the idea of the TDM-policy these are meant to specify if your data, after it’s been crawled, may be used for things such as for AI training. But, OpenAI, Anthropic etc. have different crawlers for training vs retrieval so this seems a bit moot to me when you could just specify rules for the training user agents instead and handle usage that way.</p><p>I can’t think of a case where I’d allow the training bot to come and leech my content and hammer my server resources only for me to then tell it not to use it after the fact. That’s also assuming that they’ll honour robots.txt in the first place.</p><p>For Bing &amp; Google, well don’t they use their current crawlers &amp; index to train on anyway? Yes and no.</p><p>Google-Extended demonstrates the problem: they’re trying to retrofit usage preferences (not rights) onto a protocol that was designed to say whether a bot may crawl a URL. While they may already have your content through normal Google crawling, Google-Extended is then used as a separate usage control for future model training. Although, that then begs the question of what about all the other features &amp; models in legacy search engines that have been trained on your data in the name of product improvements, ranking and relevance over the years. So that was OK but you’re not cool with it being used to train the LLMs or by other bots?</p><p>And this is where it gets a bit weirder still. For example, the current Content-Usage proposal explicitly says changes aren’t retroactive. That means that if I tell you today not to train on my content that you fetched six months ago, my updated preference only works after you’ve picked up my new robots.txt and fetched the page again which you’ve already been training on, so are you then going to un-train on my data?</p><p>We’re rapidly going from “please don’t crawl this” and “please don’t index this” to “you may already have a copy of this page but if you do decide to fetch it again after reading my newer copy of robots.txt then my preferences for what you do with that new copy are now different.”</p><p>Ok… but what should you do?</p><h3>Use the standard stuff… that’s it!</h3><p>RFC 9309 itself is pretty small, the actual protocol is basically User-agent, Allow and Disallow with sitemap as an established additional record rather than part of core REP. Boring!</p><h3>You can use the non-standard stuff but I think there’s better ways</h3><p>These may have some use for some tools &amp; crawlers but won’t work for all and I would avoid using them if I could.</p><p><strong>crawl-delay</strong> <br>A polite advance warning to try to slow down if the crawler understands it but I think it’s better to enforce the limit in the tools settings or script if you can or the server or CDN and return a status 429 Too Many Requests with a Retry-After when necessary.</p><p><strong>clean-param</strong> <br>Tells a crawler that particular query parameters don&#39;t materially change the content and is for Yandex. The better fix though is to stop generating unnecessary parameter URLs where possible and use redirects or canonical tags instead. That’s good for everyone.</p><p><strong>content-signal <br></strong>For preferences about what crawled content may be used for e.g. search, AI input or model training. This is created and implemented by Cloudflare. As I’ve said, I think that if you don’t want your content used in model training for example then it’s better to block the bots directly instead.</p><p><strong>request-rate</strong> <br>This attempts to tell a crawler how many requests it may make over a period, before the crawler overloads you and this is typically the small python based crawlers or tools that some ones running adhoc. I’d do this either in the tool or script itself or with the server/CDN rate limiting backed by the 429 and Retry-After.</p><p><strong>visit-time</strong> <br>Same idea as request-rate but for time windows: please crawl me at these hours. Like request-rate I would instead use retry-after and defer requests until the right time window. No one supports visit-time from what I see.</p><h3>Real things but mistakenly put inside robots.txt</h3><p>Adding these things in does not magically make them a robots.txt rule.</p><p><strong>noindex <br></strong>If you want a publicly accessible page excluded from search, use the meta robots tag in the HTML or the X-Robots-Tag in the header. Also, don’t block the crawler from fetching the page first because then it can’t see the noindex.</p><p><strong>nofollow <br></strong>Use robots meta / X-Robots-Tag for a page wide instruction or rel=nofollow, ugc or sponsored on individual links.</p><p><strong>host</strong> <br>This was an historical attempt to declare a preferred hostname, but just use redirects, canonicals and consistency in your URLs instead.</p><p><strong>cache-delay <br></strong>HTTP already has an entire caching and validation layer for this with Cache-Control, ETag, Last-Modified, conditional requests and 304 Not Modified. Use that, it’ll work unlike adding cache-delay in your robots.txt</p><p>Then there’s the other less frequent stuff I saw in the data that’s legit but also being used in the wrong place such as X-Robots-Tag is literally an HTTP header and in there you also have max-snippet, max-image-preview and max-video-preview though for those you can also use the robots meta. Why put them in the robots.txt file I have no idea. Similarly, tdm-policy belongs to TDMRep, IndexNow has its own submission protocol and OpenAPI, MCP and Agent Cards all have their own discovery mechanisms. Don&#39;t put this stuff in your robots.txt file, you’ll only be disappointed.</p><h3>And then there’s this shit</h3><p>Remember, some of the top ranked domains in the world are doing this and probably have paid an “expert” to come up with these things that do nothing at all under the guise of AEO, AIO, GEO or whatever-EO term you like.</p><p>Large sites and small really do have these things in their robots.txt files: llm-policy, llms, llm, llms-txt, llm-content, llmstxt, llms-full, llm-full-content, llms-full-txt, llms-file, llm-info, llm-feed, llmsconfig, llmtxt, llms index, llm-manifest, llm-instructions, llm-config, llm-txt, llm-sitemap, llm policy, llm-discovery, url llms, llm-access, llms-small, llms-text, llms-api, llm-full, llms page, llm-file, llm-friendly, ai, ai-sitemap, ai-policy, ai-manifest, ai-crawlers, ai-index, ai-directory, ai product feed, ai information, ai-content, ai-info, ai-legal, ai-feed, allow-ai, allow-ai-agent, ai-crawlers-full, ai-access, ai-access-json, usage, attribution, licence, attribution-url, allow-training, allow-generation, require-attribution, agents, signin-agent, api-docs, agent-manifest, agent-homepage, agent-homepage-txt, agent-walkthrough, agent, agente utente</p><p>None of them do anything…</p><p>Just use the basics in robots.txt and do everything else in the better, intended places and you’re good!</p><p>Next, <a href="https://medium.com/@petedainty/the-bots-did-exactly-what-you-asked-6a3540d56f37">I’ll get in to more of the rules being specified that don’t work</a> such as using a full browser UA string, or a URL, or perhaps the surprisingly large amount of people using email addresses, or the use of * in a user-agent strings (which will not work).</p><h3>Before you go</h3><ul><li>Please take a moment to like the post and follow the writer!</li><li>Did you know that over 400,000 developers share what they’re building, learning, and discovering across our platforms every month? Learn how you can contribute <a href="https://plainenglish.io/write-for-us"><strong>here</strong></a><strong>.</strong></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=642bc7b32f0b" width="1" height="1" alt=""><hr><p><a href="https://blog.stackademic.com/you-cant-just-invent-a-protocol-bruh-642bc7b32f0b">You can’t just invent a protocol, bruh</a> was originally published in <a href="https://blog.stackademic.com/">Stackademic</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why you stalking my robots.txt file, bro?]]></title>
            <link>https://blog.stackademic.com/why-you-stalking-my-robots-txt-file-bro-490aa2407642?source=rss-37b7b3f873ec------2</link>
            <guid isPermaLink="false">https://medium.com/p/490aa2407642</guid>
            <category><![CDATA[marketing]]></category>
            <category><![CDATA[aeo]]></category>
            <category><![CDATA[seo]]></category>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[web-crawling]]></category>
            <dc:creator><![CDATA[Pete]]></dc:creator>
            <pubDate>Wed, 16 Sep 2026 17:28:28 GMT</pubDate>
            <atom:updated>2026-09-18T17:30:18.746Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BDRn3tUgF5y1JxwEoLK91g.png" /></figure><p>I got side tracked… I think it was when I saw a large ecommerce site allowing every known AI crawler access to everything on their site while at the same time their CEO was saying how bad AI was and that they were blocking all of it… only they weren’t. Obviously that status update was not received across the company.</p><p>Anyway, it got me to wondering how many people think their robots.txt file is doing one thing but actually doing something else entirely. <a href="https://www.robotstxt.org/orig.html">The protocol</a> has been around 30+ years and it’s the foundation for all things SEO so you’d think everyone would get it but nope! And it actually seems to be more the case that it’s the larger sites getting it wrong and a few even comically so.</p><p>For the uninitiated, robots.txt is a public note sitting at the root of a website that tells the well behaved crawlers which parts of the site they should and shouldn&#39;t crawl. The not so well behaved crawlers, and <a href="https://blog.cloudflare.com/perplexity-is-using-stealth-undeclared-crawlers-to-evade-website-no-crawl-directives/">Cloudflare is looking at you Perplexity</a>, tend to ignore it and do whatever they want in order to scrape you in the name of all things AI probably.</p><p>This file is one of the very first things I’d argue you want to look at for search engine and agentic optimisation. But it’s OK if you have no idea what robots.txt is about because apparently neither do some of the largest sites out there.</p><p>It all started out so innocently as ‘I wonder’ and then quickly moved into the ‘oh no they didn’t!’ territory and I just had to keep going.</p><p>I ended up taking a list of ranked domains from <a href="https://tranco-list.eu/">Tranco</a>, fetching the robots.txt files of the top 250k or so domains and analysing each of the files for no other reason than my curiousity. I’m measuring what sites are telling crawlers to do, it’s not like I can know whether the crawlers obey them, only the hopes and wishes of the author of the file which quite often turn out to be misguided. I could use the HTTP Archive project for this and I still might but I only really needed to look at the robots.txt files and Big Query is Big Expensive for this initial look so I wrote a quick script to fetch the files for &lt;domain&gt;/robots.txt and to store them in an SQLite DB to then mine later.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ivGOAE7kyeiw2odxx6NZQw.png" /><figcaption>Data from a manual crawl of the top ranked domains robots.txt files for the start of September 2026</figcaption></figure><p>I got to 267,053 of the top ranked domains before stopping, of those, 58,247 weren’t actually websites or didn&#39;t resolve DNS or HTTP service when I checked them so I couldn’t directly get a robots.txt file which is interesting in and of itself: ~22% of the top domains in Trancos ranked list aren’t websites.</p><p>Of the remaining 208,806 sites, I could only fetch the robots.txt file for 189,070 of them and of those, 168,207 were available - that’s ~63% of the top ranked domains that had accessible robots.txt files that weren’t 4xx status!</p><p>Weirdly, a very large domain registrar and hosting platform who really should know better, decided to serve theirs as HTML! Which basically means it’s unreliable as to whether the rules served this way are followed. The standard is for it to be served as text/plain… I mean it’s only been around 30 years! Or… perhaps it’s some 5D chess move where they’re serving it that way for any unrecognised bot or person just to mess with them - I can’t confirm what everyone sees but I would like to believe this was on purpose just to mess with us.</p><p>Then there were the sites that served their robots.txt as 403 forbidden, one very large software company was also deciding to do that but I don’t think that strategy is going to pay off for them. “But why are the bots still crawling everything on the site?!” is what I imagine is happening in a meeting room somewhere… I do appreciate the humour though in putting up a sign for the robots to read but then forbidding them to read it.</p><p>At least one large site also redirects their robots.txt file to a login page! “I’m sorry Mr Bot but you’ll have to create an account to access that.”</p><p>A large ecommerce site couldn’t even spell Gemini correctly and are probably wondering why their rules for “Gemeni” aren’t working... and so it goes on.</p><p>I opened this box, there is definitely some weird shit in there and I’m going to be sharing the contents of that with you, the internet.</p><p>Next, the made up stuff people add to their robots.txt files: <a href="https://medium.com/@petedainty/you-cant-just-invent-a-protocol-bruh-642bc7b32f0b">https://medium.com/@petedainty/you-cant-just-invent-a-protocol-bruh-642bc7b32f0b</a></p><h3>Before you go</h3><ul><li>Please take a moment to like the post and follow the writer!</li><li>Did you know that over 400,000 developers share what they’re building, learning, and discovering across our platforms every month? Learn how you can contribute <a href="https://plainenglish.io/write-for-us"><strong>here</strong></a>.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=490aa2407642" width="1" height="1" alt=""><hr><p><a href="https://blog.stackademic.com/why-you-stalking-my-robots-txt-file-bro-490aa2407642">Why you stalking my robots.txt file, bro?</a> was originally published in <a href="https://blog.stackademic.com/">Stackademic</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>