<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Ilya Nekhay on Medium]]></title>
        <description><![CDATA[Stories by Ilya Nekhay on Medium]]></description>
        <link>https://medium.com/@ilyanekhay?source=rss-b1336ca9d703------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*9GI_jGdSy6nBfk09.</url>
            <title>Stories by Ilya Nekhay on Medium</title>
            <link>https://medium.com/@ilyanekhay?source=rss-b1336ca9d703------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 10:17:30 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@ilyanekhay/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How I Created an Infinite Website: A Journey into Search Engines and SEO]]></title>
            <link>https://medium.com/@ilyanekhay/how-i-created-an-infinite-website-a-journey-into-search-engines-and-seo-996f4e5dde63?source=rss-b1336ca9d703------2</link>
            <guid isPermaLink="false">https://medium.com/p/996f4e5dde63</guid>
            <category><![CDATA[search-engines]]></category>
            <category><![CDATA[hacks]]></category>
            <dc:creator><![CDATA[Ilya Nekhay]]></dc:creator>
            <pubDate>Sat, 27 May 2023 02:40:49 GMT</pubDate>
            <atom:updated>2023-05-27T02:40:49.132Z</atom:updated>
            <content:encoded><![CDATA[<p>My first post on Medium — a story of how I built an infinite website, got 300 million pages indexed, started getting links from Wikipedia and made a small profit!</p><p>It all began when I was a fresh graduate, starting to learn about Natural Language Processing (NLP), Search Engine Optimization (SEO), and Search Engines in general.</p><h3>The Curiosity That Sparked the Idea</h3><p>During a conference I attended back then, I got caught up in a discussion about how search engines crawl websites. It mentioned how it was necessary for a crawler to be breadth-first rather than depth-first, as link loops and infinite websites would pose a certain challenge to a depth-first crawler.</p><p>This sparked my curiosity, and I became eager to see how web search engines handled this issue. I also wondered how much resources search engines would dedicate to a single website, especially a small and unknown one.</p><h3>Infinity and Python’s int</h3><p>Coincidentally, I learned about Python’s int type natively supporting BigInts of arbitrary size. This discovery ignited an idea in my mind. I decided to dedicate each page of the website to a specific number, using that number in the URL. Each page would use the same template, populated by a simple Python script.</p><p>Here’s one of those pages preserved by the Wayback Machine (using Chrome Translate to translate from Russian):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1JYrZR7-q7omoHaifehPEw.png" /><figcaption><a href="https://web.archive.org/web/20150218133412/http://randn.ru/properties/2011">https://web.archive.org/web/20150218133412/http://randn.ru/properties/2011</a></figcaption></figure><h3>Connecting the Pages</h3><p>I wanted to make sure the pages get well interlinked, so that the crawlers could discover them all. Making a list of links, similar to what pagination links look like (1, 2, 3, …) would’ve worked pretty poorly, as that would’ve created a chain of links the crawler needs to traverse.</p><p>I wanted the crawler to see both the “nearby” pages (e.g. pages “3” and “5” when the current page is “4”), as well as some pages arbitrarily far away. The idea was simple: make every number on the page link to a corresponding page, while also adding various sections that’d offer “skips”.</p><p>That’s how, for instance, I came up with an idea to display 10–20 next multiples of the current page’s number. Thus, assuming the current page is “101”, I had a section that would link to: 202, 303, 404, 505, …</p><p>Coming up with a domain name was a bit of a struggle. Eventually I just thought “gosh, these are all just kinda random numbers!” and that’s how all of this got the <strong>randn.ru</strong> domain name.</p><h3>From Implementation to Indexing</h3><p>With the domain, hosting, and a Python script in place, I set up Google Analytics and eagerly awaited the site’s getting indexing by search engines. Being Russian, I was particularly interested in how Yandex.ru would handle it, too.</p><p>A few months later, the results arrived. To my astonishment, Google successfully crawled and indexed a staggering 300 million pages! In contrast, Yandex took a more conservative approach, stopping at approximately 4,000 pages.</p><h3>Unexpected Discoveries and Surprises</h3><p>As the site gained visibility, exciting things began to unfold. First, Google’s crawling efforts brought organic traffic to the website. It reached a plateau of around 40 daily visitors, prompting me to introduce advertisements and generate a modest income. I covered the hosting costs and made a few bucks on top of that.</p><p>Second, while monitoring Google Analytics, one day I noticed visitors coming from Wikipedia. It turned out that my site was linked from a Wikipedia page as an “authoritative source” for a particular number. Even today, you can find a copy of that page on a Wikipedia mirror: <a href="http://wiki-org.ru/wiki/183_(%D1%87%D0%B8%D1%81%D0%BB%D0%BE)">http://wiki-org.ru/wiki/183_(%D1%87%D0%B8%D1%81%D0%BB%D0%BE)</a> .</p><p>Third, to my surprise, I discovered that I had “competitors” who had embarked on similar ventures. These competitors took the concept a step further by translating their pages into over 60 languages, adding a couple of orders of magnitude to the already-infinite number of pages.</p><h3>Reflections and Lessons Learned</h3><p>While I continued to explore the possibilities of the infinite website, I eventually decided to bring this project to a close. I started it as a little experiment, and never found a way to develop it into something meaningful.</p><p>I also didn’t feel good about the environmental impacts of this. Assuming each page was around 5kb, it’s still somewhat shocking to think that Google fetched ~15 gigabytes of data from a little script, and kept storing all of those texts, indexing them and searching over them. And of course it’s even more shocking to just think about all the websites in existence and the massive amounts of trashy and spammy content indexed by search engines.</p><p>In the following years, I started a new chapter of my career as an engineer at Yandex. That’s where the reason behind Yandex only indexing 4000 pages revealed to me: I learned that Yandex uses ML at every stage of crawling and indexing to determine the pages worth keeping and reject the rest.</p><h3>Closing Thoughts</h3><p>The creation of an infinite website may have been a short-lived experiment, but it left an indelible mark on my journey of exploration and discovery. Through this endeavor, I gained a deeper understanding of SEO, and the fascinating world of search engines.</p><p>If you liked this post, please consider following me on Medium, and some day I share more of my past stories, like how I hacked the MTV’s website to promote a friend’s band, or some of my more recent experiences being an engineering manager or a data scientist working on search and NLP.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=996f4e5dde63" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>