<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Farbod Khorramvatan on Medium]]></title>
        <description><![CDATA[Stories by Farbod Khorramvatan on Medium]]></description>
        <link>https://medium.com/@farbodkhorramvatan?source=rss-147504268a3f------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*t6k29sBYIVnS9gFhR1pgaQ.jpeg</url>
            <title>Stories by Farbod Khorramvatan on Medium</title>
            <link>https://medium.com/@farbodkhorramvatan?source=rss-147504268a3f------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 10:11:04 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@farbodkhorramvatan/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[NULL != NULL: How NULLs are ruining your DB with duplicates]]></title>
            <link>https://itnext.io/null-null-how-nulls-are-ruining-your-db-with-duplicates-a0a247e79eb7?source=rss-147504268a3f------2</link>
            <guid isPermaLink="false">https://medium.com/p/a0a247e79eb7</guid>
            <category><![CDATA[postgresql]]></category>
            <category><![CDATA[database]]></category>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Farbod Khorramvatan]]></dc:creator>
            <pubDate>Sun, 07 Jun 2026 14:55:42 GMT</pubDate>
            <atom:updated>2026-06-07T14:55:42.477Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LLHovY2XR2jrCVWuOPMiIg.png" /><figcaption>Image generated by AI</figcaption></figure><h4>If you rely on UNIQUE constraints over nullable columns, you might be silently producing duplicates in your DB.</h4><p>I was working with this huge DB table with dozens of columns, and I said to myself, let’s just manually check the data quality. Then I saw something weird — two records that were exactly the same. How did this happen? I was sure we had a solid unique constraint to stop any duplicates from being imported into the table.</p><p>So I went for the systematic approach — a few DB queries — to see if there were more of these records. And voilà! looks like we have a lot of uninvited guests in the table. But HOW?</p><h4>NULL is not NULL — Postgres says</h4><p>The first thing I did was check the <strong><em>Unique Constraint</em></strong> in the table. Weird again — everything looked 100% fine. The constraint was doing exactly what it was told to do. Then I looked at the sample records again, and something caught my eye: why was this relational id column empty? It’s allowed to be empty, sure — but every other column was identical (and populated) except this one. Both records were NULL in that single column. And apparently, NULL != NULL.</p><p>I grabbed the records and tried inserting them again with the same dummy value in that column — and now the constraint kicked in and blocked the duplicate. So this whole time, we’d been importing duplicates whenever one of these columns was NULL? I asked myself. And the answer was: YES.</p><h4>Why NULL is not equal to NULL?</h4><p>Because NULL is not a value — it’s the absence of one. SQL reads it as <em>we just don’t know</em>, i.e. <strong><em>UNKNOWN</em></strong>. And SQL is designed not to lie to you, which is a completely intentional and valid decision.</p><p>Imagine two nailed-shut, sealed boxes, and you don’t know what’s inside either one. Now I ask you: <strong>are the contents equal? </strong>You can’t say <em>yes</em> — you don’t know what’s inside. But you also can’t say <em>no</em> — you don’t know they’re different either. The only honest answer is <strong>“I can’t tell.”</strong> That’s exactly what SQL does. NULL = NULL doesn&#39;t return FALSE. It returns a third thing: UNKNOWN.</p><blockquote>NULL is never <strong>known to be equal</strong> to anything — not even to another NULL.</blockquote><h4>How this fact can ruin your database?</h4><p>I mean it’s not a disaster to have NULL columns in your table, it’s actually healthy. But in my case, I was counting on one of those columns inside a unique constraint to keep duplicates out. And it CAN become a real disaster with a wide table full of columns, when you’re inserting huge amounts of data on a daily basis — duplicates at that scale can seriously affect the quality of your product.</p><h4>What can be done?</h4><p>The fix is pretty straightforward. Two options, from native to portable:</p><p>If you’re on Postgres 15+, there’s now a native switch for exactly this: declare the constraint as UNIQUE NULLS NOT DISTINCT, and the database will treat two NULLs as equal — no workaround needed.</p><p>And the portable fix that works everywhere (including older Postgres and SQLite): give the column a default value inside the constraint with COALESCE(column_name, default_value). For instance, if it&#39;s an integer, -1 works well; if it&#39;s a relational id pointing to another table, use a dummy id. Just make sure the sentinel is a value that can never legitimately appear, or you&#39;ll create false collisions.</p><p>Let me walk you through an example to make it concrete. I’ll create some tables and show the exact difference, in SQLite via Python (zero setup — sqlite3 ships with Python, so you can paste this and run it yourself). As I mentioned, this is standard SQL behavior, so Postgres, MySQL and Oracle behave the same way; SQL Server is the one exception, allowing a single NULL.</p><pre>import sqlite3<br><br>db = sqlite3.connect(&quot;:memory:&quot;, isolation_level=None)<br>cur = db.cursor()<br><br>cur.executescript(&quot;&quot;&quot;<br>    CREATE TABLE accounts (id INTEGER PRIMARY KEY, name TEXT);<br>    INSERT INTO accounts VALUES (1,&#39;Account A&#39;), (2,&#39;Account B&#39;), (3,&#39;Account C&#39;);<br><br>    CREATE TABLE products (id INTEGER PRIMARY KEY, name TEXT);<br>    INSERT INTO products VALUES (101,&#39;Analytics Pro&#39;), (102,&#39;Data Warehouse&#39;);<br><br>    -- one entitlement per (account, product)<br>    CREATE TABLE ent_plain (<br>        account_id INTEGER, product_id INTEGER, source TEXT,<br>        UNIQUE (account_id, product_id)<br>    );<br>    -- same intent, but a custom/enterprise deal (NULL product) is treated<br>    -- as a real value via COALESCE<br>    CREATE TABLE ent_guarded (account_id INTEGER, product_id INTEGER, source TEXT);<br>    CREATE UNIQUE INDEX ux_guarded<br>        ON ent_guarded (account_id, COALESCE(product_id, -1));<br>&quot;&quot;&quot;)</pre><p>Same accounts and products tables — the only difference is that one uses COALESCE in its unique index and the other doesn&#39;t.</p><pre>ACC = {1:&#39;Account A&#39;, 2:&#39;Account B&#39;, 3:&#39;Account C&#39;}<br>PROD = {101:&#39;Analytics Pro&#39;, 102:&#39;Data Warehouse&#39;, None:&#39;custom deal (no SKU)&#39;}<br><br>def trial(table, a, p, src):<br>    try:<br>        cur.execute(f&quot;INSERT INTO {table}(account_id,product_id,source) VALUES (?,?,?)&quot;,<br>                    (a, p, src))<br>        return &quot;ALLOWED&quot;<br>    except sqlite3.IntegrityError:<br>        return &quot;BLOCKED&quot;<br><br># (account_id, product_id, why) -- a sync that replays records, dupes and all<br>records = [<br>    (1, 101, &quot;first sync   - new entitlement&quot;),<br>    (1, 101, &quot;sync re-ran  - same catalog product&quot;),<br>    (1, None,&quot;first sync   - negotiated enterprise deal&quot;),<br>    (1, None,&quot;sync re-ran  - same enterprise deal&quot;),<br>    (1, None,&quot;sync re-ran  - again&quot;),<br>    (2, None,&quot;first sync   - Account B&#39;s own enterprise deal&quot;),<br>]<br><br>print(&quot;Rule A  ent_plain   : UNIQUE (account_id, product_id)&quot;)<br>print(&quot;Rule B  ent_guarded : UNIQUE (account_id, COALESCE(product_id, -1))&quot;)<br>print(&quot;=&quot;*82)<br>print(f&quot;{&#39;#&#39;:&lt;2} {&#39;account&#39;:&lt;10} {&#39;product&#39;:&lt;22} {&#39;A: plain&#39;:&lt;18} {&#39;B: COALESCE&#39;:&lt;14}&quot;)<br>print(&quot;-&quot;*82)<br>for i,(a,p,why) in enumerate(records,1):<br>    ra = trial(&quot;ent_plain&quot;, a, p, why)<br>    rb = trial(&quot;ent_guarded&quot;, a, p, why)<br>    leak = (ra==&quot;ALLOWED&quot; and rb==&quot;BLOCKED&quot;)<br>    aout = ra + (&quot;  &lt;-- LEAK&quot; if leak else &quot;&quot;)<br>    bout = rb + (&quot;  caught&quot; if leak else &quot;&quot;)<br>    print(f&quot;{i:&lt;2} {ACC[a]:&lt;10} {PROD[p]:&lt;22} {aout:&lt;18} {bout:&lt;14}&quot;)<br>    print(f&quot;   {&#39;&#39;:&lt;10} {why}&quot;)<br><br>print(&quot;=&quot;*82)<br>print(&quot;WHY THEY DIVERGE -- what each uniqueness check actually compares:&quot;)<br>print(&quot;-&quot;*82)<br>print(&quot;  Account A&#39;s enterprise deal arrives 3x as (account_id=1, product_id=NULL)&quot;)<br>print()<br>print(&quot;    plain   compares  (1, NULL) = (1, NULL)  -&gt;  UNKNOWN  -&gt;  not a dup -&gt; ALLOWED&quot;)<br>print(&quot;    guarded compares  (1,  -1 ) = (1,  -1 )  -&gt;  TRUE     -&gt;  duplicate -&gt; BLOCKED&quot;)<br>print()<br>print(&quot;  Account B&#39;s deal (2, NULL) -&gt; (2, -1) is a DIFFERENT key, so it still goes in.&quot;)<br>print(&quot;  COALESCE collides only true duplicates, never unrelated NULLs.&quot;)<br><br>print(&quot;=&quot;*82)<br>print(&quot;WHAT ENDED UP IN EACH TABLE (joined to account names):&quot;)<br>print(&quot;-&quot;*82)<br>def dump(table):<br>    rows = cur.execute(f&quot;&quot;&quot;<br>        SELECT a.name, COALESCE(p.name,&#39;custom deal (no SKU)&#39;)<br>        FROM {table} e JOIN accounts a ON a.id=e.account_id<br>        LEFT JOIN products p ON p.id=e.product_id<br>        ORDER BY a.name&quot;&quot;&quot;).fetchall()<br>    return rows<br>for label, t in [(&quot;ent_plain  &quot;,&quot;ent_plain&quot;), (&quot;ent_guarded&quot;,&quot;ent_guarded&quot;)]:<br>    rows = dump(t)<br>    print(f&quot;  {label} ({len(rows)} rows):&quot;)<br>    for nm, pr in rows:<br>        print(f&quot;       {nm:&lt;10} -&gt; {pr}&quot;)<br>print()<br>na = cur.execute(&quot;SELECT COUNT(*) FROM ent_plain   WHERE account_id=1 AND product_id IS NULL&quot;).fetchone()[0]<br>nb = cur.execute(&quot;SELECT COUNT(*) FROM ent_guarded WHERE account_id=1 AND product_id IS NULL&quot;).fetchone()[0]<br>print(f&quot;  Account A is billed for its enterprise deal {na}x in plain, {nb}x in guarded.&quot;)<br>print(f&quot;\n(SQLite {sqlite3.sqlite_version})&quot;)<br>db.close()</pre><p>And here are the results:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QbGhtsVnXcr566QYg_f8DA.png" /></figure><p>Simple, easy, and effective. It might not sound like a serious problem at first — but in a table with millions of rows and dozens of nullable columns, those uninvited guests pile up fast, and duplicate data quietly erodes the quality your product depends on. The fix is a single COALESCE. The bug is a single assumption: that NULL equals NULL.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a0a247e79eb7" width="1" height="1" alt=""><hr><p><a href="https://itnext.io/null-null-how-nulls-are-ruining-your-db-with-duplicates-a0a247e79eb7">NULL != NULL: How NULLs are ruining your DB with duplicates</a> was originally published in <a href="https://itnext.io/">ITNEXT</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Your Python Scraper Has a Tell. curl-cffi Is How You Hide It.]]></title>
            <link>https://itnext.io/your-python-scraper-has-a-tell-curl-cffi-is-how-you-hide-it-3f4ebd02516f?source=rss-147504268a3f------2</link>
            <guid isPermaLink="false">https://medium.com/p/3f4ebd02516f</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[web-scraping]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[cybersecurity]]></category>
            <dc:creator><![CDATA[Farbod Khorramvatan]]></dc:creator>
            <pubDate>Mon, 25 May 2026 15:20:06 GMT</pubDate>
            <atom:updated>2026-05-25T15:20:06.702Z</atom:updated>
            <content:encoded><![CDATA[<h4>The invisible fingerprint every requests call leaves behind — and here’s how you can fix it.</h4><p>You spent three hours crafting the perfect request. Rotating User-Agents. Real browser headers copied straight from DevTools. Residential proxies on top.</p><p>And the site still hands you a 403. Or worse — a 200 with a Cloudflare challenge page that your code happily parses as &quot;success.&quot;</p><p>So you do what most scrapers eventually do. You reach for Selenium. You spin up a headless browser. You watch your memory climb past a gigabyte to fetch a JSON endpoint that should have taken 200 milliseconds.</p><p>Here’s the uncomfortable truth I had to swallow: <strong>the site wasn’t blocking my headers. It was blocking me before my headers were even read.</strong></p><h3>The handshake snitches on you!</h3><p>Before a single byte of HTTPS leaves your machine, your TLS client sends a thing called a <strong>ClientHello</strong>. It’s basically a polite introduction:</p><blockquote><em>Hi, I support these TLS versions, here are my cipher suites </em>in this specific order<em>, here are the extensions I’d like to negotiate, and here are some elliptic curves I’m cool with.</em></blockquote><p>Here’s the catch: every TLS client does this introduction differently.</p><p>Chrome sends a particular list of extensions in a particular order, with particular GREASE values sprinkled in. Firefox sends a different list. Safari, different again. Python’s requests uses OpenSSL through the ssl module, and the list OpenSSL sends has been sitting in anti-bot databases since roughly 2017.</p><p>Hash that list with a known algorithm and you get a <strong>JA3 fingerprint</strong> (or its newer cousin, JA4). The hash for requests is a famous constant. The hash for Chrome 131 is a different famous constant. Cloudflare reads yours, compares it to your User-Agent — <em>&quot;you claim to be Chrome but your handshake screams Python urllib3, lol&quot;</em> — and quietly returns a 403 before your application code does anything at all.</p><p>You can rotate proxies until the heat death of the universe. The handshake will still snitch on you.</p><p>This is also exactly why curl from your terminal often works when requests fails. Different TLS stack, different fingerprint, less of an obvious tell.</p><p>And it’s not just TLS, JA3 is the famous part. But modern anti-bot stacks check at least two more layers, and you should know about them because they’re going to come up:</p><p><strong>HTTP/2 fingerprint.</strong> Once TLS is up, HTTP/2 has its own little handshake — SETTINGS frames, WINDOW_UPDATE, the pseudo-header order (yes, :method vs :authority vs :scheme vs :path — the order matters). Akamai&#39;s bot product was the first to popularize fingerprinting this layer. Different clients, different signatures.</p><p><strong>Header order.</strong> This one trips people up. HTTP/1.1 doesn’t define a canonical header order, but browsers always send headers the same way. Chrome’s order is consistent across sessions. requests has its own order. If your TLS is perfect but you send User-Agent before Accept when Chrome sends it after — that alone can flag you.</p><p>So three signals: TLS, HTTP/2, header order. All three are leaking from requests. All three get fixed by magical curl-cffi library!</p><h3>Enter curl-cffi: The Library That Finally Made My Scrapers Quiet</h3><p><a href="https://github.com/lexiforest/curl_cffi">curl-cffi</a> is a Python binding for curl-impersonate — a patched fork of curl that mimics, byte for byte, the TLS and HTTP/2 fingerprints of real browsers.</p><p>A simple example of using the library:</p><pre>from curl_cffi import requests<br><br>response = requests.get(&quot;https://example.com&quot;, impersonate=&quot;chrome&quot;)</pre><p>That’s it. One impersonate keyword argument. Your handshake is now indistinguishable from a real Chrome browser!</p><p>That impersonate=&quot;chrome&quot; parameter is doing a <em>lot</em> of work. Behind it:</p><ul><li>The TLS ClientHello gets rewritten to match Chrome’s exact extension list, order, and GREASE pattern.</li><li>The HTTP/2 settings frames, pseudo-header order, and window sizes match Chrome’s.</li><li>The default headers go out in Chrome’s order.</li><li>The JA3 hash you produce matches a real Chrome’s JA3 hash, byte for byte.</li></ul><p>If you are more curious:</p><pre>from curl_cffi import requests<br><br>response = requests.get(&quot;https://tls.browserleaks.com/json&quot;, impersonate=&quot;chrome&quot;)<br>print(response.json())<br># ja3n_hash (the GREASE-normalized JA3) will match a real Chrome browser</pre><p>The requests interface from curl_cffi is API-compatible on purpose. r.status_code, r.json(), r.headers, r.cookies — they all work the way you expect.</p><h4>Don’t take my word for it</h4><p>Run this yourself. It takes 30 seconds.</p><pre>import requests<br>import curl_cffi<br><br>URL = &quot;https://tls.browserleaks.com/json&quot;<br><br>def fingerprint(label, data):<br>    print(f&quot;\n{label}&quot;)<br>    print(f&quot;  ja3_hash:    {data.get(&#39;ja3_hash&#39;)  or &#39;(none)&#39;}&quot;)<br>    print(f&quot;  ja3n_hash:   {data.get(&#39;ja3n_hash&#39;) or &#39;(none)&#39;}&quot;)<br>    print(f&quot;  akamai_hash: {data.get(&#39;akamai_hash&#39;) or &#39;(no HTTP/2)&#39;}&quot;)<br><br>fingerprint(&quot;requests (no impersonation)&quot;,<br>    requests.get(URL).json())<br><br>fingerprint(&quot;curl_cffi (impersonate=&#39;chrome&#39;)&quot;,<br>    curl_cffi.get(URL, impersonate=&quot;chrome&quot;).json())<br><br># Open the same URL in Chrome and compare the hashes in your browser.</pre><p>Here’s what I got on my machine:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8sL5P4TjnLfqWrlcCaffkA.png" /></figure><p>Same target URL, three TLS stacks. curl_cffi&#39;s ja3n_hash and akamai_hash match real Chrome byte-for-byte — to any anti-bot vendor, those two requests are indistinguishable. Meanwhile requests couldn&#39;t even negotiate HTTP/2. The raw ja3_hash differs only because real browsers randomize GREASE values per-connection; ja3n_hash strips that randomness out, which is the field anti-bot systems actually fingerprint on.</p><h3>Under the Hood: What “Impersonation” Actually Means</h3><p>Just knowing that “it spoofs the fingerprint” is not enough. If you’re going to lean on this library in production, you should understand what’s happening.</p><h4>1. It’s not Python doing the handshake</h4><p>Vanilla requests and httpx use Python&#39;s ssl module, which wraps OpenSSL. OpenSSL&#39;s defaults are sensible, standards-compliant, and — for fingerprinting purposes — a dead giveaway.</p><p>curl-cffi ships precompiled native libraries (libcurl-impersonate plus patched BoringSSL/nss/wolfSSL builds) inside the Python wheel. When you call curl_cffi.get(...), Python hands the request to native code, which performs a handshake using the <em>exact</em> TLS extension order, cipher list, and GREASE values that the target browser uses.</p><p>That’s why you don’t have to compile anything. The hard work was done at build time and shipped to PyPI.</p><h4>2. JA3 is only half the story</h4><p>A lot of older guides only mention JA3. Modern anti-bot systems care about three things:</p><ul><li><strong>JA3 / JA4</strong> — the TLS ClientHello fingerprint</li><li><strong>HTTP/2 fingerprint</strong> — frame settings, header table size, pseudo-header order (Akamai-style fingerprinting)</li><li><strong>HTTP/3 fingerprint</strong> — newer, but Cloudflare already checks for it</li></ul><p>curl-cffi matches all three. As of their new version, it also supports HTTP/3 impersonation, which most of its competitors still don&#39;t.</p><h4>3. The impersonate target list is huge</h4><p>You’re not stuck with “chrome.” You can pin a specific version, or let it auto-update:</p><pre># Latest Chrome — automatically tracks the newest version as the library updates<br>curl_cffi.get(url, impersonate=&quot;chrome&quot;)<br><br># Pin to a specific build for reproducibility<br>curl_cffi.get(url, impersonate=&quot;chrome131&quot;)<br><br># Mobile<br>curl_cffi.get(url, impersonate=&quot;chrome131_android&quot;)<br>curl_cffi.get(url, impersonate=&quot;safari184_ios&quot;)<br><br># Firefox, Edge, Tor<br>curl_cffi.get(url, impersonate=&quot;firefox135&quot;)<br>curl_cffi.get(url, impersonate=&quot;edge101&quot;)<br>curl_cffi.get(url, impersonate=&quot;tor145&quot;)</pre><p>If you need something completely custom — say, you reverse-engineered a mobile app’s signature — you can pass raw JA3 and Akamai strings:</p><pre>curl_cffi.get(url, ja3=&quot;...&quot;, akamai=&quot;...&quot;)</pre><p>That’s the kind of escape hatch that turns “this almost works” into “this works.”</p><h3>The Five Features That Made Me Delete My Selenium Code</h3><p>Impersonation is the headline. But after a few months with this library, here’s what truly proves its worth:</p><h4>1. The API is requests. You already know it.</h4><pre>import curl_cffi<br><br>r = curl_cffi.get(&quot;https://example.com&quot;, impersonate=&quot;chrome&quot;)<br>r.status_code<br>r.json()<br>r.headers<br>r.cookies</pre><p>Zero learning curve. You can swap import requests as r for import curl_cffi as r in most scripts and they&#39;ll just work — but now they&#39;ll work on sites that used to block you.</p><h4>2. Sessions with cookie jars, just like you’d expect</h4><pre>s = curl_cffi.Session()<br>s.get(&quot;https://httpbin.org/cookies/set/foo/bar&quot;)<br>print(s.cookies)<br># &lt;Cookies[&lt;Cookie foo=bar for httpbin.org /&gt;]&gt;</pre><h4>3. Real asyncio support with per-request proxy rotation</h4><p>This is the one that quietly saved me thousands of hours. requests doesn&#39;t do async at all. aiohttp doesn&#39;t impersonate. httpx does async but can&#39;t impersonate. curl-cffi does all three.</p><pre>import asyncio<br>from curl_cffi import AsyncSession<br><br>urls = [&quot;https://example.com/1&quot;, &quot;https://example.com/2&quot;, &quot;https://example.com/3&quot;]<br><br>async with AsyncSession() as s:<br>    tasks = [s.get(url, impersonate=&quot;chrome&quot;) for url in urls]<br>    results = await asyncio.gather(*tasks)</pre><p>And benchmarks put it on par with aiohttp for speed — far ahead of requests and httpx. You&#39;re not paying a performance tax for the impersonation.</p><h4>4. WebSockets in the same library</h4><p>If you’ve ever tried to scrape a site that pushes live data over WebSockets — crypto exchanges, sports odds, chat apps — you know the pain of stitching together a TLS-impersonating HTTP client with a separate WebSocket library.</p><p>curl-cffi does both:</p><pre>import asyncio<br>from curl_cffi import AsyncSession<br><br><br>async with AsyncSession() as session:<br>    async with session.ws_connect(&quot;wss://echo.websocket.org&quot;) as ws:<br>        await ws.send_str(&quot;Hello, World!&quot;)<br>        async for message in ws:<br>            print(message)<br>            break</pre><p>Same TLS fingerprint. Same session cookies. One library.</p><h4>5. HTTP/3 with fingerprints — and UDP SOCKS5 proxies</h4><p>This one is for the people who keep up with anti-bot evolution. As more sites push HTTP/3, your HTTP/2 traffic starts looking suspicious by <em>omission</em>. curl-cffi is one of the few Python clients that can speak HTTP/3 with a browser-accurate fingerprint:</p><pre>r = curl_cffi.get(<br>    &quot;https://fp.impersonate.pro/api/http3&quot;,<br>    http_version=&quot;v3&quot;,<br>    impersonate=&quot;chrome&quot;<br>)</pre><h4>One Honorable Mention</h4><p>The latest beta (April 2026) added two things you should know about:</p><p><strong>A live-updatable fingerprint database.</strong> Previously, when Chrome shipped a new minor version with a slightly tweaked fingerprint, you had to wait for a new curl-cffi release to include the patched binary. Not anymore. The library can now pull fingerprint definitions from a remote database at runtime. New Chrome drops → update the DB → no pip upgrade needed.</p><p>If you maintain long-running scrapers in production, this is the kind of feature that quietly takes weeks of maintenance off your plate over a year. Docs: <a href="https://curl-cffi.readthedocs.io/en/latest/fingerprints.html">https://curl-cffi.readthedocs.io/en/latest/fingerprints.html</a></p><p><strong>CurlOpt.HTTPHEADER_ORDER.</strong> Header order, as I mentioned above, is itself fingerprintable. Now you can set it explicitly, comma-separated:</p><pre>&quot;Host,Cookie,User-Agent,Accept,...&quot;</pre><p>If you’ve ever had a scraper that <em>almost</em> works on a site but mysteriously gets flagged on certain endpoints — header order was likely the missing piece.</p><h3>Be Honest: What curl-cffi Won&#39;t Solve</h3><p>It does a lot of great work, no doubt. But assuming it will solve all the anti-bot protections and Cloudflare, is absolutely wrong. TLS fingerprinting is <em>one layer</em> of bot detection. The big vendors stack more on top.</p><p>curl-cffi does not:</p><ul><li><strong>Solve JS challenges.</strong> cf_clearance, Akamai sensor data, DataDome payloads — these are computed by JavaScript running in a real browser. Run a headless browser once to mint the token, then ride it with curl-cffi. Or pay a solver service. There&#39;s no shortcut.</li><li><strong>Beat behavioral fingerprinting.</strong> Mouse movement, scroll velocity, focus events. That’s browser-level.</li><li><strong>Save a burned IP.</strong> If your IP is on a blocklist, no handshake voodoo will save you. Pair this with decent residential or mobile proxies.</li></ul><p>Realistic stack: curl-cffi as your default HTTP client (silently dodges a whole category of blocks at zero cost), real browsers only when JS is mandatory, good proxies on top.</p><h3>Bottom Line</h3><p>curl-cffi is a technical brilliance and once you see it, you can’t unsee it. And once you can’t unsee it, you stop reaching for Selenium first.</p><p><strong>Repo:</strong> <a href="https://github.com/lexiforest/curl_cffi">https://github.com/lexiforest/curl_cffi</a></p><p><strong>Docs:</strong> <a href="https://curl-cffi.readthedocs.io/">https://curl-cffi.readthedocs.io</a></p><p><strong>Benchmarks:</strong> <a href="https://github.com/lexiforest/curl_cffi/tree/main/benchmark">https://github.com/lexiforest/curl_cffi/tree/main/benchmark</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3f4ebd02516f" width="1" height="1" alt=""><hr><p><a href="https://itnext.io/your-python-scraper-has-a-tell-curl-cffi-is-how-you-hide-it-3f4ebd02516f">Your Python Scraper Has a Tell. curl-cffi Is How You Hide It.</a> was originally published in <a href="https://itnext.io/">ITNEXT</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How I Sped Up Database Queries by Over 34,000×]]></title>
            <link>https://medium.com/@farbodkhorramvatan/how-i-sped-up-database-queries-by-over-34-000-d56b5afead51?source=rss-147504268a3f------2</link>
            <guid isPermaLink="false">https://medium.com/p/d56b5afead51</guid>
            <category><![CDATA[performance]]></category>
            <category><![CDATA[database-indexing]]></category>
            <category><![CDATA[postgresql]]></category>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[database]]></category>
            <dc:creator><![CDATA[Farbod Khorramvatan]]></dc:creator>
            <pubDate>Mon, 18 May 2026 06:42:08 GMT</pubDate>
            <atom:updated>2026-05-18T06:42:08.890Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iItO-Wrq620ljB1EEbeGxA.png" /><figcaption>Image generated by AI</figcaption></figure><h4>Same query. Same over-9m-record table. Same data. Median execution time: 2,550 ms → 0.073 ms. Here’s the receipt.</h4><p>I’m watching a script run.</p><p>It’s supposed to look up companies by name — a simple lookup, the kind that should return in milliseconds. The script is doing thousands of these in a row. And it’s slow. Not “needs tuning” slow. <strong>Painful</strong> slow. The kind of slow where you start refreshing your terminal just to confirm something is still happening.</p><p>So I check the table. Millions of rows. Big, but not absurd.</p><p>I check the column. aliases, a VARCHAR[] — a Postgres array of every name a company has ever gone by.</p><p>I check the index. A GIN index on aliases. There it is. Right where it should be.</p><p>I had an index. I felt safe. I was wrong.</p><h3>EXPLAIN tells the truth</h3><p>I take one of the queries the script is running and put EXPLAIN (ANALYZE, BUFFERS) in front of it.</p><pre>Gather  (actual time=1477..2527 rows=1)<br>  Workers Planned: 2<br>  Workers Launched: 2<br>  Buffers: shared hit=2041 read=705971<br>  I/O Timings: read=1872.895<br>  -&gt;  Parallel Seq Scan on companies  (loops=3)<br>        Filter: (&#39;google&#39;::text = ANY ((aliases)::text[]))<br>        Rows Removed by Filter: 3,109,448<br>        Buffers: shared hit=2041 read=705971<br>Execution Time: 2,527.196 ms</pre><p>A sequential scan. Across 9.3 million rows. <strong>705,971 pages read from disk.</strong> <strong>1.87 seconds of pure I/O waiting.</strong> For a query I had told myself was indexed.</p><p>Look at the filter line, though. Really look at it:</p><pre>Filter: (&#39;google&#39;::text = ANY ((aliases)::text[]))</pre><p>= ANY(...). That&#39;s not the operator I thought I was using. I&#39;d written my ORM code believing it would generate something like aliases @&gt; ARRAY[...] — the textbook GIN-friendly pattern. Instead it had quietly compiled to = ANY(), which is a completely different beast.</p><p>And here’s the part that took me an embarrassingly long time to internalize:</p><p><strong>GIN indexes on arrays only support a specific set of operators: </strong><strong>@&gt;, </strong><strong>&lt;@, </strong><strong>&amp;&amp;, and array </strong><strong>=.</strong> That&#39;s it. If you filter with anything else — including = ANY() — the planner has no choice but to seq scan. The index might as well not exist.</p><p>The GIN index was there. It was healthy. It was perfect. It just wasn’t <em>applicable</em> to the query I was actually running. And nothing about my code, my migrations, or my dashboards would have ever told me that. The only thing that told me was reading the EXPLAIN output and noticing that one tiny operator.</p><h3>The ORM is the actual villain</h3><p>The deeper sin here isn’t mine — it’s the ORM’s. SQLAlchemy’s Column.any_(value) looks like the natural way to ask &quot;does this array contain this value.&quot; It compiles to = ANY(). That&#39;s not a GIN-friendly query, and nothing in the API hints at it.</p><p>The fix in SQLAlchemy is Column.contains([value]), which compiles to @&gt;. Django ORM users on ArrayField want aliases__contains=[&#39;google&#39;] (which produces @&gt;), not aliases__contains_item=&#39;google&#39; or membership checks that fall back to = ANY(). Prisma users on Postgres arrays should reach for hasEvery / hasSome, which map to @&gt; and &amp;&amp; respectively.</p><p>If you take one thing from this post: <strong>your ORM is silently rewriting your indexed query into an unindexed one, and there is no warning when it happens.</strong></p><h3>The one-line fix</h3><p>I changed the query to use the operator the index actually supports:</p><pre>SELECT * FROM companies<br>WHERE aliases @&gt; ARRAY[&#39;google&#39;]::VARCHAR[];</pre><p>Re-ran EXPLAIN:</p><pre>Bitmap Heap Scan on companies  (actual time=0.066..0.066 rows=1)<br>  Recheck Cond: (aliases @&gt; &#39;{&quot;google&quot;}&#39;::varchar[])<br>  Heap Blocks: exact=1<br>  Buffers: shared hit=7<br>  -&gt;  Bitmap Index Scan on ix_companies_aliases<br>        Index Cond: (aliases @&gt; &#39;{&quot;google&quot;}&#39;::varchar[])<br>        Buffers: shared hit=6<br>Execution Time: 0.098 ms</pre><p>Seven cached page hits. One heap block. The index pointed straight at the row.</p><p>I benchmarked both versions properly — five measured runs each, one warmup discarded, median reported (because one autovacuum kick will distort an average and there’s no reason to let that happen):</p><pre>Original query (= ANY → seq scan):<br>  runs:   [2552, 2594, 2551, 2483, 2527] ms<br>  median: 2,550.517 ms<br>Fixed query (@&gt; → GIN bitmap scan):<br>  runs:   [0.071, 0.080, 0.071, 0.073, 0.098] ms<br>  median: 0.073 ms<br>Speedup: 34,939×</pre><p>Look at the variance. The slow runs cluster within 4% of each other. The fast runs cluster within 30 microseconds. This isn’t noise. This is two completely different access patterns producing two completely consistent steady-state costs.</p><p><strong>705,971 disk reads vs 7 cached page hits.</strong> That’s the entire 34,939×. There’s no clever algorithm, no exotic config, no new hardware. Just the difference between scanning the world and scanning one row.</p><h3>How GIN actually works (60 seconds)</h3><p>A B-tree index sorts row pointers by a single column value. Brilliant for WHERE col = &#39;x&#39;. Useless when the column is an array — there&#39;s no single value to sort on.</p><p>GIN — <strong>G</strong>eneralized <strong>In</strong>verted Index — flips the problem. For every <em>element</em> that appears in any row’s array, it stores the list of row IDs containing that element. Look up &#39;google&#39;, get the (usually tiny) list of rows that have it as an alias, fetch those rows. Done.</p><p>But the GIN structure only knows how to answer specific questions: <em>does this array contain this element</em> (@&gt;), <em>is this array contained in this set</em> (&lt;@), <em>do these arrays overlap</em> (&amp;&amp;). If you ask it a question outside that vocabulary — like = ANY(), which is technically asking <em>is this scalar value equal to any of those scalars</em> — it can&#39;t help, and the planner falls back to scanning every row.</p><p>That’s the lesson hiding inside the lesson: <strong>an index isn’t just a data structure on a column. It’s a contract between your query and the planner.</strong> Break the contract — even by one operator — and the structure becomes invisible.</p><h3>A few performance and operational caveats to consider</h3><p><strong>Selectivity is doing a lot of work here.</strong> This query returns 1 row out of 9.3M. GIN crushes it because the bitmap heap scan only fetches a handful of pages. If your query returns 100,000 rows, the bitmap scan still has to fetch all those pages, and the multiplier collapses. Still a big win — just not a 34,939× win.</p><p><strong>GIN writes aren’t free.</strong> Every insert or update can touch many entries in the inverted structure — roughly proportional to the number of distinct elements in the array. As a rule of thumb, expect inserts on a GIN-indexed array column to run 3–10× slower than the same insert on an unindexed column, and significantly slower than a B-tree on a scalar. The fastupdate option (on by default) buffers pending inserts in a side list and merges them lazily, which softens the per-insert hit but pushes work to autovacuum and to the next reader unlucky enough to trigger the flush. Read-heavy, write-rare → no-brainer. Write-heavy → benchmark before you commit.</p><h3><strong>Having an index is not the same as using an index</strong></h3><p>The index existed. It had been there for months. Every check I would have run — pg_indexes, schema migrations, the ORM model definition, the dashboard — said the table was properly indexed. And it was. The index was perfect. The query was just written in a way the planner couldn&#39;t connect to it.</p><p>It’s a humbling kind of bug. You can’t catch it by reading your migrations. You can’t catch it from your ORM. You can’t catch it from your monitoring. You catch it exactly one way: by reading the EXPLAIN output of the slow query and asking, <em>“is this index actually doing anything for me?”</em></p><p>Try this in your own database tonight! If you have any WHERE clause touching an array column, a jsonb column, or anything searchable by collection — go run EXPLAIN (ANALYZE, BUFFERS) on it right now.</p><p>Look for these red flags:</p><ul><li>Seq Scan on a large table → bad sign</li><li>Rows Removed by Filter in the millions → very bad sign</li><li>I/O Timings: read=... measured in seconds → catastrophic sign</li><li>A Filter: line whose operator isn&#39;t on your index&#39;s supported list → smoking gun</li></ul><p>If you see them, you might be one operator change away from a four-orders-of-magnitude win. The kind of win that doesn’t show up on a roadmap, that nobody asks for in a sprint planning meeting, but that quietly makes your entire product feel faster. The kind that turns an hour-long script into seconds.</p><h3>The benchmark script</h3><p>If you want to reproduce the numbers on your own table, this is the script I used. Two queries — the slow = ANY() version and the fast @&gt; version — each run five times, one warmup discarded, medians compared.</p><pre>import re<br>import statistics<br>from sqlalchemy import text<br>from your_database_module import db  # adjust import<br><br>COMPANY = &quot;google&quot;<br>RUNS = 5<br><br># Slow: `= ANY()` — operator not supported by GIN, forces seq scan<br>SLOW_QUERY = (<br>    f&quot;SELECT * FROM companies &quot;<br>    f&quot;WHERE &#39;{COMPANY}&#39; = ANY(aliases);&quot;<br>)<br><br># Fast: `@&gt;` — uses the GIN index<br>FAST_QUERY = (<br>    f&quot;SELECT * FROM companies &quot;<br>    f&quot;WHERE aliases @&gt; ARRAY[&#39;{COMPANY}&#39;]::VARCHAR[];&quot;<br>)<br><br>def parse_execution_time(plan_lines):<br>    for line in plan_lines:<br>        m = re.search(r&quot;Execution Time:\s*([\d.]+)\s*ms&quot;, line)<br>        if m:<br>            return float(m.group(1))<br>    return None<br><br>def explain_analyze(sql):<br>    return db.execute(text(f&quot;EXPLAIN (ANALYZE, BUFFERS) {sql}&quot;)).scalars().all()<br><br>def benchmark(sql, runs):<br>    # Warmup, discarded<br>    explain_analyze(sql)<br>    db.rollback()<br><br>    times = []<br>    for _ in range(runs):<br>        plan = explain_analyze(sql)<br>        db.rollback()<br>        times.append(parse_execution_time(plan))<br>    return times<br><br>slow_times = benchmark(SLOW_QUERY, RUNS)<br>fast_times = benchmark(FAST_QUERY, RUNS)<br><br>slow_median = statistics.median(slow_times)<br>fast_median = statistics.median(fast_times)<br><br>print(f&quot;Slow query (= ANY):  median {slow_median:&gt;10.3f} ms&quot;)<br>print(f&quot;Fast query (@&gt;):     median {fast_median:&gt;10.3f} ms&quot;)<br>print(f&quot;Speedup:             {slow_median / fast_median:&gt;10,.0f}x&quot;)</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d56b5afead51" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why Automation Is Inevitable — and Why Anything Can Be Automated]]></title>
            <link>https://medium.com/@farbodkhorramvatan/why-automation-is-inevitable-and-why-anything-can-be-automated-11ec79e1983c?source=rss-147504268a3f------2</link>
            <guid isPermaLink="false">https://medium.com/p/11ec79e1983c</guid>
            <category><![CDATA[automation]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[future-of-work]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[productivity]]></category>
            <dc:creator><![CDATA[Farbod Khorramvatan]]></dc:creator>
            <pubDate>Sun, 16 Nov 2025 11:31:29 GMT</pubDate>
            <atom:updated>2025-11-16T11:31:29.858Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VkkHvH01OXefJfLXkxmBjA.png" /><figcaption>Humans and machines now share the same workspace — automation has become part of how we build, think, and create.</figcaption></figure><h3>Why Automation Is Inevitable — and Why Anything Can Be Automated</h3><p>Let’s face it — the internet isn’t people anymore. By 2024, <strong>around half of all global web traffic</strong> was generated by bots or automated systems. Sounds crazy, right? But it’s true — nearly half of what moves around the web today isn’t human. Dozens of lines of code run endlessly to achieve specific outcomes.</p><p>Your favorite chatbot now searches the web, gathers fresh data, and answers your questions. The CV you spent hours crafting? It’s not checked by a person anymore — an ATS system does that. Even YouTube creators are automating entire content workflows to fill your feed with endless shorts.</p><p>Automation isn’t a prediction anymore — it’s everyday reality. Much of what you interact with online now runs through automated pipelines designed to be faster, cheaper, and always on.</p><h4>It’s Fast, Efficient, and Reliable</h4><p>Imagine doing the exact same job over and over again… everyone hates that — employees and even employers. It kills creativity and makes life dull. But for machines, it’s different — <em>they love it!</em> They can repeat the same process millions of times without getting tired or losing focus.</p><p>Almost anything that’s <strong>scriptable and repeatable </strong>can be fed to machines. They’ll do it with the highest accuracy, in a fraction of the time, without cutting corners. <strong><em>They love it when it’s scriptable.</em></strong></p><p>Bots can be tuned for almost anything. They summarize your meetings in seconds, replace customer support with 24/7 chatbots tailored to each business, and even build your favorite TL;DR newsletter from thousands of articles across the web. News outlets use AI tools to write reports and summaries in real time. What once required teams of people now happens in milliseconds — and that’s exactly why automation is so powerful.</p><p>Automation doesn’t just make things faster. It makes them <strong><em>scale</em></strong>. It frees humans from repetition so we can focus on creativity, design, and problem-solving — things machines still can’t fully replicate.</p><p>And sometimes, the most powerful automations start small!</p><h4>Small Automations, Big Impact</h4><p>Not every automation has to be massive or complex. Sometimes the smallest ones make the biggest difference.</p><p>A while ago, I built a small Chrome extension that completely changed how I work. I used to copy and paste every single webpage URL I wanted to analyze — one after another — just to feed them into my tools for summarizing their content. It wasn’t exactly time-consuming, but it was <em>annoying</em>, repetitive, and definitely not how I wanted to spend my focus.</p><p><strong><em>So, I automated it!</em></strong></p><p>Now, with a single right-click on any webpage, the extension sends its URL to my backend for processing. The server extracts the main content and turns it into a clean, structured summary — formatted neatly in Markdown. What used to be a constant irritation is now handled automatically in the background.</p><p>It’s simple, but it made a huge difference. That one click replaced a boring workflow I used every day. That’s what automation is really about — not replacing people, but removing the things that waste our time and attention.</p><h4>If You Can Describe a Process, You Can Automate It</h4><p>If you can picture a workflow in your head and explain how it works, you can probably turn it into code. The only real challenge is breaking things down clearly enough that a machine can follow the steps.</p><p>In the past, that was hard because most data was messy and unstructured — computers couldn’t really <em>understand</em> what we meant. But that problem’s basically gone now. With today’s AI tools, you can describe your outcome in plain English and the computer actually gets what you mean.</p><p>This ability to understand context changed everything. Before this, automation only worked with structured, predictable stuff. Now, even abstract or creative processes can be automated to some extent. That’s a huge shift in how we work and build.</p><p>That’s why the phrase <em>“</em><strong><em>if you can describe it, you can automate it</em></strong><em>”</em> isn’t just a catchy line — it’s literally true now. Context used to be a blocker. Today, it’s just another input.</p><h4>Automation Isn’t the Future — It’s the Infrastructure</h4><p>Automation isn’t some futuristic dream anymore — it’s the invisible engine running everything around us. It’s in your bank’s fraud detection, in your food delivery app, in your news feed, even in your smart home lights.</p><p>When you order something online, automation handles your payment, warehouse pickup, packaging, and shipping before anyone even touches it. When you apply for a job, AI screens your resume before a human ever reads it. When you scroll social media, algorithms decide what you see next.</p><p>Automation has quietly become the <strong><em>backbone</em></strong> of the digital economy. It’s no longer about replacing humans — it’s about building systems so humans can work smarter. The real shift isn’t about machines doing our jobs — it’s about humans learning to design and manage the systems that now do the work for us.</p><h4>The World Is Moving Fast — So You Should Too</h4><p>You can’t sit there doing everything manually and expect to keep up. The world’s speeding up, and those who use automation and AI are already way ahead.</p><p>Think about it: you’re a writer posting three or four times a day, tops. Meanwhile, someone who automated their writing workflow can drop twenty posts without breaking a sweat. That’s when you really start to feel the difference.</p><p>Sure, setting up automation might take time and effort, but once it’s running it’s like having an invisible team working nonstop in the background. You get speed, consistency, and time to focus on the creative parts that actually matter. When you reach that point, you’re going to have a massive advantage over your competitors.</p><p><strong><em>The world won’t wait — so start automating before you’re the one trying to catch up!</em></strong></p><h4>Bottom of The Line</h4><p>I strongly suggest everyone start learning how to use AI and automation in their work. Automation isn’t optional anymore — it’s an unstoppable force. If you don’t adapt, you’ll fall behind those who do.</p><p>We’ve even reached a point where we’re automating the <em>automation process</em> itself. Take <strong>n8n</strong>, for example — a platform that lets you build automated workflows without writing a single line of code. Fascinating, right? We’ve literally automated the <strong><em>automation-building process</em></strong>…</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=11ec79e1983c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What Web Scraping Taught Me About Data, Patience, and Adaptability]]></title>
            <link>https://medium.com/@farbodkhorramvatan/what-web-scraping-taught-me-about-data-patience-and-adaptability-6090ec7cf064?source=rss-147504268a3f------2</link>
            <guid isPermaLink="false">https://medium.com/p/6090ec7cf064</guid>
            <category><![CDATA[web-crawling]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[web-scraping]]></category>
            <category><![CDATA[data-gathering]]></category>
            <dc:creator><![CDATA[Farbod Khorramvatan]]></dc:creator>
            <pubDate>Sun, 02 Nov 2025 10:43:18 GMT</pubDate>
            <atom:updated>2025-11-02T10:43:18.858Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AY0phk_y3lKcVdbVnHW5QA.png" /></figure><p><strong>Without data, you’re guessing. And guessing doesn’t scale.</strong></p><p>Nowadays, the need for data is greater than ever. Ever wondered how ChatGPT responds to your common-knowledge questions? Or how YouTube perfectly suggests your guilty-pleasure video? Or how companies estimate your salary based on your role and location? The key to all of them is one thing: <strong>data</strong>!</p><p>Over the past years I’ve worked on web-scale data projects — crawling sites, parsing messy HTML, and wiring up nightly jobs that feed real products. Here’s what I’ve learned from my journey into web data.</p><h4>Why Public Data Matters</h4><p>I’ve seen teams with great engineering and beautiful UIs stall because they lacked one thing: usable data. Think of it like a luxury car with no fuel — impressive, but going nowhere. No matter how fancy your product is, if you don’t have enough data it would be useless.</p><p>Recommendation systems, search, and modern AI all depend on large, high-quality datasets. Models and backends can be replicated, but differentiated data is what gives companies their real edge. Most AI companies’ structures and backends can be duplicated — but what truly sets them apart from competitors is <strong>having access to more (and better) data.</strong> And the largest, most abundant source of data available is the <strong>public web.</strong></p><h4>Extract, Transform, Load (ETL)</h4><p>ETL explains it the best I guess. At the very beginning step you need to extract valuable data from the wild web! The biggest and most honest source of data — which is called <strong>Web Scraping / Web Crawling. </strong>Then of course you need to clean, polish and map the raw data into your own taxonomies. When it’s standardized and structured, it’s ready for your <strong>use case</strong>.</p><h4>Diving Deeper into Crawling</h4><p>The first thing you learn is that web scraping teaches you fast. It will make you patient, adaptable, and stubborn. You need to use your creativity to navigate site structures and anti-bot systems. You battle rate limits, dodge IP bans, and set up automated jobs that never sleep. Each problem feels like a boss fight, and each solution feels like a small victory!</p><blockquote>Heraclitus said it best: <strong>“The only constant is change.”</strong> One hour your scraper works like a dream, the next it’s dead. In the wild world of web data, chaos is the only guarantee.</blockquote><p>And then there’s the battle with <strong>The Big Boss: Cloudflare and anti-bot protections.</strong> Suddenly, 403 errors are your new best friends, your requests are slammed with CAPTCHAs, and your carefully built pipeline grinds to a halt. You spend hours — or sometimes days — devising clever workarounds: rotating proxies, throttling request rates, randomizing headers, even spinning up full browser sessions to sneak past the gatekeepers. Some days it feels less like engineering and more like playing cat-and-mouse with an opponent that keeps leveling up just as you do.</p><p>Hence, I strongly suggest: if you’re going to start your journey as a web scraper, learn patience and adaptability. If they can load that data on a web page, then you can scrape it as well!</p><h4>What makes you shine as a Data Gathering Specialist?</h4><p>On the technical side, I’d say: <strong>learn how websites work.</strong> From HTML and how JavaScript runs in the browser to how data moves between client and server — understanding the web deeply is your biggest advantage. The web is the best source of truth if you know how to look carefully!</p><p>Next, <strong>learn how APIs work.</strong> APIs are the cleanest and most reliable gateways to data, and often the difference between a fragile scraper and a stable data pipeline. Develop the habit of checking <strong>network logs</strong> constantly — that’s where many hidden API endpoints reveal themselves.</p><p>Once you master the fundamentals, focus on <strong>evolving your toolkit.</strong> The web changes fast, and so should you. From new HTTP request methods and advanced session handling to TLS fingerprinting and headless browsers, there’s always something new to learn that keeps you one step ahead. Trust me, programmers will try their best to stop you from scraping their entire business value….</p><p>And last but not least, <strong>be gentle folks</strong>! No matter your tech stack, your bots should have a sense of persona — try your best to make them behave less like bots… Use your mimic potential wisely!</p><p>At the end of the day, gathering data isn’t just about code — it’s about <strong>curiosity, patience, and creativity</strong>. If you can master those, the tools will follow.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6090ec7cf064" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>