<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Behind the Firewall - Medium]]></title>
        <description><![CDATA[Accessing data from protected sources, dealing with anti-bot measures, and the technical challenges of web scraping. - Medium]]></description>
        <link>https://medium.com/behind-the-firewall?source=rss----4619bd0a3bc2---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Behind the Firewall - Medium</title>
            <link>https://medium.com/behind-the-firewall?source=rss----4619bd0a3bc2---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 21:36:49 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/behind-the-firewall" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Best Proxy IP Changer Tools in 2026: Free and Paid Solutions]]></title>
            <link>https://medium.com/behind-the-firewall/best-proxy-ip-changer-tools-in-2026-ace61d0c5576?source=rss----4619bd0a3bc2---4</link>
            <guid isPermaLink="false">https://medium.com/p/ace61d0c5576</guid>
            <category><![CDATA[proxy]]></category>
            <category><![CDATA[proxy-ip-changer]]></category>
            <category><![CDATA[ip-change]]></category>
            <category><![CDATA[residential-proxies]]></category>
            <dc:creator><![CDATA[Lambert Watts]]></dc:creator>
            <pubDate>Mon, 14 Sep 2026 06:44:03 GMT</pubDate>
            <atom:updated>2026-09-14T06:44:02.093Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wuX_mAeh6I0bxM--jXaTEw.png" /></figure><p>Sometimes you just need a new IP address. Maybe you want more privacy online, need to get around a geo-block, are troubleshooting a flaky connection, or you’re testing something for work.</p><p>Whatever the reason, a proxy IP changer isn’t the only route. There are a handful of solid ways to change my IP address, and each one has its own trade-offs. Here are four that actually work:</p><ol><li>Using premium proxies</li><li>Changing your IP manually</li><li>Resetting it through your router</li><li>Switching onto a different network</li></ol><p>Curious how I can change my IP address with each method, and which one fits your situation best? Let’s break them down one by one.</p><h4>What is an IP address?</h4><p>Every device online gets a number from its Internet Service Provider called an IP address. There are two main types, IPv4 and IPv6, and within those you’ll find public vs. private and dynamic vs. static addresses.</p><p>Your IP address mostly gives away where you’re connecting from. Changing it hides your location and can fix certain connection problems.</p><p>Checking your current public IP takes ten seconds. Just search “what is my IP address,” and Google will show it right at the top of the results. This is also how you’ll confirm your “change my IP address” attempt actually worked.</p><p><strong>Changing IP address: different methods compared, tl;dr for quick readers</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*W4dM-65DEzQBW6o35oYhKQ.png" /></figure><h4>Free proxy IP changer tools</h4><p>Free proxy IP changers can work fine for casual use, like checking if a page loads differently in another country.</p><p>Tools like hide.me and ProxySite offer decent no-cost options for quick, low-stakes browsing.</p><p>Just keep security in mind: skip anything that doesn’t use HTTPS, and avoid free proxies for logins, banking, or anything you wouldn’t want logged. For that kind of use, a premium provider is a much better choice — most even offer a free trial.</p><h4>How can I change my IP address: set up instructions for mobile and desktop devices</h4><p>The easiest way to switch your IP on the fly is with a proxy server. Most operating systems have built-in proxy settings, so integration is usually just a few clicks. If you go this route, pick a provider with rotation support, since automatic IP rotation is ideal for testing and development.</p><p>Setup varies slightly by OS, so here’s a quick breakdown for each one below.</p><h4>How to change my IP address on macOS</h4><p>macOS gives you fine-grained control over proxy type, whether that’s HTTP, HTTPS, or SOCKS5, and it plays nicely with PAC files for automated rotation. You can also set different proxy configurations for each network you connect to.</p><p><a href="https://oxylabs.io/resources/integrations/mac">macOS proxy integration</a></p><h4>How to change my IP address on iOS</h4><p>iOS (the software running your iPhone or iPad) sticks to two options: PAC files or manual proxy entry. For rotation, you’ll need extra help. Native iOS won’t cycle through IPs on its own, so apps with built-in proxy support pick up the slack.</p><p><a href="https://oxylabs.io/resources/integrations/iphone">iOS proxy integration</a></p><h4>How to change my IP address on Windows 10 and 11</h4><p>Windows 10’s proxy settings are straightforward but basic. You can set things up manually or automate configuration with a PAC (Proxy Auto-Config) file, though it lacks built-in rotation features.</p><p>Windows 11 has the same functionality, just easier to find thanks to a cleaner Settings menu. It also applies proxy settings globally across apps.</p><p><a href="https://oxylabs.io/resources/integrations/windows">Windows 10 and 11 proxy integration</a> guide.</p><h4><strong>How to change my IP address on Android</strong></h4><p>Android’s built-in proxy setup is manual only, and there’s no native support for automatic rotation. Same story as iOS: apps like ProxyDroid or custom scripts fill that gap when you need rotation on the fly.</p><p><a href="https://oxylabs.io/resources/integrations/android">Android proxy integration</a></p><h4>Which type should I choose when I change my IP address?</h4><p>Not all proxies serve the same purpose, so start with your goal. Here’s a quick rundown of each type:</p><ul><li>Residential Proxies: IPs tied to real residential ISPs, so they blend in as ordinary users. Great for getting around geo-restrictions or handling detection-sensitive work like scraping or ad verification.</li><li>Datacenter Proxies: Hosted in data centers, fast, and cheap. Good for high-volume tasks like bulk scraping or managing multiple accounts, though easier to flag than residential IPs.</li><li>ISP Proxies: A mix of both worlds. ISP-issued IPs hosted in data centers, giving you speed plus a more trustworthy footprint. Solid for e-commerce and automation work.</li><li>Mobile Proxies: Route traffic through real mobile carrier networks. Best when you need high trust, like social automation or bypassing tough geo-blocks.</li></ul><p>For privacy, security, or serious testing and development work, a paid proxy service beats free options every time. You can never go wrong with Oxylabs, Decodo, and Webshare — these are known as the most reliable, ethical providers on the market.</p><p>If the budget’s tight, stick to free proxies from providers with a solid reputation, and check whether your paid provider also offers a free tier to test first. Of course, this will only work for small-scale projects.</p><h4>How can I change my IP address manually?</h4><p>You can also change your IP address manually. We went through proxy setup instructions above, so you might be thinking — is there a difference?</p><ul><li>A proxy setup routes your traffic through another server, so websites see the proxy’s IP instead of yours. Your device’s real IP doesn’t change.</li><li>A manual IP change edits the actual IP address your device uses on the network. No proxy involved, no traffic gets rerouted.</li></ul><p>Below are the instructions for each OS.</p><h4>How can I change my IP address manually on macOS</h4><p>Open the Apple menu and click System Settings. Select Wi-Fi from the sidebar.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-0a9g2NXy1OIaEFaFCN7cQ.png" /></figure><p>You’ll see your current network along with other known ones. Click <strong>Details</strong> for more options.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UjN8PcfJa3_PNmhK2OGkmA.png" /></figure><p>Go to the TCP/IP tab, then choose Manually under Configure IPv4.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WzfJ1kTqDIu8ufJPbIt_DQ.png" /></figure><p>Enter your new IP address and click OK.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4gBcG0xi8KKqxXP5PSzNLg.png" /></figure><h4>How can I change my IP address manually on iOS</h4><p>Open Settings, tap Wi-Fi, then tap the “i” icon next to your connected network.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lIk0olDWFmLVbAcFG0li9A.png" /></figure><p>Scroll down to IPv4 Address and tap Configure IP. Choose Manual, then enter your new IP address.</p><p>Tap Save to lock it in.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TWB8076Q9-UNg11jhk-Ezw.png" /></figure><h4>How can I change my IP address manually on Android</h4><p>Open Settings, go to Connections, then Wi-Fi. Tap the gear icon next to your network.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5EM4Xkvd8M6vYnstng7nGg.png" /></figure><p>On the Network details screen, tap Edit, then Advanced options. Under IP settings, choose Static.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3melojiRZ1P89Bm1ilQmlA.png" /></figure><p>Enter your new IP address and tap Save</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/504/1*4UZAW9VJLbtQx31vpFrQ6Q.png" /></figure><h4>How can I change my IP address manually on Windows 10</h4><p>Click Start, then open Settings and head to Network &amp; Internet.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/912/1*Y4an_1MbRjS1S-n3-2frCQ.png" /></figure><p>Pick Ethernet or Wi-Fi depending on how you’re connected. We’ll use Wi-Fi for this example.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/1*SwpL2LNDXoDF4ba2tKs9Kg.png" /></figure><p>Click your active network on the Wi-Fi screen.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/934/1*24XnT8Ysy96vk6hk9CDDHw.png" /></figure><p>Find IP assignment and click Edit.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/902/1*lozWIG3dEkez-5lEf7ERvw.png" /></figure><p>Select Manual from the dropdown.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/916/1*ibOwrMWtvLNlbZndqft1Mw.png" /></figure><p>Flip IPv4 to “On,” enter your new IP details, and hit Save.</p><h4>How can I change my IP address manually on Windows 11</h4><p>The process on Windows 11 mirrors Windows 10 with one extra step.</p><p>Open Network &amp; Internet in Settings, then select Wi-Fi and click Manage known networks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/932/1*gSg-Rwn1I-ilneH4VrQP5A.png" /></figure><p>Pick your active network from the list.</p><p>Click Edit next to the IP assignment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/908/1*85_Wx1bUeRjePoMiawDryw.png" /></figure><p>Switch to manual entry.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/914/1*lbeBEmUPchZwWbPT43I0RQ.png" /></figure><p>Enter your new IP details and click Save.</p><h4>Can I change my IP address on my router?</h4><p>Yes, you can — the simplest fix is to reboot your router or modem. Many ISPs assign a fresh IP address when your device reconnects.</p><ol><li>Check your current public IP</li><li>Turn the router off, then on</li><li>Wait about 10 minutes</li><li>Check your public IP again</li></ol><p>Good for occasional, one-off changes. Too slow and impractical if you need a new IP often.</p><h4>Can I change my IP address by switching networks?</h4><p>Yes, you can connect to a different network and get a different public IP — simple as that.</p><ul><li>Switch Wi-Fi networks — public Wi-Fi or a mobile hotspot both work</li><li>Switch ISPs — gets you an entirely new IP range, though it’s a bigger commitment</li><li>Call your current ISP — ask them to reassign your IP; most will do this on request</li></ul><p>Drawbacks worth knowing: random public Wi-Fi is risky if privacy matters, and switching ISPs can get expensive. Stick to networks you trust.</p><h4>In summation</h4><p>Need a quick, occasional IP change? Reboot your router or switch networks, free and fast. Want more control? Manual configuration works on any device.</p><p>For anything detection-sensitive or serious, skip free proxies and go with a premium provider like Oxylabs, offering reliable rotating proxy IPs.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ace61d0c5576" width="1" height="1" alt=""><hr><p><a href="https://medium.com/behind-the-firewall/best-proxy-ip-changer-tools-in-2026-ace61d0c5576">Best Proxy IP Changer Tools in 2026: Free and Paid Solutions</a> was originally published in <a href="https://medium.com/behind-the-firewall">Behind the Firewall</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AWS Web Scraping with Scrapy and Lambda 2026]]></title>
            <link>https://medium.com/behind-the-firewall/aws-web-scraping-with-scrapy-and-lambda-2026-60557bac12ad?source=rss----4619bd0a3bc2---4</link>
            <guid isPermaLink="false">https://medium.com/p/60557bac12ad</guid>
            <category><![CDATA[datacenter-proxies]]></category>
            <category><![CDATA[aws-lambda]]></category>
            <category><![CDATA[residential-proxies]]></category>
            <category><![CDATA[web-scraping]]></category>
            <category><![CDATA[aws-web-services]]></category>
            <dc:creator><![CDATA[Lambert Watts]]></dc:creator>
            <pubDate>Fri, 04 Sep 2026 09:11:43 GMT</pubDate>
            <atom:updated>2026-09-07T06:31:08.236Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KqLOmvCKUSUCWGY9loW69Q.png" /></figure><p>Running your own server fleet just to scrape data gets expensive fast, and most engineers don’t have time to babysit infrastructure. That’s where serverless scraping comes in.</p><p>This guide walks you through pairing Scrapy with AWS Lambda scraping, Amazon’s serverless compute platform, so you can build and deploy your own AWS web scraping setup without managing a single server.</p><h4>AWS Lambda defined</h4><p>AWS Lambda runs your code without you touching a single server. Amazon handles the infrastructure behind the scenes, spinning up whatever compute power your job needs and scaling automatically as demand shifts.</p><p>You just upload your code. Lambda takes care of allocation, scaling, and availability, so your scraper runs reliably without you managing a fleet of machines.</p><h4>AWS Lambda scraping — what is serverless scraping?</h4><p>Resources scale up when demand spikes and back down when things go quiet. That makes serverless scraping a solid fit for projects with unpredictable traffic or occasional high-volume runs. If you’d rather skip Lambda entirely, Scrapy Cloud is another option worth a look.</p><p>Scrapy itself brings the muscle. It gives you fine-grained control over complex extraction jobs and lets you export scraped data in whatever format you need.</p><h4>What is Scrapy?</h4><p>Scrapy is an open-source Python framework built for web crawling. It handles the full job: pulling data off pages and processing it into whatever structure you need.</p><p>For best results, pair it with Oxylabs’ <a href="https://oxylabs.io/products/residential-proxy-pool">Residential</a> or <a href="https://oxylabs.io/products/datacenter-proxies">Datacenter</a> Proxies. Integrating proxies with Scrapy masks your real IP address from the sites you’re scraping. That keeps you off the radar and away from bans that come with running an automated tool instead of clicking around manually.</p><p><strong>Why use Oxylabs proxies with Scrapy</strong></p><ul><li>Massive IP pool. 175M+ residential and 2M+ datacenter IPs across 195+ countries.</li><li>Low block rates. Automatic rotation keeps you under the radar.</li><li>Fast response times. Averages around 0.6 seconds, well within Lambda’s execution window.</li><li>High uptime. 99.9%+ uptime keeps jobs running without interruption.</li></ul><h4>AWS web scraping tutorial</h4><p><strong>Step 1: Create a Scrapy spider</strong></p><p>Before touching AWS Lambda scraping setup, you need a working Scrapy crawler. Scrapy runs on spiders, self-contained crawlers built from a set of instructions you define.</p><p>Start by installing Scrapy for your environment — run this in your terminal:</p><pre>pip install scrapy</pre><p><strong>Step 2: Build the spider</strong></p><p>For this example, we’ll keep it simple. Create a new Python file named games.py and paste in the code below. This spider scrapes products from a<a href="https://sandbox.oxylabs.io/products"> demo e-commerce site</a>:</p><pre>#====# games.py #====#<br>import scrapy<br><br><br>class GamesSpider(scrapy.Spider):<br>    name = &quot;games&quot;<br>    allowed_domains = [&quot;sandbox.oxylabs.io&quot;]<br>    start_urls = [&quot;https://sandbox.oxylabs.io/products&quot;]<br><br>    def parse(self, response):<br>        for card in response.css(&quot;.product-card&quot;):<br>            yield {<br>                &quot;title&quot;: card.css(&quot;h4::text&quot;).get(),<br>                &quot;price&quot;: card.css(&quot;.price-wrapper::text&quot;).get(),<br>            }<br>        next_page = response.css(&quot;li.next &gt; a::attr(href)&quot;).get()<br>        if next_page:<br>            yield scrapy.Request(response.urljoin(next_page))</pre><p>This spider starts at<a href="https://sandbox.oxylabs.io/products"> https://sandbox.oxylabs.io/products</a>, pulls the title and price for every game, then moves to the next page and repeats. Run it to completion, and you end up with data from 3,000 game products.</p><p>To save the output locally, use the -o switch. This command writes results straight to a games.json file, no JSON import needed in your code:</p><pre>scrapy runspider games.py -o games.json</pre><h4>Step 3: Modify the Scrapy spider for AWS</h4><p>Running Scrapy as an AWS Lambda function means no terminal access and no persistent file system. Writing output locally won’t work here.</p><p>Instead, send the scraped data straight to an S3 bucket. You’ll need an AWS account set up before this step.</p><p>Update your spider code with these custom settings:</p><pre>class GamesSpider(scrapy.Spider):<br>    name = &quot;games&quot;<br>    allowed_domains = [&quot;sandbox.oxylabs.io&quot;]<br>    start_urls = [&quot;https://sandbox.oxylabs.io/products&quot;]<br><br>    custom_settings = {<br>        &quot;FEED_URI&quot;: &quot;s3://YOUR-BUCKET-NAME/items.json&quot;,<br>        &quot;FEED_FORMAT&quot;: &quot;json&quot;,<br>    }<br>#rest of the code</pre><p>Of course, replace YOUR-BUCKET-NAME with the actual S3 bucket you created for your AWS account.</p><p><strong>Step 4: Preparing the environment for Lambda function</strong></p><p>You’ll need four tools installed before configuring AWS Lambda:</p><ul><li>Docker. Packages your Scrapy spider and its dependencies into a container that runs the same way everywhere, with no dependency mismatches between your machine and the Lambda server. Download it from docker.com and confirm you can run it from the command line.</li><li>AWS CLI. Lets you manage AWS services from your terminal instead of the web console. Download it from aws.amazon.com/cli.</li><li>Serverless Framework. Handles packaging and deploying your Docker image to Lambda. Install it via npm. If you don’t have Node.js yet, grab the LTS version from nodejs.org first, then run:</li><li>boto3. The Python SDK for AWS, used to connect your spider to S3.</li></ul><pre>$ npm install -g serverless</pre><p>Botocore is the low-level library behind Boto3, AWS’s Python SDK. It handles the core functionality for talking to AWS services from your code.</p><p>Create and activate a virtual environment, then run:</p><pre>(venv) $ pip install boto3</pre><p>Make sure Scrapy is installed in this same virtual environment. If it isn’t yet, install it with pip:</p><pre>(venv) $ pip install scrapy</pre><p><strong>Step 5: Prepare the code for Lambda function deployment</strong></p><p>Keep your Docker container lean by listing only the dependencies your spider actually needs. Create a requirements.txt file with your Python packages. It should look something like this:</p><pre>requests==2.31.0<br>requests-file==1.5.1<br>Scrapy==2.6.0<br>boto3==1.28.14<br>service-identity==21.1.0<br>cryptography==38.0.4<br># more packages</pre><p>Next, create a Dockerfile. Use the <strong>Create a file</strong> option and name it Dockerfile. It should contain the following:</p><pre>FROM public.ecr.aws/lambda/python:3.10<br><br># Required for lxml<br>RUN yum install -y gcc libxml2-devel libxslt-devel<br>COPY . ${LAMBDA_TASK_ROOT}<br>RUN pip3 install -r requirements.txt<br>CMD [ &quot;lambda_function.handler&quot; ]</pre><p>This Dockerfile starts from a basic Python image, sets /app as the working directory, copies your application into the container, installs the requirements, and sets the run command to entrypoint.sh, the script that acts as your entry point.</p><p>Next, create a new file named lambda_function.py and add the following:</p><pre>import sys<br><br><br>def handler(event, context):<br>    # Run the Scrapy spider<br>    import subprocess<br>    subprocess.run([&quot;scrapy&quot;, &quot;runspider&quot;, &quot;games.py&quot;])<br>    return { <br>        &#39;statusCode&#39;: &#39;200&#39;,   # a valid HTTP status code<br>        &#39;body&#39;: &#39;Lambda function invoked&#39;,        <br>    }</pre><p>Lastly, you’ll need a YML file for deployment. Add a new file, save it as serverless.yml, and use the following code:</p><pre>service: scrapy-lambda<br><br>provider:<br>  name: aws<br>  runtime: python3.9<br>  stage: dev<br>  region: us-east-1<br>  environment:<br>    BUCKET: my-bucket<br>  iamRoleStatements:<br>    - Effect: &quot;Allow&quot;<br>      Action:<br>        - &quot;s3:*&quot;<br>      Resource: &quot;arn:aws:s3:::${self:provider.environment.BUCKET}/*&quot;<br><br>functions:<br>  scrapyFunction:<br>    image: YOUR_REPO_URI:latest<br>    events:<br>      - http:<br>          path: scrape<br>          method: post<br>          cors: true</pre><p>We’ll update YOUR_REPO_URI in the next section. Note that scrapy-lambda is just the name of the image we’ll create there.</p><p><strong>Step 6: Deploying Docker image</strong></p><p>First, create a user through AWS IAM. Save the Access Key and Secret Access Key it gives you.</p><p>Then run the following and enter those keys when prompted:</p><pre>aws configure</pre><p>Then, create a new ECR repository:</p><pre>$ aws ecr create-repository --repository-name YOUR_REPO_NAME</pre><p>From the JSON output, grab the repositoryUri value. It’ll look something like 76890223446.dkr.ecr.us-east-1.amazonaws.com/scrapy-images.</p><p>Paste that value into YOUR_REPO_URI in your serverless.yml file.</p><p>If you haven’t created an S3 bucket yet, do that now in the AWS console. Take the bucket name and update games.py, replacing YOUR-BUCKET-NAME with the actual name.</p><p>Now build your Docker image:</p><pre>$ docker build -t scrapy-lambda</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/915/1*-Y9ue6FK0ZWJSXbxIqTPeQ.png" /></figure><p>Tag and push your Docker image to Amazon ECR:</p><pre>$ aws ecr get-login-password --region region | docker login --username AWS --password-stdin YOUR_REPO_URI<br><br>$ docker tag scrapy-lambda:latest YOUR_REPO_URI:latest<br>$ docker push YOUR_REPO_URI:latest</pre><p>Swap region with your AWS region and YOUR_REPO_NAME with your Amazon ECR repository.</p><p>Lastly, deploy the images:</p><pre>$ sls deploy</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/914/1*c9_ecfdkxL9QMHUg1YOUng.png" /></figure><p>Here’s what your output should look like:</p><pre>Service deployed to stack scrapy-lambda-dev (79s)<br>endpoint: POST - https://abcde.execute-api.us-east-1.amazonaws.com/dev/scrape</pre><p><strong>Step 7: Run the Lambda function</strong></p><p>After running sls deploy, you’ll see a service URL endpoint in the output.</p><p>To trigger the function, send a POST request to that URL:</p><pre>curl -X POST https://abcde.execute-api.us-east-1.amazonaws.com/dev/scrape</pre><p>The Lambda function will start running, and the output lands directly in your S3 bucket.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/913/1*czRBRFikqqUU4Gi6xMK31A.png" /></figure><h4><strong>Debugging Python code in AWS Lambda: practical tips</strong></h4><p>Lambda has no terminal, no local file system, and no traditional breakpoints.</p><p>A few ways to make that easier to work with:</p><ul><li>Deploy after every change. Run sls deploy after each adjustment to your serverless config or spider code. This catches errors early. For faster iteration without a full deployment, use sls invoke local.</li><li>Set up logging. Add print statements or use Python’s logging module in your script. CloudWatch automatically streams this output to a log group named /aws/lambda/&lt;function-name&gt;. High logging volume can add extra cost.</li><li>Invoke with logs attached. Once logging is set up, run sls invoke -f &lt;function-name&gt; — log to execute the function and see the logs directly in your terminal.</li></ul><h4>Final thoughts</h4><p>Pairing Scrapy with AWS Lambda scraping cuts out server management entirely. You get a scraper that scales on demand, runs on a schedule, and only costs you when it’s actually working.</p><p>If you made it this far, feel free to read more of my content on related topics :)</p><p><a href="https://medium.com/python-in-plain-english/scrape-craigslist-data-with-python-2026-guide-343e4115461b">How to Scrape Craigslist Data With Python: 2026 Guide<br></a><a href="https://medium.com/@lambert.watts.809/scrape-google-scholar-in-2026-python-de2070b5552d?sharedUserId=lambert.watts.809">How to Scrape Google Scholar in 2026: Python Tools and Proxy Setup<br></a><a href="https://medium.com/@lambert.watts.809/scrape-google-images-with-python-2026-guide-89a5f5a6fd42">How to Scrape Google Images With Python ( 2026 Guide)<br></a><a href="https://medium.com/python-in-plain-english/curl-with-python-and-proxies-the-2026-guide-64f63c010ec2">cURL with Python and Proxies: The 2026 Practical Guide</a><a href="https://blog.stackademic.com/python-web-scraping-complete-guide-2026-027c54e9f1b4"><br></a><a href="https://medium.com/@lambert.watts.809/httpx-vs-requests-in-2026-1ff6079252f0">HTTPX vs Requests in 2026: Which Python HTTP Client Should You Use?<br></a><a href="https://medium.com/@lambert.watts.809/scraping-walmart-using-python-and-rotating-proxies-2026-1fdc9a5d62dd">Scraping Walmart Using Python and Rotating Proxies: Full Guide for 2026</a><br><a href="https://medium.com/behind-the-firewall/scrape-google-gemini-answers-4-methods-ranked-27f119ce1dfe">How to Scrape Google Gemini Answers (4 Methods, Ranked)</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=60557bac12ad" width="1" height="1" alt=""><hr><p><a href="https://medium.com/behind-the-firewall/aws-web-scraping-with-scrapy-and-lambda-2026-60557bac12ad">AWS Web Scraping with Scrapy and Lambda 2026</a> was originally published in <a href="https://medium.com/behind-the-firewall">Behind the Firewall</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AI Web Scraping Agents in 2026: Beyond ChatGPT-Generated Scripts]]></title>
            <link>https://medium.com/behind-the-firewall/ai-web-scraping-agents-in-2026-beyond-chatgpt-generated-scripts-dc582e7d4ca8?source=rss----4619bd0a3bc2---4</link>
            <guid isPermaLink="false">https://medium.com/p/dc582e7d4ca8</guid>
            <category><![CDATA[chatgpt-scraping]]></category>
            <category><![CDATA[beautifulsoup]]></category>
            <category><![CDATA[web-scraping]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[ai-web-scraping]]></category>
            <dc:creator><![CDATA[Lambert Watts]]></dc:creator>
            <pubDate>Tue, 25 Aug 2026 12:56:32 GMT</pubDate>
            <atom:updated>2026-08-26T08:22:43.790Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iYuDZW6fYpd2v58EFHJMhQ.png" /></figure><p>Ask ChatGPT to write you a scraper and it will. Fast, too. But there’s a gap between “ChatGPT scraper” and “a scraper that still works next week.” Sites change their layout, anti-bot systems get smarter, and a static Python script doesn’t notice any of that until it breaks.</p><p>That gap is exactly why AI scraping agents exist now. They don’t just generate code once. They run, adapt, and fix themselves when something changes. Here’s what that actually looks like in 2026, and where ChatGPT web scraping still fits in.</p><h4>Can ChatGPT scrape websites?</h4><p>Yes, but only at a small scale. ChatGPT web scraping works through its Web Search feature, which lets you pull data directly for quick, one-off lookups. That’s fine if you need a handful of data points.</p><p>It’s not fine if you need real volume. For that, you still need code. Python or JavaScript still does the heavy lifting, running actual requests and parsing actual HTML at scale. If you’re going the Python route, a tool like BeautifulSoup handles the parsing side well.</p><p>So where does ChatGPT scraping fit in? Think of it as your coding partner, not your scraper, and ask it to:</p><ul><li>Generate a ChatGPT scraper script</li><li>Explain scraping frameworks</li><li>Walk you through implementation steps</li></ul><p>Next up: how to actually prompt ChatGPT so it writes a scraper worth using.</p><h4>ChatGPT web scraping: the tutorial</h4><p>Before we get into the ChatGPT web scraping of it all, let’s set up a demo target. We’ll pull data from the<a href="https://sandbox.oxylabs.io/products"> Oxylabs Sandbox</a>, a dummy e-commerce store stocked with video game listings across several categories.</p><p>Here’s what the landing page looks like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9Gua_duM8uhLXqMQSUZcPA.png" /></figure><p>Now, let’s explore the ChatGPT web scraping process to build a scraper that works out of the box.</p><h4>1. Create a ChatGPT account</h4><p>If you don’t have an account, visit ChatGPT’s<a href="https://chat.openai.com/auth/login"> login page</a> and hit Sign up. Then, start a chat by typing your query into the text field.</p><h4>2. Locate the elements to scrape</h4><p>Before prompting ChatGPT, locate the elements you need to extract from the target page. Say you only need the video game titles and prices.</p><ul><li>Right-click one of the game titles and select Inspect. This opens the HTML code for that element in the Developer Tools window.</li><li>Right-click the element and select Copy selector with the game title in it. The figure below shows this.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2V3tJ15FqqeS1aCXG5Ew_w.png" /></figure><ul><li>Write down the selector and repeat the same to find the selector for the price element.</li></ul><h4>3. Prepare the ChatGPT prompt</h4><p>ChatGPT scraping will work best if the prompt is well-explained.</p><p>Specify the programming language, the tools and libraries to use, the element selectors, the output format, and any special instructions the code must follow.</p><p>Here’s a sample prompt you can use to build the scraper with Python and BeautifulSoup:</p><pre>Write a web scraper using Python and BeautifulSoup.<br>Sample Target: https://sandbox.oxylabs.io/products <br><br>Rationale: Scrape the video game titles and prices of all the games on the target page.<br><br>CSS selectors are as follows:<br><br>1. Title: #__next &gt; main &gt; div &gt; div &gt; div &gt; div:nth-child(2) &gt; div &gt; div:nth-child(1) &gt; a.card-header.css-o171kl.eag3qlw2 &gt; h4<br><br>2. Price: #__next &gt; main &gt; div &gt; div &gt; div &gt; div:nth-child(2) &gt; div &gt; div:nth-child(1) &gt; div.price-wrapper.css-li4v8k.eag3qlw4<br><br>Output: Save all the Titles and Prices for all the video games in a CSV file<br><br>Additional Instructions: Handle character encoding and remove undesirable symbols in the output CSV.</pre><p>Notice we’ve provided CSS selectors for prices and titles, copied in the earlier step.</p><p>The scraped data might contain a few odd characters if not handled properly, due to encoding issues. This happens when the web page and the Python script interpret character encoding differently. That’s why we’ve added the encoding fix as an Additional Instruction in the prompt above.</p><h4>4. Review the ChatGPT scraper code</h4><p>Once you get ChatGPT scraping code, review it:</p><ul><li>Check that it does what you actually asked for</li><li>Double-check the libraries and packages it pulls in, in case you want to swap any out</li></ul><p>If you spot glitches, tell ChatGPT and ask it to regenerate the code.</p><h4>5. Execute and test</h4><p>Make sure you’ve installed Python from the<a href="https://www.python.org/downloads/"> official website</a>. After that, create a new folder, and inside it, set up a virtual environment. You can handle all three steps by running these lines in your terminal:</p><pre>mkdir web_scraper<br>cd web_scraper<br>python -m venv venv</pre><p>Activate your venv afterward:</p><pre># On macOS &amp; Linux<br>source venv/bin/activate<br><br># On Windows<br>venv\Scripts\activate.bat</pre><p>After that, install the necessary dependencies that your code uses, for instance:</p><pre>pip install requests beautifulsoup4</pre><p>Next, copy the ChatGPT-generated code, run it, and check that it executes cleanly. Here’s the code ChatGPT generated for us:</p><pre>import requests<br>from bs4 import BeautifulSoup<br>import csv<br><br># URL of the target page<br>url = &quot;https://sandbox.oxylabs.io/products&quot;<br><br># Send a GET request to the URL<br>response = requests.get(url)<br><br># Parse the HTML content using BeautifulSoup<br>soup = BeautifulSoup(response.content, &quot;html.parser&quot;)<br><br># CSS selectors for title and price<br>title_selector = &quot;a.card-header h4&quot;<br>price_selector = &quot;div.price-wrapper&quot;<br><br># Find all game titles and prices<br>titles = soup.select(title_selector)<br>prices = soup.select(price_selector)<br><br># Create a list to store the data<br>data = []<br><br># Iterate through the titles and prices<br>for title, price in zip(titles, prices):<br>    game_title = title.get_text(strip=True)<br>    game_price = price.get_text(strip=True)<br>    data.append((game_title, game_price))<br><br># Define the CSV filename<br>filename = &quot;game_data.csv&quot;<br><br># Write the data to a CSV file<br>with open(filename, &quot;w&quot;, newline=&quot;&quot;, encoding=&quot;utf-8&quot;) as file:<br>    writer = csv.writer(file)<br>    writer.writerow([&quot;Title&quot;, &quot;Price&quot;])  # Write header<br>    for game in data:<br>        writer.writerow(game)<br><br>print(f&quot;Data scraped successfully and saved to &#39;{filename}&#39;.&quot;)</pre><p>Here’s a snippet of the output CSV file generated after executing the code:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/886/1*18NsrI-m5iWUK8Mpcd5CMA.png" /></figure><p>Voilà! You’ve just successfully used a ChatGPT scraper<strong> </strong>to collect data.</p><p>But what if the code throws exceptions, or the output isn’t what you wanted? Let’s address that next.</p><h4>ChatGPT web scraping best practices</h4><p>A few tips help you get more accurate, more useful output from ChatGPT.</p><h4>1. Get code editing assistance</h4><p>Code editing is one of ChatGPT’s strongest features. If your ChatGPT scraping code misses the mark or throws bad output, ask it to edit the code based on what you actually need.</p><p>Be specific. Tell it what to change: the elements you’re scraping, how efficient the code should be, or how the extraction logic works. ChatGPT can then offer new code or targeted fixes to improve the scraper.</p><h4>2. Lint your code for quality</h4><p>Linting makes code easier to read and maintain. ChatGPT can lint for you: it flags syntax issues, recommends best practices, and cleans up readability.</p><p>Paste your code and ask ChatGPT to review it against coding standards. You can also add “lint the code” as an additional instruction directly in your prompt.</p><h4>3. Get help optimizing your code</h4><p>Efficiency matters in web scraping, especially with large datasets or tough targets. ChatGPT can suggest ways to speed things up.</p><p>Ask it about frameworks and packages built for performance, caching techniques, concurrency or parallel processing, and how to cut unnecessary network calls.</p><h4>4. Handle dynamic content</h4><p>Some sites build content with JavaScript, or load it in through <a href="https://aws.amazon.com/what-is/ajax/">AJAX requests</a>. ChatGPT scraping can help you handle that complexity, too. Ask ChatGPT how to pull content from JavaScript-rendered pages. It can point you toward headless browsers, dynamic content scraping techniques, or automating interactions by simulating real user actions.</p><h4>The specifics of ChatGPT scraping</h4><p>LLMs like GPT-5, which powers ChatGPT, are prone to hallucination. That means ChatGPT can hand you code or answers that sound right but aren’t. Always review and test what it gives you before you run it.</p><p><a href="https://chatgpt.com/">ChatGPT</a> scraping also runs into limits once bot protection enters the picture. Many sites use CAPTCHAs and rate-limiting to block automated traffic, and a simple ChatGPT-generated scraper won’t get past them.</p><p>That’s where a tool like <a href="https://oxylabs.io/products/web-unblocker">Web Unblocker</a> by Oxylabs comes in, handling rotating proxies and CAPTCHAs so your scraper keeps running.</p><p>Here’s what <a href="https://oxylabs.io/products/web-unblocker">Web Unblocker</a> can do for you:</p><ul><li>Manages tough anti-bot systems automatically</li><li>Rotates proxies with ML to boost success rates</li><li>Emulates browser behavior to dodge CAPTCHAs</li><li>Renders JavaScript, no headless browser needed</li><li>Pay only for successful requests</li><li>Drops into existing code, no rebuild required</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*J9E0dZvM_uQdqfcXrWIFTA.png" /></figure><p>Alternatively, you can look into <a href="https://decodo.com/proxies/site-unblocker#gref">Decodo’s<strong> </strong>Site Unblocker</a><strong> </strong>tool.</p><h4>Wrapping up</h4><p><a href="https://chatgpt.com/">ChatGPT</a> gets you a working scraper fast, but it’s a starting point, not a finish line. Real scraping projects still need testing and a plan for when sites fight back.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dc582e7d4ca8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/behind-the-firewall/ai-web-scraping-agents-in-2026-beyond-chatgpt-generated-scripts-dc582e7d4ca8">AI Web Scraping Agents in 2026: Beyond ChatGPT-Generated Scripts</a> was originally published in <a href="https://medium.com/behind-the-firewall">Behind the Firewall</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Build an eBay Scraper for Product Data and Prices in 2026]]></title>
            <link>https://medium.com/behind-the-firewall/ebay-scraper-for-product-data-and-prices-in-2026-edd8da82653d?source=rss----4619bd0a3bc2---4</link>
            <guid isPermaLink="false">https://medium.com/p/edd8da82653d</guid>
            <category><![CDATA[web-scraping]]></category>
            <category><![CDATA[ebay-scraper]]></category>
            <category><![CDATA[residential-proxies]]></category>
            <category><![CDATA[ebay-web-scraping]]></category>
            <category><![CDATA[ebay-web-scraper]]></category>
            <dc:creator><![CDATA[Lambert Watts]]></dc:creator>
            <pubDate>Wed, 19 Aug 2026 05:58:30 GMT</pubDate>
            <atom:updated>2026-08-20T06:33:31.502Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qNvGZDjjAkE4TkT_S1bd7A.png" /></figure><p>eBay generates more pricing and product data than almost any other marketplace. That’s why so many businesses turn to an <a href="https://github.com/oxylabs/ebay-scraper">eBay scraper</a> to pull listings, track competitor prices, and catch market trends early.</p><p>Running price monitoring? Building a competitive analysis dashboard? Just want e-commerce insights? <a href="https://github.com/oxylabs/ebay-scraper">eBay web scraping</a> turns scattered listing pages into usable numbers.</p><p>Below, we’ll walk through how eBay web scraping works and where it pays off most for your business.</p><h4>eBay web scraping challenges</h4><p>eBay holds valuable product and pricing data. But pulling it out cleanly isn’t easy, even if it’s publicly available.</p><p>eBay has spent years defending against automated access. When you run an <a href="https://oxylabs.io/products/scraper-api/ecommerce/ebay">eBay web scraper</a> without the right setup, you’ll likely hit:</p><ul><li>Rate limits that throttle or cut off your requests</li><li>CAPTCHAs that stop your scraper mid-task</li><li>JavaScript-rendered content that hides data from basic tools</li></ul><p>Miss any one of these, and you’re stuck with failed requests, gaps in your data, or a blocked IP.</p><p>A <a href="https://oxylabs.io/products/scraper-api/ecommerce/ebay">dedicated eBay scraper</a> fixes this. For example, <a href="https://oxylabs.io/products/scraper-api/ecommerce/ebay">Oxylabs’ eBay Scraper API</a> handles the roadblocks for you:</p><ul><li>Rotates <a href="https://oxylabs.io/products/residential-proxy-pool">proxies</a> so you avoid IP bans</li><li>Manages CAPTCHAs so your requests keep moving</li><li>Renders JavaScript so you capture dynamic content</li></ul><p>There are other solid options on the market — Decodo, Webshare, and ScrapingBee, just to name a few.</p><p>But for the sake of this tutorial, we’ll be using the Oxylabs API — eBay is a difficult target, so we need the cream of the crop.</p><h4>What data can you get with an <a href="https://github.com/oxylabs/ebay-scraper">eBay scraper</a>?</h4><p>Before we get into the technical part of the eBay web scraping tutorial, let’s look at what type of data you can gather with your eBay scraper tool:</p><h4><strong>Search results</strong></h4><p>You’ll be able to gather the search result data, a.k.a., the data you see after putting in a query — <strong>’</strong>cat tree’, in this case. To be more specific, you’ll get:</p><ul><li>Product links</li><li>Titles</li><li>Prices</li><li>Ratings</li><li>Images</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/922/1*Qt-mg2nlSd8nViCzFzMraw.png" /></figure><h4><strong>Product page</strong></h4><p>Using the cat tree page example shown below, we can see a multitude of categories which can be scraped, such as:</p><ul><li>Price</li><li>Availability</li><li>Product title</li><li>Product rating</li><li>Product image</li><li>Shipping cost</li><li>Item specifics</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/898/1*F2qit4VyvWoRaTZGiLN8Lw.png" /></figure><h4><a href="https://oxylabs.io/products/scraper-api/ecommerce/ebay">eBay web scraping</a> use cases</h4><p>A standard eBay page holds plenty of useful data: images, prices, stock status, page numbers, URLs, status codes, and more. Pulling this data out unlocks several use cases:</p><ul><li>Competitive analysis. See competitors’ products, pricing, and sales volume. Use it to shape your own pricing strategy and spot ways to stand out.</li><li>Market research. Find out which products sell well, and how often.</li><li>Product development. Spot which products are in demand, then use that insight to guide your design and development decisions.</li></ul><p>Now that we have all that out of the way, let’s get into the technical part of our eBay web scraping tutorial.</p><h4>eBay web scraping: step-by-step tutorial</h4><p>For this tutorial, we’ll be collecting data from <a href="https://www.ebay.com/itm/225313054444">this product page</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/734/1*HKDEJiCGeXESgNxyn7HsFA.png" /></figure><h4>1. Install and import libraries</h4><p>If you haven’t yet, start by installing the requests library in your terminal/console with the command below:</p><pre>pip install requests</pre><p>Start by opening your preferred IDE and creating a new Python module file. Import the requests, json, and pprint libraries, then add your API credentials.</p><p>Don’t have credentials yet? Grab a free trial of Oxylabs’ eBay Scraper API by registering on the<a href="https://dashboard.oxylabs.io/en/"> dashboard</a>.</p><pre>import requests, json<br>from pprint import pprint<br><br>credentials = (&#39;USERNAME&#39;, &#39;PASSWORD&#39;)</pre><h4>2. Create a payload dictionary</h4><p>Next, define a new dictionary called payload. Set the source value to universal_ecommerce, then add the eBay URL for the product data you want to pull.</p><pre>import requests, json<br>from pprint import pprint<br><br>credentials = (&#39;USERNAME&#39;, &#39;PASSWORD&#39;)<br><br>payload = {<br>    &#39;source&#39;: &#39;universal_ecommerce&#39;,<br>    &#39;url&#39;: &#39;https://www.ebay.com/itm/225313054444&#39;,<br>}</pre><h4>3. Localize eBay results</h4><p>If you like, you can also specify a geo-location to collect localized eBay product data:</p><pre>payload = {<br>    &#39;source&#39;: &#39;universal_ecommerce&#39;,<br>    &#39;url&#39;: &#39;https://www.ebay.com/itm/225313054444&#39;,<br>    &#39;geo_location&#39;: &#39;Canada&#39;, # For fetching localized product data<br>}</pre><p><a href="https://developers.oxylabs.io/products/web-scraper-api/features/localization/proxy-location?_gl=1*1hzsjk8*_gcl_au*NjIzMjAxMTgzLjE3ODQxNDE2MDU.">Here’s</a> more information about extracting data from specific countries.</p><h4>4. Parse eBay product data</h4><p>Next, pull the eBay product data using Custom Parser, a free Oxylabs API feature. Add the ‘parse’: True and ‘parsing_instructions’ parameters, then define your custom parsing logic with CSS or XPath selectors:</p><pre>payload = {<br>    &#39;source&#39;: &#39;universal_ecommerce&#39;,<br>    &#39;url&#39;: &#39;https://www.ebay.com/itm/225313054444&#39;,<br>    &#39;geo_location&#39;: &#39;Canada&#39;, # For fetching localized product data<br>    &#39;parse&#39;: True,<br>    &#39;parsing_instructions&#39;: {<br>        &#39;title&#39;: {<br>            &#39;_fns&#39;: [<br>                {<br>                    &#39;_fn&#39;: &#39;xpath_one&#39;,<br>                    &#39;_args&#39;: [&#39;//h1/span/text()&#39;]<br>                }<br>            ]<br>        },<br>        &#39;price&#39;: {<br>            &#39;_fns&#39;: [<br>                {<br>                    &#39;_fn&#39;: &#39;xpath_one&#39;,<br>                    &#39;_args&#39;: [&#39;//div[contains(@class, &quot;price-primary&quot;)]/span/text()&#39;]<br>                }<br>            ]<br>        },<br>        &#39;seller_url&#39;: {<br>            &#39;_fns&#39;: [<br>                {<br>                    &#39;_fn&#39;: &#39;xpath_one&#39;,<br>                    &#39;_args&#39;: [&#39;//div[contains(@class, &quot;about-seller&quot;)]//a/@href&#39;]<br>                }<br>            ]<br>        },<br>        &#39;availability&#39;: {<br>            &#39;_fns&#39;: [<br>                {<br>                    &#39;_fn&#39;: &#39;xpath_one&#39;,<br>                    &#39;_args&#39;: [&#39;//div[contains(@class, &quot;availability&quot;)]/span/text()&#39;]<br>                }<br>            ]<br>        },<br>        &#39;item_specifics&#39;: {<br>            &#39;_fns&#39;: [<br>                {<br>                    &#39;_fn&#39;: &#39;xpath&#39;,<br>                    &#39;_args&#39;: [&#39;//div[contains(@class, &quot;evo__col&quot;)]&#39;]<br>                }<br>            ],<br>            &#39;_items&#39;: {<br>                &#39;key&#39;: {<br>                    &#39;_fns&#39;: [<br>                        {<br>                            &#39;_fn&#39;: &#39;xpath_one&#39;,<br>                            &#39;_args&#39;: [&#39;.//div[contains(@class, &quot;labels-content&quot;)]//text()&#39;]<br>                        }<br>                    ]<br>                },<br>                &#39;value&#39;: {<br>                    &#39;_fns&#39;: [<br>                        {<br>                            &#39;_fn&#39;: &#39;xpath_one&#39;,<br>                            &#39;_args&#39;: [&#39;.//div[contains(@class, &quot;values-content&quot;)]//text()&#39;]<br>                        }<br>                    ]<br>                }<br>            }<br>        }<br>    }<br>}</pre><p>These instructions parse the product title, price, seller URL, availability, and item specifics table. Add more instructions if you need them.</p><h4>5. Send a request</h4><p>With the configuration done, send a POST request to the API endpoint using the requests library:</p><pre>response = requests.post(<br>    &#39;https://realtime.oxylabs.io/v1/queries&#39;,<br>    json=payload,<br>    auth=credentials # Pass your API credentials<br>)</pre><h4>6. Save eBay data to a JSON file</h4><p>Start by printing the full API response, which includes the scraped eBay results along with details about the submitted job. To get to the actual product data, access the content key from the response. From there, save the data to a JSON file:</p><pre>pprint(response.json()) # Get a pretty-print of the API&#39;s response<br><br>with open(&#39;ebay_product_data.json&#39;, &#39;w&#39;) as f: # Save the results to a JSON file<br>    json.dump(response.json()[&#39;results&#39;][0][&#39;content&#39;], f, indent=4)</pre><h4>Full eBay scraper code</h4><p>Put it all together, and you’ll have this Python code for eBay web scraping:</p><pre>import requests, json<br>from pprint import pprint<br><br>credentials = (&#39;USERNAME&#39;, &#39;PASSWORD&#39;)<br><br>payload = {<br>    &#39;source&#39;: &#39;universal_ecommerce&#39;,<br>    &#39;url&#39;: &#39;https://www.ebay.com/itm/225313054444&#39;,<br>    &#39;geo_location&#39;: &#39;Canada&#39;, # For fetching localized product data<br>    &#39;parse&#39;: True,<br>    &#39;parsing_instructions&#39;: {<br>        &#39;title&#39;: {<br>            &#39;_fns&#39;: [<br>                {<br>                    &#39;_fn&#39;: &#39;xpath_one&#39;,<br>                    &#39;_args&#39;: [&#39;//h1/span/text()&#39;]<br>                }<br>            ]<br>        },<br>        &#39;price&#39;: {<br>            &#39;_fns&#39;: [<br>                {<br>                    &#39;_fn&#39;: &#39;xpath_one&#39;,<br>                    &#39;_args&#39;: [&#39;//div[contains(@class, &quot;price-primary&quot;)]/span/text()&#39;]<br>                }<br>            ]<br>        },<br>        &#39;seller_url&#39;: {<br>            &#39;_fns&#39;: [<br>                {<br>                    &#39;_fn&#39;: &#39;xpath_one&#39;,<br>                    &#39;_args&#39;: [&#39;//div[contains(@class, &quot;about-seller&quot;)]//a/@href&#39;]<br>                }<br>            ]<br>        },<br>        &#39;availability&#39;: {<br>            &#39;_fns&#39;: [<br>                {<br>                    &#39;_fn&#39;: &#39;xpath_one&#39;,<br>                    &#39;_args&#39;: [&#39;//div[contains(@class, &quot;availability&quot;)]/span/text()&#39;]<br>                }<br>            ]<br>        },<br>        &#39;item_specifics&#39;: {<br>            &#39;_fns&#39;: [<br>                {<br>                    &#39;_fn&#39;: &#39;xpath&#39;,<br>                    &#39;_args&#39;: [&#39;//div[contains(@class, &quot;evo__col&quot;)]&#39;]<br>                }<br>            ],<br>            &#39;_items&#39;: {<br>                &#39;key&#39;: {<br>                    &#39;_fns&#39;: [<br>                        {<br>                            &#39;_fn&#39;: &#39;xpath_one&#39;,<br>                            &#39;_args&#39;: [&#39;.//div[contains(@class, &quot;labels-content&quot;)]//text()&#39;]<br>                        }<br>                    ]<br>                },<br>                &#39;value&#39;: {<br>                    &#39;_fns&#39;: [<br>                        {<br>                            &#39;_fn&#39;: &#39;xpath_one&#39;,<br>                            &#39;_args&#39;: [&#39;.//div[contains(@class, &quot;values-content&quot;)]//text()&#39;]<br>                        }<br>                    ]<br>                }<br>            }<br>        }<br>    }<br>}<br><br>response = requests.post(<br>    &#39;https://realtime.oxylabs.io/v1/queries&#39;,<br>    json=payload,<br>    auth=credentials # Pass your API credentials<br>)<br><br>pprint(response.json()) # Get a pretty-print of the API&#39;s response<br><br>with open(&#39;ebay_product_data.json&#39;, &#39;w&#39;) as f: # Save the results to a JSON file<br>    json.dump(response.json()[&#39;results&#39;][0][&#39;content&#39;], f, indent=4)</pre><p>After running the eBay scraper code, the final saved JSON file should look like this:</p><pre>{<br>    &quot;price&quot;: &quot;US $549.99&quot;,<br>    &quot;title&quot;: &quot;15.6\&quot; TOUCH Dell laptop Precision 5530 i7 8850H P2000 32GB 1TB SSD Win 11 pro&quot;,<br>    &quot;seller_url&quot;: &quot;https://www.ebay.com/str/kl0?_trksid=p4429486.m3561.l161211&quot;,<br>    &quot;availability&quot;: &quot;Last One&quot;,<br>    &quot;item_specifics&quot;: [<br>        {<br>            &quot;key&quot;: &quot;Condition&quot;,<br>            &quot;value&quot;: &quot;Excellent - Refurbished&quot;<br>        },<br>        {<br>            &quot;key&quot;: &quot;Seller Notes&quot;,<br>            &quot;value&quot;: &quot;\u201cOne Year Warranty\u201d&quot;<br>        },<br>        {<br>            &quot;key&quot;: &quot;Processor&quot;,<br>            &quot;value&quot;: &quot;Intel Core i7 8th Gen.&quot;<br>        },<br>        {<br>            &quot;key&quot;: &quot;Screen Size&quot;,<br>            &quot;value&quot;: &quot;15.6 in&quot;<br>        },<br>        {<br>            &quot;key&quot;: &quot;Graphics Processing Type&quot;,<br>            &quot;value&quot;: &quot;Integrated/On-Board Graphics&quot;<br>        },<br>        {<br>            &quot;key&quot;: &quot;RAM Size&quot;,<br>            &quot;value&quot;: &quot;32 GB&quot;<br>        },<br>        {<br>            &quot;key&quot;: &quot;Color&quot;,<br>            &quot;value&quot;: &quot;Black&quot;<br>        },<br>        {<br>            &quot;key&quot;: &quot;Custom Bundle&quot;,<br>            &quot;value&quot;: &quot;No&quot;<br>        },<br>        {<br>            &quot;key&quot;: &quot;SSD Capacity&quot;,<br>            &quot;value&quot;: &quot;1tb&quot;<br>        },<br>        {<br>            &quot;key&quot;: &quot;Most Suitable For&quot;,<br>            &quot;value&quot;: &quot;Casual Computing, Graphic Design, Parts&quot;<br>        },<br>        {<br>            &quot;key&quot;: &quot;GPU&quot;,<br>            &quot;value&quot;: &quot;Nvdia p2000&quot;<br>        },<br>        {<br>            &quot;key&quot;: &quot;Processor Speed&quot;,<br>            &quot;value&quot;: &quot;2.70 GHz&quot;<br>        },<br>        {<br>            &quot;key&quot;: &quot;Brand&quot;,<br>            &quot;value&quot;: &quot;Dell&quot;<br>        },<br>        {<br>            &quot;key&quot;: &quot;Series&quot;,<br>            &quot;value&quot;: &quot;Precision&quot;<br>        },<br>        {<br>            &quot;key&quot;: &quot;Type&quot;,<br>            &quot;value&quot;: &quot;Notebook/Laptop&quot;<br>        },<br>        {<br>            &quot;key&quot;: &quot;Maximum Resolution&quot;,<br>            &quot;value&quot;: &quot;FHD&quot;<br>        },<br>        {<br>            &quot;key&quot;: &quot;Model&quot;,<br>            &quot;value&quot;: &quot;Dell Precision 5530&quot;<br>        },<br>        {<br>            &quot;key&quot;: &quot;Connectivity&quot;,<br>            &quot;value&quot;: &quot;Gigabit Ethernet, USB 2.0, USB 3.0&quot;<br>        },<br>        {<br>            &quot;key&quot;: &quot;Hard Drive Capacity&quot;,<br>            &quot;value&quot;: &quot;1 TB&quot;<br>        },<br>        {<br>            &quot;key&quot;: &quot;Features&quot;,<br>            &quot;value&quot;: &quot;Bluetooth, Wi-Fi&quot;<br>        },<br>        {<br>            &quot;key&quot;: &quot;Operating System&quot;,<br>            &quot;value&quot;: &quot;Windows 11 Pro&quot;<br>        },<br>        {<br>            &quot;key&quot;: &quot;Storage Type&quot;,<br>            &quot;value&quot;: &quot;SSD (Solid State Drive)&quot;<br>        }<br>    ],<br>    &quot;parse_status_code&quot;: 12000<br>}</pre><h4>Wrapping up</h4><p>That’s it — you now have a working Python setup for fetching, parsing, and saving eBay product data at scale.</p><p><strong>Continue reading related articles</strong></p><p><a href="https://python.plainenglish.io/scrape-craigslist-data-with-python-2026-guide-343e4115461b">How to Scrape Craigslist Data With Python: 2026 Guide</a><br><a href="https://blog.stackademic.com/bypass-captcha-with-puppeteer-2026-feab91e31ec3?sharedUserId=lambert.watts.809">How to Bypass CAPTCHA With Puppeteer ( 2026 )</a><br><a href="https://medium.com/python-in-plain-english/curl-with-python-and-proxies-the-2026-guide-64f63c010ec2">cURL with Python and Proxies: The 2026 Practical Guide</a><br><a href="https://medium.com/@lambert.watts.809/web-scraping-headers-you-should-know-e78dea707ad8">Web Scraping Headers You Should Know to Avoid Blocks in 2026</a><br><a href="https://medium.com/behind-the-firewall/gemini-for-web-scraping-a-python-scraper-6d29cde8cf33?sharedUserId=lambert.watts.809">Gemini for Web Scraping: Building a Python Scraper with Google’s AI</a><br><a href="https://medium.com/behind-the-firewall/replaced-tavily-in-my-agent-setup-for-oxylabs-fast-search-api-f82d34f8943f">I replaced Tavily in my agent setup for Oxylabs Fast Search API. Here’s what changed</a><br><a href="https://medium.com/behind-the-firewall/chatgpt-web-scraping-the-complete-2026-guide-python-6ca20161a060?sharedUserId=lambert.watts.809">ChatGPT Web Scraping: The Complete 2026 Guide (Python)</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=edd8da82653d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/behind-the-firewall/ebay-scraper-for-product-data-and-prices-in-2026-edd8da82653d">Build an eBay Scraper for Product Data and Prices in 2026</a> was originally published in <a href="https://medium.com/behind-the-firewall">Behind the Firewall</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Your AI Coding Agent Has Two Context Problems, and You’re Probably Only Fixing One]]></title>
            <link>https://medium.com/behind-the-firewall/your-ai-coding-agent-has-two-problems-and-youre-fixing-one-a396d593bfd9?source=rss----4619bd0a3bc2---4</link>
            <guid isPermaLink="false">https://medium.com/p/a396d593bfd9</guid>
            <category><![CDATA[web-scraping]]></category>
            <category><![CDATA[data-collection]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[aistudio]]></category>
            <category><![CDATA[ai-coding]]></category>
            <dc:creator><![CDATA[Lambert Watts]]></dc:creator>
            <pubDate>Wed, 12 Aug 2026 07:19:34 GMT</pubDate>
            <atom:updated>2026-08-14T06:36:47.102Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*x3HvN1vSxGmpqkCCkgKArg.png" /></figure><p>An AI coding agent is a tool that helps generate and understand code, but it only becomes reliably useful when it can combine fresh external information with your team’s internal context. Everybody’s context engineering advice these days boils down to one of two camps. Camp one says: your agent’s knowledge cutoff is the problem, wire up a good search tool and stop trusting stale training data. Camp two says: your agent has no idea how <em>your</em> team writes code, so build a RAG pipeline over your repo and docs before you let it touch a pull request.</p><p>Both camps are right. That’s the annoying part. An agent that can search the live web but knows nothing about your internal conventions will confidently generate a REST endpoint that follows this week’s Spring Boot best practices and completely ignores your hexagonal architecture and your ADRs. An agent that’s stuffed full of your internal patterns via RAG but has no search layer will happily scaffold an integration against an API version your company retired eight months ago.</p><p>For software teams, AI engineers, and technical decision-makers trying to make an ai coding agent actually usable in production, those are the two failure modes that matter most: stale model knowledge and missing codebase understanding. This is about how to solve both with search tools and RAG pipelines, where products like <a href="https://aistudio.oxylabs.io/">Oxylabs AI Studio</a> and <a href="https://oxylabs.io/products/scraper-api/fast-search">Fast Search API </a>fit into the stack, and what to watch for when configuring them inside agent tooling.</p><h4>Problem one: the world moved on without your large language model</h4><p>Every model has a knowledge cutoff. That’s not news. What’s less obvious is how confidently a model will paper over that gap. An ai coding agent is an autonomous software system that can execute multi-step workflows, so if you ask it about a library’s current API, it won’t say “I don’t know, that might have changed.” It’ll generate code against the version it remembers, formatted exactly like correct code, and you won’t find out it’s wrong until the build fails or, worse, until it doesn’t fail and just behaves subtly incorrectly in production.</p><p>The fix everyone reaches for is “give the agent a search tool.” Fine. But which search tool, configured how, matters a lot more than people assume. These systems analyze requirements and formulate a plan before acting, which is exactly why weak search configuration distorts downstream software tasks. A raw search API handing back ten links and snippets isn’t the same as a search layer that actually synthesizes an answer, and a synthesis layer running at low retrieval depth can be worse than no synthesis at all, because a wrong answer stated confidently is harder to catch than ten links you have to sort through yourself.</p><p>This is the part that trips people up: adding search doesn’t automatically add reliable context. You still have to decide how much retrieval depth a given query needs, whether it needs raw results or reasoned synthesis, and whether it needs freshness filtering. Treat that as a solved problem and you’ll get plausible-looking answers built on dead API docs.</p><h4>Problem two: the model doesn’t know your codebase either</h4><p>Say you fix the freshness problem. Your agent can now tell you, correctly, that a library’s API changed last month. Great. Now ask it to write a new endpoint for your service, and it writes clean, idiomatic code that has nothing to do with how your team actually structures a microservice. That is the part it gets wrong, even though AI coding agents can still be useful partners for developers rather than just autocomplete tools. It doesn’t know your package layout. It doesn’t know you require an ADR for every architectural decision. It doesn’t know that “policy” means something specific and non-obvious in your domain glossary.</p><p>This is a different failure mode from stale training data. It’s not that the model’s information is out of date, it’s that the model was never given your information in the first place, because it isn’t public and it never will be. No amount of web search fixes this. You need your own codebase, your ADRs, your test patterns, and your domain vocabulary indexed and retrievable, so that when the agent gets a task it’s grounded in “how we do this” and not just “how this is generically done.” In practice, these systems are genuinely helpful for boilerplate, documentation, unit tests, and quick prototypes.</p><p>The honest version of this problem is a data engineering problem before it’s a model problem. What counts as a canonical example versus legacy cruft you don’t want the model imitating. How you strip secrets and PII before anything goes near an embedding index. How you keep the index from going stale as your patterns evolve. None of that is glamorous, and all of it matters more than which framework you pick to do the retrieval. Better internal retrieval can lower barriers for junior developers, but very large codebases still create context limitations.</p><h4>Where the two problems actually meet</h4><p>Here’s the part that doesn’t get said often enough: your internal RAG index has to be built from something, and a lot of what feeds it isn’t sitting in your own git history. It’s public documentation, competitor implementations, regulatory pages, framework changelogs, forum threads where someone already solved the exact edge case you’re stuck on. Building your internal knowledge base is itself a web data collection problem, on top of being a search-routing problem at inference time.</p><p>So the two “camps” converge on the same underlying requirement: reliable, structured access to web data, both live at query time and in bulk at indexing time. That’s less a coincidence and more the actual shape of the problem. An agent’s context has an internal half and an external half, and both halves need a pipe to the web, they just need different flow rates and different levels of synthesis.</p><h4>What web scraping looks like with Oxylabs in the stack</h4><p>This is where Oxylabs <a href="https://aistudio.oxylabs.io/">AI Studio</a> and the <a href="https://oxylabs.io/products/scraper-api/fast-search">Fast Search API</a> are worth a closer look, because they sit at both ends of that pipe.</p><p>Fast Search API is built for the query-time half of the problem. It’s a lightweight, high-throughput SERP scraper, sub-second latency, built to hold up under millions of queries a day, with AI helping improve the speed of data retrieval for high-frequency lookups. It doesn’t do AI synthesis on top of the results the way a reasoning-tier search model does. It gives you clean, structured search data fast, which is exactly what you want for the high-frequency, low-ambiguity slice of an agent’s search traffic: current release numbers, whether a package still exists, what a status page says right now. You don’t need chain-of-thought reasoning to answer “is this API still up,” you need speed and a result you can trust.</p><p>AI Studio covers more of the surface. It ships as a Python and JavaScript SDK (oxylabs-ai-studio on PyPI) with an AiSearch app that gives you both a synthesized search() call and an instant_search() call for the fast, no-frills path, plus AI-Scraper, AI-Crawler, and AI-Browser-Agent apps for turning arbitrary pages, including JavaScript-heavy ones, into structured or markdown output. This kind of automation supports scraping at scale, including anti-bot bypass and automated proxy management for better reliability. That crawler and scraper pairing is the part that matters for the internal-context problem: it’s what you’d point at your own documentation site, your competitors’ public docs, or a stack of regulatory pages you need indexed, before any of that ever reaches your vector database. In other words, the same product family that answers “what does this error code mean right now” is also the thing you’d use to build the corpus that teaches your model what “our way of doing it” looks like.</p><pre>from oxylabs_ai_studio.apps.ai_search import AiSearch</pre><pre>search = AiSearch(api_key=&quot;&lt; API_KEY&gt;&quot;)</pre><pre># fast, structured, no synthesis - good for high-frequency lookups<br>fast = search.instant_search(query=&quot;current Bing Search API status&quot;, limit=10)</pre><pre># slower, content-returning, better for building a RAG corpus<br>deep = search.search(<br>    query=&quot;Spring Boot hexagonal architecture reference implementation&quot;,<br>    limit=5,<br>    render_javascript=True,<br>    return_content=True,<br>)</pre><p>These tools are trusted by 15,000+ clients globally, can scrape over 5TB of data daily, and come with 99.99% service uptime.</p><p>The render_javascript and return_content flags matter more than they look. A lot of internal documentation lives behind client-side rendering, Confluence exports, or gated developer portals with JS-heavy front ends, and a scraper that can’t execute JavaScript will hand you an empty shell instead of the page you actually wanted indexed, which is exactly where a managed platform makes a difference to extraction reliability.</p><h4>Wiring it into ai agents: the <a href="https://www.youtube.com/watch?v=yv7PDbVb0Ww">Hermes plugin</a></h4><p>The clearest illustration of “one provider, both halves” is the <a href="https://github.com/oxylabs/hermes-web-oxylabs">oxylabs/hermes-web-oxylabs</a> plugin for <a href="https://www.youtube.com/watch?v=yv7PDbVb0Ww">Hermes Agent</a>. It’s a small plugin, a provider file and a manifest, but it’s a useful reference for how a single web-data vendor slots into an agent’s tool layer as both the search backend and the extraction backend at once, rather than stitching together two separate providers for two separate jobs. Teams used to inspect or collect this kind of data manually, which is time consuming compared with automated web scraping.</p><p>Install and config are about as minimal as this gets:</p><pre>hermes plugins install oxylabs/hermes-web-oxylabs --enable<br>export OXYLABS_API_KEY=&quot;...&quot;</pre><pre>hermes config set web.search_backend oxylabs<br>hermes config set web.extract_backend oxylabs</pre><p>or, in config form:</p><pre>web:<br>  backend: &quot;oxylabs&quot;</pre><p>Once it’s wired in, the agent gets two tools backed by the same provider: web search that returns ranked titles, URLs, and descriptions, and web extraction that turns a target page into clean markdown instead of raw, noisy HTML. Modern web scraping software usually automates extraction rather than doing it by hand, often by driving a browser with tools such as Selenium or Playwright. Worth flagging honestly: <a href="https://www.youtube.com/watch?v=yv7PDbVb0Ww">Hermes</a> core’s default web_extract tool currently requests markdown with JavaScript rendering off and doesn’t expose geo-location as a model-callable parameter yet. The Oxylabs provider already accepts output-format, JS-rendering, and geo-location options under the hood, so the forward compatibility is there, but until Hermes core exposes those knobs to the model, you’re relying on the provider defaults for anything JS-rendered. On harder sites, CAPTCHA solvers are one of the techniques used to get past scraping restrictions.</p><p>That’s a small thing, but it’s the kind of detail that separates “we added a search tool” from “we configured a search layer,” which is the whole point Haberlah’s Perplexity-routing writeup makes about Claude Code and Brave: the default connection isn’t the interesting part, the configuration on top of it is. The same is true here. Installing the plugin gets you a connection. Deciding when your agent should reach for instant_search versus a slower, content-returning crawl, and what happens when a JS-heavy internal wiki page needs indexing, is where the actual engineering is.</p><h4>A rough decision tree for data collection, if you’re setting this up</h4><p>None of this needs to be complicated, but it’s worth writing down rather than leaving to the model’s judgment:</p><ul><li>Fast, structured, no ambiguity (“is this still the current version,” “what’s the status of X”): route to a fast, non-synthesizing search call. Speed and freshness matter more than reasoning here.</li><li>Comparisons and synthesis (“compare these two approaches,” “what changed and why”): route to a reasoning-capable search or a full-content search call, and accept the extra latency.</li><li>Building or refreshing an internal knowledge base: route to a crawler or scraper with JavaScript rendering and content extraction on, aimed at your own docs, ADRs, and any external references worth indexing, on a schedule, not on demand; the same setup also supports vertical aggregation platforms that automate data harvesting for specific industries.</li><li>Everything domain-specific: don’t reach for the web at all. Retrieve from your own indexed codebase and docs. If the agent is inventing your architecture patterns from general knowledge instead of retrieving them, that’s a signal your internal index is missing something, not a signal to search harder. In the extraction process, some methods can also use regular expressions to pull structured data from web pages before indexing.</li></ul><h4>The actual takeaway</h4><p>An agent’s context has an outside and an inside, and legal boundaries around collecting it vary by jurisdiction and by the terms of the websites involved. The outside is what changed in the world since the model finished training. The inside is what was never public in the first place. Treating either one as solved by “add a search tool” or “add a RAG pipeline” and stopping there leaves the other half of the agent’s context exactly as broken as it was before. The tooling to fix both increasingly comes from the same place, because both problems are, underneath the framing, a web data acquisition problem with different latency and synthesis requirements. In the U.S., scraping can implicate the Computer Fraud and Abuse Act, though the Ninth Circuit ruled in 2019 that some scraping did not violate the CFAA. Worth building your decision tree around that, rather than around whichever half of the problem you noticed first. Copyright law distinguishes copying facts from copying original expression, publicly available data can still be personal data under GDPR, and Australia’s Spam Act 2003 restricts some web-harvesting practices.</p><p>Related articles that might be of interest to you:</p><p><a href="https://blog.cubed.run/search-api-for-ai-agent-66b03339d38c?sharedUserId=lambert.watts.809">Search API for AI Agents: Fast Search API vs Tavily vs Firecrawl (And When You Need AI Studio Instead)</a><br><a href="https://medium.com/behind-the-firewall/chatgpt-vs-gemini-vs-perplexity-which-ones-actually-hardest-to-scrape-for-geo-tracking-0adcb96357e7?sharedUserId=lambert.watts.809">ChatGPT vs. Gemini vs. Perplexity: Which One’s Actually Hardest to Scrape for GEO Tracking</a><br><a href="https://medium.com/behind-the-firewall/scrape-google-gemini-answers-4-methods-ranked-27f119ce1dfe">How to Scrape Google Gemini Answers (4 Methods, Ranked)</a><br><a href="https://blog.stackademic.com/python-web-scraping-complete-guide-2026-027c54e9f1b4">Python Web Scraping: A Complete Step-by-Step Guide (2026)</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a396d593bfd9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/behind-the-firewall/your-ai-coding-agent-has-two-problems-and-youre-fixing-one-a396d593bfd9">Your AI Coding Agent Has Two Context Problems, and You’re Probably Only Fixing One</a> was originally published in <a href="https://medium.com/behind-the-firewall">Behind the Firewall</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Gemini for Web Scraping: Building a Python Scraper with Google’s AI]]></title>
            <link>https://medium.com/behind-the-firewall/gemini-for-web-scraping-a-python-scraper-6d29cde8cf33?source=rss----4619bd0a3bc2---4</link>
            <guid isPermaLink="false">https://medium.com/p/6d29cde8cf33</guid>
            <category><![CDATA[gemini]]></category>
            <category><![CDATA[google-ai]]></category>
            <category><![CDATA[web-scraping]]></category>
            <category><![CDATA[gemini-web-scraping]]></category>
            <category><![CDATA[residential-proxies]]></category>
            <dc:creator><![CDATA[Lambert Watts]]></dc:creator>
            <pubDate>Fri, 07 Aug 2026 08:44:37 GMT</pubDate>
            <atom:updated>2026-08-07T08:47:33.589Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3IpfKw4GP7VYYOKMbPHigg.png" /></figure><p>Web scraping used to mean writing brittle CSS selectors that broke every time a site changed its layout. Then AI models got good enough to just… read the page and figure out what you want.</p><p>That’s where Gemini web scraping comes in. Google’s Gemini can parse messy HTML, pull out structured data, and even help you write the scraper itself. No more babysitting XPath expressions every time a developer tweaks a div.</p><p>In this guide, we’ll build a working Python Gemini scraper from the ground up. You’ll see how Gemini fits into your scraping stack, where it saves you time, and where you still need traditional tools to do the heavy lifting.</p><h4>How does Gemini web scraping work?</h4><p>Gemini is Google’s AI model, built to understand and generate natural language, code, and HTML. On its own, it can’t open a browser or visit a URL.</p><p>Here’s where it gets useful: you feed it raw HTML, and it reads that page the way a person would. It picks out the product name, the price, the review count, whatever you need, without you writing a single selector.</p><p>That’s the core of Gemini web scraping. Python handles the fetching, Gemini handles the understanding.</p><h4>Building a Gemini scraper: the tutorial</h4><p>Here’s how to put it all together with <a href="https://oxylabs.io/products/scraper-api/serp/gemini">Gemini web scraping</a>, Python, and Oxylabs Residential Proxies.</p><p>Alternatively, you can look into providers like Decodo, Webshare, or ScrapingBee, just to name a few solid ones. But for the sake of this tutorial, we’ll be using Oxylabs proxy servers, as they’re known for:</p><ul><li>High success rates on tough, anti-bot-protected sites</li><li>Global IP coverage across residential and datacenter pools</li><li>Reliable uptime for large-scale, long-running scraping jobs</li><li>Simple integration with existing Python scraping setups</li></ul><h4>Set up your environment</h4><p>Grab the latest Python version from the<a href="https://www.python.org/downloads/"> official site</a> if you don’t have it yet.</p><p>Next, create a project folder, then set up your script file and a virtual environment:</p><pre>python -m venv env<br>source env/bin/activate<br>touch main.py</pre><h4>Installing libraries</h4><p>With your virtual environment running, install what the script needs.</p><p>Here’s the toolkit for this build:</p><ul><li>requests — sends the HTTP requests to the target site</li><li>pydantic — defines the response format you want back from the LLM</li><li>bs4 — parses the raw HTML</li><li>google-genai — connects to the Gemini API</li></ul><p>You’ll also import the built-in json module later to save the scraped data.</p><p>Install everything with:</p><pre>pip install requests pydantic bs4 google-genai</pre><p>The libraries are now installed, so you can open up your previously created Python file and import the libraries:</p><pre>import json<br>from pydantic import BaseModel<br>import requests<br>from bs4 import BeautifulSoup<br>from google import genai</pre><h4>Gemini scraper with proxies</h4><p>Before writing any Gemini web scraping code, get access to Gemini through Google AI Studio.</p><p>Log in with your Google account, then head to the<strong> </strong>API key page<strong>. </strong>Create a project if you don’t already have one, and generate your key.</p><p>With the key in hand, set it as an environment variable inside your virtual environment. This keeps it out of your actual script:</p><pre>export GEMINI_API_KEY=&quot;&lt;your_api_key&gt;&quot;</pre><p>Head to the Oxylabs dashboard and log in. From there, grab your <a href="https://oxylabs.io/products/residential-proxy-pool">Residential Proxies</a> user credentials.</p><p>Once you have them, drop them at the top of your script:</p><pre>OXYLABS_USERNAME=&quot;USERNAME&quot;<br>OXYLABS_PASSWORD=&quot;PASSWORD&quot;</pre><h4>Using a Gemini scraper for basic scraping tasks</h4><p>With your libraries imported and credentials in place, it’s time to actually put our Gemini scraper to practice.</p><p><strong>Fetching the page with </strong><a href="https://oxylabs.io/products/residential-proxy-pool"><strong>Residential Proxies</strong></a></p><p>Start with a function that does one job: take a URL, return the HTML. Call it scrape_website.</p><p>This keeps your scraping logic separate from your Gemini logic, which makes the script easier to debug later. Here’s the setup using <a href="https://oxylabs.io/products/residential-proxy-pool">Oxylabs Residential Proxies</a>:</p><pre>def scrape_website(url: str) -&gt; str:<br>   ...</pre><p>Now let’s build out the function itself.</p><p>First up: the proxy definition.</p><p>We’re using a rotating residential proxy with a country entry point. Set up the proxy URL, then the dictionary you’ll pass into the HTTP request:</p><pre>proxy_url = f&quot;@pr.oxylabs.io:7777&quot;&gt;http://customer-{OXYLABS_USERNAME}:{OXYLABS_PASSWORD}@pr.oxylabs.io:7777&quot;<br><br>proxies = {<br>    &quot;http&quot;: proxy_url,<br>    &quot;https&quot;: proxy_url,<br>}</pre><p>Last piece of this function: fire off the request.</p><p>Make it a GET request, passing in the URL and the proxies dictionary you just built:</p><pre>response = requests.get(url, proxies=proxies)<br>response.raise_for_status()</pre><p>Add response.raise_for_status() right after the request. This stops execution immediately if something goes wrong with the scrape, instead of letting bad data slip through silently.</p><p>Then return the response text. Here’s the complete function:</p><pre>def scrape_website(url: str) -&gt; str:<br>    proxy_url = f&quot;@pr.oxylabs.io:7777&quot;&gt;http://customer-{OXYLABS_USERNAME}:{OXYLABS_PASSWORD}@pr.oxylabs.io:7777&quot;<br><br>    proxies = {<br>        &quot;http&quot;: proxy_url,<br>        &quot;https&quot;: proxy_url,<br>    }<br><br>    response = requests.get(url, proxies=proxies)<br>    response.raise_for_status()<br>    return response.text</pre><h4>Extracting data from the HTML with Gemini scraper</h4><p>Now for the fun part: turning raw HTML into structured data.</p><p>Modern LLMs are good enough to just read a prompt and pull out what you ask for. If the page is small, you can dump the whole HTML in and tell Gemini what you want back.</p><p>But Gemini’s client also gives you structured outputs. Instead of hoping the model formats its answer the way you expect, you define the exact shape of data you want in advance.</p><p>For this tutorial, we’re scraping an example e-commerce site.</p><p>Define a Product model with pydantic, containing title, price, and description fields. That tells Gemini exactly what to look for and how to structure the response:</p><pre>class Product(BaseModel):<br>    title: str<br>    price: str<br>    description: str</pre><p>Then, we can define the function we’ll be using to parse out product data from a scraped website’s HTML.</p><p>We shall call it extract_products_from_html.</p><pre>def extract_products_from_html(html: str) -&gt; list[Product]:<br>    ...</pre><p>The function returns a list of Product objects, so anyone reading the code knows exactly what data comes out the other end.</p><p>Next, set up the Gemini client and the prompt you’ll send it. Keep the prompt short. Gemini doesn’t need elaborate instructions, just the basics of what to extract:</p><pre>client = genai.Client()<br>prompt = f&quot;Extract information about all products in the following HTML: {html}.&quot;</pre><p>Time to send the actual request to the LLM.</p><p>We’re using gemini-2.5-flash-lite, Google’s fastest and most cost-efficient model right now. It has more than enough power for this job.</p><p>Pass in your list of Product models as the response_schema parameter. This tells Gemini to use structured outputs for the generation:</p><pre>response = client.models.generate_content(<br>    model=&quot;gemini-2.5-flash-lite&quot;,<br>    contents=prompt,<br>    config={<br>        &quot;response_mime_type&quot;: &quot;application/json&quot;,<br>        &quot;response_schema&quot;: list[Product],<br>    },<br>)</pre><p>The function wraps up by returning response.parsed, giving you back a clean list of Product objects.</p><p>Here’s the complete function, start to finish:</p><pre>def extract_products_from_html(html: str) -&gt; list[Product]:<br>    client = genai.Client()<br><br>    prompt = f&quot;Extract information about all products in the following HTML: {html}.&quot;<br>    response = client.models.generate_content(<br>        model=&quot;gemini-2.5-flash-lite&quot;,<br>        contents=prompt,<br>        config={<br>            &quot;response_mime_type&quot;: &quot;application/json&quot;,<br>            &quot;response_schema&quot;: list[Product],<br>        },<br>    )<br><br>    return response.parsed</pre><h4>Running the code</h4><p>Both Gemini scraping functions are built. Time to see them work.</p><p>Set the URL for the site you’re scraping, then call both functions at the bottom of your script.</p><p>We’re using a <a href="https://sandbox.oxylabs.io/">scraping sandbox</a> hosted by Oxylabs for this example, so you can test freely without hitting a live production site.</p><pre>url = &quot;https://sandbox.oxylabs.io/products&quot;<br><br>content = scrape_website(url)<br>products = extract_products_from_html(content)</pre><p>We’ll store the scraped products in a JSON file — this way, we can view the data in a structured format.</p><pre>products_json = [product.model_dump() for product in products]<br><br>with open(&quot;products.json&quot;, &quot;w&quot;) as f:<br>    json.dump(products_json, f, indent=2)</pre><p>Run the script, and you’ll get a new file called products.json in your current directory.</p><p>Open it up, and the data should look something like this:</p><pre>[<br>  {<br>    &quot;title&quot;: &quot;Super Mario Odyssey&quot;,<br>    &quot;price&quot;: &quot;89,99&quot;,<br>    &quot;description&quot;: &quot;New Evolution of Mario Sandbox-Style Gameplay. Mario embarks on a new journey through unknown worlds, running and jumping through huge 3D worlds in the first sandbox-style Mario game since Super Mario 64 and Super Mario Sunshine. Set sail between expansive worlds aboard an airship, and perform all-new actions, such as throwing Mario&#39;s cap.&quot;<br>  },<br>  {<br>    &quot;title&quot;: &quot;Halo: Combat Evolved&quot;,<br>    &quot;price&quot;: &quot;87,99&quot;,<br>    &quot;description&quot;: &quot;Enter the mysterious world of Halo, an alien planet shaped like a ring. As mankind&#39;s super soldier Master Chief, you must uncover the secrets of Halo and fend off the attacking Covenant. During your missions, you&#39;ll battle on foot, in vehicles, inside, and outside with alien and human weaponry. Your objectives include attacking enemy outposts, raiding underground labs for advanced technology, rescuing fallen comrades, and sniping enemy forces. Halo also lets you battle three other players via intense split screen combat or fight cooperatively with a friend through the single-player missions.&quot;<br>  },<br>]</pre><p>Gemini parsed everything without a single explicit instruction. No selectors, no manual mapping.</p><p>But run this at scale, and the LLM calls add up fast. Let’s look at how to keep costs down</p><h4>Parsing CSS selectors with Gemini</h4><p>Here’s a smarter approach to Gemini web scraping — instead of feeding Gemini the full HTML every time, use it once to find the CSS selectors for you. Then reuse those selectors with regular parsing tools on every future run.</p><p>That means calling the API occasionally instead of on every single job.</p><p>Start with a pydantic model for the selectors themselves:</p><pre>class ProductSelectors(BaseModel):<br>    title_selector: str<br>    price_selector: str<br>    description_selector: str<br>    product_card_selector: str</pre><p>One selector per field you want to extract, plus a selector for a single product card, which you’ll need shortly.</p><p>Now define get_product_selectors. It takes an html argument and returns a ProductSelectors object. The structure mirrors your earlier extract_products_from_html function, just with a different prompt and response_schema:</p><pre>def get_product_selectors(html: str) -&gt; ProductSelectors:<br>    client = genai.Client()<br><br>    prompt = f&quot;Extract CSS selectors for retrieving information about all products in the following HTML: {html}&quot;<br>    response = client.models.generate_content(<br>        model=&quot;gemini-2.5-flash-lite&quot;,<br>        contents=prompt,<br>        config={<br>            &quot;response_mime_type&quot;: &quot;application/json&quot;,<br>            &quot;response_schema&quot;: ProductSelectors,<br>        },<br>    )<br>    return response.parsed</pre><p>Now put those selectors to work. Use BeautifulSoup, the library you installed earlier, to parse the data yourself this time.</p><p>Write a function called parse_products_with_selectors. It takes the HTML and the selectors, and returns a list of Product objects:</p><pre> def parse_products_with_selectors(html: str, selectors: ProductSelectors) -&gt; list[Product]:<br>    ...</pre><p>Inside the function, select every product card in the HTML, loop through each one, and pull out the fields you need.<br>It looks something like this:</p><pre>def parse_products_with_selectors(html: str, selectors: ProductSelectors) -&gt; list[Product]:<br><br>    soup = BeautifulSoup(html, &quot;html.parser&quot;)<br>    products = soup.select(selectors.product_card_selector)<br>    parsed_products = []<br><br>    for product in products:<br>        title_element = product.select_one(selectors.title_selector)<br>        title = title_element.text if title_element else &quot;&quot;<br><br>        price_element = product.select_one(selectors.price_selector)<br>        price = price_element.text if price_element else &quot;&quot;<br><br>        description_element = product.select_one(selectors.description_selector)<br>        description = description_element.text if description_element else &quot;&quot;<br><br>        parsed_product = Product(title=title, price=price, description=description)<br>        parsed_products.append(parsed_product)<br><br>    return parsed_products</pre><p>Now bring it all together. Run the script once more, and save the scraped products to a JSON file, same as before.</p><pre>url = &quot;https://sandbox.oxylabs.io/products&quot;<br><br>content = scrape_website(url)<br>selectors = get_product_selectors(content)<br>products = parse_products_with_selectors(content, selectors)<br><br>products_json = [product.model_dump() for product in products]<br><br>with open(&quot;products.json&quot;, &quot;w&quot;) as f:<br>    json.dump(products_json, f, indent=2)</pre><p>If you run it, you’ll see the same products as before.</p><p>The output hasn’t changed, but the process has. You call the LLM once, save the selectors, and reuse them on every future scrape. Fewer API calls, same results.</p><h4>Gemini scraper full code</h4><p>Here’s the complete Gemini scraper<strong> </strong>code with both examples combined:</p><pre>import json<br>from pydantic import BaseModel<br>import requests<br>from bs4 import BeautifulSoup<br>from google import genai<br><br>OXYLABS_USERNAME = &quot;USERNAME&quot;<br>OXYLABS_PASSWORD = &quot;PASSWORD&quot;<br><br><br>class ProductSelectors(BaseModel):<br>    title_selector: str<br>    price_selector: str<br>    description_selector: str<br>    product_card_selector: str<br><br><br>class Product(BaseModel):<br>    title: str<br>    price: str<br>    description: str<br><br><br>def get_product_selectors(html: str) -&gt; ProductSelectors:<br>    client = genai.Client()<br><br>    prompt = f&quot;Extract CSS selectors for retrieving information about all products in the following HTML: {html}&quot;<br>    response = client.models.generate_content(<br>        model=&quot;gemini-2.5-flash-lite&quot;,<br>        contents=prompt,<br>        config={<br>            &quot;response_mime_type&quot;: &quot;application/json&quot;,<br>            &quot;response_schema&quot;: ProductSelectors,<br>        },<br>    )<br><br>    return response.parsed<br><br><br>def extract_products_from_html(html: str) -&gt; list[Product]:<br>    client = genai.Client()<br><br>    prompt = f&quot;Extract information about all products in the following HTML: {html}.&quot;<br>    response = client.models.generate_content(<br>        model=&quot;gemini-2.5-flash-lite&quot;,<br>        contents=prompt,<br>        config={<br>            &quot;response_mime_type&quot;: &quot;application/json&quot;,<br>            &quot;response_schema&quot;: list[Product],<br>        },<br>    )<br><br>    return response.parsed<br><br><br>def scrape_website(url: str) -&gt; str:<br>    proxy_url = (<br>        f&quot;@pr.oxylabs.io:7777&quot;&gt;http://customer-{OXYLABS_USERNAME}:{OXYLABS_PASSWORD}@pr.oxylabs.io:7777&quot;<br>    )<br><br>    proxies = {<br>        &quot;http&quot;: proxy_url,<br>        &quot;https&quot;: proxy_url,<br>    }<br><br>    response = requests.get(url, proxies=proxies)<br>    response.raise_for_status()<br><br>    return response.text<br><br>def parse_products_with_selectors(<br>    html: str, selectors: ProductSelectors<br>) -&gt; list[Product]:<br><br>    soup = BeautifulSoup(html, &quot;html.parser&quot;)<br>    products = soup.select(selectors.product_card_selector)<br>    parsed_products = []<br><br>    for product in products:<br>        title_element = product.select_one(selectors.title_selector)<br>        title = title_element.text if title_element else &quot;&quot;<br><br>        price_element = product.select_one(selectors.price_selector)<br>        price = price_element.text if price_element else &quot;&quot;<br><br>        description_element = product.select_one(selectors.description_selector)<br>        description = description_element.text if description_element else &quot;&quot;<br><br>        parsed_product = Product(title=title, price=price, description=description)<br>        parsed_products.append(parsed_product)<br><br>    return parsed_products<br><br>url = &quot;https://sandbox.oxylabs.io/products&quot;<br><br># First example<br><br>content = scrape_website(url)<br>products = extract_products_from_html(content)<br>products_json = [product.model_dump() for product in products]<br><br>with open(&quot;products_1.json&quot;, &quot;w&quot;) as f:<br>    json.dump(products_json, f, indent=2)<br><br># Second example<br><br>content = scrape_website(url)<br>selectors = get_product_selectors(content)<br>products = parse_products_with_selectors(content, selectors)<br>products_json = [product.model_dump() for product in products]<br><br>with open(&quot;products_2.json&quot;, &quot;w&quot;) as f:<br>    json.dump(products_json, f, indent=2)</pre><h4>Real-world use cases for Gemini web scraping</h4><p>1. E-commerce product data</p><p>Gemini pulls names, prices, images, and reviews from sites like Amazon or Shopify. When a layout changes, it adapts instead of breaking.</p><p>2. Content aggregation</p><p>No custom selectors needed per site. Gemini reads headlines, authors, and timestamps, and summarizes as it goes.</p><p>3. RAG applications</p><p>Fresh web data keeps chatbots grounded. Gemini pulls live content from sites or APIs so responses stay current instead of stale.</p><p>4. Business intelligence</p><p>Price monitoring, market research, competitor monitoring, all without custom logic per source. Gemini adds context too, flagging market gaps instead of just dumping numbers.</p><h4>Challenges and limits of Gemini for web scraping</h4><p>Of course, Gemini web scraping isn’t free of tradeoffs — here are some challenges to be aware of:</p><p>1. Speed</p><p>AI processing takes 2–5 seconds per query, versus milliseconds for a regular parser. Fine for small jobs, a real bottleneck at scale.</p><p>2. Cost</p><p>Token-based pricing means large HTML inputs add up fast. Batch requests or trim what you extract to keep costs sane. Custom Parser by Oxylabs is worth a look if you want to cut down on manual parser work while keeping costs predictable.</p><p>3. Accuracy</p><p>Gemini predicts text, but it doesn’t enforce strict rules. That means occasional hallucinations or misreads on messy layouts. Pair it with schema validation to catch errors before they hit production.</p><p>This tutorial covered building a Gemini scraper from scratch: environment setup, structured outputs, Residential Proxies, and the tradeoffs that come with Gemini web scraping at scale.</p><h4>To sum up</h4><p>That’s the full build: a Gemini scraper that fetches HTML through Residential Proxies and hands it off to Gemini for structured extraction, plus the selector-caching trick to keep costs down.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6d29cde8cf33" width="1" height="1" alt=""><hr><p><a href="https://medium.com/behind-the-firewall/gemini-for-web-scraping-a-python-scraper-6d29cde8cf33">Gemini for Web Scraping: Building a Python Scraper with Google’s AI</a> was originally published in <a href="https://medium.com/behind-the-firewall">Behind the Firewall</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AI Training Data: How to Source Quality Datasets at Scale]]></title>
            <link>https://medium.com/behind-the-firewall/ai-training-data-how-to-source-quality-datasets-at-scale-b97e50018e00?source=rss----4619bd0a3bc2---4</link>
            <guid isPermaLink="false">https://medium.com/p/b97e50018e00</guid>
            <category><![CDATA[api-web-scraping]]></category>
            <category><![CDATA[dataset]]></category>
            <category><![CDATA[web-scraping]]></category>
            <category><![CDATA[ai-data-collection]]></category>
            <category><![CDATA[ai-training]]></category>
            <dc:creator><![CDATA[Lambert Watts]]></dc:creator>
            <pubDate>Mon, 03 Aug 2026 07:24:40 GMT</pubDate>
            <atom:updated>2026-08-05T07:13:21.054Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LbYpawKfc5lEl3mJaps8GQ.png" /></figure><p>Good AI models need good data, simple as that. Most teams learn this the hard way, after training on datasets full of duplicates or mislabeled samples.</p><p>AI data collection at scale is all about sourcing clean, diverse, and relevant data without burning months on manual collection. As a result, API web scraping has become the go-to method for teams that need fresh, structured data on a schedule.</p><p>This guide covers where to find quality datasets, how to collect them efficiently, and what separates a usable dataset from, well, a useless one.</p><h4>AI data collection defined</h4><p>AI data collection is the process of collecting information that machine learning models can actually learn from. You might be thinking, it’s just another data scraping process; unfortunately, it’s much more difficult than that.</p><p>The specific AI data collection process involves finding data across multiple sources, pulling it out, and structuring it into datasets a model can train on.</p><p>When done well, it gives an AI system the raw material to see patterns and make accurate predictions. On the contrary, if done poorly, it gives the model bad habits that show up later as bad outputs.</p><h4>Different types of AI data can be split into</h4><ul><li>Structured — organized data in databases, spreadsheets, and other predefined formats.</li><li>Unstructured — raw, unorganized data that needs interpretation: text, images, videos, social media posts, product reviews, etc.</li><li>Semi-structured — partially organized data with identifiable patterns (JSON, XML, logs, metadata).</li></ul><h4>Different AI data collection methods — which one’s best?</h4><p>Web scraping is the most scalable way to gather AI training data from the internet. It pulls structured and unstructured data from e-commerce sites, social platforms, news portals, and thousands of other sources, turning public web data into usable datasets for ML applications.</p><p>There are multiple ways to collect this data, such as:</p><ul><li>Build your own scraper — the most time-consuming and resource-heavy option</li><li>Integrate proxies into existing infrastructure</li><li>Use an API web scraper — the fastest, lowest-effort option</li></ul><p>Let’s take a closer look at how proxies and scraper APIs stack up for AI data collection.</p><ul><li><strong>AI data collection with proxies</strong></li></ul><p>Premium proxy servers keep your web scraping running without interruptions. You route requests through different IP addresses across multiple locations, which helps you avoid rate limits, IP address blocks, and CAPTCHAs.</p><p>Residential proxies are especially useful here. Since they route your traffic through real device IPs, you blend in with normal user behavior instead of getting flagged as a bot.</p><p>The payoff: a data collection pipeline that doesn’t break down halfway through your training run.</p><ul><li><strong>API web scraping for AI data collection</strong></li></ul><p>API web scraping tools take large-scale scraping and strip out the technical complexity. You get ready-to-use infrastructure instead of building your own from scratch. They handle anti-bot challenges for you and hand back clean, structured data through a simple API call.</p><p>That means you spend your time on model development, not on fighting website defenses to get the data in the first place.</p><h4>How proxies and API web scraping tools help with AI data collection</h4><p>Here’s where AI data collection and API web scraping actually come together in practice.</p><ol><li><strong>Extract responses from AI search engines</strong> — some dedicated LLM scrapers pull responses straight from AI-driven search platforms and chatbots. You get a clear view into how AI systems interpret queries, so you can benchmark your own models against what’s already leading the market.</li><li><strong>Gather multimodal data at scale</strong> — Specialized APIs and high-bandwidth proxies give you access to video, audio, subtitles, and metadata without the manual grind. This multimodal data gathering approach helps you build models that handle more than one data type at once.</li><li><strong>Use ready-made datasets</strong> — video datasets give you instant access to usually creator-approved video data. You skip the collection process entirely and get the contextual detail your model needs to sound more natural.</li><li><strong>Avoid blocks with a proxy network</strong> — proxies from premium providers like <a href="https://oxylabs.io/products/residential-proxy-pool">Oxylabs rotate through thousands of IPs</a> across multiple locations, so you keep collecting data even from heavily protected sites. Your training timeline stays intact instead of stalling on incomplete datasets.</li></ol><p>Though Oxylabs is considered probably the most reliable option in the market, it isn’t the only name in this space.</p><p>Providers like Decodo, Webshare, and ScrapingBee are worth knowing too, especially if you’re comparing options on price or specific use cases. But for enterprise-scale AI data collection, where uptime and clean data matter most, Oxylabs is the one built for that job.</p><h4>Best practices for AI data collection</h4><p>Getting the data is only half the job. Here’s how you keep it clean, legal, and usable at scale.</p><ol><li><strong>Data quality</strong> — Web scraping for AI model training only works if you validate the data as it comes in. Set up multi-layer checks, automated cleaning pipelines, and real-time monitoring, so you’re not training on corrupted or incomplete datasets.</li><li><strong>Ethical and legal compliance</strong> — Data privacy laws and website terms of service aren’t optional reading. Build clear data governance policies, stay aligned with GDPR and CCPA, document where your data comes from, and audit your practices regularly. Of course, it’s always best to consult a professional for your specific case.</li><li><strong>Data governance</strong> — Encrypt your data in transit, secure your storage, control who has access, and audit regularly. It’s what keeps your data usable and trustworthy over the long run, and it’s not just about ticking compliance boxes.</li><li><strong>Scalability and automation</strong> — Proxies and scraper APIs are what let you run large-scale operations without manual babysitting. They keep fresh, relevant data flowing into your pipeline continuously, so your AI systems never run dry.</li></ol><h4>To summarize: quality data beats fast data</h4><p>Good AI models start with good data, and good data doesn’t happen by accident. AI data collection takes the right sourcing methods, quality checks, and compliance work to build a pipeline that holds up at scale.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b97e50018e00" width="1" height="1" alt=""><hr><p><a href="https://medium.com/behind-the-firewall/ai-training-data-how-to-source-quality-datasets-at-scale-b97e50018e00">AI Training Data: How to Source Quality Datasets at Scale</a> was originally published in <a href="https://medium.com/behind-the-firewall">Behind the Firewall</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[I replaced Tavily in my agent setup for Oxylabs Fast Search API. Here’s what changed]]></title>
            <link>https://medium.com/behind-the-firewall/replaced-tavily-in-my-agent-setup-for-oxylabs-fast-search-api-f82d34f8943f?source=rss----4619bd0a3bc2---4</link>
            <guid isPermaLink="false">https://medium.com/p/f82d34f8943f</guid>
            <category><![CDATA[web-scraping]]></category>
            <category><![CDATA[fast-search-api]]></category>
            <category><![CDATA[web-crawling]]></category>
            <category><![CDATA[residential-proxies]]></category>
            <category><![CDATA[data]]></category>
            <dc:creator><![CDATA[Lambert Watts]]></dc:creator>
            <pubDate>Mon, 27 Jul 2026 06:20:50 GMT</pubDate>
            <atom:updated>2026-08-24T09:21:33.605Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*auX94wujtGH7PQI6sPmsBQ.png" /></figure><p>If you’re looking for a fast search api, <a href="https://oxylabs.io/products/scraper-api/fast-search">Oxylabs’ Fast Search API </a>is a pure SERP retrieval API built to return the top 10 organic search results with minimal latency, which makes it a strong fit for agents and AI systems that need web search results fast. If you’ve built anything agentic in the last year, you’ve probably wired up Tavily or Firecrawl without thinking too hard about it. They’re the default answer. Ask in any AI engineering Discord “what do I use for web search in my agent” and you’ll get the same three names back every time.</p><p>I’ve shipped both into production. I also spent the last few weeks running Oxylabs’ new <a href="https://oxylabs.io/products/scraper-api/fast-search">Fast Search API</a> against the same workloads to see where it fits and where it doesn’t. This is for AI engineers, developers building agentic applications or chatbots, and anyone trying to add a fast, reliable web search API to their setup. It’s not a “look how much better this is” post. It’s closer to a field note on what changes when you swap out the search step in your setup, with side-by-side notes on Tavily, Firecrawl, and Oxylabs across latency under load, output format, pricing, data retention, use cases, and the practical tradeoffs that show up during integration.</p><p>That matters when search is on the critical path: slow retrieval turns into slow agents, worse UX, and more brittle systems under load. A fast API with sub-second responses — and options like zero data retention for enterprise teams — can change whether search feels usable in production or becomes the bottleneck.</p><h4>First, the thing nobody says clearly enough</h4><p>Tavily and Firecrawl solve different problems, and half the confusion in this space comes from people treating them as interchangeable.</p><p>Tavily is a search-and-rank API. You send a query, it goes and finds relevant pages, scores them, and hands back summarized results with citations. It’s built for the RAG loop: question in, ranked context out.</p><p>Firecrawl is a crawl-and-extract engine. You point it at a domain or a URL list and it comes back with clean markdown, structured JSON, screenshots, whatever format you asked for. It’s built for the “I need the actual content of this page” problem, not the “what pages should I even look at” problem.</p><p>Fast Search API sits closer to Tavily’s lane than Firecrawl’s. It’s a pure SERP retrieval layer, not a scraper for arbitrary pages. If your bottleneck is “give me the top 10 organic results for this query, fast, and let the model handle the reasoning,” that’s the exact shape of problem it’s built for. If your bottleneck is “crawl this competitor’s whole product catalog into structured JSON,” this isn’t the tool, and Oxylabs isn’t pretending it is. Their own Web Scraper API is the one built for that job.</p><p>So the honest framing is: this is a Tavily and Serper alternative first, and a Firecrawl alternative only for the discovery step of a setup that also does scraping somewhere else.</p><h4>What <a href="https://oxylabs.io/products/scraper-api/web">web scraping</a> software I tested</h4><p>I ran the same batch of 500 queries, a mix of long-tail product research terms and generic informational queries, through three setups:</p><p>Tavily’s standard search endpoint, Firecrawl’s search tool (the lighter-weight one, not the full scrape/extract combo), and Oxylabs Fast Search API.</p><p>I wasn’t watching for accuracy, since all three pull from the same underlying search engines and the result sets were broadly comparable. What mattered was latency under load and how clean the JSON came back, because that’s what decides whether your setup feels responsive or feels like it’s waiting on a page load every time it needs context, and scale becomes just as important once query volume grows in production applications.</p><p>Fast Search API came back under a second on the vast majority of requests, even if modern search APIs in optimized environments can deliver sub-100-millisecond response time. That tracks with what’s on the product page, sub-1s at the 90th percentile, and it held up when I pushed concurrent requests into the hundreds per second range. Low latency at scale usually depends on techniques like in-memory caching and inverted indexes. Tavily was solid too, typically in the 1 to 2 second range, but noticeably slower once concurrency went up. Firecrawl’s search tool, to be fair to it, isn’t really built for this at all, it’s a secondary feature bolted onto a scraping-first product, and it showed. That kind of search infrastructure is also useful for ai workflows that need real-time retrieval and may need to handle millions of queries per day.</p><p>The response payload from Fast Search API is deliberately thin: title, link, description, rank, plus a top stories block. No ads, no featured snippets, no knowledge panels. That’s a real tradeoff. Fast search APIs often add advanced features like autocomplete to improve user experience, but this test was about raw retrieval speed rather than UI extras. If your downstream logic depends on pulling structured data out of a featured snippet, you’ll want the full Web Scraper API instead, or you’ll be missing signal. But if all you need is the ten best pages for a given query, you’re not paying any extra parsing overhead for information you were going to throw away anyway.</p><h4>A quick side-by-side on proxy management</h4><p>Common use cases include e-commerce search, documentation search, and AI assistants, and different teams may compare managed search services with DIY tools depending on workflow needs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-RvHRKdd88z-7P3nwrhoxw.png" /></figure><p>For example, broader search API evaluation often also covers filtering and sorting, full-text search across unstructured text, personalized results, and AI-driven semantic or neural relevance.</p><h4>Where I’d actually use it: a use case perspective</h4><p>If you’re building an AI chatbot, an answer engine, or any agent that needs to ground responses in current search results before it starts reasoning, and speed under load matters more to you than every last SERP feature, this is worth trying for business use cases where teams need current results quickly for assistants or answer engines. The zero data retention part also carries more weight than people give it credit for once you’re dealing with enterprise clients who ask pointed questions about where their prompts and results get logged. High-reliability search APIs are also useful solutions when companies need to retrieve and structure data efficiently, including from multiple search engines in broader search setups.</p><p>Where I wouldn’t reach for it: if you currently lean on Firecrawl for the actual page extraction step, nothing here replaces that, and you shouldn’t try to bend it into doing so. And if you need multi-engine coverage or heavy geo-targeting down to the city or coordinate level for SEO tracking work, Oxylabs’ own SERP Scraper API, the bigger sibling product, is the better fit than Fast Search API, since that one keeps ads, featured snippets, and full localization intact.</p><h4>The pricing gap, web scraping legal issues, and why it matters</h4><p>The one place I’d push back a little: Fast Search API doesn’t publish pricing anywhere. Tavily and Serper both put numbers on a page you can read in ten seconds. Oxylabs wants you to talk to sales first. For something aimed at developers who want to move fast and self-serve, that’s friction, and it’s the main reason I’d tell anyone evaluating this to budget an extra day for the back-and-forth before they can weigh total cost against Tavily’s published tiers.</p><h4>Bottom line</h4><p>Don’t read this as “drop Tavily, drop Firecrawl, use this instead.” Read it as: if the search step is the part making your latency numbers look bad, there’s an option built specifically for that job, and it’s fast enough that I’ve started sending a part of my own traffic through it while keeping Firecrawl around for the crawling work. Two different jobs, same setup. Worth testing side by side with whatever you’re running now before you decide either way.</p><p><strong>Continue reading related articles</strong></p><p><a href="https://medium.com/behind-the-firewall/chatgpt-web-scraping-the-complete-2026-guide-python-6ca20161a060?sharedUserId=lambert.watts.809">ChatGPT Web Scraping: The Complete 2026 Guide (Python)</a><br><a href="https://ai.plainenglish.io/the-best-web-scraper-in-2026-5c6a8758e066?sharedUserId=lambert.watts.809">How to Choose the Best Web Scraper in 2026 Based on Your Project Type</a><br><a href="https://medium.com/@lambert.watts.809/langchain-web-scraping-with-apis-2026-9181a96e797a">LangChain Web Scraping with APIs: Automate Data Extraction in 2026</a><br><a href="https://medium.com/@lambert.watts.809/web-scraping-headers-you-should-know-e78dea707ad8">Web Scraping Headers You Should Know to Avoid Blocks in 2026</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f82d34f8943f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/behind-the-firewall/replaced-tavily-in-my-agent-setup-for-oxylabs-fast-search-api-f82d34f8943f">I replaced Tavily in my agent setup for Oxylabs Fast Search API. Here’s what changed</a> was originally published in <a href="https://medium.com/behind-the-firewall">Behind the Firewall</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ChatGPT vs. Gemini vs. Perplexity: Which One’s Actually Hardest to Scrape for GEO Tracking]]></title>
            <link>https://medium.com/behind-the-firewall/chatgpt-vs-gemini-vs-perplexity-which-ones-actually-hardest-to-scrape-for-geo-tracking-0adcb96357e7?source=rss----4619bd0a3bc2---4</link>
            <guid isPermaLink="false">https://medium.com/p/0adcb96357e7</guid>
            <category><![CDATA[chatgpt]]></category>
            <category><![CDATA[web-scraping]]></category>
            <category><![CDATA[gemini]]></category>
            <category><![CDATA[perplexity]]></category>
            <category><![CDATA[residential-proxies]]></category>
            <dc:creator><![CDATA[Lambert Watts]]></dc:creator>
            <pubDate>Fri, 24 Jul 2026 05:40:26 GMT</pubDate>
            <atom:updated>2026-07-24T05:40:23.246Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*P77a5Jl5SR2P5n7MWIE60w.png" /></figure><p>If you’re running any kind of GEO program right now, you’ve probably already hit the same wall from three different directions. Every AI platform your brand needs to show up in has its own frontend, its own anti-bot posture, and its own idea of what a “response” even contains. Tracking one of them is a project. Tracking all three on an ongoing basis is closer to maintaining three separate scraping pipelines that happen to serve the same report.</p><p>We’ve covered ChatGPT, Gemini, and Perplexity separately before. This one’s about how they stack up against each other when you’re the one who has to actually go get the data, on a schedule, across markets, without it turning into a full-time job.</p><h4>Why compare them at all</h4><p>Each platform answers a slightly different question for GEO purposes. Perplexity is citation-heavy and search-native, so it’s the closest thing to a rankings page for AI answers. Gemini is baked into Google Search through AI Overviews and sits on every Android phone by default, which makes it the one with the widest reach even when nobody thinks of it as “an AI chatbot” first. ChatGPT is still the biggest single surface by user count, and it’s the one that just picked up a commercial layer, sponsored cards and product discovery, that the other two don’t have yet.</p><p>If your tracking setup only covers one of these, you’re missing a meaningfully different audience with a different answer format each time.</p><h4>How hard is each one to scrape for financial data, realistically</h4><p>ChatGPT. No official way to reach it besides the API, and the API doesn’t reflect what a logged-in user actually sees, since the web app layers in memory, browsing, and continuity from previous inputs in a conversation that a raw model call won’t return. Scraping the actual chat interface means dealing with a modern SPA, session state, and Cloudflare-grade bot detection on OpenAI’s frontend. Doable, but the DOM shifts often enough that a homemade parser needs regular upkeep, and even polished product answers can sound credible while still being inaccurate.</p><p>Gemini. Sits behind Google’s own infrastructure, which is not a lightly defended surface. Expect device and session fingerprinting on top of the usual JS rendering problem. The upside is that once you’re past the wall, the response is richer. Gemini can return citations, related content, and multimodal elements that some other platforms don’t expose in a scrapable form.</p><p>Perplexity. Generally the most forgiving frontend of the three to render, but it’s really a guided AI search product aimed at deeper exploration, so the payoff is also the most complex to parse. A single answer can carry cited sources, related searches, images, and tables all in one response, and it can also generate structured summaries that aggregate cited sources, while the HTML structure around all of that isn’t especially stable between updates. Its answer stack can also draw on models from OpenAI, Anthropic, and Meta, which adds to variation in output structure.</p><p>None of these are “easy.” They’re differently hard, which is exactly the problem with building three bespoke scrapers instead of one.</p><h4>Head-to-head comparison of ai models</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RHnFrbNFFYYwGTpdixrZog.png" /></figure><h4>The pattern that shows up every time with up to date answers</h4><p>Manual checks don’t scale past a handful of prompts a week. Calling the underlying model’s API is cheap and consistent, but it measures the model, not the product your audience actually sees, since none of the three raw APIs include the citations, sponsored content, or UI-layer ranking that show up in the real app. Building your own scraper gets you the real experience, but now you’re maintaining three separate proxy setups, three separate parsers, and three separate sets of anti-bot workarounds that all break on their own schedule.</p><p>That’s the cost of tracking GEO across platforms properly. It’s not the scraping logic, that part’s straightforward. It’s keeping three unrelated maintenance jobs alive at once, indefinitely, for data that goes stale within a week if the pipeline breaks; products also differ in how they preserve context across multiple tasks, as with Perplexity’s assistant, which is another reason raw outputs and real app behavior diverge.</p><h4>One API, three targets for free app syncs</h4><p>This is the part that makes a shared scraper API worth it once you’re tracking more than one platform. <a href="https://oxylabs.io/products/scraper-api">Oxylabs’ Web Scraper API</a> has dedicated targets for ChatGPT, Gemini, and Perplexity that all take the same shape of request and hand back the same shape of parsed response, regardless of which platform’s answer you’re pulling.</p><pre>import requests<br>from pprint import pprint</pre><pre>targets = [<br>    {&#39;source&#39;: &#39;openai&#39;, &#39;prompt&#39;: &#39;best residential proxy providers 2026&#39;, &#39;geo_location&#39;: &#39;United States&#39;},<br>    {&#39;source&#39;: &#39;gemini&#39;, &#39;prompt&#39;: &#39;best residential proxy providers 2026&#39;, &#39;geo_location&#39;: &#39;Germany&#39;},<br>    {&#39;source&#39;: &#39;perplexity&#39;, &#39;prompt&#39;: &#39;best residential proxy providers 2026&#39;, &#39;geo_location&#39;: &#39;United Kingdom&#39;},<br>]</pre><pre>for payload in targets:<br>    payload[&#39;parse&#39;] = True<br>    response = requests.post(<br>        &#39;https://realtime.oxylabs.io/v1/queries&#39;,<br>        auth=(&#39;USERNAME&#39;, &#39;PASSWORD&#39;),<br>        json=payload,<br>    )<br>    pprint(response.json())</pre><p>Same credentials, same endpoint, same JSON shape coming back each time, just a different source value per platform. Each response is already parsed into the answer text, a markdown copy, and a citations or references list where the platform provides one, so a single downstream script can build one GEO report across all three instead of reconciling three different data formats by hand.</p><h4>What this actually saves you</h4><p>The proxy rotation, CAPTCHA handling, and browser fingerprinting work is the same underlying problem no matter which platform you’re targeting, so paying for it once across all three targets is a lot cheaper than building and babysitting three separate stacks that each solve it slightly differently. The bigger saving is honestly just headcount and attention. A scraper that breaks on one platform’s UI update is a Tuesday afternoon problem. Three scrapers that each break on their own schedule is a standing item on someone’s calendar every week.</p><h4>Which setup makes sense?</h4><p>If you’re only watching one platform, a purpose-built scraper for that one target is a reasonable lift. The moment you’re tracking two or more, and most GEO programs end up needing all three within a quarter, a shared scraper API stops being a convenience and starts being the only version of this that doesn’t eat an engineer’s time every time OpenAI, Google, or Perplexity ships a frontend change.</p><h4>FAQs</h4><p>What is ChatGPT? ChatGPT is a generative AI chatbot developed by OpenAI. It works by leveraging a large language model built on a transformer architecture, trained on massive datasets, and improved with reinforcement learning from human feedback.</p><p>What is ChatGPT mainly used for? ChatGPT is mainly used for text generation, information retrieval, practical guidance, and research. It assists with writing, editing, summarizing, drafting, refining, improving written content, and brainstorming. It also helps with learning, academic tasks like explaining concepts and homework help, and language-practice conversation. For programming, it can write code and help with debugging. It can also draft customer support replies. Many people use it for planning and organizing in daily life. Its applications span professional work, personal tasks, and creative projects for many users.</p><p>Can ChatGPT work with images or files? Yes, when those capabilities are available, ChatGPT can analyze images or files. It is designed to understand and generate human-like text, code, and images, and it can create image outputs from text prompts through integrated tools like DALL-E, including support for some video-related workflows.</p><p>Is ChatGPT reliable enough for important decisions? It can help with data analysis by providing insights and forecasts, but its answers are generated from learned statistical patterns rather than true understanding. That means it can sound confident and still be wrong, so it should not be your sole source for important medical, legal, or financial decisions.</p><p>Which AI platform is hardest to scrape? ChatGPT and Gemini sit behind the heaviest anti-bot infrastructure. Perplexity’s frontend is generally more forgiving to render, but its responses are more complex to parse correctly.</p><p>Can I track all three platforms with one tool? Yes. A scraper API with dedicated targets for each platform, like Oxylabs’ Web Scraper API, takes the same request shape and returns the same parsed structure regardless of which platform you’re querying.</p><p>Does calling each platform’s own API work for GEO tracking? Only partially. A direct model API call measures the model’s raw output, not what a real user sees in the free app, since sponsored content, citations, and UI-layer ranking are added on top of the base model response.</p><p>How often should GEO tracking run? Weekly at minimum across your core prompt set and markets. Daily if you’re tracking anything time-sensitive, like sponsored placements or a recent product launch.</p><p>What does Perplexity Pro cost, and what are the main limits? Perplexity Pro is a paid service that costs $20 per month. It also offers in-app purchases for additional credits, and the Pro plan gives access to upload and search through up to 500 files. If you want a quick features check before paying, it also helps to read the latest review details in the app store.</p><p>What privacy and account controls does Perplexity offer? Perplexity says its Pro tier includes stronger security, and user data is encrypted in transit. At the same time, data may be shared with third parties, and privacy practices can vary by region and age. Users can request that their data be deleted, and subscriptions can be canceled anytime in the app settings.</p><p><strong>Learn more by exploring the articles below:</strong></p><p><a href="https://medium.com/@lambert.watts.809/perplexity-web-scraping-methods-f19367832bc1">Perplexity Web Scraping: Methods, Tools, and GEO Data Insights</a><br><a href="https://medium.com/behind-the-firewall/scrape-google-gemini-answers-4-methods-ranked-27f119ce1dfe">How to Scrape Google Gemini Answers (4 Methods, Ranked)</a><br><a href="https://medium.com/behind-the-firewall/chatgpt-web-scraping-the-complete-2026-guide-python-6ca20161a060">ChatGPT Web Scraping: The Complete 2026 Guide (Python)</a><br><a href="https://medium.com/@lambert.watts.809/chatgpt-web-scraping-complete-python-guide-661f7ec87754">ChatGPT Web Scraping: Complete Python Guide for 2025</a><br><a href="https://medium.com/@lambert.watts.809/ai-scraper-6450bd4b81bf">How to Build an AI Scraper With Crawl4AI and ChatGPT<br></a><a href="https://medium.com/@lambert.watts.809/web-scraping-with-claude-ai-1c94a6351d0c">Web Scraping with Claude AI: Python Tutorial<br></a><a href="https://medium.com/@lambert.watts.809/what-is-a-residential-proxy-how-does-it-work-54781190f534">What Is a Residential Proxy &amp; How Does It Work?<br></a><a href="https://medium.com/@lambert.watts.809/how-to-scrape-youtube-6abb8340ae54?postPublishedType=repub">How to Scrape YouTube Data with Python<br></a><a href="https://medium.com/@lambert.watts.809/what-is-ai-scraping-8ab38a3b18b1">What Is AI Scraping? A Modern Guide in 2025<br></a><a href="https://medium.com/@lambert.watts.809/best-proxy-services-c63ce09310d0">Explore the Best Proxy Services: 8 BrightData Alternatives for 2025<br></a><a href="https://medium.com/@lambert.watts.809/http-vs-socks-proxies-7cd07d019bf5">HTTP vs. SOCKS Proxies: What’s the Difference?<br></a><a href="https://medium.com/@lambert.watts.809/how-to-use-curl-with-proxy-step-by-step-guide-7e0b57602292">How to Use cURL with Proxy: Best Practices<br></a><a href="https://medium.com/@lambert.watts.809/how-to-scrape-google-news-3a99e937c132">How to Scrape Google News Using Python<br></a><a href="https://medium.com/@lambert.watts.809/how-to-bypass-amazon-captcha-when-scraping-fec20546abf9">How to Bypass Amazon CAPTCHA When Scraping<br></a><a href="https://medium.com/@lambert.watts.809/best-google-maps-scrapers-47253b701c84">10 Best Google Maps Scrapers In 2025</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0adcb96357e7" width="1" height="1" alt=""><hr><p><a href="https://medium.com/behind-the-firewall/chatgpt-vs-gemini-vs-perplexity-which-ones-actually-hardest-to-scrape-for-geo-tracking-0adcb96357e7">ChatGPT vs. Gemini vs. Perplexity: Which One’s Actually Hardest to Scrape for GEO Tracking</a> was originally published in <a href="https://medium.com/behind-the-firewall">Behind the Firewall</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ChatGPT Web Scraping: The Complete 2026 Guide (Python)]]></title>
            <link>https://medium.com/behind-the-firewall/chatgpt-web-scraping-the-complete-2026-guide-python-6ca20161a060?source=rss----4619bd0a3bc2---4</link>
            <guid isPermaLink="false">https://medium.com/p/6ca20161a060</guid>
            <category><![CDATA[html]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[web-scraping]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[residential-proxies]]></category>
            <dc:creator><![CDATA[Lambert Watts]]></dc:creator>
            <pubDate>Tue, 21 Jul 2026 06:31:20 GMT</pubDate>
            <atom:updated>2026-07-21T06:39:55.689Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0idtoolcbM70OKuvW1FuJA.png" /></figure><p>ChatGPT has changed shape a lot since last year. It writes better code, it holds longer context, and it now sits inside a much bigger commercial ecosystem: sponsored cards show up in free-tier answers, and a discovery-first shopping layer sends millions of product queries through the model every day. None of that changes the basic fact developers keep running into: ChatGPT still can’t reach out and grab data from a live website on its own. What it does very well is write the code that does.</p><p>This guide walks through how to use ChatGPT to build a working Python scraper in 2026, what’s changed with ChatGPT Ads and ChatGPT Shopping, and where the model fits next to Gemini, Perplexity, and Google’s AI Mode if you’re weighing your options.</p><h4>Can ChatGPT scrape websites?</h4><p>No, and this hasn’t changed. ChatGPT is a Generative Pre-trained Transformer. It’s built to predict and generate text and code based on patterns learned from training data, not to reach across the internet and pull structured data off a live page. Web scraping is a different job entirely: fetching a page, parsing its structure, and pulling out the pieces you actually want.</p><p>Without a connected tool or plugin, ChatGPT has no way to send an HTTP request to a target site and hand you back the HTML. What it can do is write the script that sends that request for you. That distinction is the whole game here.</p><h4>ChatGPT web scraping workflows</h4><p>There are two workflows most people end up using:</p><ul><li>Data collection: prompting ChatGPT to generate scraping code</li><li>Data extraction and analysis: feeding scraped data back into ChatGPT to summarize or process it</li></ul><h4>Data collection</h4><p>This is the core use case. You describe what you want and ChatGPT writes the scraper, replacing a manual approach that lacks automation and becomes time consuming and inefficient for large datasets. A good prompt usually covers three things: the language and libraries you want used, any tools you plan to pair it with (a scraping API, proxies, a headless browser), and the format you want the output saved in, whether that’s CSV, JSON, or a database write.</p><h4>Data extraction and processing</h4><p>Once you’ve got the raw data, you can hand it back to ChatGPT and ask it to summarize, categorize, or flag anomalies. Part of that analysis quality comes from supervised training and reinforcement learning from human feedback, where reviewers collect comparison data and feedback to improve responses over time. This is where a lot of teams get real value: not from the scraping step itself, but from what happens after, when the model turns a spreadsheet of product listings or news headlines into something you can actually act on. OpenAI’s own developer docs are worth a read if you want to push further into the API side of this.</p><h4>How to use ChatGPT to scrape a website</h4><p>Here’s a walkthrough using a sandbox site built for scraping practice.</p><h4>Step 1: Grab the HTML</h4><p>Open the target page in your browser, right-click, choose Save As, and save it as Web Page, HTML Only. You’ll end up with a local .html file you can hand to ChatGPT.</p><h4>Step 2: Ask ChatGPT to write the scraper</h4><p>Upload the file and prompt the model to study the structure, build CSS selectors, and turn that into a working script. If the markup is messy, you can also ask for a regular-expression pattern-matching method for data extraction.</p><p>Prompt example:</p><pre>Analyze the HTML structure of the attached page and produce the most resilient and logical CSS selectors to extract each product&#39;s title, price, and URL.</pre><pre>Then, write a Python script using requests and BeautifulSoup to scrape all products from all available pages. Use the CSS selectors to gather product titles, prices, and URLs.</pre><pre>First page URL: <a href="https://books.toscrape.com/catalogue/page-1.html">https://books.toscrape.com/catalogue/page-1.html</a><br>Next page URL: <a href="https://books.toscrape.com/catalogue/page-2.html">https://books.toscrape.com/catalogue/page-2.html</a></pre><pre>Iterate over each page until there are no more pages left. Save all data to a CSV file. Use logging where useful.</pre><p>Semantic annotations or other structured page hints can help locate specific snippets more reliably when selectors alone are brittle.</p><p>ChatGPT will typically hand back something close to this:</p><pre>import csv<br>import logging<br>from urllib.parse import urljoin</pre><pre>import requests<br>from bs4 import BeautifulSoup</pre><pre>START_URL = &#39;https://books.toscrape.com/catalogue/page-1.html&#39;<br>OUTPUT_CSV = &#39;books.csv&#39;</pre><pre>PRODUCT = &#39;article.product_pod&#39;<br>TITLE_ANCHOR = &#39;h3 &gt; a[title]&#39;<br>PRICE = &#39;p.price_color&#39;<br>NEXT = &#39;ul.pager li.next &gt; a&#39;</pre><pre>def make_session():<br>    s = requests.Session()<br>    s.headers.update({<br>        &#39;User-Agent&#39;: &#39;Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 &#39;<br>                      &#39;(KHTML, like Gecko) Chrome/129.0 Safari/537.36&#39;<br>    })<br>    return s</pre><pre>def parse_page(html: str, base_url: str):<br>    soup = BeautifulSoup(html, &#39;html.parser&#39;)<br>    items = []<br>    for card in soup.select(PRODUCT):<br>        a = card.select_one(TITLE_ANCHOR)<br>        if not a:<br>            logging.debug(&#39;Skipping a product without title anchor&#39;)<br>            continue<br>        title = a.get(&#39;title&#39;) or a.get_text(strip=True)<br>        href = a.get(&#39;href&#39;, &#39;&#39;).strip()<br>        prod_url = urljoin(base_url, href)<br>        price_el = card.select_one(PRICE)<br>        price = price_el.get_text(strip=True) if price_el else &#39;&#39;<br>        items.append({&#39;title&#39;: title, &#39;price&#39;: price, &#39;url&#39;: prod_url})<br>    next_a = soup.select_one(NEXT)<br>    next_url = urljoin(base_url, next_a[&#39;href&#39;]) if next_a and next_a.has_attr(&#39;href&#39;) else None<br>    return items, next_url</pre><pre>def scrape_all(start_url: str, out_csv: str):<br>    session = make_session()<br>    visited = set()<br>    url = start_url<br>    total = 0<br>    with open(out_csv, &#39;w&#39;, newline=&#39;&#39;, encoding=&#39;utf-8&#39;) as f:<br>        writer = csv.DictWriter(f, fieldnames=[&#39;title&#39;, &#39;price&#39;, &#39;url&#39;])<br>        writer.writeheader()<br>        while url and url not in visited:<br>            visited.add(url)<br>            logging.info(&#39;Fetching: %s&#39;, url)<br>            resp = session.get(url, timeout=30)<br>            resp.raise_for_status()<br>            resp.encoding = &#39;utf-8&#39;<br>            items, next_url = parse_page(resp.text, resp.url)<br>            if not items:<br>                logging.warning(&#39;No products found on %s&#39;, url)<br>            for it in items:<br>                writer.writerow(it)<br>            total += len(items)<br>            logging.info(&#39;Found %d products on this page (running total: %d)&#39;, len(items), total)<br>            url = next_url<br>    logging.info(&#39;Done. Wrote %d products to %s&#39;, total, out_csv)</pre><pre>if __name__ == &#39;__main__&#39;:<br>    logging.basicConfig(level=logging.INFO, format=&#39;%(asctime)s %(levelname)s %(message)s&#39;)<br>    scrape_all(START_URL, OUTPUT_CSV)</pre><p>That’s a working scraper. It’ll run right out of the box on a practice site like this one.</p><h4>Step 3: Add proxies once you hit real anti-bot defenses</h4><p>The moment you point a script like this at a live commercial site, you’ll start running into IP bans, CAPTCHAs, geo-restrictions, or throttling tied to request frequency, and many live pages also load content through JavaScript and Ajax, which changes how scraping software needs to fetch the page. Rotating proxies are the standard fix. They mask your real IP and let you send requests from different locations, which helps you avoid detection and reach region-locked content. You can manage a proxy list yourself or lean on a managed provider that handles rotation, authentication, and uptime.</p><p>Prompt example:</p><pre>Modify this script with the following improvements:<br>- Use proxy servers (such as residential proxies)<br>- Add common HTTP headers to mimic a browser</pre><p>ChatGPT will typically add something like this:</p><pre>PROXY_URL = &#39;https://USERNAME:PASSWORD@proxy.provider.com:1234&#39;<br>PROXIES = {&#39;http&#39;: PROXY_URL, &#39;https&#39;: PROXY_URL}</pre><pre>HEADERS = {<br>    &#39;Accept&#39;: &#39;text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9&#39;,<br>    &#39;Accept-Language&#39;: &#39;en-US,en;q=0.9&#39;,<br>    &#39;Connection&#39;: &#39;keep-alive&#39;,<br>    &#39;Referer&#39;: &#39;https://books.toscrape.com/&#39;,<br>    &#39;Upgrade-Insecure-Requests&#39;: &#39;1&#39;,<br>    &#39;User-Agent&#39;: &#39;Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36&#39;<br>}</pre><pre>def make_session():<br>    s = requests.Session()<br>    s.headers.clear()<br>    s.headers.update(HEADERS)<br>    s.proxies.update(PROXIES)<br>    return s</pre><p>Selenium is one option when you need to control a real browser for dynamic pages.</p><p>ChatGPT won’t hand you proxy IPs itself, it just writes the fields you need to fill in. For anything past a hobby project, a premium proxy service that offers automated rotation, geo-targeting, and CAPTCHA handling will save you a lot of debugging time.</p><p>You can also tighten the prompt further:</p><pre>Additional code requirements:<br>- Handle exceptions where needed<br>- Break logic into small and reusable functions or classes<br>- Retry on errors (429, 5xx, timeouts) with exponential backoff<br>- Stop on non-retryable errors<br>- Pause after too many failures</pre><h4>Step 4: Scale it up</h4><p>From here, scaling usually means moving from one-off scripts to automated scraping across multiple sites, often through vertical aggregation platforms that can monitor collection in real time. Maintaining that stack yourself gets expensive fast, and it eats engineering time that’s usually better spent elsewhere. A managed <a href="https://oxylabs.io/products/scraper-api/serp/chatgpt">scraping API</a> can take that whole layer off your plate, and some platforms also use machine learning to improve extraction quality on messy layouts. Here’s an example using Oxylabs’ Web Scraper API to pull Amazon search results at scale:</p><pre>import json<br>import requests</pre><pre>payload = {<br>    &#39;source&#39;: &#39;amazon_search&#39;,<br>    &#39;domain&#39;: &#39;com&#39;,<br>    &#39;query&#39;: &#39;smart watches&#39;,<br>    &#39;start_page&#39;: 1,<br>    &#39;pages&#39;: 3,<br>    &#39;parse&#39;: True,<br>    &#39;sort_by&#39;: &#39;featured&#39;,<br>    &#39;geo_location&#39;: &#39;90210&#39;<br>}</pre><pre>response = requests.request(<br>    &#39;POST&#39;,<br>    &#39;https://realtime.oxylabs.io/v1/queries&#39;,<br>    auth=(&#39;USERNAME&#39;, &#39;PASSWORD&#39;),<br>    json=payload,<br>)</pre><pre>print(response.json())</pre><pre>with open(&#39;amazon_data.json&#39;, &#39;w&#39;) as f:<br>    json.dump(response.json(), f, indent=2)</pre><h4>ChatGPT web scraping methods and tools</h4><p>Three approaches cover most use cases: a dedicated Web Scraper API, proxy-based DIY scraping, and a custom agent built with a framework like LangChain.</p><h4>Web Scraper API</h4><p><a href="https://oxylabs.io/products/scraper-api/serp/chatgpt">Oxylabs’ Web Scraper API</a> is built to handle the messy parts for you: IP blocks, CAPTCHAs, proxy management, dynamic rendering, and parsing. It takes a handful of lines of code to get running, and it’s fast, with a straightforward interface, and dedicated APIs may also use computer vision techniques to analyze a page visually when DOM-based extraction is unreliable. The documentation includes sample code for most common use cases, which cuts setup time down considerably.</p><h4>Proxy-based scraping</h4><p>This is the DIY route covered above: prompt ChatGPT to write scraping code that routes through proxies to dodge IP blocks, reach geo-restricted pages, and target specific regions. The generated code usually leaves clearly commented fields for your credentials and proxy URL, which makes it easy to plug in a provider like Oxylabs.</p><h4>Custom GPT agent + LangChain</h4><p>For more advanced setups, you can build a custom agent using LangChain’s framework, then connect it to tools like Oxylabs Web Unblocker (which pairs a headless browser with anti-blocking logic) and to whichever LLM backs the agent, whether that’s GPT-5.x, Anthropic’s Claude, or Google’s Gemini. This is the most expensive route by a wide margin. It takes real engineering time to build and maintain, so it only tends to make sense for teams running serious, ongoing scraping operations.</p><h4>What’s new: ChatGPT Ads and ChatGPT Shopping</h4><p>Two things changed the ChatGPT landscape enough in 2026 that anyone doing SEO, GEO, or competitive research work needs to account for them.</p><h4>ChatGPT Ads</h4><p>OpenAI began testing sponsored placements in ChatGPT’s Free and Go tiers in February 2026, and opened self-serve access through Ads Manager in May. Ads show up as a single “sponsored” card below the model’s organic answer, clearly labeled and visually separated, and OpenAI has stuck to its “answer independence” line: advertisers can’t buy their way into what ChatGPT actually says. Targeting works off the conversation’s context rather than a keyword list, and buying happens on CPM for reach or CPC for clicks, both through a second-price auction. Plus, Pro, Business, and Enterprise users don’t see any of this. It’s a Free and Go tier experience only.</p><p>For anyone tracking brand visibility across AI platforms, this matters for two reasons. First, if you’re doing competitive or SEO research through a ChatGPT scraper, sponsored cards are a new data point worth capturing alongside organic answers. Second, if robots.txt rules or bot management on your site are blocking OAI-AdsBot along with GPTBot, you could be shutting yourself out of an ad channel without meaning to. Worth checking your rules if this is relevant to your work.</p><h4>ChatGPT Shopping</h4><p>This one’s had a bumpier ride. OpenAI launched Instant Checkout in late 2025, letting US users buy directly from Etsy sellers (and eventually Shopify merchants) without leaving the chat, built on the Agentic Commerce Protocol it developed with Stripe. By March 2026, OpenAI had pulled it back. Merchant onboarding was slow, product data was inconsistent, and the flexibility just wasn’t there yet.</p><p>What replaced it is a discovery-first model. ChatGPT now focuses on surfacing product recommendations, organic and unsponsored, ranked by relevance, and routes shoppers out to merchant apps or storefronts to actually complete the purchase. ACP itself didn’t go away. It’s become the underlying infrastructure that a growing number of retailers plug into, alongside Google’s competing Universal Commerce Protocol announced around the same time.</p><p>For scraping work, this shift is worth knowing about mainly because it changes what “ChatGPT shopping data” actually looks like right now: product feeds and recommendation rankings rather than completed transactions. If your scraper or GEO tracking setup was built around the checkout flow, it’s worth revisiting.</p><h4>ChatGPT scraper example use cases</h4><p>A ChatGPT scraper, paired with something like the Web Scraper API for the heavy lifting, is often used alongside ChatGPT as an instant personal assistant available 24/7 to create usable outputs from scraped data, and it tends to show up in a handful of recurring jobs:</p><ol><li>Live price monitoring: pulling pricing, stock levels, reviews, and product descriptions from e-commerce sites on a schedule.</li><li>News sentiment tracking: scraping coverage of your brand or industry and having ChatGPT tag it positive, negative, or neutral; it can also process and generate text, audio, and image content, which broadens post-scrape analysis workflows.</li><li>SEO and GEO research: keyword research, content gap analysis, and tracking how often and how favorably your brand shows up in ChatGPT answers, including any sponsored placements now in the mix, while also generating ideas for reports, experiments, or content angles.</li><li>Social listening: pulling data from social platforms to track sentiment, trends, and competitor activity, with voice assistant support for hands-free interaction.</li><li>RAG pipelines: scraping and embedding data into a vector database so an LLM has fresh, accurate context to draw from, and ChatGPT can also interpret images when visual context from a page matters.</li><li>Shopping and product discovery tracking: monitoring how your products get surfaced (or don’t) in ChatGPT’s discovery-first shopping results, now that ranking is organic rather than checkout-driven.</li></ol><h4>Where Gemini, Perplexity, and Google AI Mode fit in</h4><p>Worth a quick mention if you’re weighing platforms rather than locked into ChatGPT specifically. Gemini leans on its tie-in with Google’s own index and infrastructure, which shows up in how it handles grounded, up-to-date queries. Perplexity is built around citation-heavy, search-native answers, which makes it a different kind of target if you’re tracking brand mentions across AI platforms rather than building scrapers with it. Google’s AI Mode inside Search is its own animal again, folding AI-generated summaries directly into the search results page rather than a standalone chat product. None of these change the core mechanics of this guide much. Code generation, proxy handling, and structured extraction work about the same regardless of which model wrote the script. But if GEO tracking is part of your workflow, all three are worth including alongside ChatGPT rather than treating it as the only surface that matters.</p><h4>Best practices for ChatGPT web scraping</h4><p>Use structured outputs. Have ChatGPT format results as JSON or CSV from the start. It makes downstream analysis and trend-spotting much easier.</p><p>Respect robots.txt. If a site’s instructions say not to crawl certain pages, follow them.</p><p>Respect rate limits. Sites and APIs both cap how many requests you can send in a window. Proxies help you stay under those limits during larger jobs, but they’re not a substitute for pacing your requests sensibly.</p><p>Stay on the right side of legal and ethical guidelines. Know the data laws relevant to your use case, review site terms, and keep your collection process transparent. In the U.S., scraping disputes can involve copyright claims, the Computer Fraud and Abuse Act, and even programs meant to enforce compliance around access and data use. For example, a 2012 case found that scraping Craigslist could violate the CFAA. Rules also vary around the world: publicly available data can still be restricted, Australia’s Spam Act 2003 outlaws some web harvesting practices, and Indian courts enforce contract law against unauthorized scraping.</p><p>Pair the ChatGPT scraper with dedicated tools. Whether that’s the Oxylabs Web Scraper API or a prompt that adds proxies and a headless browser, don’t rely on ChatGPT alone for the heavy lifting.</p><p>Clean your data before feeding it back into ChatGPT. Not everything you scrape is usable as-is. Strip out irrelevant or malformed entries before asking the model to summarize or analyze.</p><h4>Conclusion</h4><p>ChatGPT still can’t scrape a website by itself, and that hasn’t changed heading into 2026. What has changed is the ecosystem around it: sponsored placements now sit inside free-tier answers, and shopping has moved from an in-chat checkout experiment to a discovery-and-recommendation layer. If you’re building a ChatGPT scraper, the fundamentals are the same as they’ve always been. Prompt it to generate code, pair that code with proxies or a dedicated scraping API, and feed the cleaned results back in for analysis. Oxylabs’ Web Scraper API and proxy infrastructure can handle the parts that get expensive and fragile to maintain on your own, so you can spend your time using the data instead of babysitting the pipeline.</p><h4>FAQs</h4><p>Can ChatGPT scrape websites? No. It can’t directly pull data from a live site. It can generate the code that does, usually paired with a tool like the Oxylabs Web Scraper API, proxies, or the Web Unblocker.</p><p>How do I use ChatGPT to scrape a website? Prompt it to write code that fetches and parses the target page. Be specific about the language, the elements you want extracted, the libraries to use, and whether you want the output parsed into a structured format.</p><p>Is ChatGPT scraping legal? There’s no law against using ChatGPT to help build a scraper. It’s generally legal as long as you follow a site’s terms of service, its robots.txt instructions, and any applicable rate limits.</p><p>What tools pair well with ChatGPT for web scraping? Oxylabs Web Scraper API, headless browsers, proxy servers with rotation, and LangChain for agent-based setups. The Web Scraper API in particular handles CAPTCHA solving, fingerprinting, and IP blocks so you don’t have to build that layer yourself.</p><p>Does ChatGPT Shopping affect how I should scrape product data? A bit. Since OpenAI moved away from in-chat checkout toward a discovery-first model, what matters now is how your products get ranked in ChatGPT’s organic recommendations, not transaction data. Worth tracking if product visibility in AI answers matters to your business.</p><p>Can ChatGPT act as a scraper without any coding? No. It’s built to understand language and generate text or code, not to independently reach out and extract data from a page. It was trained on billions of words from books, websites, articles, and code to build broad knowledge, learning patterns and statistical relationships between words and phrases during training. That is why it generates coherent sentences through iterative word prediction rather than directly accessing a site. Coding is still part of the process, even if ChatGPT is the one writing most of it.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6ca20161a060" width="1" height="1" alt=""><hr><p><a href="https://medium.com/behind-the-firewall/chatgpt-web-scraping-the-complete-2026-guide-python-6ca20161a060">ChatGPT Web Scraping: The Complete 2026 Guide (Python)</a> was originally published in <a href="https://medium.com/behind-the-firewall">Behind the Firewall</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>