<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Zenrows - Medium]]></title>
        <description><![CDATA[Web data infrastructure for AI agents, data, and automation teams. Fetch, extract, batch, or drive a browser, all through one platform. - Medium]]></description>
        <link>https://medium.com/zenrows?source=rss----e3809efdcbcf---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Zenrows - Medium</title>
            <link>https://medium.com/zenrows?source=rss----e3809efdcbcf---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 14:01:13 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/zenrows" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Build an AI Lead Generation Agent With OpenAI Agents SDK and Zenrows]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/zenrows/build-an-ai-lead-generation-agent-with-openai-agents-sdk-and-zenrows-e206aa5e289c?source=rss----e3809efdcbcf---4"><img src="https://cdn-images-1.medium.com/max/1500/1*pFuSZzCZz19RnsoRr31yig.png" width="1500"></a></p><p class="medium-feed-snippet">How to build an AI lead generation agent that reads live, bot-protected directories with the OpenAI Agents SDK and Zenrows.</p><p class="medium-feed-link"><a href="https://medium.com/zenrows/build-an-ai-lead-generation-agent-with-openai-agents-sdk-and-zenrows-e206aa5e289c?source=rss----e3809efdcbcf---4">Continue reading on Zenrows »</a></p></div>]]></description>
            <link>https://medium.com/zenrows/build-an-ai-lead-generation-agent-with-openai-agents-sdk-and-zenrows-e206aa5e289c?source=rss----e3809efdcbcf---4</link>
            <guid isPermaLink="false">https://medium.com/p/e206aa5e289c</guid>
            <category><![CDATA[openai]]></category>
            <category><![CDATA[web-scraping]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[lead-generation]]></category>
            <dc:creator><![CDATA[Benny Ifeanyi Iheagwara]]></dc:creator>
            <pubDate>Wed, 14 Oct 2026 21:47:57 GMT</pubDate>
            <atom:updated>2026-10-14T21:47:55.998Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Build a Web-Aware TypeScript Agent With Mastra and Zenrows]]></title>
            <link>https://medium.com/zenrows/build-a-web-aware-typescript-agent-with-mastra-and-zenrows-7654767e2e13?source=rss----e3809efdcbcf---4</link>
            <guid isPermaLink="false">https://medium.com/p/7654767e2e13</guid>
            <category><![CDATA[web-scraping]]></category>
            <category><![CDATA[mastra-ai]]></category>
            <category><![CDATA[javascript]]></category>
            <category><![CDATA[typescript]]></category>
            <category><![CDATA[ai-agent]]></category>
            <dc:creator><![CDATA[Benny Ifeanyi Iheagwara]]></dc:creator>
            <pubDate>Wed, 14 Oct 2026 20:52:08 GMT</pubDate>
            <atom:updated>2026-10-14T20:52:07.405Z</atom:updated>
            <content:encoded><![CDATA[<blockquote><em>This article was originally published on the </em><a href="https://www.zenrows.com/blog/web-aware-typescript-agent-mastra-zenrows"><em>Zenrows blog</em></a><em>.</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kLidC2z1Aa7EzjMyddNqAw.png" /></figure><p>Ask a Mastra agent for the price of a product on a protected retail page, and it will usually give you an answer. The trouble is that the answer may come from a Cloudflare challenge page. The agent never saw the product, and nothing in its response tells you so.</p><p>Mastra itself is easy to work with. Its built-in webFetchTool is the weak point, because it sends plain HTTP requests, doesn’t run JavaScript, and returns raw HTML. That’s fine for simple pages and unreliable everywhere else.</p><p>This guide replaces it with a custom <a href="https://docs.zenrows.com/fetch/api-reference">Zenrows Fetch</a> tool. By the end, your agent will read protected pages as clean Markdown or structured JSON, and a workflow will fetch whole lists of URLs in parallel.</p><h3>What you’ll learn</h3><ul><li>You’ll learn why Mastra’s built-in fetch tool fails silently on protected pages.</li><li>You’ll build a typed Zenrows Fetch tool with Mastra’s createTool.</li><li>You’ll extend the tool to return structured JSON fields like price and rating.</li><li>You’ll wire the tool into a market research agent and test it in Mastra Studio.</li><li>You’ll add a workflow that fetches a list of URLs with controlled concurrency.</li></ul><h3>Why TypeScript AI agents need a smarter web fetch tool</h3><p>When webFetchTool hits a page protected by <a href="https://www.zenrows.com/blog/bypass-cloudflare">Cloudflare</a>, Cloudflare returns a challenge page with a normal HTTP status code. Nothing errors. The agent receives the challenge, treats it as the result, and reasons over it.</p><p>For a market research agent, that means comparing prices that were never on the page.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*51MDTTmrKs0I2CKRudViMA.png" /><figcaption>what the agent actually reads</figcaption></figure><p>The problem lives in the transport layer, which controls how bytes reach the agent through requests, headers, proxies, and JavaScript execution. Zenrows Fetch with mode=auto handles <a href="https://www.zenrows.com/blog/selenium-datadome">DataDome</a>, <a href="https://www.zenrows.com/blog/bypass-akamai">Akamai</a>, and Cloudflare in one call. Wrapping it in Mastra’s createTool keeps the typed schemas Mastra relies on.</p><h3>Prerequisites</h3><p>You’ll need Node.js 22.13 or later and an API key for one model provider. This guide uses OpenAI by default and shows the Anthropic alternative, so get a key from your <a href="https://platform.openai.com/api-keys">OpenAI dashboard</a> or your <a href="https://platform.claude.com/docs/en/get-api-key">Anthropic dashboard</a>.</p><p>You’ll also need a Zenrows API key from your <a href="https://app.zenrows.com/register">Zenrows dashboard</a>. The finished code is on <a href="https://github.com/ZenRows/web-aware-agent-mastra-zenrows">GitHub</a> if you’d like to compare as you go.</p><h3>Setting up the Mastra project</h3><p>Mastra’s CLI scaffolds the project structure, config files, and a starter agent. Run it and follow the prompts to name the project and pick your LLM provider.</p><pre>npm create mastra@latest</pre><p>The CLI installs @mastra/core and its dependencies. Add Zod if it isn’t already there.</p><pre>npm install zod</pre><p>Next, create a .env file in your project root with your keys. You only need the key for the provider you plan to use, and .env belongs in your .gitignore.</p><pre>ZENROWS_API_KEY=your_zenrows_key<br>OPENAI_API_KEY=your_openai_key<br>ANTHROPIC_API_KEY=your_anthropic_key</pre><p>Mastra loads .env on its own. The standalone test scripts in this guide run outside Mastra’s loader, so they need dotenv.</p><pre>npm install dotenv — save</pre><p>Your project should now look something like this.</p><pre>src/<br>└── mastra/<br>    ├── agents/<br>    │   └── agent.ts<br>    ├── tools/<br>    │   └── schedule-tools.ts<br>    └── index.ts</pre><p>Start the dev server to check that everything works.</p><pre>npm run dev</pre><p>Mastra Studio opens at <a href="http://localhost:4111.">http://localhost:4111.</a> It lists your registered agents and gives you a chat window for each one.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AFuJ9-ZOhWB_tVoybjr11A.png" /><figcaption>Mastra Studio</figcaption></figure><h3>Building the Zenrows Fetch tool</h3><p>A Mastra tool comes from a single createTool definition with five parts. The id names it. The description tells the model when to call it. The inputSchema and outputSchema validate what goes in and out, and execute does the work.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lKeWvVirrckIq6wyhVUxZg.png" /><figcaption>What the tool definition holds</figcaption></figure><p>The description deserves the most care. It’s the only signal the model uses to decide whether to call the tool, so keep it brief and exact.</p><p>Create src/mastra/tools/zenrows-fetch.ts next to the template’s example tools and add the following.</p><pre>import { createTool } from &#39;@mastra/core/tools&#39;;<br>import { z } from &#39;zod&#39;;<br><br>export const zenrowsFetchTool = createTool({<br>  id: &#39;zenrows-fetch&#39;,<br><br>  description:<br>  &#39;Fetches web pages using Zenrows. Use this tool when you need information from a specific URL. Set extractJson to true when you need structured data such as product details, pricing, availability, ratings, or other fields. Leave extractJson false when you need the page content as Markdown.&#39;,<br><br>  inputSchema: z.object({<br>    url: z.string().url(),<br>    extractJson: z.boolean().optional(),<br>  }),<br><br>  outputSchema: z.object({<br>    content: z.union([z.string(), z.record(z.string(), z.unknown())]),<br>    statusCode: z.number(),<br>    url: z.string(),<br>  }),<br><br>  execute: async ({ url, extractJson }) =&gt; {<br>    const zenrowsUrl = new URL(&#39;https://api.zenrows.com/v1/&#39;);<br><br>    zenrowsUrl.searchParams.set(&#39;url&#39;, url);<br>    zenrowsUrl.searchParams.set(<br>      &#39;apikey&#39;,<br>      process.env.ZENROWS_API_KEY!,<br>    );<br>    zenrowsUrl.searchParams.set(&#39;mode&#39;, &#39;auto&#39;);<br><br>    if (extractJson) {<br>      zenrowsUrl.searchParams.set(&#39;extract&#39;, &#39;auto&#39;);<br>    } else {<br>      zenrowsUrl.searchParams.set(&#39;response_type&#39;, &#39;markdown&#39;);<br>    }<br><br>    const response = await fetch(zenrowsUrl);<br><br>    if (!response.ok) {<br>      const error = await response.text();<br><br>      throw new Error(<br>        `Zenrows request failed with status ${response.status}: ${error}`,<br>      );<br>    }<br><br>    if (extractJson) {<br>      const body = await response.json();<br><br>      return {<br>        content: body.parsed,<br>        statusCode: response.status,<br>        url,<br>      };<br>    }<br><br>    const content = await response.text();<br><br>    return {<br>      content,<br>      statusCode: response.status,<br>      url,<br>    };<br>  },<br>});</pre><p>The input schema does the validation. z.string().url() rejects a malformed URL before it ever reaches the API. extractJson is optional, and when it’s left undefined, the tool takes the Markdown path.</p><p>You can test the tool without the dev server or the agent. Create src/mastra/tools/test-tool.ts and point it at a Walmart search page.</p><pre>import &#39;dotenv/config&#39;;<br>import { zenrowsFetchTool } from &#39;./zenrows-fetch&#39;;<br><br>if (!zenrowsFetchTool.execute) {<br>  throw new Error(&#39;Zenrows tool does not have an execute function&#39;);<br>}<br><br>const result = await zenrowsFetchTool.execute(<br>  {<br>    url: &#39;https://www.walmart.com/search?q=bike&#39;,<br>    extractJson: false,<br>  },<br>  {} as any<br>);<br><br>if (!result || &#39;statusCode&#39; in result === false) {<br>  throw new Error(&#39;Zenrows tool returned an unexpected result&#39;);<br>}<br><br>console.log(result.statusCode);<br>console.log(<br>  typeof result.content === &#39;string&#39;<br>    ? result.content.slice(0, 300)<br>    : result.content<br>);</pre><p>Run it in your terminal.</p><pre>npx tsx src/mastra/tools/test-tool.ts</pre><p>The output prints the status code, then the opening lines of the page as Markdown.</p><pre>200<br>[](https://www.walmart.com)<br><br>[Skip to Main Content](https://www.walmart.com#maincontent)<br><br>[](https://www.walmart.com/all-departments)[](https://www.walmart.com/)<br><br>[![Walmart](https://i5.walmartimages.com/dfw/63fd9f59-14e2/9d304ce6-96de-4331-b8ec-c5191226d378/v1/spark-icon.svg)](https://www.walmart.com/)</pre><p>If you’d prefer not to maintain a custom tool, the <a href="https://docs.zenrows.com/mcp/overview">Zenrows MCP server</a> is an alternative. This walkthrough of the <a href="https://www.zenrows.com/blog/mcp-server">Zenrows MCP server and its implementation</a> covers the setup.</p><h3>Extending the tool with structured extraction</h3><p>Markdown works when the agent needs to read a page and think about it. Product listings are different, because the agent needs to sort and compare specific fields.</p><p>That’s what extract=auto is for. <a href="https://docs.zenrows.com/extract/setup">Zenrows Extract</a> parses the page on the server and returns named fields, and the tool uses it whenever extractJson is true.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eLsw9U1VPs4wvW2t2ey4mQ.png" /><figcaption>Zenrow extractJson</figcaption></figure><p>The tool you already wrote handles both paths. This part of it picks the request parameters.</p><pre>// part of the previous script<br>zenrowsUrl.searchParams.set(&#39;mode&#39;, &#39;auto&#39;);<br><br>if (extractJson) {<br>  // server side parsing, returns fields instead of page text<br>  zenrowsUrl.searchParams.set(&#39;extract&#39;, &#39;auto&#39;);<br>} else {<br>  zenrowsUrl.searchParams.set(&#39;response_type&#39;, &#39;markdown&#39;);<br>}</pre><p>And this part reads the response in the matching format, as JSON for extraction and as text for Markdown.</p><pre>// parses as JSON<br>if (extractJson) {<br>  const body = await response.json();<br><br>  return {<br>    content: body.parsed,<br>    statusCode: response.status,<br>    url,<br>  };<br>}<br><br>const content = await response.text();<br><br>return {<br>  content,<br>  statusCode: response.status,<br>  url,<br>};</pre><p>To see the JSON path in action, create src/mastra/tools/test-tool-extend-extract.ts with extractJson set to true.</p><pre>import &#39;dotenv/config&#39;;<br>import { zenrowsFetchTool } from &#39;./zenrows-fetch&#39;;<br><br>if (!zenrowsFetchTool.execute) {<br>  throw new Error(&#39;Zenrows tool does not have an execute function&#39;);<br>}<br><br>const result = await zenrowsFetchTool.execute(<br>  {<br>    url: &#39;https://www.walmart.com/search?q=bike&#39;,<br>    extractJson: true,<br>  },<br>  {} as any,<br>);<br><br>if (!result || &#39;statusCode&#39; in result === false) {<br>  throw new Error(&#39;Zenrows tool returned an unexpected result&#39;);<br>}<br><br>console.log(result.statusCode);<br>console.log(<br>  typeof result.content === &#39;string&#39;<br>    ? result.content.slice(0, 300)<br>    : result.content<br>);</pre><p>Run it.</p><pre>npx tsx src/mastra/tools/test-tool-extend-extract.ts</pre><p>The parsed response comes back as clean product records. Here are the first two.</p><pre>{<br>  &quot;products&quot;: [<br>    {<br>      &quot;badge&quot;: null,<br>      &quot;current_price&quot;: 149.99,<br>      &quot;fulfillment_label&quot;: null,<br>      &quot;is_sponsored&quot;: true,<br>      &quot;item_id&quot;: &quot;10479873441&quot;,<br>      &quot;low_stock&quot;: null,<br>      &quot;options_from_price&quot;: 149.99,<br>      &quot;original_price&quot;: 207.99,<br>      &quot;rating&quot;: 3.9,<br>      &quot;review_count&quot;: 130,<br>      &quot;title&quot;: &quot;Ktaxon 20\&quot; Mountain Bike, 7 Speed Bike with Disc Brakes, White&quot;,<br>      &quot;walmart_plus_savings&quot;: null<br>    },<br>    {<br>      &quot;badge&quot;: null,<br>      &quot;current_price&quot;: 227,<br>      &quot;fulfillment_label&quot;: null,<br>      &quot;is_sponsored&quot;: true,<br>      &quot;item_id&quot;: &quot;475992717&quot;,<br>      &quot;low_stock&quot;: null,<br>      &quot;options_from_price&quot;: null,<br>      &quot;original_price&quot;: null,<br>      &quot;rating&quot;: 4.4,<br>      &quot;review_count&quot;: 739,<br>      &quot;title&quot;: &quot;Mongoose Rebel X1 BMX Bike, 20-in. Wheels, Kids Ages 7-14 Years, Gray Child Bicycle&quot;,<br>      &quot;walmart_plus_savings&quot;: null<br>    }<br>  ],<br>  &quot;search_query&quot;: {<br>    &quot;query&quot;: &quot;bike&quot;,<br>    &quot;total_results&quot;: null<br>  }<br>}</pre><h3>Wiring the tool into a Mastra agent</h3><p>With the tool working, it’s time to give it to an agent. Create a file under src/mastra/agents/ called market-research-agent.ts.</p><pre>import { Agent } from &#39;@mastra/core/agent&#39;;<br>import { zenrowsFetchTool } from &#39;../tools/zenrows-fetch&#39;;<br><br>export const marketResearchAgent = new Agent({<br>  id: &#39;market-research-agent&#39;,<br>  name: &#39;Market Research Agent&#39;,<br><br>  description:<br>    &#39;An agent that researches and compares products using live web data.&#39;,<br><br>  instructions: `<br>You are a product market research agent.<br><br>Your job is to research and compare products using the zenrows_fetch tool.<br><br>When the user provides one or more product URLs:<br><br>1. Fetch each URL using zenrows_fetch.<br>2. When product information such as name, price, availability, or rating is needed, use structured extraction.<br>3. Extract the relevant product information from each result.<br>4. Never invent or assume information that was not returned by the tool.<br>5. If information is missing, clearly state that it is unavailable.<br>6. Preserve the currency returned by the source.<br>7. Do not directly compare prices when the products use different currencies unless a reliable currency conversion is available.<br>8. When comparing products, present the results clearly in a table when appropriate.<br>9. After presenting the data, provide useful observations based only on the retrieved information.<br><br>Do not use web search when the user asks you to use zenrows_fetch only.<br>`,<br><br>  model: &#39;openai/gpt-5.6-terra&#39;,<br><br>  tools: {<br>    zenrows_fetch: zenrowsFetchTool,<br>  },<br>});</pre><p>The instructions are where most of the reliability comes from. A vague prompt produces a confident answer with a price from training data and no way to tell which fields came from the page. These instructions forbid invented values and make the agent say when data is missing.</p><p>You can use any supported model by changing one line. Just make sure the matching key is in your .env file.</p><pre>// changing model. Replace it in the previous script<br>// openai<br>model: &#39;openai/gpt-5.6-terra&#39;,<br><br>// anthropic<br>model: &#39;anthropic/claude-sonnet-4-5&#39;,</pre><p>Next, register the agent in src/mastra/index.ts. Registration exposes it to Mastra Studio and the server.</p><pre>import { Mastra } from &#39;@mastra/core/mastra&#39;;<br>import { LibSQLStore } from &#39;@mastra/libsql&#39;;<br>import { DuckDBStore } from &#39;@mastra/duckdb&#39;;<br>import { MastraCompositeStore } from &#39;@mastra/core/storage&#39;;<br>import {<br>  MastraStorageExporter,<br>  MastraPlatformExporter,<br>  Observability,<br>  SensitiveDataFilter,<br>} from &#39;@mastra/observability&#39;;<br>import { agent } from &#39;./agents/agent&#39;;<br>import { marketResearchAgent } from &#39;./agents/market-research-agent&#39;;<br>import { startScheduleTool, stopScheduleTool } from &#39;./tools/schedule-tools&#39;;<br><br>export const mastra = new Mastra({<br>  bundler: {<br>    externals: [&#39;@duckdb/node-bindings&#39;],<br>  },<br>  agents: {<br>    agent,<br>    marketResearchAgent,<br>  },<br>  tools: { startScheduleTool, stopScheduleTool },<br>  storage: new MastraCompositeStore({<br>    id: &#39;composite-storage&#39;,<br>    default: new LibSQLStore({<br>      id: &#39;mastra-storage&#39;,<br>      url: process.env.TURSO_DATABASE_URL || &#39;file:./mastra.db&#39;,<br>      authToken: process.env.TURSO_AUTH_TOKEN || undefined,<br>    }),<br>    domains: {<br>      observability: await new DuckDBStore().getStore(&#39;observability&#39;),<br>    },<br>  }),<br>  observability: new Observability({<br>    configs: {<br>      default: {<br>        serviceName: &#39;mastra&#39;,<br>        exporters: [new MastraStorageExporter(), new MastraPlatformExporter()],<br>        spanOutputProcessors: [new SensitiveDataFilter()],<br>      },<br>    },<br>  }),<br>});</pre><p>Now test the agent with a standalone script at src/mastra/tools/test-agent.ts.</p><pre>import &#39;dotenv/config&#39;;<br>import { mastra } from &#39;..&#39;;<br><br>const agent = mastra.getAgent(&#39;marketResearchAgent&#39;);<br><br>const result = await agent.generate(<br>  &#39;Compare the bikes on https://www.walmart.com/search?q=bike. Give me the five cheapest with their ratings, and flag any that are sponsored.&#39;,<br>);<br><br>console.log(JSON.stringify(result.toolCalls, null, 2));<br>console.log(result.text);</pre><p>Run it.</p><pre>npx tsx src/mastra/tools/test-agent.ts</pre><p>The script prints the tool call first. Notice that the agent chose structured extraction on its own.</p><pre>[<br>  {<br>    &quot;type&quot;: &quot;tool-call&quot;,<br>    &quot;runId&quot;: &quot;018d9545-eddd-4fde-8cd1-1884316f4212&quot;,<br>    &quot;from&quot;: &quot;AGENT&quot;,<br>    &quot;payload&quot;: {<br>      &quot;toolCallId&quot;: &quot;call_3H76SowFUi3SRknDFYXrJu7W&quot;,<br>      &quot;toolName&quot;: &quot;zenrows_fetch&quot;,<br>      &quot;args&quot;: {<br>        &quot;url&quot;: &quot;https://www.walmart.com/search?q=bike&quot;,<br>        &quot;extractJson&quot;: true<br>      },<br>      &quot;providerMetadata&quot;: {<br>        &quot;openai&quot;: {<br>          &quot;itemId&quot;: &quot;fc_06708f1f9910751e006a84956972bc81a084b7ebc5cc25f70d&quot;<br>        }<br>      }<br>    }<br>  }<br>]</pre><p>Then comes the agent’s answer.</p><pre>The five lowest-priced bikes in the retrieved Walmart search results are:<br><br>| Rank | Bike | Price* | Rating | Reviews | Sponsored |<br>|---:|---|---:|---:|---:|---|<br>| 1 | Dynacraft 16 Inch Suspect Boys BMX Bike for Child 5-7 Years | 108 | 4.2/5 | 772 | No sponsorship flag returned |<br>| 2 | 18&quot; Kent Bicycle Abyss Boy&#39;s Freestyle BMX Child Bicycle, Blue | 128 | 4.2/5 | 2,325 | No sponsorship flag returned |<br>| 3 | 20&quot; Kent Tempest BMX Bicycle, Fits Riders 4&#39;2&quot;-5&#39;, Black/Aqua, Child, Unisex | 138 | 4.3/5 | 1,496 | No sponsorship flag returned |<br>| 4 | Ktaxon 24&quot; Women&#39;s 7-Speed Cruiser Bike with Basket &amp; Rack, Green | 189.99 | 4.2/5 | 78 | **Sponsored** |<br>| 5 | Mongoose Rebel X1 BMX Bike, 20-in. Wheels, Kids Ages 7-14, Gray | 227 | 4.4/5 | 740 | **Sponsored** |</pre><p>The agent ranked the five cheapest bikes, all from the live page. Where is_sponsored came back null, it said so and left the guesswork out. This run happened later than the first tool test, so some values differ.</p><h3>Testing the agent in Mastra Studio</h3><p>You can run the same test through the Studio chat window. Start the dev server.</p><pre>npm run dev</pre><p>Open <a href="http://localhost:4111,">http://localhost:4111,</a> click the market research agent, and send it this prompt.</p><pre>Compare the bikes on Walmart. Give me the five cheapest with their ratings, and flag any that are sponsored.</pre><p>Studio shows the tool call the agent made and the ranked table it built from the returned fields.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DMjYp3DZTe-YQz9TrLzsmQ.png" /><figcaption>Mastra Studio: Market Research Agent</figcaption></figure><p>To give the agent memory, follow the steps in the <a href="https://mastra.ai/docs/memory/overview">Mastra memory documentation</a>.</p><h3>Adding a workflow for bulk URL processing</h3><p>An agent plans each call. When you already know which URLs to check, a Mastra workflow is a better fit because its steps are deterministic.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xzqHLPLTGusYIj6RUSIzfw.png" /><figcaption>A workflow for a list of URLs</figcaption></figure><p>Create src/mastra/workflow/bulk-fetch.ts. The first step turns the input list into one task per URL.</p><pre>import { createStep, createWorkflow } from &#39;@mastra/core/workflows&#39;;<br>import { z } from &#39;zod&#39;;<br>import { zenrowsFetchTool } from &#39;../tools/zenrows-fetch&#39;;<br><br>const prepareUrlsStep = createStep({<br>  id: &#39;prepare-urls&#39;,<br><br>  inputSchema: z.object({<br>    urls: z.array(z.string().url()),<br>  }),<br><br>  outputSchema: z.array(<br>    z.object({<br>      url: z.string().url(),<br>    }),<br>  ),<br><br>  execute: async ({ inputData }) =&gt; {<br>    return inputData.urls.map(url =&gt; ({<br>      url,<br>    }));<br>  },<br>});</pre><p>The second step calls the Zenrows tool for each URL with structured extraction on. createTool types execute as optional and as taking two arguments, so the step narrows it first and passes both.</p><pre>const fetchProductStep = createStep({<br>  id: &#39;fetch-product&#39;,<br><br>  inputSchema: z.object({<br>    url: z.string().url(),<br>  }),<br><br>  outputSchema: z.object({<br>    url: z.string(),<br>    content: z.record(z.string(), z.unknown()),<br>    statusCode: z.number(),<br>  }),<br><br>  execute: async ({ inputData }) =&gt; {<br>    // createTool types execute as optional, so narrow it before calling<br>    if (!zenrowsFetchTool.execute) {<br>      throw new Error(&#39;Zenrows tool does not have an execute function&#39;);<br>    }<br><br>    const result = await zenrowsFetchTool.execute(<br>      {<br>        url: inputData.url,<br>        extractJson: true,<br>      },<br>      {} as any,<br>    );<br><br>    if (!result || &#39;statusCode&#39; in result === false) {<br>      throw new Error(&#39;Zenrows tool returned an unexpected result&#39;);<br>    }<br><br>    return {<br>      url: inputData.url,<br>      // the tool returns a union, this workflow always extracts<br>      content: result.content as Record&lt;string, unknown&gt;,<br>      statusCode: result.statusCode,<br>    };<br>  },<br>});</pre><p>The last block chains the steps into a workflow. foreach runs the fetch step on five URLs at a time.</p><pre>export const productResearchWorkflow = createWorkflow({<br>  id: &#39;product-research-workflow&#39;,<br><br>  inputSchema: z.object({<br>    urls: z.array(z.string().url()),<br>  }),<br><br>  outputSchema: z.array(<br>    z.object({<br>      url: z.string(),<br>      content: z.record(z.string(), z.unknown()),<br>      statusCode: z.number(),<br>    }),<br>  ),<br>})<br>  .then(prepareUrlsStep)<br>  // concurrency controls how many urls run at once<br>  .foreach(fetchProductStep, { concurrency: 5 })<br>  .commit();</pre><p>Register the workflow in src/mastra/index.ts alongside the agent.</p><pre>// add to the imports<br>import { productResearchWorkflow } from &#39;./workflow/bulk-fetch&#39;;<br><br>// add to the Mastra constructor, alongside agents<br>workflows: {<br>  productResearchWorkflow,<br>},</pre><p>Then test it with src/mastra/workflow/test-workflow.ts, which sends three Walmart pages through the workflow.</p><pre>import &#39;dotenv/config&#39;;<br>import { mastra } from &#39;..&#39;;<br><br>const workflow = mastra.getWorkflow(&#39;productResearchWorkflow&#39;);<br><br>const run = await workflow.createRun();<br><br>const result = await run.start({<br>  inputData: {<br>    urls: [<br>      &#39;https://www.walmart.com/search?q=tanktop&#39;,<br>      &#39;https://www.walmart.com/search?q=shirts&#39;,<br>      &#39;https://www.walmart.com/browse/home/shop-water-bottles/4044_623679_639999_7751805_4269055_2976621&#39;,<br>    ],<br>  },<br>});<br><br>console.log(JSON.stringify(result, null, 2));</pre><p>Run it.</p><pre>npx tsx src/mastra/workflow/test-workflow.ts</pre><p>Here are two of the three results. Every URL returned status 200, and the results arrived in the same order as the input.</p><pre>{<br>  &quot;result&quot;: [<br>    {<br>      &quot;url&quot;: &quot;https://www.walmart.com/search?q=tanktop&quot;,<br>      &quot;content&quot;: {<br>        &quot;products&quot;: [<br>          {<br>            &quot;badge&quot;: &quot;rollback&quot;,<br>            &quot;current_price&quot;: 11.98,<br>            &quot;original_price&quot;: 29.98,<br>            &quot;fulfillment_label&quot;: &quot;Shipping, arrives today&quot;,<br>            &quot;is_sponsored&quot;: true,<br>            &quot;item_id&quot;: &quot;342153793&quot;,<br>            &quot;low_stock&quot;: true,<br>            &quot;rating&quot;: 4.5,<br>            &quot;review_count&quot;: 15482,<br>            &quot;title&quot;: null<br>          }<br>          // 8 more products<br>        ],<br>        &quot;search_query&quot;: { &quot;query&quot;: &quot;tanktop&quot;, &quot;total_results&quot;: null }<br>      },<br>      &quot;statusCode&quot;: 200<br>    },<br>    {<br>      &quot;url&quot;: &quot;https://www.walmart.com/browse/home/shop-water-bottles/...&quot;,<br>      &quot;content&quot;: {<br>        &quot;products&quot;: [<br>          {<br>            &quot;badge&quot;: null,<br>            &quot;current_price&quot;: 15.95,<br>            &quot;original_price&quot;: 17.98,<br>            &quot;is_sponsored&quot;: null,<br>            &quot;item_id&quot;: &quot;1604723&quot;,<br>            &quot;rating&quot;: 4.7,<br>            &quot;review_count&quot;: 19927,<br>            &quot;title&quot;: &quot;Brita Standard Replacement Water Filter (3 Pack)&quot;<br>          }<br>          // 11 more products<br>        ],<br>        &quot;search_query&quot;: null<br>      },<br>      &quot;statusCode&quot;: 200<br>    }<br>  ]<br>}</pre><p>You can run the workflow from Mastra Studio too.</p><h3>What you built</h3><p>You now have a Mastra agent backed by Zenrows that fetches data from protected sites and returns structured results. It reads pages as Markdown when it needs to reason and as JSON when it needs fields, and a workflow handles known URL lists in parallel.</p><p>To go further, the <a href="https://mastra.ai/docs">Mastra documentation</a> covers advanced patterns. For larger workloads, <a href="https://www.zenrows.com/products/batch">Zenrows Batch</a> handles big URL lists so the agent doesn’t send every request at once.</p><h3>Key takeaways</h3><ul><li>Mastra’s built-in webFetchTool can hand your agent a challenge page with a normal status code, and the agent won’t know.</li><li>A custom createTool definition keeps typed schemas around a Zenrows Fetch call that handles protected access.</li><li>One boolean lets the same tool return Markdown for reasoning or structured JSON for comparison.</li><li>Specific agent instructions stop the model from filling gaps with training data.</li><li>Workflows suit known URL lists because their steps run deterministically with controlled concurrency.</li></ul><h3>FAQ</h3><p><strong>What is Mastra?</strong></p><p>Mastra is an open-source TypeScript framework for building AI agents. It includes built-in support for tools, workflows, memory, tool calling, and observability.</p><p><strong>Does the Zenrows tool work with both OpenAI and Anthropic models?</strong></p><p>It does. Only the agent’s model line changes when you switch, because Mastra passes your Zod inputSchema to the AI SDK in whatever format the provider expects.</p><p><strong>Can I connect Zenrows through MCP?</strong></p><p>You can. Zenrows has an <a href="https://docs.zenrows.com/mcp/overview">official MCP server</a>, and Mastra’s MCPClient works with it like any other MCP server. It gives agents Fetch and browser tools without a wrapper, and you still need an API key.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7654767e2e13" width="1" height="1" alt=""><hr><p><a href="https://medium.com/zenrows/build-a-web-aware-typescript-agent-with-mastra-and-zenrows-7654767e2e13">Build a Web-Aware TypeScript Agent With Mastra and Zenrows</a> was originally published in <a href="https://medium.com/zenrows">Zenrows</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Best web scraping API for AI agents in 2026]]></title>
            <link>https://medium.com/zenrows/best-web-scraping-api-for-ai-agents-in-2026-c4406a2452b2?source=rss----e3809efdcbcf---4</link>
            <guid isPermaLink="false">https://medium.com/p/c4406a2452b2</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[web-scraping]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[ai-agents-in-action]]></category>
            <category><![CDATA[web-development]]></category>
            <dc:creator><![CDATA[Goodness E. Eboh]]></dc:creator>
            <pubDate>Wed, 14 Oct 2026 20:50:31 GMT</pubDate>
            <atom:updated>2026-10-14T20:50:29.839Z</atom:updated>
            <content:encoded><![CDATA[<p><em>This article was originally published on the </em><a href="https://www.zenrows.com/blog/best-web-scraping-api-for-ai-agents"><em>Zenrows blog</em></a><em>.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nrbwQIPTHdync248hc3VPw.png" /></figure><p>Picture an AI agent built to track prices or watch job listings. In testing, it reads documentation and news sites without a hitch. Then it meets its first production target, a job board behind DataDome, and every answer it gives turns wrong.</p><p>The agent’s <a href="https://www.zenrows.com/glossary/web-data-access/what-is-a-web-scraping-api">web scraping API</a> probably looked great on paper. It returned clean Markdown, shipped an MCP server, and rendered JavaScript. Those are the features these APIs usually get compared on. The one feature that decides whether the agent sees the page at all rarely makes the list.</p><p>So we tested that feature directly. We sent 800 requests through four scraping APIs against DataDome, Cloudflare, and Akamai, then compared five APIs on everything else. Two of the four never returned a single usable page from the DataDome target.</p><h3>What you’ll learn</h3><ul><li>You’ll see how four scraping APIs performed against three major anti-bot systems and an unprotected baseline.</li><li>You’ll learn why speed and cost figures mislead when you read them apart from success rate.</li><li>You’ll see where Zenrows, Firecrawl, Jina AI Reader, CrawlForge, and Bright Data each fit best.</li><li>You’ll get working code to connect a scraping API to Claude Desktop, the OpenAI Agents SDK, and LangChain.</li></ul><h3>Why web scraping APIs for AI agents differ</h3><p>Public docs, company blogs, and news sites rarely put up a fight. E-commerce pricing pages, job boards, and industry directories are a different story. Many sit behind <a href="https://www.zenrows.com/glossary/web-data-access/what-is-an-anti-bot-system">anti-bot systems</a>.</p><p>A protected site might return a challenge page, an empty response, or partial content. To the agent, each one looks like a page. It reads whatever comes back, finds nothing useful, and carries on as if the source were empty.</p><p>That’s why this comparison starts with <a href="https://www.zenrows.com/glossary/web-data-access/what-is-protected-web-access">protected web access</a>. Output format and tooling matter only after the agent has the page.</p><h3>Benchmark methodology</h3><p>We tested Zenrows (using Fetch with mode=auto), Jina AI Reader, and CrawlForge on August 20, 2026. Firecrawl followed on August 25, 2026, in proxy=auto mode, which starts in standard mode and retries in stealth mode.</p><p>Each provider got 50 requests per target at two requests per second, for 800 requests in total. The targets were JobLookup behind DataDome, Chrono24 behind <a href="https://www.zenrows.com/blog/bypass-cloudflare">Cloudflare</a>, Reuters behind Akamai, and Wikipedia with no protection at all.</p><p>A request only counted as a success if it returned an HTTP 200 status code and a valid page title or meaningful data. A 200 with a challenge page inside counted as a failure.</p><p>Bright Data sat this one out because it requires enterprise KYC. Every figure below was recomputed on October 1, 2026, from the raw per-request results in the public <a href="https://github.com/ZenRows/ai-agent-web-scraping-benchmark">benchmark repository</a>, so you can check any of them yourself.</p><h3>Benchmark results</h3><p>On Wikipedia, every API scored 100 percent. The easy page proved nothing, and the protected targets told the story.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6OgJ-cd2wq3IWnWo82Y0Iw.png" /><figcaption>Protected-target success rate for Zenrows, Jina, Firecrawl and CrawlForge</figcaption></figure><p>Zenrows returned a usable page on all 200 of its requests, for a 100 percent average across the three protected targets. It was the only API to clear every request in the benchmark.</p><p>Jina came a close second at 99.33 percent. It failed exactly once, on JobLookup, and cleared every request to Chrono24 and Reuters.</p><p>Firecrawl averaged 66.67 percent. It was perfect on Chrono24 and Reuters, but all 50 JobLookup requests returned a reCAPTCHA page, even with stealth retries.</p><p>CrawlForge averaged zero percent on the protected targets. It got 403s from JobLookup and Chrono24 and a <a href="https://www.zenrows.com/glossary/http-errors/what-is-a-401-error">401</a> from Reuters.</p><h3>Response time and consumption</h3><p>Across all four targets, CrawlForge averaged 867 ms per request, Jina 1,257 ms, Firecrawl 1,497 ms, and Zenrows 5,215 ms. Read alone, those numbers would put CrawlForge first.</p><p>Read next to the success rates, they paint a different picture. CrawlForge was fast because it was turned away fast. Jina was the quickest API that returned content from protected targets. Zenrows took the longest on the protected pages, and those are the pages where it was the only API to clear every request.</p><p>Cost works the same way. Each provider bills in its own unit, so the figures don’t compare directly. Zenrows averaged 4.59 credits per request, Jina 24,922.58 tokens, and Firecrawl and CrawlForge one credit each. A one-credit request buys you little when the page it returns is a challenge.</p><h3>How Zenrows credits scale with protection</h3><p>The Zenrows average hides a pattern worth seeing. mode=auto escalates only as far as each target forces it to, so different pages cost very different amounts.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*80b-Kw8kqhGjhc580IkGRw.png" /><figcaption>Zenrows credits per successful request by target protection</figcaption></figure><p>The unprotected Wikipedia page cost one credit. Reuters, behind Akamai, cost ten. Chrono24, behind Cloudflare, averaged 2.36 because not every request needed the full escalation.</p><p>For an agent whose targets mix easy and hard pages, that matters. You pay in proportion to the protection each page puts up.</p><h3>Best web scraping APIs for AI agents compared</h3><h3>Zenrows</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8ivY82O2gIweO6PrcnrU0Q.png" /><figcaption><em>Zenrows homepage</em></figcaption></figure><p>Zenrows is the strongest fit for agents that need protected and dynamic sources. Beyond the benchmark, Zenrows publishes a 99.93 percent success rate across its wider target set. <a href="https://docs.zenrows.com/fetch/api-reference">Zenrows Fetch</a> handles Cloudflare, DataDome, Akamai, and PerimeterX in one call, along with <a href="https://www.zenrows.com/glossary/web-data-access/what-is-javascript-rendering">dynamic website support</a>.</p><p>It returns <a href="https://www.zenrows.com/glossary/web-data-extraction/what-is-llm-ready-content">LLM-ready</a> Markdown, plain text, or PDF. Zenrows Extract returns <a href="https://www.zenrows.com/glossary/web-data-extraction/what-is-schema-based-extraction">structured JSON</a> without pre-written CSS selectors.</p><p>Its <a href="https://www.zenrows.com/agents">Agent Toolkit</a> ships on every plan. The <a href="https://www.zenrows.com/blog/zenrows-mcp-server">MCP server</a> exposes scrape, extract, and more than 30 browser_* tools, with setup guides for hosts like Claude Desktop, Cursor, and VS Code and frameworks like the OpenAI Agents SDK and Google ADK. A <a href="https://www.zenrows.com/cli">CLI</a> serves terminal-native agents.</p><p>The most unusual piece is <a href="https://agents.zenrows.com/">agentic payments</a>. An agent can read a machine-readable price catalogue, settle through x402, MPP, or a card, and finish a purchase when it hits a 402 challenge. For an autonomous agent, that’s the gap between a run that finishes and one that stalls overnight waiting for a human.</p><p><a href="https://docs.zenrows.com/batch/setup">Zenrows Batch</a> handles async jobs of up to 100,000 URLs. The free tier gives you 5,000 credits a month with no card, and paid plans start at $16/month. Billing is <a href="https://www.zenrows.com/glossary/web-data-access/what-does-pay-per-successful-request-mean">pay per successful request</a>, so blocked requests cost nothing.</p><h3>Firecrawl</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RlwZ15B5DfCwdKMau7oeQw.png" /><figcaption><em>Firecrawl homepage</em></figcaption></figure><p><a href="https://www.zenrows.com/blog/best-firecrawl-alternative-for-anti-bot-bypass">Firecrawl</a> suits agents whose sources are unprotected or lightly protected. Its output is built for LLMs, with Markdown, JSON Schema generation, and metadata extraction designed for <a href="https://www.zenrows.com/glossary/ai-agents-and-the-web/what-is-web-scraping-for-rag">RAG</a> and chunking.</p><p>It also offers /agent and /interact endpoints for multi-page exploration and browser control. Its AI Extract feature needs a separate subscription. The free plan includes 1,000 credits a month, and paid plans start at $19/month.</p><h3>Jina AI Reader</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*R55rid7qPWVTa5QSo8B0nQ.png" /><figcaption><em>Jina AI Reader homepage</em></figcaption></figure><p>Jina converts any URL into <a href="https://www.zenrows.com/glossary/web-data-extraction/what-is-html-to-markdown-conversion">LLM-ready Markdown</a> when you prepend r.jina.ai/ to it. It needs no setup and includes image captioning for multimodal agents.</p><p>Its token usage tracks the amount of content returned, making large crawls harder to budget for. Jina offers 10 million free tokens, and paid usage starts at $50. It’s a strong match for quick lookups of documentation, news, and public content.</p><h3>CrawlForge</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NrnA3z-wxpuT0BBRkYzIMQ.png" /><figcaption><em>CrawlForge homepage</em></figcaption></figure><p>CrawlForge is MCP-native, with 31 specialized tools an agent can discover without wrapper code. It produces token-efficient Markdown, can run extraction schemas locally through Ollama, and keeps pricing predictable per tool.</p><p>Its free tier includes 1,000 credits, and paid plans start at $19/month. It works best for agents on MCP-native hosts that mostly read unprotected sources.</p><h3>Bright Data</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3Hv8MULr2OnkhAGSz3_VJA.png" /><figcaption><em>Bright Data homepage</em></figcaption></figure><p><a href="https://www.zenrows.com/blog/best-bright-data-alternative-for-self-service-scraping">Bright Data</a> is the enterprise option, with more than 150 million IPs and SOC 2 Type II and ISO 27001 certifications. Its Web Unlocker, Browser API, and Residential Proxies products each have $499/month Scale plans.</p><p>That pricing puts it beyond a typical individual developer or small team. It fits enterprise teams with five-figure data budgets, compliance requirements, and dedicated data engineers.</p><h3>Integrating Zenrows into your agent</h3><p>If protected access matters for your targets, connecting Zenrows takes a few lines. All three patterns below need a Zenrows API key from the <a href="https://app.zenrows.com/register">signup page</a>.</p><h3>Claude Desktop</h3><p>The quickest route is an MCP host. Add this block to claude_desktop_config.json, then restart <a href="https://docs.zenrows.com/integrations/claude-desktop">Claude Desktop</a>.</p><pre>{<br>  &quot;mcpServers&quot;: {<br>    &quot;zenrows&quot;: {<br>      &quot;command&quot;: &quot;npx&quot;,<br>      &quot;args&quot;: [&quot;-y&quot;, &quot;@zenrows/mcp&quot;],<br>      &quot;env&quot;: {<br>        &quot;ZENROWS_API_KEY&quot;: &quot;YOUR_ZENROWS_API_KEY&quot;<br>      }<br>    }<br>  }<br>}</pre><p>Claude will now call the scrape and browser_* tools whenever a task needs live web data. If you’d like to compare this server with the alternatives, we tested seven <a href="https://www.zenrows.com/blog/best-scraping-mcp-servers">MCP servers for web scraping</a>.</p><h3>OpenAI Agents SDK</h3><p>The <a href="https://docs.zenrows.com/integrations/openai-agents-sdk">OpenAI Agents SDK</a> supports MCP natively through HostedMCPTool. That means you can point an agent at the hosted Zenrows server with no custom function-calling code.</p><pre>import asyncio<br>import os<br>from agents import Agent, HostedMCPTool, Runner</pre><pre>ZENROWS_API_KEY = os.environ[&quot;ZENROWS_API_KEY&quot;]</pre><pre>agent = Agent(<br>    name=&quot;Web Research Assistant&quot;,<br>    instructions=&quot;Use live web data when you need current information.&quot;,<br>    tools=[<br>        HostedMCPTool(<br>            tool_config={<br>                &quot;type&quot;: &quot;mcp&quot;,<br>                &quot;server_label&quot;: &quot;zenrows&quot;,<br>                &quot;server_description&quot;: &quot;Web data access through ZenRows.&quot;,<br>                &quot;server_url&quot;: &quot;https://mcp.zenrows.com/mcp&quot;,<br>                &quot;authorization&quot;: ZENROWS_API_KEY,<br>                &quot;require_approval&quot;: &quot;never&quot;,<br>            }<br>        )<br>    ],<br>)</pre><pre>async def main():<br>    result = await Runner.run(<br>        agent,<br>        &quot;Visit <a href="https://news.ycombinator.com/">https://news.ycombinator.com/</a> and summarize the three most recent posts.&quot;,<br>    )<br>    print(result.final_output)</pre><pre>if __name__ == &quot;__main__&quot;:<br>    asyncio.run(main())</pre><p>If you’d prefer to register Fetch as a plain function tool, we walk through that in <a href="https://www.zenrows.com/blog/web-aware-agent-openai-agents-sdk-zenrows">building a web-aware agent with the OpenAI Agents SDK</a>.</p><h3>LangChain</h3><p>For <a href="https://www.zenrows.com/blog/langchain-web-scraping">LangChain</a>, the langchain-zenrows package turns Zenrows into a tool the agent calls whenever it needs live data.</p><pre>from langchain_zenrows import ZenRowsUniversalScraper<br>from langchain_openai import ChatOpenAI<br>from langgraph.prebuilt import create_react_agent<br><br>llm = ChatOpenAI(model=&quot;gpt-4o-mini&quot;)<br><br>zenrows_tool = ZenRowsUniversalScraper()<br><br>agent = create_react_agent(llm, [zenrows_tool])<br><br>result = agent.invoke({<br>    &quot;messages&quot;: [<br>        {<br>            &quot;role&quot;: &quot;user&quot;,<br>            &quot;content&quot;: &quot;Scrape https://www.etsy.com/ in Markdown and return the four cheapest products in JSON.&quot;<br>        }<br>    ]<br>})</pre><p>Set ZENROWS_API_KEY as an environment variable before you run it.</p><h3>What the benchmark settles</h3><p>Output format and tool count are the easy part of this category, and every API here does them well. Protected access decides whether your agent returns an answer or a challenge page, and that’s where the tools disagree most.</p><p>Across 800 requests, Zenrows was the only API that returned a usable page every time, and it charged one credit for the page that needed nothing. Jina came closest and runs faster when the target is open. Firecrawl and CrawlForge are reasonable picks for agents whose targets stay public.</p><p>So choose based on the targets your agent will hit. If any of them sit behind an anti-bot system, <a href="https://app.zenrows.com/register">start with 5,000 free credits</a> and run your own list through Fetch before you commit. The full per-target tables and the side-by-side comparison are in the <a href="https://www.zenrows.com/blog/best-web-scraping-api-for-ai-agents">original post</a>.</p><h3>Key takeaways</h3><ul><li>Every API scored 100 percent on an unprotected page, so test on the protected targets your agent will hit.</li><li>Zenrows was the only API to return a usable page on all 200 benchmark requests, and Jina came second at 99.33 percent.</li><li>Response time and cost figures only mean something next to success rate, because a fast failure still fails.</li><li>With mode=auto, Zenrows charged one credit for an unprotected page and ten for an Akamai-protected one.</li><li>You can connect a scraping API to Claude Desktop, the OpenAI Agents SDK, or LangChain in a few lines of code.</li></ul><h3>FAQ</h3><p><strong>Can AI agents scrape websites protected by Cloudflare or DataDome?</strong></p><p>They can, as long as the API they call handles the protection. In our benchmark, two of the four APIs tested returned nothing usable from the DataDome target across 50 requests each. Zenrows returned a valid page on every request to the Cloudflare, DataDome, and Akamai targets.</p><p><strong>Is there a free web scraping API for AI agents?</strong></p><p>Several have free tiers. Zenrows offers 5,000 credits a month with no card, Firecrawl offers 1,000 credits, Jina AI Reader offers 10 million tokens, and CrawlForge offers 1,000 credits. Bright Data also offers 5,000 free credits a month, shared across several products.</p><p><strong>What’s the difference between a web scraping API and a scraping MCP server?</strong></p><p>A web scraping API fetches the page. An MCP server hands that service to an agent as a callable tool. Zenrows offers both, so the same capability works over REST, through the CLI, or as a tool your agent discovers automatically.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c4406a2452b2" width="1" height="1" alt=""><hr><p><a href="https://medium.com/zenrows/best-web-scraping-api-for-ai-agents-in-2026-c4406a2452b2">Best web scraping API for AI agents in 2026</a> was originally published in <a href="https://medium.com/zenrows">Zenrows</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Best Browserbase Alternative in 2026 for Protected Web Access]]></title>
            <link>https://medium.com/zenrows/best-browserbase-alternative-in-2026-for-protected-web-access-4ff068b06294?source=rss----e3809efdcbcf---4</link>
            <guid isPermaLink="false">https://medium.com/p/4ff068b06294</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[zenrows]]></category>
            <category><![CDATA[web-scraping]]></category>
            <category><![CDATA[browser-agent]]></category>
            <dc:creator><![CDATA[Praise James]]></dc:creator>
            <pubDate>Mon, 05 Oct 2026 11:00:11 GMT</pubDate>
            <atom:updated>2026-10-05T11:00:09.967Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9-vGXY799CmOUyZW0cbMHQ.png" /></figure><blockquote>This article was originally published on the Zenrows blog. Read the original here: https://www.zenrows.com/blog/best-browserbase-alternative-for-protected-web-access</blockquote><p>We benchmarked Zenrows against Browserbase on four protected targets and one unprotected page. Zenrows returned a 100 percent success rate. Browserbase Fetch returned 62.25 percent, and on the DataDome-protected TripAdvisor target it returned a 403 on all 100 requests.</p><p>That gap is the whole argument, and it comes down to a design difference. Browserbase gives AI agents a real browser to drive, which is the right tool when an agent has to navigate, click, and hold session state across steps. For plain data extraction, running that browser adds a configuration layer you manage yourself, and the bill tracks browser time rather than successful data.</p><p>This compares both platforms on success rate, cost, and developer experience, and marks where each one actually fits.</p><p><strong>What you’ll learn</strong></p><ul><li>What Browserbase costs in 2026 once failed sessions are counted</li><li>Why success-based billing changes the math on protected extraction</li><li>How the two platforms performed on four protected targets</li><li>When Browserbase is still the right call</li></ul><h3>What Browserbase costs once failures are counted</h3><p>Browserbase bills for browser session time and proxy bandwidth, so a failed session still consumes infrastructure. That makes the cost of high-volume extraction hard to predict, because the final bill depends on session durations and bandwidth you have to estimate before you can calculate it.</p><p>Take a concrete workload. 10,000 page attempts per day on a Cloudflare-protected site with a 40 percent failure rate consumes roughly 5,000 browser-hours and 300 GB of proxy bandwidth per month. On the $20/month Developer plan that comes to an estimated <strong>$4,196</strong> for browser execution and proxy bandwidth alone. On the $99/month Startup plan it’s <strong>$3,499</strong>. Both figures exclude Fetch, Extract, Search, and model tokens.</p><p>Zenrows handles the same workload through success-based billing. Using the 100 percent success rate from our benchmark, all 10,000 daily attempts succeed and consume credits. Some Cloudflare-protected requests in our test used both JavaScript rendering and premium proxies, a 25x credit multiplier. If every request required both, the workload consumes 7.5 million credits per month, which runs <strong>$622</strong> on the Scale plan.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/812/1*zlNoJzj8fhjoWvQKfj3XBA.png" /></figure><p>The difference is where the calculation starts. Browserbase starts from session time and bandwidth. Zenrows starts from the number of successful requests and the configuration each one required.</p><h3>Why Zenrows fits protected extraction</h3><p>Zenrows is the web data infrastructure layer, covering protected access, structured output, and job orchestration. Fetch starts at $16/month on the Build plan and charges only for successful requests. A basic request is one credit, JavaScript rendering adds a 5x multiplier, proxies add 10x, and both together add 25x. Those weights hold on every plan.</p><p>Fetch uses mode=autoto adapt the request configuration to the target’s protection level, activating JavaScript rendering, residential proxies, or both when a page requires it. That removes the browser-lifecycle and per-request recovery logic you’d otherwise build. Fetch also returns Markdown, plain text, or structured JSON through Extract, selected with the response_typeparameter, so LLM and RAG pipelines skip manual HTML parsing.</p><p>For bulk work, Batch processes up to 100,000 URLs per job and handles queuing, concurrency, retries, tracking, storage, and completion notifications. When a page genuinely needs a browser, <a href="https://www.zenrows.com/products/browser-sessions">Zenrows Browser Sessions</a> supports navigation, clicking, and persistent state through Playwright, Puppeteer, and other CDP-compatible tools, so you swap your local browser launch for a Zenrows WebSocket URL and keep your workflow.</p><h3>Migrating from Browserbase Fetch</h3><p>Moving a basic Fetch request from Browserbase to Zenrows changes the endpoint, authentication, HTTP method, and parameters. The target URL and how your application handles the returned content stay the same. Browserbase Fetch uses a POST to <a href="https://api.browserbase.com/v1/fetch">https://api.browserbase.com/v1/fetch</a> with the URL in the JSON body, and returns raw content by default, with Markdown or structured JSON living behind the separate Extract feature. Zenrows uses a GET to <a href="https://api.zenrows.com/v1/">https://api.zenrows.com/v1/</a> with the API key, target URL, and config as query parameters:</p><pre># pip install requests<br>import os<br>import requests<br><br>API_KEY = os.environ.get(&quot;ZENROWS_API_KEY&quot;)<br>TARGET_URL = &quot;https://www.ikea.com/us/en/p/kallax-shelf-unit-white-80275887/&quot;<br><br>params = {<br>    &quot;url&quot;: TARGET_URL,<br>    &quot;apikey&quot;: API_KEY,<br>    &quot;mode&quot;: &quot;auto&quot;,<br>}<br><br>response = requests.get(&quot;https://api.zenrows.com/v1/&quot;, params=params)<br>print(f&quot;status code: {response.status_code}&quot;)</pre><p>Zenrows keeps protected retrieval and structured output inside the Fetch workflow, where Browserbase splits those across Fetch and Extract.</p><h3>How the two performed on protected targets</h3><p>We tested Zenrows Fetch and Browserbase Fetch on September 24, 2026, with 100 requests per target at 2 req/s across four protected targets and one unprotected page, each on its default configuration. The five targets covered different protection systems, an IKEA page (Cloudflare), TripAdvisor listings (DataDome), a Walmart page (PerimeterX), an Amazon page (Akamai), and Python docs (unprotected). The benchmark script and CSV results are on <a href="https://github.com/ZenRows/zenrows-vs-browserbase-benchmark">GitHub</a>.</p><p>Zenrows returned 100 percent across the four protected sites, because mode=auto escalated only when a target needed more than the default path. Browserbase Fetch returned 62.25 percent across the protected targets, 100 percent from IKEA and Walmart, 49 percent from Amazon, and zero from TripAdvisor, where all 100 requests came back 403. That result follows from the default Fetch configuration we tested, which executes no JavaScript and disables proxies by default. Reaching DataDome-protected TripAdvisor needs browser fingerprinting through the Sessions API, which adds a browser-session management layer to a data extraction task.</p><p>Zenrows averaged 5.36s per target against Browserbase’s 1.18s. But response time has to be read alongside retrieval. Browserbase averaged 0.61s on TripAdvisor while returning a 403 on every request, and Zenrows retrieved the target successfully. A faster response is only faster if it returns the content you asked for.</p><h3>When Browserbase is the right call</h3><p>Browserbase fits when an agent needs to operate a browser rather than retrieve page content. Its managed browser infrastructure gives agents direct control over pages, sessions, and state without maintaining a headless fleet. Four cases make it the right tool.</p><ul><li>You’re building an AI agent that navigates inside websites, authenticates, and fills forms, where Stagehand’s act, observe, and extract support natural-language browser actions</li><li>You need session replay and visual debugging through Live View and Session Inspector for workflows that are hard to reproduce</li><li>Your existing automation already runs on Playwright with a custom proxy stack you want to keep</li><li>Your workflow depends on Chrome extensions or persistent browser state across requests</li></ul><h3>Other alternatives worth knowing</h3><p>The right fit depends on whether you need browser control, managed web access, structured extraction, or self-hosted infrastructure. Playwright (self-hosted) gives maximum browser control at the cost of owning access, proxies, and scaling. <a href="https://www.zenrows.com/blog/best-bright-data-alternative-for-self-service-scraping">Bright Data’s</a> Scraping Browser runs managed Chromium with proxy and CAPTCHA handling but bills by proxy bandwidth at $8/GB. Oxylabs targets enterprise scale with a large proxy network. <a href="https://www.zenrows.com/blog/best-firecrawl-alternative-for-anti-bot-bypass">Firecrawl</a> is API-first for LLM-ready Markdown but scored 33.69 percent on Proxyway’s protected-site benchmark. Browserless suits Docker-based self-hosting. ScrapingBee handles JavaScript rendering and proxies for lightly protected sites.</p><h3>The bottom line</h3><p>Zenrows is the stronger Browserbase alternative when the job is reliable access to protected web data without building a workflow around a browser. Fetch handles retrieval and protection, Extract returns structured JSON, and Batch coordinates high-volume jobs, all billed on success. Browserbase remains the fit when an agent needs to drive a real browser, replay sessions visually, or hold persistent browser state. <a href="https://app.zenrows.com/register">Sign up for Zenrows</a> and test your own targets.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4ff068b06294" width="1" height="1" alt=""><hr><p><a href="https://medium.com/zenrows/best-browserbase-alternative-in-2026-for-protected-web-access-4ff068b06294">Best Browserbase Alternative in 2026 for Protected Web Access</a> was originally published in <a href="https://medium.com/zenrows">Zenrows</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[10 Best Web Scraping Tools for Data Extraction in 2026]]></title>
            <link>https://medium.com/zenrows/10-best-web-scraping-tools-for-data-extraction-in-2026-6b76a29b82f6?source=rss----e3809efdcbcf---4</link>
            <guid isPermaLink="false">https://medium.com/p/6b76a29b82f6</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-scraping]]></category>
            <category><![CDATA[web-scraping]]></category>
            <category><![CDATA[python-webscraping]]></category>
            <category><![CDATA[web-scraping-tools]]></category>
            <dc:creator><![CDATA[Tioluwani Oyedele]]></dc:creator>
            <pubDate>Mon, 05 Oct 2026 10:56:39 GMT</pubDate>
            <atom:updated>2026-10-05T10:56:38.731Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qNDytnqEY_A-4EnLaCq7dA.png" /></figure><p>This article was originally published on the Zenrows blog. Read the original here: <a href="https://www.zenrows.com/blog/best-web-scraping-too">https://www.zenrows.com/blog/best-web-scraping-too</a></p><p>The right scraping tool comes down to three questions, and getting any one wrong means paying for infrastructure that doesn’t solve your problem. What are your target sites? What output do you need? And how much maintenance can you afford?</p><p>Target sites eliminate options fastest. If your targets sit behind Cloudflare, DataDome, or Akamai, most open-source frameworks are off the table before you write a line of code, because they render JavaScript but provide no anti-bot bypass on their own. Output determines whether you need a framework, an API, or an AI-native extraction layer. And maintenance is the hidden cost. Open-source frameworks license for free and cost everything to maintain, while managed APIs charge for access and absorb the upkeep.</p><p>Here are the ten tools worth knowing in 2026, ranked on how well they solve both halves of the job, getting past defenses and turning the page into usable data.</p><figure><img alt="How to match your target to a tool category" src="https://cdn-images-1.medium.com/max/1024/1*pD1cTWk3BkrlSD6hipYBxQ.png" /></figure><h3>1. Zenrows</h3><p>Zenrows is the lead pick for developers who need structured data from protected sites without building a separate access layer. Most tools solve one half of the problem. Zenrows does both through one API call.</p><p>Fetch with mode=auto handles Cloudflare, DataDome, Akamai, and PerimeterX automatically, starting with the cheapest viable request and escalating only when a site needs more. Zenrows reports a 99.93 percent success rate across supported targets. Extract turns a protected page into structured JSON with a single parameter:</p><pre># pip install requests<br>import os<br>import requests<br><br>params = {<br>&quot;url&quot;: &quot;https://www.chrono24.com/rolex/index.html&quot;,<br>&quot;apikey&quot;: os.environ[&quot;ZENROWS_API_KEY&quot;],<br>&quot;mode&quot;: &quot;auto&quot;, # handles the Cloudflare bypass<br>&quot;extract&quot;: &quot;auto&quot;, # returns structured JSON, no selectors<br>}<br><br>response = requests.get(&quot;https://api.zenrows.com/v1/&quot;, params=params)<br><br>data=response.json()</pre><p>A plain request to that Cloudflare-protected page returns a 403. With mode=auto and extract=auto, it returns clean per-item fields with no selectors written. Fetch also supports response_type=markdown for LLM pipelines and response_type=plaintext for NLP, and Batch handles up to 100,000 URLs per job for production runs. The free tier includes 5,000 credits a month; paid plans start at $16/month.</p><p><strong>Honest limitation:</strong> Credit-based pricing means a request’s cost depends on the access features a target needs, so estimate your mix of open and protected pages before choosing a plan.</p><h3>2. Bright Data</h3><p><a href="https://www.zenrows.com/blog/best-bright-data-alternative-for-self-service-scraping">Bright Data</a> is built for enterprise scale and scored 93.14 percent on Proxyway’s 2025 benchmark of 15 heavily protected sites, the highest of the providers tested. It runs 72 million+ residential IPs across 195 countries, and its SOC 2 Type II and ISO 27001 certifications matter to procurement teams with compliance requirements. A free tier and pay-as-you-go pricing now cover Web Unlocker, SERP API, and Web Scraper API without a committed plan.</p><p><strong>Honest limitation</strong>: KYC verification is mandatory for residential and mobile proxy access and reviewers report it taking one to three business days. Entry-level committed plans start around $499/month.</p><h3>3. Scrapy</h3><p><a href="https://www.zenrows.com/alternatives/scrapy">Scrapy</a> is the open-source Python standard for custom crawlers, built on an async architecture that handles dozens of concurrent requests out of the box. Its middleware system covers proxy rotation, retries, and headers, and item pipelines give you a clean stage for cleaning and storing data. It fits Python developers who want full control and have the DevOps capacity to manage proxies and anti-bot handling separately.</p><p><strong>Honest limitation</strong>: No built-in browser rendering, anti-bot handling, or residential proxies. On JavaScript-heavy or defended sites, total cost of ownership often runs higher than a managed API once proxy fees and engineering hours are counted. Scrapy is free; Scrapy Cloud starts at $9/month per unit.</p><h3>4. Playwright</h3><p><a href="https://www.zenrows.com/compare/playwright">Playwright</a> is Microsoft’s open-source browser automation framework for Chromium, Firefox, and WebKit, and the standard choice for single-page applications and sites that require clicking, scrolling, or form interaction. It renders JavaScript the way a real browser does, which makes it a fit for infinite-scroll pages that plain HTTP requests can’t touch.</p><p><strong>Honest limitation</strong>: Playwright alone provides no anti-bot protection, and Cloudflare, DataDome, and Akamai all detect a standard session. Stealth libraries close some of the gap, but independent testing finds DIY approaches plateauing short of a managed anti-bot API on the hardest targets. Zenrows Browser Sessions offers the same browser-level control as a managed service.</p><h3>5. ScrapingBee</h3><p><a href="https://www.zenrows.com/alternatives/scrapingbee">ScrapingBee</a> is a managed API with JavaScript rendering, CAPTCHA handling, and residential proxy rotation behind one set of keys, plus an AI extraction mode that returns structured data from a plain-English description. It handles lightly to moderately protected sites well.</p><p><strong>Honest limitation</strong>: Pricing runs on a steep credit multiplier. A basic scrape is one credit, JavaScript rendering is five, premium proxies 25, and stealth mode 75, so a $49/month plan advertised at 250,000 credits shrinks to roughly 3,333 requests once stealth is needed. Proxyway scored it 84.47 percent and called the credit model “evidently not ideal for opening protected websites.” Entry pricing starts at $49/month.</p><h3>6. ScraperAPI</h3><p><a href="https://www.zenrows.com/alternatives/scraperapi">ScraperAPI</a> is built around structured data endpoints for Amazon, Google, Walmart, and eBay, returning parsed JSON or CSV instead of raw HTML. It handles proxy rotation and CAPTCHA bypass automatically, and its pre-built endpoints skip setup entirely for those targets.</p><p><strong>Honest limitation</strong>: It offers no general-purpose AI extraction, so protected sites outside its pre-built list still return raw HTML you parse yourself. On Proxyway’s benchmark it scored 68.95 percent, solid but a step below the top performers on the hardest general targets. The Hobby plan starts at $49/month.</p><h3>7. Firecrawl</h3><p><a href="https://www.zenrows.com/blog/best-firecrawl-alternative-for-anti-bot-bypass">Firecrawl</a> is an AI-native scraping API that converts any URL into clean Markdown or structured data for LLM consumption, built around scrape, crawl, map, and extract endpoints, with an MCP server shipped alongside. It fits developers building RAG pipelines who want clean Markdown by default. The free tier gives 1,000 credits a month.</p><p><strong>Honest Limitation</strong>: Anti-bot coverage on heavily protected sites is limited. On Proxyway’s benchmark it scored 33.69 percent, last among providers tested, though it’s built for crawling the long tail rather than hardened targets. Structured extraction costs roughly 5x a standard scrape, and credits don’t roll over. Paid plans start at $16/month on annual billing</p><h3>8. Apify</h3><p><a href="https://www.zenrows.com/blog/best-apify-alternative-for-large-scale-scraping">Apify</a> is a full-stack platform built around a marketplace of over 30,000 pre-built Actors covering targets like Google Maps, LinkedIn, and Amazon, with serverless compute, scheduling, and storage included. It fits teams that want a pre-built scraper for a specific popular target.</p><p><strong>Honest limitation</strong>: Apify is no general-purpose anti-bot solution, and handling quality varies by Actor since each is maintained independently. Residential proxy access is billed separately at roughly $8/GB. The free tier gives $5 in monthly platform credit; paid plans start at $29/month.</p><h3>9. Browserless</h3><p>Browserless is a cloud-managed headless browser service exposing Chromium through a REST API and BQL, a GraphQL-based stealth API. CAPTCHA solving, WebGL fingerprint randomization, and residential proxies come built in, with a self-hosted Docker option. It fits developers who want Playwright-compatible infrastructure without running their own browser pool.</p><p><strong>Honest limitation</strong>: Structured extraction means CSS selectors or DOM queries you write yourself. There’s no prompt-to-schema extraction, so pulling clean fields from a new page still means writing selectors first. The free tier includes 1,000 units a month; paid plans start at $25/month on annual billing.</p><h3>10. Octoparse</h3><p>Octoparse is a no-code, point-and-click scraper built around a Windows desktop app with 500+ pre-built templates. You click the fields you want and it infers the extraction logic without selectors or code. It fits marketers, researchers, and small teams collecting from open, well-structured sites without hiring a developer.</p><p><strong>Honest limitation:</strong> Octoparse is a visual scraper rather than an anti-bot specialist. Cloudflare, DataDome, and Akamai block it, and reviewers report paid-plan proxy add-ons struggling on heavily protected targets. The free desktop version costs nothing; cloud plans start at $69/month.</p><h3>Which tool fits your workload</h3><ul><li><strong>Match your workload to one of four scenarios. Scraping protected sites and need structured JSON without writing selectors? </strong>Zenrows handles access and extraction in one call, with CLI and MCP support for terminal and agent workflows.</li><li><strong>Building a RAG or LLM ingestion layer?</strong> Fetch returns clean Markdown so protected and unprotected sources run through one pipeline, with Firecrawl a reasonable alternative when nothing sits behind Cloudflare, DataDome, or Akamai.</li><li><strong>A Python developer wanting full control over unprotected targets?</strong> Scrapy gives you the pipeline architecture, paired with Playwright when a page needs rendering.</li><li><strong>Need a pre-built scraper for a popular e-commerce or mapping target?</strong> Apify’s marketplace already has one.</li></ul><p>The number on a pricing page is rarely the full budget. Factor in proxy multipliers, structured-extraction premiums, and the engineering hours a target’s defenses will cost you before you commit.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6b76a29b82f6" width="1" height="1" alt=""><hr><p><a href="https://medium.com/zenrows/10-best-web-scraping-tools-for-data-extraction-in-2026-6b76a29b82f6">10 Best Web Scraping Tools for Data Extraction in 2026</a> was originally published in <a href="https://medium.com/zenrows">Zenrows</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Run Large-Scale Web Scraping Jobs With Zenrows Batch]]></title>
            <link>https://medium.com/zenrows/how-to-run-large-scale-web-scraping-jobs-with-zenrows-batch-b3abc8f4392c?source=rss----e3809efdcbcf---4</link>
            <guid isPermaLink="false">https://medium.com/p/b3abc8f4392c</guid>
            <category><![CDATA[data-scraping]]></category>
            <category><![CDATA[web-scraping-tools]]></category>
            <category><![CDATA[web-scraping]]></category>
            <category><![CDATA[python-webscraping]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Tioluwani Oyedele]]></dc:creator>
            <pubDate>Mon, 05 Oct 2026 10:51:03 GMT</pubDate>
            <atom:updated>2026-10-05T10:51:01.941Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TYKCpXr5qdJGVsA_8pbOGA.png" /></figure><p>This article was originally published on the Zenrows blog. Read the original here: <a href="https://www.zenrows.com/blog/large-scale-web-scraping-zenrows-batch">https://www.zenrows.com/blog/large-scale-web-scraping-zenrows-batch</a></p><p>A scraper that runs fine on 500 URLs starts falling apart at 50,000. The target rate-limits you, 429 becomes the most common line in your logs, thousands of open connections eat memory, and failed requests stay failed because nothing was built to retry them. By the time you notice, you often can’t even tell which URLs succeeded.</p><p>The reason is that concurrency was never the hard part. asyncio and aiohttp solve concurrency well on a single machine, and so does Scrapy. What none of them solve is the infrastructure underneath: the queue, the retry logic, the failure tracking, and the result delivery. At scale, owning all of that is the real engineering cost.</p><p>This walks through running a large scraping job as a single managed call, from one Cloudflare-protected page to a 100,000-URL job, without building that infrastructure yourself.</p><h4>What you’ll learn</h4><ul><li>Where the DIY approach breaks down and what six pieces every large-scale system needs</li><li>How to submit a Batch job, attach a webhook, and verify the delivery with HMAC</li><li>How to handle real failures and rerun only what failed</li><li>How to submit 100,000 URLs with a credit safety monitor, CSV input, and scheduling</li></ul><p>Every example ran against the live Batch API. The runnable code is in the <a href="https://github.com/ZenRows/large-scale-web-scraping-zenrows-batch">GitHub repo</a>.</p><h4>Where the DIY approach breaks down</h4><p>Large-scale web scraping is any job where the infrastructure required to run it reliably competes with the value of the data it produces. The threshold varies by team, but most developers hit it somewhere between 10,000 and 100,000 URLs per job.</p><p>Every large-scale scraping system ends up needing the same six pieces. A job queue that decouples discovery from fetching. Concurrency management that keeps requests from overwhelming targets. Proxy rotation that avoids IP bans. Retry logic with backoff so you can resubmit failures without rerunning the whole list. Monitoring that catches silent failures before they distort a dataset. And a delivery mechanism that gets the data where it’s used.</p><p>Each one is its own engineering project. Together they are the orchestration layer, and Batch replaces all six with a single API call.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GbiVTFkN63e1qpVlMijRyQ.png" /><figcaption>DIY vs. Zenrows Batch</figcaption></figure><h4>How Batch works</h4><p>Batch is an async scraping API. You submit a list of URLs and a scraping config in one request, and it returns a job ID you can check on later. It queues each URL, handles protected access with mode=auto, and retries anything that fails for transient reasons.</p><p>When a run finishes, you find out one of two ways. Zenrows calls your webhook automatically, or you poll the job’s status endpoint yourself until it reports back as done.</p><p>A submission needs a few things. It sets whether the job is regular or scheduled, whether it’s closed with all URLs known up front or open with URLs added over time, and a list of URLs, each with an optional external_id for matching results back to your own records. Scraping options go in a separate zenrows_params object. `mode: “auto” for anti-bot handling, response_type for Markdown or plaintext instead of HTML, and css_extractor for a specific structure. Set these once at the job level to apply them to every URL, or override them per task.</p><p>A job takes up to 100,000 URLs. There are three ways to load them. A closed job takes every URL inline in the submission, the simplest path when you already have the full list. An open job lets you add URLs in batches over time, up to 10,000 per add-tasks call, and close it when you’re done. A CSV upload skips inlining URLs entirely, which becomes the practical choice once a list gets too large to paste into a request body.</p><h4>Running your first Batch job</h4><p>A Batch job is one request built from your URLs and a scraping config. The simplest real example submits a Cloudflare-protected page with mode=auto.</p><p>Every example calls into client.py, a small helper in the repo that wraps authentication, the Batch endpoints, job submission, polling, results, reruns, webhooks, HMAC verification, CSV uploads, and scheduling, so each script avoids repeating the same HTTP boilerplate.</p><pre><br># refer to 01_cloudflare_target.py in repo<br>import client<br>payload = {<br> &quot;type&quot;: &quot;regular&quot;,<br> &quot;status&quot;: &quot;closed&quot;,<br> &quot;zenrows_params&quot;: {&quot;mode&quot;: &quot;auto&quot;},<br> &quot;tasks&quot;: [<br> {&quot;url&quot;: &quot;https://www.scrapingcourse.com/cloudflare-challenge&quot;, &quot;external_id&quot;: &quot;cloudflare-challenge&quot;}<br> ],<br>}<br>submitted = client.submit_job(payload)<br>job_id = submitted[&quot;job_id&quot;]<br>print(job_id, submitted[&quot;latest_run&quot;][&quot;status&quot;])</pre><p>We ran this against a real page. With mode=auto, the job used dynamic website support and advanced network access to retrieve the content, consuming 25 credits. The resulting content contains the page’s own confirmation text, reading “You bypassed the Cloudflare challenge! :D”</p><h4>Getting notified when a job finishes</h4><p>Attaching a webhook means Zenrows pushes the scraped data to your server automatically. It POSTs to your endpoint the moment a run completes, instead of you polling for it.</p><p>Signed webhooks use an HMAC key on your account. Generate one before your first signed job.</p><pre><br># refer to 05_webhook.py in repo<br>key = client.rotate_hmac_key()<br># store key[“secret”] now, it’s returned exactly once<br></pre><p>With that in place, attach the webhook to the submission with `”webhook”: {“url”: “https://your-app.com/webhooks/zenrows&quot;, “signature”: True}`. Here’s the completed run delivery we captured:</p><pre>json<br>{<br> “schema”: “zenrows.webhook/v1”,<br> “event_type”: “run.completed”,<br> “job_id”: “JOB_ID”,<br> “run_id”: “RUN_ID”,<br> “stats”: {“total”: 3, “completed”: 3, “successful”: 3, “failed”: 0, “spend”: {“credits”: 3, “cost”: 0.00038997}}<br>}<br></pre><p>Verify the delivery came from Zenrows using the `X-Signature` header and the secret from the HMAC step.</p><pre><br>import hmac<br>import hashlib<br>import base64<br><br>def verify(raw_body: bytes, t: str, v1: str, secret_b64: str) -&gt; bool:<br> secret = base64.b64decode(secret_b64)<br> expected = hmac.new(secret, f&quot;{t}.&quot;.encode() + raw_body, hashlib.sha256).hexdigest()<br> return hmac.compare_digest(expected, v1)</pre><h4>Handling a job with real failures</h4><p>Some URLs fail, and Batch reports exactly why. We submitted a batch mixing five real product pages with three URLs built to fail on purpose, a nonexistent domain, a dead page, and an unroutable IP. Five succeeded and three failed. Once the run reaches a terminal state, query the results and filter for the failures.</p><pre><br># refer to 03_failure_and_rerun.py in repo<br>results = client.list_results(job_id, status=”all”)<br>failed_rows = [r for r in results[“results”] if r[“status”] == “failed”]<br>for row in failed_rows:<br> print(row[“external_id”], row[“url”], row[“error”])<br></pre><p>Each failure carries a real error code, RESP002 for the 404, RESP007 for the domain that could not be resolved, and REQS004 for the IP-based URL rejected before the fetch. Resubmitting only the failures is one call, and it doesn’t charge you for what already succeeded.</p><pre><br>rerun = client.rerun_job(job_id, status=”failed”)<br>print(rerun[“retried_tasks”], rerun[“inherited_tasks”])<br># 3 retried, 5 inherited<br></pre><p>One thing to know. A URL that’s malformed at the syntax level never becomes a per-task failure. It rejects the entire submission with a 400. Fix the malformed URL and resubmit the whole batch. Only well-formed URLs that fail at scrape time show up as individual failed tasks you can inspect and retry.</p><h4>What this replaces</h4><p>Reaching the same Cloudflare result with a plain asyncio/aiohttp scraper takes a concurrency semaphore and retry logic built by hand. When we ran that DIY version against the same page, all five requests came back HTTP 403. Cloudflare’s challenge is a normal HTTP response rather than a connection error, so retry logic built to catch TimeoutError and ClientError never triggers. mode=auto cleared it by selecting the configuration the challenge required, at the same 25-credit tier.</p><h4>Submitting a 100,000-URL job</h4><p>We submitted a job with 100,000 URLs and monitored its credit use as it ran. The monitor stopped the job after 14,105 URLs had completed successfully, at a total of 300,787 credits. Batch reports spend at the run level, with your account statement authoritative above it.</p><p>The cost climbed enough during our run to justify watching it as a job progresses. Here’s a safety check that stops the job at a credit ceiling.</p><pre><br># refer to 10_scale_run.py in repo<br>import time<br>import client<br><br>job_id = &quot;YOUR_JOB_ID&quot;<br>credit_ceiling = 300_000<br><br>while True:<br> job = client.get_job(job_id)<br> stats = job[&quot;latest_run&quot;][&quot;stats&quot;]<br> spend = stats.get(&quot;spend&quot;, {}).get(&quot;credits&quot;, 0)<br><br>if spend &gt;= credit_ceiling:<br> client.stop_job(job_id)<br> print(f&quot;stopped at {spend} credits, {stats[&#39;completed&#39;]}/{stats[&#39;total&#39;]} completed&quot;)<br> break<br><br>if job[&quot;latest_run&quot;][&quot;status&quot;] in {&quot;completed&quot;, &quot;stopped&quot;}:<br> break<br><br>time.sleep(5)</pre><h4>CSV upload for large URL lists</h4><p>For a list too large to paste into a request body, a CSV upload takes over. Create an upload slot, upload the file, then submit a job that references it.</p><pre><br># refer to 07_csv_upload.py in repo<br>slot = client.create_csv_input_slot(fields={“url”: “URL”, “external_id”: “Customer Ref”}, header=True)<br>client.upload_csv(slot, csv_bytes)<br>payload = {<br> &quot;type&quot;: &quot;regular&quot;,<br> &quot;status&quot;: &quot;closed&quot;,<br> &quot;file_input_id&quot;: slot[&quot;file_input_id&quot;],<br> &quot;zenrows_params&quot;: {&quot;mode&quot;: &quot;auto&quot;},<br>}<br>submitted = client.submit_job(payload)</pre><p>CSV uploads cap at 50MB or 100,000 rows, the same ceiling as an inline submission.</p><h4>Scheduling recurring jobs</h4><p>Price tracking, inventory monitoring, and content-change detection all need the same job to run repeatedly. A scheduled job stores your URLs and config as a template, and each firing creates a new run.</p><pre><br># refer to 08_scheduling.py in repo<br>payload = {<br> “type”: “scheduled”,<br> “status”: “closed”,<br> “schedule”: {“rate”: {“every”: 24, “unit”: “hour”}},<br> “tasks”: [{“url”: u, “external_id”: f”sched-{i}”} for i, u in enumerate(urls, start=1)],<br>}<br><br>submitted = client.submit_job(payload)<br></pre><p>We tested pausing and replacing a schedule on a real job. Both worked as documented, and replacing a schedule leaves a paused job paused rather than quietly reactivating it.</p><h4>Structured extraction instead of raw HTML</h4><p>Batch can return structured JSON using css_extractor for fields you select, passed as a JSON-encoded string. The result comes back as a flat object keyed by the field name. For automatic structured output without a selector map, use Zenrows Extract with Batch.</p><h4>What Batch leaves to you</h4><p>Batch submits URLs and gets back results. It holds no browser session open between requests, so a target that needs a login flow, clicking through steps, or filling a form before the data appears calls for [Browser Sessions](https://www.zenrows.com/products/browser-sessions) instead.</p><p>What’s left is what should be left to you. Choosing targets, setting zenrows_params for each job, and deciding what happens with the data once it’s back, whether that’s an ETL pipeline, a price-monitoring table, or <a href="https://www.zenrows.com/blog/web-data-llm-fine-tuning">dataset creation for fine-tuning</a>.</p><h4>Key takeaways</h4><ul><li>Concurrency is the easy part. The queue, retries, failure tracking, and delivery are the real cost, and Batch handles all of them.</li><li>mode=auto escalates only for URLs that need it, so protected and ordinary pages run in one job at the right cost each.</li><li>rerun(status=”failed”) resubmits only failures without recharging for what already succeeded.</li><li>A credit ceiling watch loop keeps an unattended 100,000-URL run from overspending.</li><li>CSV upload and scheduling cover bulk lists and recurring jobs without extra infrastructure.</li></ul><p>For running the collected data into a training pipeline, see <a href="https://www.zenrows.com/blog/web-data-llm-fine-tuning">web data for LLM fine-tuning</a>. For the single-URL path, <a href="https://www.zenrows.com/products/fetch">Fetch</a> scrapes one page per request with the same zenrows_params.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b3abc8f4392c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/zenrows/how-to-run-large-scale-web-scraping-jobs-with-zenrows-batch-b3abc8f4392c">How to Run Large-Scale Web Scraping Jobs With Zenrows Batch</a> was originally published in <a href="https://medium.com/zenrows">Zenrows</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Build a web-aware agent with OpenAI Agents SDK and Zenrows]]></title>
            <link>https://medium.com/zenrows/build-a-web-aware-agent-with-openai-agents-sdk-and-zenrows-99451a3755bc?source=rss----e3809efdcbcf---4</link>
            <guid isPermaLink="false">https://medium.com/p/99451a3755bc</guid>
            <category><![CDATA[openai-agents-sdk]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[zenrows]]></category>
            <category><![CDATA[agent-web]]></category>
            <category><![CDATA[web-data-extraction]]></category>
            <dc:creator><![CDATA[Praise James]]></dc:creator>
            <pubDate>Thu, 10 Sep 2026 10:06:46 GMT</pubDate>
            <atom:updated>2026-09-10T10:06:45.615Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*C5i1AMzH0Jr4LbOER94Vww.png" /></figure><p>Your agent can search the web. What it can’t do reliably is read the live page behind the result. The OpenAI Agents SDK ships a built-in WebSearchTool that handles discovery well, but it returns indexed snapshots of content captured when a crawler last visited, not what the page shows right now. On JavaScript-rendered sites and pages with bot protection, the answer you get back is incomplete by design.</p><p>The fix isn’t to replace the built-in search. It’s to give the agent a second retrieval path, one that reaches the live page when discovery alone isn’t enough. This guide shows you how to register Zenrows Fetch as a @function_tool next to WebSearchTool in a single agent, so the model routes between them on its own each turn.</p><h3>What you’ll learn</h3><ul><li>Why built-in web search falls short on live and protected pages</li><li>How to wrap Zenrows Fetch as a typed @function_tool</li><li>How to run both tools in one agent loop without any routing code of your own</li><li>When to use each tool and how to extend this pattern to multi-agent setups</li></ul><p>All the code is on <a href="https://github.com/ZenRows/web-aware-agent-openai-agents-sdk-zenrows">GitHub</a>.</p><h3>Prerequisites</h3><p>You need two API keys and one SDK.</p><ul><li>Python 3.9 or later</li><li>An OpenAI API key from your <a href="https://platform.openai.com/api-keys">OpenAI developer dashboard</a></li><li>A Zenrows API key from your <a href="https://app.zenrows.com/register">Zenrows dashboard</a></li></ul><p>Install the Agents SDK and python-dotenv:</p><pre>python3 -m pip install openai-agents python-dotenv</pre><p>Create a .env file in your project root with both keys, and add .env to .gitignore so the credentials never reach version control:</p><pre>OPENAI_API_KEY=your_openai_api_key_here<br>ZENROWS_API_KEY=your_zenrows_api_key_here</pre><p>Then load them in your script:</p><pre>from dotenv import load_dotenv<br>load_dotenv()</pre><h3>Start with the built-in search</h3><p>WebSearchTool is a hosted tool. It runs on OpenAI&#39;s infrastructure through the Responses API, so enabling it takes one line and no configuration.</p><pre>import asyncio<br>from dotenv import load_dotenv<br>from agents import Agent, Runner, WebSearchTool<br><br>load_dotenv()<br><br>agent = Agent(<br>    name=&quot;Product Researcher&quot;,<br>    instructions=(<br>        &quot;You research products and web pages. &quot;<br>        &quot;Report exactly what you find, and state plainly when &quot;<br>        &quot;specific information is missing from your results.&quot;<br>    ),<br>    tools=[WebSearchTool()],<br>)<br><br>async def main():<br>    result = await Runner.run(<br>        agent,<br>        &quot;Which laptops did Apple release most recently, and where can I buy them?&quot;,<br>    )<br>    print(result.final_output)<br><br>asyncio.run(main())</pre><p>On a broad discovery question, this works. The agent finds relevant pages, pulls indexed content, and returns answers with source links. The problem appears when you ask for something live.</p><h3>Where the built-in search stops</h3><p>There are two gaps that matter in practice.</p><p><strong>The index lags the live page.</strong> When a site allows crawling, the built-in search returns a snippet of OpenAI’s indexed copy. For documentation, that’s usually fine. For anything where freshness matters like an exchange rate, a stock level, or a price, the indexed copy lags. The page changes faster than the index refreshes.</p><p><strong>JavaScript-rendered content never makes it in.</strong> The initial HTML of many sites is a shell. Prices, inventory, and reviews only appear once the page’s JavaScript runs in a browser. The index stores that initial HTML, so the fields you want were never captured. Crawling more often doesn’t fix this.</p><p>Asking the built-in search for a live price on a JavaScript-heavy retailer shows the gap clearly:</p><pre>result = await Runner.run(<br>    agent,<br>    &quot;What is the current price and stock status of &quot;<br>    &quot;https://www.amazon.com/dp/B0GR1JKMBV/ref=fs_a_mbt2_us1?th=1&quot;,<br>)<br>print(result.final_output)</pre><p>The agent finds the page and returns the product title. Then it tells you exactly what it couldn’t get:</p><pre>I visited the Amazon product page for ASIN B0GR1JKMBV... The page displays<br>a prompt: &quot;To see product details, add this item to your cart.&quot; That message<br>appears several times in place of visible pricing or availability information.<br>Therefore I could not retrieve the current price or availability.</pre><p>No error or crash, the agent gives a confident explanation that the data wasn’t there.</p><h3>Add Zenrows Fetch as a @function_tool</h3><p>This is the retrieval layer the built-in search can’t provide. One Python function, one decorator, and the model gets access to live pages.</p><pre>import os<br>import requests<br>from agents import function_tool<br><br>@function_tool<br>def fetch_page_content(url: str) -&gt; str:<br>    &quot;&quot;&quot;Fetch the full, current content of a specific web page as Markdown.<br><br>    Use this when you need the complete content of a known URL rather than<br>    a search result summary, such as live prices, stock levels, or data<br>    that loads via JavaScript after the page opens.<br><br>    Args:<br>        url: The full URL of the page to retrieve.<br>    &quot;&quot;&quot;<br>    response = requests.get(<br>        &quot;https://api.zenrows.com/v1/&quot;,<br>        params={<br>            &quot;url&quot;: url,<br>            &quot;apikey&quot;: os.getenv(&quot;ZENROWS_API_KEY&quot;),<br>            &quot;mode&quot;: &quot;auto&quot;,<br>            &quot;response_type&quot;: &quot;markdown&quot;,<br>        },<br>        timeout=90,<br>    )<br><br>    if response.status_code != 200:<br>        return f&quot;Zenrows returned {response.status_code} for {url}&quot;<br><br>    return response.textFour things make this work.</pre><p>The function itself accepts a URL and returns page content as a string. To the agent it’s one more tool in the list.</p><p>The request hits the <a href="https://www.zenrows.com/products/fetch">Zenrows Fetch</a> endpoint. mode=auto activates Adaptive Stealth Mode, which turns on JavaScript rendering and escalates to premium proxies only when the target site requires them. response_type=markdown strips the markup so the model reads clean text.</p><p>The @function_tool decorator builds the tool schema from the function signature automatically. No manual JSON schema.</p><p>The docstring is what the model reads every turn to decide whether the tool applies. The phrase “rather than a search result summary” is load-bearing: it’s what stops the agent from defaulting to built-in search when it has a URL and needs the live page.</p><h3>Run both tools in a single agent loop</h3><p>Put both tools in the same tools list. The model selects between them from their descriptions alone.</p><pre>import asyncio<br>import os<br>import requests<br>from dotenv import load_dotenv<br>from agents import Agent, Runner, WebSearchTool, function_tool<br><br>load_dotenv()<br><br>@function_tool<br>def fetch_page_content(url: str) -&gt; str:<br>    &quot;&quot;&quot;Fetch the full, current content of a specific web page as Markdown.<br><br>    Use this when you need the complete content of a known URL rather than<br>    a search result summary, such as live prices, stock levels, or data<br>    that loads via JavaScript after the page opens.<br><br>    Args:<br>        url: The full URL of the page to retrieve.<br>    &quot;&quot;&quot;<br>    response = requests.get(<br>        &quot;https://api.zenrows.com/v1/&quot;,<br>        params={<br>            &quot;url&quot;: url,<br>            &quot;apikey&quot;: os.getenv(&quot;ZENROWS_API_KEY&quot;),<br>            &quot;mode&quot;: &quot;auto&quot;,<br>            &quot;response_type&quot;: &quot;markdown&quot;,<br>        },<br>        timeout=90,<br>    )<br><br>    if response.status_code != 200:<br>        return f&quot;Zenrows returned {response.status_code} for {url}&quot;<br><br>    return response.text<br><br>agent = Agent(<br>    name=&quot;Web-Aware Researcher&quot;,<br>    instructions=(<br>        &quot;You research products on the web. &quot;<br>        &quot;Use web search to find the product page URL. &quot;<br>        &quot;Then call fetch_page_content on that URL and report the price &quot;<br>        &quot;and stock status from the fetched page content. &quot;<br>        &quot;State which tool each figure came from.&quot;<br>    ),<br>    tools=[WebSearchTool(), fetch_page_content],<br>)<br><br>def print_trace(items):<br>    for item in items:<br>        raw = getattr(item, &quot;raw_item&quot;, None)<br>        name = getattr(raw, &quot;name&quot;, None) or getattr(raw, &quot;type&quot;, &quot;&quot;)<br>        print(f&quot;[{item.type}] {name}&quot;)<br><br>        if item.type == &quot;tool_call_item&quot; and name == &quot;fetch_page_content&quot;:<br>            print(&quot;  args:&quot;, getattr(raw, &quot;arguments&quot;, &quot;&quot;))<br><br>        if item.type == &quot;tool_call_output_item&quot;:<br>            out = str(getattr(item, &quot;output&quot;, &quot;&quot;))<br>            print(f&quot;  output: {len(out)} chars&quot;)<br>            print(f&quot;  body: {out[:500]}&quot;)<br><br>async def main():<br>    result = await Runner.run(<br>        agent,<br>        &quot;Find the PriceOye page for the iPhone 17 Pro &quot;<br>        &quot;and report its current price and stock status.&quot;,<br>    )<br><br>    print_trace(result.new_items)<br>    print(&quot;\n---\n&quot;)<br>    print(result.final_output)<br><br>asyncio.run(main())</pre><p>The trace makes the two-step routing visible. Turn one is a web_search_call for discovery. Turn two calls fetch_page_content with the URL it found:</p><pre>[tool_call_item] web_search_call<br>[message_output_item] message<br>[tool_call_item] fetch_page_content<br>  args: {&quot;url&quot;:&quot;https://priceoye.pk/mobiles/apple/apple-iphone-17-pro&quot;}<br>[tool_call_output_item]<br><br>---<br><br>Here are the details for the iPhone 17 Pro from its PriceOye product page:<br><br>Current Price: Rs 471,999<br>Stock Status: Only 1 left in stock<br><br>Price and stock status are directly extracted from the fetched page content<br>(functions.fetch_page_content tool).The agent discovers the URL through search, then reads the live page through Zenrows. No conditionals. No orchestration code. The routing lives entirely in the tool descriptions.</pre><h3>When to use each tool</h3><p>Use WebSearchTool when the agent needs to discover something: open-ended research, finding relevant pages, or answering questions from indexed public content.</p><p>Use Zenrows as the retrieval layer. It gives the agent dependable access to pages the built-in search can’t reach like protected targets and JavaScript-rendered sites, and returns content the model can read directly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/982/1*KrrNGPwE7Fxhkd0iVQ28tQ.png" /></figure><p>If you need specific fields rather than a whole page, <a href="https://docs.zenrows.com/extract/introduction">Zenrows Extract</a> returns structured data so the model never has to parse Markdown to find a price. For recurring retrieval across many URLs, <a href="https://www.zenrows.com/products/batch">Zenrows Batch</a> handles that without you managing concurrency.</p><p>The same pattern extends to multi-agent setups. The guide on <a href="https://www.zenrows.com/blog/web-research-multi-agent-ag2-zenrows">building a web research multi-agent system with AG2 and Zenrows</a> walks through that. The wrapper carries over to <a href="https://www.zenrows.com/blog/zenrows-smolagents">smolagents</a> too, under a @tool decorator.</p><h3>Key takeaways</h3><ul><li>The built-in WebSearchTool handles discovery. Zenrows handles retrieval. They do different jobs and both belong in the same agent.</li><li>The @function_tool decorator builds the tool schema from your Python function signature automatically.</li><li>The docstring is how the model decides which tool to call. Write it for the model, not for a human reader.</li><li>mode=auto in the Zenrows request handles JavaScript rendering and proxy escalation per request, so you only pay for what each site actually needs.</li><li>The routing between tools requires no orchestration code on your side.</li></ul><p><em>This article was originally published on the Zenrows blog. Read the original here: </em><a href="https://www.zenrows.com/blog/web-aware-agent-openai-agents-sdk-zenrows"><em>https://www.zenrows.com/blog/web-aware-agent-openai-agents-sdk-zenrows</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=99451a3755bc" width="1" height="1" alt=""><hr><p><a href="https://medium.com/zenrows/build-a-web-aware-agent-with-openai-agents-sdk-and-zenrows-99451a3755bc">Build a web-aware agent with OpenAI Agents SDK and Zenrows</a> was originally published in <a href="https://medium.com/zenrows">Zenrows</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Build a web research multi-agent system with AG2 and Zenrows]]></title>
            <link>https://medium.com/zenrows/build-a-web-research-multi-agent-system-with-ag2-and-zenrows-ff1e645a0752?source=rss----e3809efdcbcf---4</link>
            <guid isPermaLink="false">https://medium.com/p/ff1e645a0752</guid>
            <category><![CDATA[ag2]]></category>
            <category><![CDATA[web-research-agent]]></category>
            <category><![CDATA[zenrows]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[web-data-extraction]]></category>
            <dc:creator><![CDATA[Praise James]]></dc:creator>
            <pubDate>Thu, 10 Sep 2026 10:05:59 GMT</pubDate>
            <atom:updated>2026-09-10T10:05:58.289Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XyqS52zkphqn1qk_3vnQCw.png" /></figure><p>In a multi-agent pipeline, a broken fetch doesn’t stay broken. It enters the shared history and every agent that runs after reads it, reasons around it, and forwards it to the next turn. You end up paying for agents that had nothing real to work with.</p><p>AG2’s group chat architecture makes this visible: every tool response enters a shared context window that all agents read on their turn. When the fetch layer works, clean content flows through the pipeline and agents produce useful output. When it doesn’t, the failure circulates and agents spend their turns explaining why they can’t help.</p><p>This guide shows how to register <a href="https://www.zenrows.com/products/fetch">Zenrows Fetch</a> as a typed tool inside an AG2 group chat, so clean Markdown enters the history from the first turn. It also runs both configurations, with and without the fetch tool, and compares token spend to show what changes when the retrieval layer is reliable.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*E3TkitJoLGTWBAURveYUpQ.png" /></figure><h3>What you’ll learn</h3><ul><li>How AG2’s group chat history flows between agents and why a failed fetch is expensive</li><li>How to define and register a typed Zenrows tool that AG2 can call and execute</li><li>How to wire a researcher, analyst, and critic into a group chat pipeline</li><li>How token usage compares between a failed fetch and a working one</li></ul><p>All the code is on <a href="https://github.com/ZenRows/ag2-zenrows-research-pipeline">GitHub</a>.</p><h3>Prerequisites</h3><ul><li>Python 3.10 or later</li><li>AG2 v0.12.2</li><li>A <a href="https://app.zenrows.com/register">Zenrows account</a> and API key</li><li>An <a href="https://platform.openai.com/home">OpenAI account</a> and API key</li></ul><h3>Set up an AG2 group chat</h3><p>Install AG2 with its OpenAI client:</p><pre>pip install &quot;ag2[openai]==0.12.2&quot;</pre><p>Two things to know before you run anything.</p><p>The package name and the import name differ. You install with pip install &quot;ag2[openai]&quot; and import with import autogen. Using import ag2 raises a ModuleNotFoundError in v0.12.2. AG2 is deprecating the classic framework at v1.0, at which point import ag2 becomes correct.</p><p>AG2 doesn’t bundle the OpenAI client, which is why the [openai] extra is required.</p><p>Load your API keys and set up the LLM config:</p><pre>import os<br>from dotenv import load_dotenv<br><br>import autogen<br>from autogen import AssistantAgent, GroupChat, GroupChatManager, UserProxyAgent<br><br>load_dotenv()<br><br>llm_config = {<br>    &quot;config_list&quot;: [<br>        {<br>            &quot;model&quot;: &quot;gpt-4o-mini&quot;,<br>            &quot;api_key&quot;: os.getenv(&quot;OPENAI_API_KEY&quot;),<br>        }<br>    ],<br>    &quot;temperature&quot;: 0,<br>}</pre><p>Next, define the four agents. The structure gives each agent a distinct job: the researcher fetches and summarizes, the analyst extracts structured fields, the critic validates and terminates.</p><pre>user_proxy = UserProxyAgent(<br>    name=&quot;user_proxy&quot;,<br>    human_input_mode=&quot;NEVER&quot;,<br>    code_execution_config=False,<br>    is_termination_msg=lambda msg: &quot;TERMINATE&quot; in (msg.get(&quot;content&quot;) or &quot;&quot;),<br>)<br><br>researcher = AssistantAgent(<br>    name=&quot;researcher&quot;,<br>    system_message=(<br>        &quot;You are a research agent. When given a URL, fetch its web content and &quot;<br>        &quot;summarize what you find. Be factual and note anything that looks &quot;<br>        &quot;incomplete or missing from the page. Pass your summary to the analyst.&quot;<br>    ),<br>    llm_config=llm_config,<br>)<br><br>analyst = AssistantAgent(<br>    name=&quot;analyst&quot;,<br>    system_message=(<br>        &quot;You are a data analyst. Extract structured fields, such as plan names &quot;<br>        &quot;and prices, from the content the researcher provides. Present the &quot;<br>        &quot;result as a clean list. If the researcher&#39;s content has no usable &quot;<br>        &quot;data, say so explicitly instead of guessing.&quot;<br>    ),<br>    llm_config=llm_config,<br>)<br><br>critic = AssistantAgent(<br>    name=&quot;critic&quot;,<br>    system_message=(<br>        &quot;You are a critic. Validate the analyst&#39;s structured output against &quot;<br>        &quot;the researcher&#39;s original content. Flag any gaps, missing fields, or &quot;<br>        &quot;inconsistencies. When the output checks out or no further progress is &quot;<br>        &quot;possible, say so and end your message with TERMINATE.&quot;<br>    ),<br>    llm_config=llm_config,<br>)</pre><p>user_proxy runs any registered function the LLM requests and ends the chat on TERMINATE. Wire them into a group chat and run it:</p><pre>group_chat = GroupChat(<br>    agents=[user_proxy, researcher, analyst, critic],<br>    messages=[],<br>    max_round=6,<br>)<br><br>manager = GroupChatManager(<br>    groupchat=group_chat,<br>    llm_config=llm_config,<br>)<br><br>chat_result = user_proxy.initiate_chat(<br>    manager,<br>    message=(<br>        &quot;Fetch the product listing from this page and extract the &quot;<br>        &quot;product name and price: https://www.walmart.com/ip/AirPods-Pro-3/17835006350&quot;<br>    ),<br>)</pre><p>To get reliable token counts, call autogen.gather_usage_summary() on the full agent list. chat_result.cost reports zero for group chats and isn&#39;t usable.</p><pre>usage_summary = autogen.gather_usage_summary(<br>    [user_proxy, researcher, analyst, critic, manager]<br>)<br>print(usage_summary)</pre><h3>What happens without a fetch tool</h3><p>The researcher’s system message says to fetch web content, but nothing registered can actually do that. The model admits it can’t reach the URL. That admission enters the shared history, the analyst reads it, admits the same, and the critic terminates cleanly:</p><pre>researcher: I&#39;m unable to access external URLs directly to fetch content.<br>analyst: I cannot access external URLs or fetch content from them.<br>critic: Both parties acknowledge the inability to access external URLs. TERMINATE</pre><p>The pipeline completes in four turns and costs almost nothing because it retrieved nothing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/676/1*vNnjpvnyltzAQOyWxxGERA.png" /></figure><p>Low cost and zero output. The next step is what changes that.</p><h3>Register Zenrows as an AG2 tool</h3><p>To give the researcher access to JavaScript-rendered and protected pages, register Zenrows Fetch as a typed tool. AG2 builds the tool schema from your type hints, so annotating the parameters is required.</p><p>Add your Zenrows API key to .env, then define the fetch function:</p><pre>import requests<br>from typing import Annotated<br>from autogen.tools import Tool<br><br>ZENROWS_API_KEY = os.getenv(&quot;ZENROWS_API_KEY&quot;)<br>ZENROWS_ENDPOINT = &quot;https://api.zenrows.com/v1/&quot;<br><br>def fetch_page_content(<br>    url: Annotated[str, &quot;The target URL to fetch through Zenrows using adaptive stealth mode.&quot;],<br>) -&gt; str:<br>    &quot;&quot;&quot;Fetch a URL through Zenrows Fetch and return clean Markdown.&quot;&quot;&quot;<br>    params = {<br>        &quot;url&quot;: url,<br>        &quot;apikey&quot;: ZENROWS_API_KEY,<br>        &quot;mode&quot;: &quot;auto&quot;,<br>        &quot;response_type&quot;: &quot;markdown&quot;,<br>    }<br>    response = requests.get(ZENROWS_ENDPOINT, params=params, timeout=60)<br>    response.raise_for_status()<br>    return response.text</pre><p>mode=auto activates <a href="https://docs.zenrows.com/fetch/features/adaptive-stealth-mode">Adaptive Stealth Mode</a>. Zenrows evaluates each target individually and escalates to JavaScript rendering or premium proxies only when the site requires them. response_type=markdown strips markup so the model reads clean text, keeping the token count proportional to content rather than inflated by raw HTML.</p><p>Wrap the function in an AG2 Tool and register it with both agents:</p><pre>fetch_tool = Tool(<br>    name=&quot;fetch_page_content&quot;,<br>    description=(<br>        &quot;Fetch a URL through Zenrows and return clean Markdown content from &quot;<br>        &quot;JavaScript-rendered and anti-bot-protected pages.&quot;<br>    ),<br>    func_or_tool=fetch_page_content,<br>)<br><br>fetch_tool.register_for_llm(researcher)<br>fetch_tool.register_for_execution(user_proxy)</pre><p>register_for_llm exposes the schema to the researcher so it knows the tool exists. register_for_execution tells AG2 which agent runs the function in a group chat, that&#39;s always the UserProxyAgent.</p><p>Also update the researcher’s system message to name the tool explicitly:</p><pre>researcher = AssistantAgent(<br>    name=&quot;researcher&quot;,<br>    system_message=(<br>        &quot;You are a research agent. When given a URL, use the fetch_page_content &quot;<br>        &quot;tool to retrieve its web content, then summarize what you find. Be &quot;<br>        &quot;factual and note anything that looks incomplete or missing from the &quot;<br>        &quot;page. Pass your summary to the analyst.&quot;<br>    ),<br>    llm_config=llm_config,<br>)</pre><h3>Run the pipeline and compare token spend</h3><p>With Zenrows registered, the researcher calls fetch_page_content, Zenrows retrieves the page, and clean Markdown enters the shared history. The analyst extracts the product name and price. The critic validates and terminates.</p><pre>researcher: ***** Suggested tool call: fetch_page_content *****<br>Arguments: {&quot;url&quot;:&quot;https://www.walmart.com/ip/AirPods-Pro-3/17835006350&quot;}<br>&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt; EXECUTING FUNCTION fetch_page_content...<br><br>analyst:<br>- Product Name: Apple AirPods Pro 3<br>- Price: $189.99 (was $249.00, you save $59.01)<br><br>critic: The analyst&#39;s output is consistent with the researcher&#39;s content. TERMINATE</pre><p>The working run costs more because the Markdown from a fully fetched page is far larger than the researcher’s refusal in the baseline. That token spend is the cost of actual content. The pipeline terminates in about the same number of turns, just with real data at the end.</p><h3>Keeping costs predictable in production</h3><p>Content volume in the shared history is the primary cost driver. Every agent turn re-reads and forwards whatever is in the history, so page size multiplies across the pipeline. max_round sets a ceiling on your worst case, not a cost lever by itself.</p><p>The best way to keep round counts down is to make the fetch layer reliable. Every failed or empty fetch that enters the history costs tokens on every subsequent turn and can push the pipeline toward its round limit without producing anything useful.</p><p>For pipelines that fetch the same URL more than once per session, caching the Zenrows response after the first call cuts both token spend and credit usage. For pipelines that research several URLs in one run, <a href="https://www.zenrows.com/products/batch">Zenrows Batch</a> takes the full list as one managed job and handles concurrency and retries on your behalf.</p><h3>Key takeaways</h3><ul><li>In an AG2 group chat, every tool response enters the shared history. A failed fetch costs tokens on every agent turn that reads it, not just the one that called it.</li><li>AG2 builds the tool schema from type hints. Annotating parameters with Annotated[type, description] is required, not optional.</li><li>mode=auto in the Zenrows request handles JavaScript rendering and proxy escalation per target, without configuration on your side.</li><li>chat_result.cost reports zero for group chats. Use autogen.gather_usage_summary() on the full agent list for accurate token counts.</li><li>The higher token spend in the working run is the cost of real content. The baseline was cheaper because it retrieved nothing.</li></ul><p><em>This article was originally published on the Zenrows blog. Read the original here: </em><a href="https://www.zenrows.com/blog/web-research-multi-agent-ag2-zenrows"><em>https://www.zenrows.com/blog/web-research-multi-agent-ag2-zenrows</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ff1e645a0752" width="1" height="1" alt=""><hr><p><a href="https://medium.com/zenrows/build-a-web-research-multi-agent-system-with-ag2-and-zenrows-ff1e645a0752">Build a web research multi-agent system with AG2 and Zenrows</a> was originally published in <a href="https://medium.com/zenrows">Zenrows</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Web data for LLM fine-tuning, a practical 2026 guide]]></title>
            <link>https://medium.com/zenrows/web-data-for-llm-fine-tuning-a-practical-2026-guide-21db53a70407?source=rss----e3809efdcbcf---4</link>
            <guid isPermaLink="false">https://medium.com/p/21db53a70407</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[web-data-extraction]]></category>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[fine-tuning]]></category>
            <dc:creator><![CDATA[Praise James]]></dc:creator>
            <pubDate>Thu, 10 Sep 2026 10:05:14 GMT</pubDate>
            <atom:updated>2026-09-10T10:05:13.637Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*J5zo6uHs3wjY8ds-Gvcocg.png" /></figure><p>Every fine-tuning cycle that runs on synthetic data alone moves the model slightly further from the real knowledge it was supposed to learn. By the third or fourth retrain, you have a model that reasons well about how domain text sounds rather than about what the domain actually says.</p><p>The fix is a real-data seed, a small, hand-verified set of examples pulled from authoritative sources before any synthetic expansion begins. Small because you can still verify it. Authoritative because the model will take these examples as ground truth. And pulled from sources that standard crawlers cannot reach, because the most valuable domain knowledge lives behind access controls.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/1*PQkFgPrCWuJ2a4PJMDrFoA.png" /></figure><p>This guide covers how to collect that seed, format it, validate it, and measure whether it worked.</p><p><strong>What you’ll learn:</strong></p><ul><li>Why fine-tuning on synthetic data without a real seed leads to drift</li><li>How to pull clean Markdown from protected sources using Zenrows Fetch</li><li>How to handle PDFs and non-HTML domain documents</li><li>How to format scraped content as instruction-output pairs, preference pairs, and domain text</li><li>How to validate the seed and measure whether the fine-tune held</li></ul><p>All the code is on <a href="https://github.com/ZenRows/web-data-for-llm-fine-tuning">GitHub</a>.</p><h3>Why the seed matters more than the volume</h3><p>Fine-tuning follows different statistical dynamics than pretraining. Pretraining spreads noise across billions of tokens, so one bad paragraph disappears into the average. In a fine-tuning set of a few hundred rows, one mislabeled extraction is a measurable percentage of your entire training signal.</p><p>In the standard synthetic pipeline, a generator produces training examples using the teacher model’s patterns, a judge filters them against those same patterns, and the fine-tuned model absorbs both the patterns and their biases. Without a real-data anchor, each retrain compounds the last. The model learns to sound like domain text rather than to reflect domain truth.</p><p>A real-data seed breaks that cycle. It anchors the generator and gives the judge something to filter against that is not its own output. The seed does not have to be large. A few hundred high-quality examples is usually enough for a single extraction task. What matters is that each row came from a source you can vouch for and an extraction you can verify.</p><p>The catch is that authoritative sources are usually the hardest to retrieve. They sit behind JavaScript rendering, IP-based rate limits, and access controls that block standard crawlers. Which is the whole problem this guide solves.</p><h3>Choosing your domain sources</h3><p>Before any code runs, decide what goes into the seed. For an ecommerce model, that means product catalogs and verified customer reviews. For a financial model: SEC filings, earnings transcripts, regulatory guidance. For medical: clinical practice guidelines, drug labels, peer-reviewed research.</p><p>The criteria in every case are the same. Authoritative means trusted by practitioners in that domain, not by a search engine. Structured means the content has consistent patterns your extraction schema can target. Current means the knowledge reflects the domain as it is, not as it was two years ago. And relevant means directly tied to the task, not adjacent to it.</p><p>Skip SEO-driven content farms, AI-generated content, and anything from a source you cannot name when someone asks where the training data came from.</p><h3>Collecting from protected sources</h3><p>A plain requests.get() carries a bare TLS fingerprint and no browser headers. Protected sites flag it immediately. If the source renders with JavaScript, you get an empty shell back.</p><p>Zenrows Fetch handles rendering, access, and Markdown conversion in one call. mode=auto is Adaptive Stealth Mode. Zenrows alternates JavaScript rendering and proxy escalation per request and picks the configuration the target actually needs.</p><pre>import os<br>import requests<br>from dotenv import load_dotenv<br><br>load_dotenv()<br><br>apikey = os.getenv(&quot;ZENROWS_API_KEY&quot;)<br>url = &quot;https://www.scrapingcourse.com/ecommerce/&quot;<br><br>params = {<br>    &quot;url&quot;: url,<br>    &quot;apikey&quot;: apikey,<br>    &quot;mode&quot;: &quot;auto&quot;,<br>    &quot;response_type&quot;: &quot;markdown&quot;,<br>}<br><br>response = requests.get(&quot;https://api.zenrows.com/v1/&quot;, params=params)<br>print(response.text)</pre><p>The output is a smaller, cleaner payload than raw HTML, with most navigation, footer, and boilerplate stripped. That cleaner input means less noise for the model to learn from.</p><h3>Handling PDFs and non-HTML documents</h3><p>Some of the most authoritative domain sources are not web pages. Drug labels, regulatory filings, clinical guidance, and technical standards live in PDFs. Use response_type=pdf for these, and js_render=true which is required for PDF responses.</p><pre>params = {<br>    &quot;url&quot;: url,<br>    &quot;apikey&quot;: apikey,<br>    &quot;js_render&quot;: &quot;true&quot;,<br>    &quot;response_type&quot;: &quot;pdf&quot;,<br>}<br><br>response = requests.get(&quot;https://api.zenrows.com/v1/&quot;, params=params)<br><br>with open(&quot;source-document.pdf&quot;, &quot;wb&quot;) as f:<br>    f.write(response.content)</pre><p>Zenrows retrieves the file but does not extract the text inside it. Use pypdf or pdfplumber to pull the text, then chunk it before it becomes a training example.</p><p>Chunk on structure rather than a fixed character count. The document’s own headings and sections are the natural place to cut, so each chunk keeps its context rather than splitting mid-argument.</p><h3>Tracking provenance</h3><p>For an authoritative seed, you need to know where each row came from. A metadata field with the source URL, retrieval date, and license keeps the dataset auditable and defensible.</p><pre>source = {<br>    &quot;url&quot;: pdf_url,<br>    &quot;retrieved&quot;: date.today().isoformat(),<br>    &quot;license&quot;: license_url,<br>}</pre><p>This matters most when the model’s outputs are questioned. You need to be able to say which row came from which source and whether the source permits the use.</p><h3>Formatting the seed as training examples</h3><p>There are three shapes a fine-tuning example can take, and they teach different things.</p><p><strong>Instruction-output pairs</strong> are the standard for supervised fine-tuning. Each example pairs the page content (the Markdown you pulled) with the correct structured output. The model learns what to extract and how to format it.</p><pre>def to_training_example(instruction, markdown_content, extracted_json):<br>    return {<br>        &quot;messages&quot;: [<br>            {&quot;role&quot;: &quot;system&quot;, &quot;content&quot;: instruction},<br>            {&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: markdown_content},<br>            {&quot;role&quot;: &quot;assistant&quot;, &quot;content&quot;: json.dumps(extracted_json)},<br>        ]<br>    }</pre><p><strong>Preference pairs</strong> are used in DPO and RLHF. Two outputs for the same input, one marked preferred. This is the method for nudging a model that already produces decent outputs toward better ones.</p><pre>def to_preference_pair(prompt, chosen, rejected):<br>    return {<br>        &quot;prompt&quot;: prompt,<br>        &quot;chosen&quot;: chosen,<br>        &quot;rejected&quot;: rejected,<br>    }</pre><p><strong>Domain text continuation</strong> is raw domain text with no input-output split. It familiarizes the model with domain vocabulary and style but is weak for extraction tasks, because the model is never shown what should be extracted.</p><p>For extraction tasks, instruction-output pairs are the right shape. The other two serve different objectives.</p><h3>Validating the seed</h3><p>Not every scraped page belongs in the dataset. Every row must clear three checks before it enters the seed.</p><pre>def keep(markdown, extracted):<br>    if len(markdown.strip()) &lt; 200:<br>        return False<br>    try:<br>        validate(instance=extracted, schema=schema)<br>    except ValidationError:<br>        return False<br>    h = hashlib.sha256(&quot; &quot;.join(markdown.lower().split()).encode()).hexdigest()<br>    if h in seen:<br>        return False<br>    seen.add(h)<br>    return True</pre><p>Length filters out pages that rendered to almost nothing. Schema validation catches wrong types and empty fields. The hash catches exact and formatting-only duplicates. A page has to clear all three before it reaches the training set.</p><h3>Measuring whether it worked</h3><p>Hold out 10 to 20 percent of the real seed before any synthetic expansion and keep those examples out of training. After fine-tuning, run the model against the holdout and measure how many outputs match the expected result.</p><pre>random.seed(42)<br>random.shuffle(seed_examples)<br><br>split = int(len(seed_examples) * 0.85)<br>train_examples = seed_examples[:split]<br>holdout_examples = seed_examples[split:]</pre><p>Run this check after each retrain and record the score over time. A stable or improving score means the seed is doing its job. A declining score is an early sign of drift before it compounds.</p><h3>Key takeaways</h3><ul><li>A fine-tuning seed of a few hundred high-quality real-world examples is more valuable than a large synthetic dataset without one.</li><li>Authoritative sources are almost always behind access controls. Standard crawlers do not reach them.</li><li>mode=auto handles JavaScript rendering and bot detection per request without per-site configuration.</li><li>Chunk PDFs on structure, not on character count, so each chunk stays coherent.</li><li>Validate every row before it enters the seed. One bad extraction is a measurable share of a small training set.</li><li>Hold out real data before synthetic expansion and measure against it after fine-tuning.</li></ul><p>For the wider picture on training data collection, see <a href="https://www.zenrows.com/blog/extract-web-data-for-ai-training">how to extract web data for AI training</a>. For running collection at scale, <a href="https://docs.zenrows.com/batch/introduction">Zenrows Batch</a> manages concurrency and retries across large URL lists.</p><blockquote><em>This article was originally published on the Zenrows blog. Read the original here: </em><a href="https://www.zenrows.com/blog/web-data-llm-fine-tuning"><em>https://www.zenrows.com/blog/web-data-llm-fine-tuning</em></a></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=21db53a70407" width="1" height="1" alt=""><hr><p><a href="https://medium.com/zenrows/web-data-for-llm-fine-tuning-a-practical-2026-guide-21db53a70407">Web data for LLM fine-tuning, a practical 2026 guide</a> was originally published in <a href="https://medium.com/zenrows">Zenrows</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Scraping 2026 FIFA World Cup data with Python and Zenrows]]></title>
            <link>https://medium.com/zenrows/scraping-2026-fifa-world-cup-data-with-python-and-zenrows-20369f98cb5b?source=rss----e3809efdcbcf---4</link>
            <guid isPermaLink="false">https://medium.com/p/20369f98cb5b</guid>
            <category><![CDATA[web-data-extraction]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[2026-fifa-world-cup]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[python-webscraping]]></category>
            <dc:creator><![CDATA[Praise James]]></dc:creator>
            <pubDate>Tue, 08 Sep 2026 13:57:24 GMT</pubDate>
            <atom:updated>2026-09-08T13:57:23.117Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IV-FyEH7LuPQk3qpD1Z2dQ.png" /></figure><p>Ousmane Dembélé finished with six goals from an expected goals figure of just over two. FIFA gave the Golden Glove to Unai Simón, who doesn’t appear in the top ten goalkeepers in our dataset. Japan recorded the highest shot conversion rate in the tournament while taking the lowest share of its shots from inside the penalty area.</p><p>We found all of that by building a Python pipeline over Sofascore, FIFA, and Wikipedia. Getting the data was the hard part.</p><p><strong>What you’ll learn</strong></p><ul><li>How we retrieved data from a JavaScript endpoint, an authenticated API, and server-rendered HTML using three different extraction strategies</li><li>Where Zenrows handled what plain HTTP requests could not</li><li>15 findings from the 2026 World Cup that challenged what we expected</li></ul><p>The full pipeline is on <a href="https://github.com/ZenRows/scraping-world-cup-data">GitHub</a>.</p><h3>The pipeline problem</h3><p>No single site exposed every statistic we wanted. FIFA had team-level metrics. Sofascore had player-level statistics. Wikipedia had tournament metadata. Each delivered its data differently, so each needed a different extraction strategy.</p><p>Standard Python scraping tools handle one of these well. None handle all three consistently, because the access patterns are fundamentally different.</p><h3>Retrieving Sofascore data from a JavaScript endpoint</h3><p>Sofascore is built as a single-page application. Watching the network tab while navigating the stats page showed the front end wasn’t rendering HTML server-side. It was calling an internal API, and JavaScript was painting the result into the browser.</p><p>requests alone would have returned the application shell with no data inside it. We used Zenrows Fetch to pull JSON straight from the endpoint, switching group and order parameters to pull different statistical categories.</p><p>Because the endpoint returns JSON rather than HTML, there was no parsing step. The response was already structured data.</p><pre>response = requests.get(<br>    &quot;https://api.zenrows.com/v1/&quot;,<br>    params={<br>        &quot;apikey&quot;: ZENROWS_API_KEY,<br>        &quot;url&quot;: target_url,<br>        &quot;mode&quot;: &quot;auto&quot;,<br>    },<br>    timeout=120,<br>)<br>return response.json()</pre><p>mode=autohandled JavaScript rendering and premium proxy escalation without us specifying which the endpoint needed per request.</p><h3>Retrieving FIFA data from an authenticated API</h3><p>The FIFA data looked similar to Sofascore’s but the access pattern was different. The data came from a backend API, and that API was called with authentication.</p><p>Every request carried an authorization header with a JWT bearer token issued for the current browsing session. Every visitor gets a temporary token and the front end attaches it to subsequent calls. The endpoint couldn’t be queried as a public URL, so Fetch alone wasn’t enough.</p><p>We used Zenrows Browser Sessions with Playwright to follow the same authenticated flow a normal browser follows, and read the API’s JSON off the responses. The browser session maintained the token across requests so the authentication held throughout the scrape.</p><pre>async with async_playwright() as p:<br>    browser = await p.chromium.connect_over_cdp(self.connection_url)<br>    context = browser.contexts[0]<br>    page = await context.new_page()<br>    page.on(&quot;response&quot;, self._handle_response)<br>    await page.goto(self.URL)</pre><h3>Retrieving Wikipedia tournament metadata</h3><p>Wikipedia gave us tournament metadata covering host countries, stadiums, and venues. Unlike the other two it’s server-rendered, so everything we needed was in the HTML the server returned. Fetch retrieved it with no extra capabilities.</p><p>The normalization step needed BeautifulSoup to locate the stadium summary table by its headers and extract clean records from it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/892/1*ZF2cm7AfS74Grr7IOTSlBQ.png" /></figure><h3>What the data revealed</h3><p>Fifteen findings came out of the normalized dataset. Several challenged what we expected going in.</p><p><strong>Dembélé was the most efficient of the top strikers.</strong> Kylian Mbappé finished with 10 goals backed by the highest expected goals figure in the tournament. Dembélé had considerably fewer expected goals than any of the other top scorers and still finished with six, close to three times his xG.</p><p><strong>Midfielders dominated the defensive statistics.</strong> You’d expect centre-backs to top defensive actions. Instead Jude Bellingham, Rodri, Pedro Vite, and Aurélien Tchouaméni took the top four places once tackles, interceptions, and clearances were counted together.</p><p><strong>The Golden Glove went to a goalkeeper outside our top ten.</strong> FIFA gave Unai Simón the award as the tournament’s best goalkeeper. He doesn’t appear among the ten highest-rated goalkeepers in our dataset. Orlando Gill tops our ratings despite Paraguay going out in the Round of 16, and he also led all goalkeepers in total saves.</p><p><strong>Michael Olise held a top passing accuracy while creating the most chances.</strong> Creative players usually trade accuracy for risk because the passes that break defensive lines are the ones most likely to fail. Olise held one of the tournament’s highest passing accuracies and finished second for key passes. He also created more big chances than any other player and finished with seven assists, a new World Cup record.</p><p><strong>Brian Brobbey didn’t miss a single big chance.</strong> The Netherlands striker recorded a 75% goal conversion rate without missing one clear-cut opportunity. In a tournament where most leading forwards needed several chances per goal, that stands out.</p><p><strong>Japan recorded the highest shot conversion rate while taking the fewest shots from inside the box.</strong> Japan wasn’t counted among the attacking heavyweights and still led the tournament in conversion rate. Only 50% of their shots came from inside the penalty area, the lowest share of the top ten.</p><p><strong>More pressing didn’t mean quicker ball recovery.</strong> France, England, and Argentina recorded the most direct defensive pressures and also some of the longest ball recovery times. Türkiye applied far fewer pressures and recovered the ball among the fastest.</p><p><strong>Argentina led the tournament in bookings and reached the final without a VAR intervention against them.</strong> They applied the most defensive pressure and collected the most bookings. The VAR record is the part that stands out.</p><p><strong>Spain broke more defensive lines than anyone, without losing passing accuracy.</strong> The usual assumption is that attempting more line-breaking passes costs you accuracy. Spain recorded the most defensive line-break attempts while holding around 90% passing accuracy.</p><p><strong>The first three-country World Cup used fewer stadiums than 1982.</strong> The 2026 tournament was hosted by the USA, Canada, and Mexico combined and used 16 stadiums, fewer than the 17 Spain used alone in 1982.</p><p>The remaining findings on England’s ball recovery, France and Spain’s contrasting styles, Spain’s off-ball movement, goalkeeper workloads, and Brobbey’s efficiency are in the <a href="https://www.zenrows.com/blog/scrape-2026-fifa-world-cup-data-python">full article</a> with the charts.</p><h3>What the pipeline taught us</h3><p>Building reliable data collection across three sources with three different access patterns took more than sending HTTP requests.</p><p>Fetch covered the JavaScript-rendered JSON endpoint and the server-rendered HTML. Browser Sessions covered the authenticated API, because following an authenticated request flow needs a real browser. From there it was normalization into a consistent format and visualizations over the result.</p><p>The thing that would have broken a standard Python scraper at each stage was the thing the pipeline was designed around from the start. JavaScript rendering on Sofascore, session-based authentication on FIFA, and HTML table parsing on Wikipedia each needed its own approach.</p><p>The full source is on <a href="https://github.com/ZenRows/scraping-world-cup-data">GitHub</a>. From here you could extend it with match-level event data or feed the normalized datasets into dashboards or an AI application.</p><blockquote><em>This article was originally published on the Zenrows blog. Read the original here: </em><a href="https://www.zenrows.com/blog/scrape-2026-fifa-world-cup-data-python"><em>https://www.zenrows.com/blog/scrape-2026-fifa-world-cup-data-python</em></a></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=20369f98cb5b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/zenrows/scraping-2026-fifa-world-cup-data-with-python-and-zenrows-20369f98cb5b">Scraping 2026 FIFA World Cup data with Python and Zenrows</a> was originally published in <a href="https://medium.com/zenrows">Zenrows</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>