Cloudflare & Anti-Bot Fetcher – Gap-Filler for Crawlers

Reliable Markdown from difficult URLs for AI agents and RAG. Give it a list of URLs -- known-hard ones, or just a normal batch -- and get clean Markdown back, using fast HTTP by default and automatic stealth escalation only when a page is actually blocked.

Works two ways, equally well:

• Standalone: hand it any list of URLs. Most resolve on the cheap fast tier; the hard ones (Cloudflare, DataDome, PerimeterX-style protection) automatically escalate to a stealth browser, without you paying stealth prices for pages that didn't need it.
• As a fallback step: already running Website Content Crawler, RAG Web Browser, or any other scraper, and some pages came back blocked or empty? Point this Actor at that run's dataset instead of re-running your whole crawl -- see Gap-filler mode below.

Have a Website Content Crawler or RAG Web Browser dataset with failed or blocked pages sitting in Storage? Skip straight to Gap-filler mode below -- pick that dataset and get results in under 90 seconds, no need to re-run your whole crawl.

🎬 See it in action

Fast → blocked → automatic stealth → HTTP 200 → clean Markdown

The Actor starts with a lightweight fetch and automatically escalates to a stealth browser only when a site actually blocks it. You can also force stealth mode for sites known to require a browser.

---

Why use it?

⚡ Fast first, stealth only when needed

URL
 ↓
Fast HTTP fetch
 ↓
Works? ─────────────── Yes → Clean Markdown
 ↓ No
Stealth browser (Cloudflare/anti-bot escalation)
 ↓
Clean Markdown

Verified results from real Actor runs:

Task	Result	Tier used	Extraction
example.com	HTTP 200	fast	title → "Example Domain"
MDN	HTTP 200	fast	title → "Add JavaScript to your web page"
Cloudflare Challenge page	HTTP 200	stealth	title → "Cloudflare Challenge"
Cloudflare Challenge, via WCC's dataset (Gap-filler)	HTTP 200	stealth	Recovered after WCC itself failed -- 10 retries + UNBLOCKER proxy escalation, still blocked
Cloudflare Challenge, via RAG Web Browser's dataset (Gap-filler)	HTTP 200	stealth	Recovered after RAG Web Browser itself failed -- blocked in under 3 seconds

Every row above is a successful extraction — the Cloudflare-protected page returned a real HTTP 200 via automatic stealth escalation, not a placeholder or partial result.

You can also set forceStealth: true for sites already known to require a browser.

---

💰 Pricing

Currently free to use while this Actor builds up usage history and reviews. Pay-per-event pricing (below) reflects what it will move to -- watch this Actor's Pricing tab in Console for its current live status.

Free allowance, permanent even after PPE is enabled: 300 fast-tier pages and 25 stealth-tier pages every month, on every account — enough to fully evaluate this Actor before spending anything.

What triggers a charge	Price	Free allowance / month
Actor start	$0.005	—
Fast-tier page (no anti-bot resistance needed)	$0.002	300
Stealth-tier page (Cloudflare/anti-bot escalation)	$0.02	25
Structured field extracted (optional, via extractionSchema)	$0.001	—

You're only ever charged the stealth price for pages that actually needed it — a page resolved on the fast tier is never billed at the stealth rate.

Example volume (typical mix)	Estimated cost
1,000 pages, ~80% fast / 20% stealth	~$5.60
10,000 pages, ~80% fast / 20% stealth	~$56.00

How this compares: the fast tier is priced close to general-purpose crawlers' cheapest tier, while including automatic escalation as a built-in option rather than a separate manual step. The stealth tier costs more than a generic headless-browser tier elsewhere -- that difference reflects automatic Cloudflare/DataDome-style challenge solving with a residential proxy, not just JS rendering.

---

🧹 Clean, LLM-ready Markdown

Turn web pages into useful Markdown instead of feeding your AI pipeline raw HTML, navigation, and page chrome.

The Markdown output preserves headings, paragraphs, links, lists/tables, and code blocks — suitable for RAG pipelines, embeddings, AI agents, research, content monitoring, and knowledge-base ingestion.

---

🎯 Optional structured extraction

Request additional fields using simple CSS selectors:

[
  { "name": "title", "selector": "h1" },
  { "name": "author", "selector": "[rel='author']" }
]

{
  "extracted": {
    "title": "Example Domain",
    "author": "John Doe"
  }
}

Adaptive (self-healing) extraction

Fields extracted via extractionSchema use Scrapling's adaptive-relocation mechanism: when the exact selector stops matching, the field is relocated by DOM similarity instead of returning null. Verified end-to-end in Showcase 5, including the actual similarity score behind a real recovery (100% match). Still validate adaptive matching against your own target sites before relying on it unattended in production -- one verified site is a proof of the mechanism, not a guarantee across every possible page structure.

---

🔌 Gap-filler mode: catch what other crawlers missed

This is the fastest way to get value from this Actor if you already use a general-purpose crawler. Website Content Crawler and RAG Web Browser are excellent at broad site ingestion, but neither automatically solves Cloudflare-style managed challenges -- WCC's own troubleshooting docs recommend manually switching crawler type and proxy group when blocked, rather than doing it automatically. That gap is exactly what this Actor is built to close.

Tool	Best for
Website Content Crawler	Broad, multi-page site crawls -- deep-crawling a whole domain
RAG Web Browser	Search-to-content workflows and quick single-URL fetches
This Actor	The pages either of those got blocked on -- Cloudflare/anti-bot escalation, not a general-purpose crawler

Not a replacement for either -- built to catch what they can't reach.

Point this Actor at that run's dataset, and it will scan for pages that came back blocked or thin, then retry just those URLs through the fast->stealth escalation ladder -- no need to re-run your whole crawl, and no need to switch tools for the pages that failed.

In the Apify Console input form, sourceDatasetId is a picker field -- but its search list generally only shows datasets you've explicitly named. Default/unnamed run datasets (which is what most actors, including WCC and RAG Web Browser, actually produce) usually won't appear in the dropdown even though they work correctly. If yours isn't listed, copy the dataset ID from Console (that run's Storage tab, or defaultDatasetId in the run's API response) and paste it directly into the picker's search box -- a valid pasted ID is accepted even when it's not shown as a suggestion. Via the API/JSON input, just pass the dataset ID as a plain string:

{
  "sourceDatasetId": "your-prior-run-dataset-id",
  "sourceActorHint": "website-content-crawler"
}

You can combine this with your own startUrls in the same run -- both are merged (deduplicated) before processing.

Works out of the box with:

• Website Content Crawler
• RAG Web Browser

Works with anything else via manual field mapping -- specify exactly which fields hold the URL and content:

{
  "sourceDatasetId": "your-prior-run-dataset-id",
  "sourceFieldMapping": {
    "urlField": "pageUrl",
    "contentField": "body",
    "minContentLength": 50
  }
}

urlField/contentField support dotted paths for nested fields (e.g. "metadata.url"). If you provide neither a recognized sourceActorHint nor a sourceFieldMapping, a generic fallback scans the item for a URL-shaped field and treats the longest text field as content -- less precise, but works on schemas we've never seen.

Scope limitation, stated plainly: this only works on datasets your own Apify account already has access to -- your own prior runs, not other users' private data. The workflow is: you run WCC (or similar) yourself first, then point this Actor at that same run's dataset ID.

---

🚀 Quick start

{ "startUrls": [{ "url": "https://example.com" }] }

Force stealth for a site known to require a browser:

{ "startUrls": [{ "url": "https://example.com" }], "forceStealth": true }

With structured extraction:

{
  "startUrls": [{ "url": "https://example.com" }],
  "extractionSchema": [{ "name": "title", "selector": "h1" }]
}

---

🧪 Showcases

Each showcase includes the copyable input and the actual output from a real Actor run, so you can compare your own run against it.

1. Normal page → fast fetch

Input

{
  "startUrls": [{ "url": "https://example.com" }],
  "extractionSchema": [{ "name": "title", "selector": "h1" }],
  "forceStealth": false,
  "maxConcurrency": 1,
  "maxUrlsPerRun": 1,
  "fetchTimeoutSecs": 150,
  "proxyConfiguration": { "useApifyProxy": false }
}

Output

{
  "url": "https://example.com",
  "success": true,
  "statusCode": 200,
  "fetchTier": "fast",
  "extracted": { "title": "Example Domain" }
}

2. MDN → fast fetch + structured extraction

Input

{
  "startUrls": [{ "url": "https://developer.mozilla.org/en-US/docs/Web/HTML/How_to/Add_JavaScript_to_your_web_page" }],
  "extractionSchema": [{ "name": "title", "selector": "h1" }],
  "forceStealth": false,
  "maxConcurrency": 1,
  "maxUrlsPerRun": 1,
  "fetchTimeoutSecs": 150,
  "proxyConfiguration": { "useApifyProxy": false }
}

Output

{
  "url": "https://developer.mozilla.org/en-US/docs/Web/HTML/How_to/Add_JavaScript_to_your_web_page",
  "success": true,
  "statusCode": 200,
  "fetchTier": "fast",
  "extracted": { "title": "Add JavaScript to your web page" }
}

3. Cloudflare Challenge → automatic stealth escalation

Input

{
  "startUrls": [{ "url": "https://www.scrapingcourse.com/cloudflare-challenge" }],
  "extractionSchema": [{ "name": "title", "selector": "h1" }],
  "forceStealth": false,
  "markdownThreshold": 45,
  "maxConcurrency": 1,
  "maxUrlsPerRun": 1,
  "fetchTimeoutSecs": 150,
  "proxyConfiguration": { "useApifyProxy": false }
}

Output

{
  "url": "https://www.scrapingcourse.com/cloudflare-challenge",
  "success": true,
  "statusCode": 200,
  "fetchTier": "stealth",
  "protectionDetected": "none",
  "extracted": { "title": "Cloudflare Challenge" },
  "markdown": "# Cloudflare Challenge\n## You bypassed the Cloudflare challenge! :D"
}

Note on proxy settings above: examples 1–3 deliberately run with useApifyProxy: false to demonstrate the escalation ladder working even in the hardest case — no proxy at all. For production use, set useApifyProxy: true (the input schema default) — see Example 4. Real testing has shown this matters: without a proxy, the same Cloudflare-protected URL has succeeded in as little as 5-10 seconds in one run, and needed multiple internal solve-retry cycles (spanning nearly a minute) in another -- both against the exact same target. A residential proxy makes this far more consistent, not just theoretically safer.

4. Recommended production config (residential proxy)

Input

{
  "startUrls": [{ "url": "https://www.scrapingcourse.com/cloudflare-challenge" }],
  "extractionSchema": [{ "name": "title", "selector": "h1" }],
  "forceStealth": false,
  "markdownThreshold": 45,
  "maxConcurrency": 1,
  "maxUrlsPerRun": 1,
  "fetchTimeoutSecs": 150,
  "proxyConfiguration": { "useApifyProxy": true }
}

Output

{
  "url": "https://www.scrapingcourse.com/cloudflare-challenge",
  "success": true,
  "statusCode": 200,
  "fetchTier": "stealth",
  "protectionDetected": "none",
  "extracted": { "title": "Cloudflare Challenge" },
  "markdown": "# Cloudflare Challenge\n## You bypassed the Cloudflare challenge! :D"
}

Same successful result as Example 3, confirmed with the recommended proxy setting.

5. Self-healing in action -- verified

Two-run test: Run A seeds a fingerprint with a working selector, Run B repeats the same URL and field name with a deliberately broken selector that matches nothing on the page.

Run A -- seed a fingerprint

{
  "startUrls": [{ "url": "https://developer.mozilla.org/en-US/docs/Web/HTML/How_to/Add_JavaScript_to_your_web_page" }],
  "extractionSchema": [{ "name": "heading", "selector": "h1" }],
  "forceStealth": false,
  "maxConcurrency": 1,
  "maxUrlsPerRun": 1,
  "fetchTimeoutSecs": 150,
  "proxyConfiguration": { "useApifyProxy": true }
}

{ "success": true, "statusCode": 200, "fetchTier": "fast", "extracted": { "heading": "Add JavaScript to your web page" } }

Run B -- same URL/field, selector changed to h1.this-class-no-longer-exists

{
  "startUrls": [{ "url": "https://developer.mozilla.org/en-US/docs/Web/HTML/How_to/Add_JavaScript_to_your_web_page" }],
  "extractionSchema": [{ "name": "heading", "selector": "h1.this-class-no-longer-exists" }],
  "forceStealth": false,
  "maxConcurrency": 1,
  "maxUrlsPerRun": 1,
  "fetchTimeoutSecs": 150,
  "proxyConfiguration": { "useApifyProxy": true }
}

{ "success": true, "statusCode": 200, "fetchTier": "fast", "extracted": { "heading": "Add JavaScript to your web page" } }

The field was recovered even though the selector matched nothing on the page. This isn't just a pass/fail result -- the run log shows the actual relocation mechanism at work:

DEBUG: Highest probability was 100.0%
DEBUG: Top 5 best matching elements are:
100.0 -> [<data='<h1>Add JavaScript to your web page</h1>' parent='<div class="layout__header...'>]
62.24 -> [<data='<mdn-survey></mdn-survey>' ...>]
49.28 -> [<data='<p> Your blueprint for a better internet...' ...>]

The correct <h1> scored a perfect 100% match against the fingerprint saved in Run A, comfortably ahead of the next-best candidate (62%) -- a clean, unambiguous relocation, not a lucky guess near the threshold.

6. Graceful failure handling

Input

{
  "startUrls": [{ "url": "https://this-domain-does-not-exist-abcxyz123.com" }],
  "extractionSchema": [],
  "forceStealth": false,
  "maxConcurrency": 1,
  "maxUrlsPerRun": 1,
  "fetchTimeoutSecs": 30,
  "proxyConfiguration": { "useApifyProxy": true }
}

Output

{
  "url": "https://this-domain-does-not-exist-abcxyz123.com",
  "success": false,
  "error": "Error: Page.goto: net::ERR_TUNNEL_CONNECTION_FAILED at https://this-domain-does-not-exist-abcxyz123.com/"
}

The run completed normally with success: false and a real, actionable error message — it did not crash or block other URLs in the same run.

Re-verified after protectionDetected was added: the field is correctly absent here, not defaulted to "none". This connection never got far enough to receive any response at all (a DNS/tunnel-level failure), so there was nothing to inspect for anti-bot challenges -- "none" would incorrectly claim "checked, found clean," when the honest answer is "never got that far." statusCode and fetchTier are absent for the same reason.

A failed or timed-out URL is recorded with success: false and an error message — it never stops the rest of a multi-URL run.

7. Gap-filler mode — verified against two real crawlers

Both tested end-to-end against the same Cloudflare-gated URL, each blocked by a different crawler first, then correctly detected and recovered via sourceDatasetId -- not a coincidence of also passing the URL in startUrls.

Case A: Website Content Crawler

WCC's own log shows it hit a real wall: 10 retries, escalation to Apify's UNBLOCKER proxy group, still 403 on every attempt, then a hard Timeout 60000ms exceeded -- 0 succeeded, 1 failed, ~135 seconds spent.

Pointing this Actor at that run's dataset:

{ "sourceDatasetId": "<WCC run's dataset ID>", "sourceActorHint": "website-content-crawler" }

Gap-filler: found 1 failed/thin-content URL(s) in source dataset '...', 1 of which are new (not already in startUrls).
...
Done: 1/1 succeeded. Stealth tier used on 1/1 URLs.

{ "success": true, "statusCode": 200, "fetchTier": "stealth", "extracted": { "title": "Cloudflare Challenge" } }

Case B: RAG Web Browser

RAG Web Browser also failed on the same URL -- a clean 403 in under 3 seconds (it doesn't retry as aggressively as WCC), recorded in its dataset as crawl.requestStatus: "failed" with empty text.

Same gap-filler call, sourceActorHint: "rag-web-browser":

Gap-filler: found 1 failed/thin-content URL(s) in source dataset '...', 1 of which are new (not already in startUrls).
...
Done: 1/1 succeeded. Stealth tier used on 1/1 URLs.

{ "success": true, "statusCode": 200, "fetchTier": "stealth", "extracted": { "title": "Cloudflare Challenge" } }

What this confirms: gap-filler mode correctly reads both crawlers' native dataset formats without manual field mapping, correctly identifies the specific failed item (not a lucky guess), and successfully retries just that URL through the stealth escalation ladder -- recovering a real 200 where both general-purpose crawlers returned nothing.

Stealth-tier success against hard-gated sites is not guaranteed on every attempt -- see the FAQ below ("Will this work on any Cloudflare-protected site?").

---

📤 Output

Each URL produces a result containing url, fetchedAt, and success. On a response being received, it also includes statusCode, fetchTier (fast or stealth), protectionDetected (cloudflare | datadome | perimeterx | bot-challenge | none), markdown, and extracted. On failure, error is included instead -- and if the failure happened before any response was received at all (e.g. DNS/connection errors), statusCode, fetchTier, and protectionDetected are omitted entirely rather than defaulted to a guessed value -- see Showcase 6 for a real example.

protectionDetected tells you which anti-bot system, if any, was identified on the final response -- useful for measuring block rates and spotting which domains need special handling across a batch. On a successful stealth-tier fetch, none confirms the challenge was actually cleared, not just that the solve attempt didn't error out.

{
  "url": "https://example.com",
  "fetchedAt": "2026-08-16T14:21:22.097540+00:00",
  "success": true,
  "statusCode": 200,
  "fetchTier": "fast",
  "protectionDetected": "none",
  "markdown": "# Example Domain\n\nThis domain is for use in documentation examples...",
  "extracted": {
    "title": "Example Domain",
    "paragraph1": "This domain is for use in documentation examples..."
  }
}

Note on the showcases above: Showcases 3 and 4 (Cloudflare bypass) have been re-run and verified with real protectionDetected values ("none", confirming the challenge was actually cleared). Showcase 6 has also been re-verified -- its correct behavior is the field being absent, not "none" (see Showcase 6 for why). Showcases 1, 2, 5, and 7 were captured before this field was added and don't show it yet -- the detection logic itself is deterministic and unit-tested, but those specific output blocks will only be updated here once re-run and captured for real, not backfilled with a plausible-looking guess.

---

When should I use this?

Good fit

• AI / RAG — ingest documentation, public information, and web content
• AI agents — provide agents with clean web-page content
• Research — collect readable content from multiple websites
• Competitive intelligence — monitor pages that may be difficult to fetch
• SEO & agencies — recurring content and page audits
• Sales & RevOps — collect public company and product information
• Automation — use the results in n8n, LangChain, CrewAI, MCP, or custom workflows
• Already using Website Content Crawler, RAG Web Browser, or similar — use Gap-filler mode to catch the pages they couldn't reach, without re-running your whole crawl

When to use something else

If you need...	Consider instead
Login-only / account-gated pages	A custom Actor with session/auth handling — this Actor only accesses publicly reachable pages
Structured data from a specific major site (LinkedIn, Amazon, etc.)	A dedicated site-specific Actor — usually cheaper and more structured than generic CSS selectors
Guaranteed success on every request	No tool can promise this — sites change and anti-bot systems evolve
Pages with no anti-bot protection at all	A simpler/cheaper HTTP-only crawler may be more cost-effective than paying for stealth-tier headroom you won't use

---

❓ FAQ

Do I need my own proxy? No — Apify Proxy is used by default and is strongly recommended, especially for the stealth tier (see Example 4). You can supply your own proxy if you have an existing contract.

Is this free to try? Yes — every account gets 300 free fast-tier pages and 25 free stealth-tier pages every month, permanently, not just during a trial window.

What happens if a page fails or times out? It's recorded with success: false and an error field (see Showcase 6). The rest of the run continues normally.

Will this work on any Cloudflare-protected site? The stealth tier is built to solve Cloudflare-style managed and embedded challenges automatically. Results can vary by target — validate against your specific site before relying on it for unattended production jobs.

Can this fill in the gaps from a Website Content Crawler or RAG Web Browser run? Yes -- see "Gap-filler mode" above. Point sourceDatasetId at a prior run's dataset and it retries the failed/thin-content pages through the stealth escalation ladder. Only works on datasets your own account already has access to.

Is the "self-healing" claim real, or marketing? Real and verified -- see Showcase 5 for a reproducible before/after test, including the actual relocation confidence score (100% match) from a real run log, not just a claimed pass/fail. It's not infallible on every possible page: validate it against your own target sites, especially pages with many visually similar elements.

---

⚙️ Input

Field	Purpose
startUrls	URLs to fetch
extractionSchema	Optional {name, selector} fields, matched adaptively
forceStealth	Skip directly to the stealth tier
markdownThreshold	Markdown boilerplate pruning aggressiveness
maxConcurrency	Parallel URL limit
maxUrlsPerRun	Maximum URLs processed per run
fetchTimeoutSecs	Per-URL timeout
proxyConfiguration	Apify Proxy (recommended) or your own proxy
sourceDatasetId	Optional. Dataset ID from a prior run of another actor -- enables gap-filler mode
sourceActorHint	Optional. auto (default), website-content-crawler, or rag-web-browser
sourceFieldMapping	Optional. Manual {urlField, contentField, minContentLength} for unrecognized source actors

---

🧩 Built for AI and automation

Web page → Clean Markdown → RAG / embeddings / AI agent / automation

Compatible with n8n, LangChain, CrewAI, MCP, and custom Python/JavaScript workflows.

---

Open source

Markdown generation uses Crawl4AI (Apache 2.0).

This product includes software developed by UncleCode (https://x.com/unclecode) as part of the Crawl4AI project (https://github.com/unclecode/crawl4ai).

Stealth fetching and adaptive extraction use Scrapling (BSD-3-Clause), with Camoufox (MPL-2.0) powering the stealth-browser tier.

---

Acceptable use

You are responsible for complying with each target site's Terms of Service, robots.txt, applicable laws, and Apify's Acceptable Use Policy. This Actor provides a general-purpose fetching capability — use stealth features only where you have the right to access the target content.

---

Try it

Give it a normal URL and let it stay fast. Give it a difficult or protected URL and let it escalate when necessary.

Fast when possible. Stealth when needed. Clean Markdown every time.