# Web Scraper For Llms (`abotapi/web-scraper-for-llms`) Actor

Stealth web scraping engine built for LLMs. Converts any web page to clean markdown or HTML

- **URL**: https://apify.com/abotapi/web-scraper-for-llms.md
- **Developed by:** [Abot API](https://apify.com/abotapi) (community)
- **Categories:** AI, Developer tools, Automation
- **Stats:** 46 total users, 0 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.00 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Web Scraper for LLMs

Stealth web scraping engine built for LLMs. Converts any web page to clean markdown or HTML, ready for RAG pipelines, AI knowledge bases, and content analysis. Automatically bypasses Cloudflare and anti-bot protection using a stealth browser with undetectable fingerprints.

### Quick Start

Scrape a list of URLs:

```json
{
  "urls": ["https://example.com/", "https://medium.com/"]
}
```

Crawl a website and scrape all discovered pages:

```json
{
  "urls": ["https://docs.example.com/"],
  "crawl": true,
  "crawlDepth": 2,
  "crawlMaxPages": 50
}
```

### Input Parameters

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `urls` | Array | *required* | URLs to scrape or crawl from |
| `crawl` | Boolean | `false` | Follow links to discover additional pages |
| `crawlDepth` | Integer | `1` | Link hops from seed URL (crawl only) |
| `crawlMaxPages` | Integer | `20` | Max pages to discover per seed (crawl only) |
| `formats` | Array | `["markdown"]` | Output formats: `markdown`, `html`, or both |
| `concurrency` | Integer | `3` | Parallel URL processing |
| `maxRetries` | Integer | `2` | Retry attempts for failed URLs (scrape only) |
| `timeoutMs` | Integer | `30000` | Timeout per URL in milliseconds |
| `onlyMainContent` | Boolean | `true` | Strip nav/header/footer/sidebar (scrape only) |
| `removeAds` | Boolean | `true` | Remove ads and tracking elements |
| `removeBase64Images` | Boolean | `true` | Remove inline base64 images |
| `includeTags` | Array | - | CSS selectors to keep (scrape only) |
| `excludeTags` | Array | - | CSS selectors to remove (scrape only) |
| `includePatterns` | Array | - | Regex URL filters (include only matching) |
| `excludePatterns` | Array | - | Regex URL filters (skip matching) |
| `waitForSelector` | String | - | Wait for CSS selector before extraction (scrape only) |
| `proxyConfiguration` | Object | - | Apify proxy settings |

### Output

```json
{
  "url": "https://medium.com/",
  "title": "Medium: Read and write stories.",
  "description": null,
  "markdown": "## Human stories & ideas\n\nA place to read, write, and deepen your understanding...",
  "html": null,
  "metadata": {
    "title": "Medium: Read and write stories.",
    "language": "en",
    "favicon": "https://miro.medium.com/...",
    "canonical": "https://medium.com/",
    "openGraph": null,
    "twitter": null
  },
  "duration": 5725,
  "scrapedAt": "2026-02-24T03:36:28.990Z",
  "success": true,
  "error": null
}
```

### Use Cases

- **RAG pipelines** - Feed clean markdown into LLM knowledge bases
- **Content monitoring** - Track changes across a set of pages
- **Research** - Bulk extract articles, documentation, or product pages
- **Site migration** - Crawl and export an entire site as markdown
- **Data extraction** - Scrape structured content from specific CSS selectors

# Actor input Schema

## `urls` (type: `array`):

URLs to scrape. When crawl is enabled, each URL is used as a seed to discover linked pages.

## `crawl` (type: `boolean`):

When enabled, the scraper will follow links from each URL to discover and scrape additional pages. Use Crawl Depth and Max Pages below to control how far it goes.

## `crawlDepth` (type: `integer`):

How many link hops from the starting URL. Ignored when crawl is off.

## `crawlMaxPages` (type: `integer`):

Maximum pages to discover per seed URL. Ignored when crawl is off.

## `formats` (type: `array`):

Which content formats to extract.

## `concurrency` (type: `integer`):

Number of URLs to process in parallel.

## `maxRetries` (type: `integer`):

Maximum retry attempts for failed URLs. Ignored when crawl is on.

## `timeoutMs` (type: `integer`):

Request timeout per URL in milliseconds.

## `onlyMainContent` (type: `boolean`):

Extract only main content, removing nav/header/footer/sidebar. Ignored when crawl is on.

## `removeAds` (type: `boolean`):

Remove ads and tracking elements.

## `removeBase64Images` (type: `boolean`):

Remove base64-encoded images to reduce output size.

## `includeTags` (type: `array`):

CSS selectors for elements to include. If set, only matching elements are kept. Ignored when crawl is on.

## `excludeTags` (type: `array`):

CSS selectors for elements to exclude from output. Ignored when crawl is on.

## `includePatterns` (type: `array`):

Regex patterns for URL filtering. In crawl mode, controls which discovered links to follow.

## `excludePatterns` (type: `array`):

Regex patterns for URL filtering. In crawl mode, controls which discovered links to skip.

## `waitForSelector` (type: `string`):

CSS selector to wait for before considering page loaded. Ignored when crawl is on.

## `proxyConfiguration` (type: `object`):

Apify proxy configuration.

## Actor input object example

```json
{
  "urls": [
    "https://example.com/"
  ],
  "crawl": false,
  "crawlDepth": 1,
  "crawlMaxPages": 20,
  "formats": [
    "markdown"
  ],
  "concurrency": 3,
  "maxRetries": 2,
  "timeoutMs": 30000,
  "onlyMainContent": true,
  "removeAds": true,
  "removeBase64Images": true
}
```

# Actor output Schema

## `scrapedPages` (type: `string`):

Each scraped page as a dataset row with URL, content, metadata, and status.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "urls": [
        "https://example.com/"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("abotapi/web-scraper-for-llms").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "urls": ["https://example.com/"] }

# Run the Actor and wait for it to finish
run = client.actor("abotapi/web-scraper-for-llms").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "urls": [
    "https://example.com/"
  ]
}' |
apify call abotapi/web-scraper-for-llms --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,abotapi/web-scraper-for-llms"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/fkaqHAofsFCxthymB/builds/Fa0mud17dW94AxfLs/openapi.json