# Website Content Crawler (`jasondev/website-content-crawler`) Actor

A powerful web crawler that extracts text content from websites, optimized for AI models, Large Language Models (LLMs), vector databases, and Retrieval-Augmented Generation (RAG) pipelines.

- **URL**: https://apify.com/jasondev/website-content-crawler.md
- **Developed by:** [Jason Giang](https://apify.com/jasondev) (community)
- **Categories:** Developer tools, Automation, AI
- **Stats:** 76 total users, 4 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

$10.00 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Web Content Crawler

A powerful web crawler that extracts text content from websites, optimized for AI models, Large Language Models (LLMs), vector databases, and Retrieval-Augmented Generation (RAG) pipelines.

### Features

- **Multiple Crawling Engines**: Choose between Playwright (Chrome/Firefox), Cheerio (fast HTTP client), or JSDOM based on your needs
- **Markdown Output**: Automatically converts HTML content to clean Markdown format
- **Smart Content Extraction**: Removes unwanted elements like cookie banners, navigation, ads, and more
- **Customizable Selectors**: Keep or remove specific elements using CSS selectors
- **Deep Crawling**: Recursively crawl websites with configurable depth limits
- **AI-Ready Output**: Structured data perfect for feeding into AI models and vector databases
- **Proxy Support**: Built-in proxy configuration for reliable crawling
- **Screenshot Capture**: Optional screenshot capture for visual documentation (Playwright only)
- **File Downloads**: Download and save linked files like PDFs and documents

### Use Cases

- **Knowledge Base Extraction**: Crawl documentation sites and help centers
- **Content Aggregation**: Collect articles, blog posts, and web content at scale
- **AI Training Data**: Extract clean text for training or fine-tuning language models
- **RAG Pipelines**: Feed content into retrieval-augmented generation systems
- **Vector Database Population**: Prepare text content for embedding and semantic search
- **Website Migration**: Extract content from existing websites for migration
- **Competitive Analysis**: Monitor and analyze competitor content

### Input Parameters

#### Required

- **Start URLs** (`startUrls`): Array of URLs where the crawler will begin. The crawler will only process pages under these URLs.

#### Crawler Configuration

- **Crawler Type** (`crawlerType`): Select the crawling engine
  - `cheerio` (default): Fast HTTP client, best for static websites
  - `playwright:chrome`: Chrome browser with full JavaScript support
  - `playwright:firefox`: Firefox browser, useful for sites with anti-bot measures
  - `jsdom`: Experimental JavaScript-capable crawler

- **Max Crawling Depth** (`maxCrawlDepth`): Maximum link depth from start URLs (default: 1)
  - 0 = Only crawl start URLs
  - 1 = Crawl start URLs and pages directly linked from them
  - 2+ = Continue crawling to specified depth

- **Max Pages** (`maxCrawlPages`): Maximum number of pages to crawl (default: 100)

- **Max Requests Per Minute** (`maxRequestsPerMinute`): Rate limiting (default: 0 = unlimited)

#### Content Extraction

- **Readable Text Char Threshold** (`readableTextCharThreshold`): Minimum characters required to save a page (default: 100)

- **Remove Cookie Warnings** (`removeCookieWarnings`): Automatically remove cookie consent dialogs (default: true)

- **Click Elements CSS Selector** (`clickElementsCssSelector`): CSS selector for elements to click before extraction (e.g., "Show more" buttons)

- **HTML Transformer** (`htmlTransformer`): How to process HTML
  - `readableText` (default): Remove scripts, styles, navigation
  - `none`: Keep original HTML

- **Remove Elements CSS Selector** (`removeElementsCssSelector`): CSS selector for elements to remove (e.g., `nav, footer, .ads`)

- **Keep Elements CSS Selector** (`keepElementsCssSelector`): CSS selector for elements to keep (removes everything else)

#### Output Options

- **Save Markdown** (`saveMarkdown`): Convert content to Markdown format (default: true)

- **Save HTML** (`saveHtml`): Save raw HTML to key-value store (default: false)

- **Save Screenshots** (`saveScreenshots`): Capture page screenshots (Playwright only, default: false)

- **Save Files** (`saveFiles`): Download linked files like PDFs (default: false)

#### Advanced Options

- **Max Scroll Height** (`maxScrollHeightPixels`): Scroll down pages with infinite scroll (default: 0 = disabled)

- **Proxy Configuration** (`proxyConfiguration`): Proxy settings for the crawler

- **Max Request Retries** (`maxRequestRetries`): Number of retry attempts for failed requests (default: 3)

- **Debug Mode** (`debugMode`): Enable detailed logging (default: false)

### Output Format

Each crawled page produces a dataset item with the following structure:

```json
{
  "url": "https://example.com/page",
  "title": "Page Title",
  "description": "Page meta description",
  "canonicalUrl": "https://example.com/page",
  "text": "Extracted plain text content...",
  "markdown": "# Page Title\n\nExtracted content in Markdown...",
  "crawl": {
    "loadedUrl": "https://example.com/page",
    "depth": 1,
    "httpStatusCode": 200,
    "loadedAt": "2024-01-01T12:00:00.000Z"
  }
}
```

### Example Usage

#### Basic Crawl

```json
{
  "startUrls": [
    { "url": "https://example.com/docs" }
  ],
  "crawlerType": "cheerio",
  "maxCrawlDepth": 2,
  "maxCrawlPages": 50
}
```

#### Advanced Configuration

```json
{
  "startUrls": [
    { "url": "https://example.com/blog" }
  ],
  "crawlerType": "playwright:chrome",
  "maxCrawlDepth": 3,
  "maxCrawlPages": 200,
  "removeElementsCssSelector": "nav, footer, .sidebar, .comments",
  "removeCookieWarnings": true,
  "saveMarkdown": true,
  "saveScreenshots": false,
  "maxRequestRetries": 5,
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}
```

#### Extract Specific Content

```json
{
  "startUrls": [
    { "url": "https://example.com/" }
  ],
  "keepElementsCssSelector": "article, .content, main",
  "htmlTransformer": "readableText",
  "readableTextCharThreshold": 500,
  "saveMarkdown": true
}
```

### How It Works

The crawler starts from your specified URLs and:

1. Fetches and processes each page using your selected crawling engine
2. Extracts and cleans the content by removing unwanted elements
3. Converts the content to your preferred format (Markdown, plain text, or HTML)
4. Follows links to discover and crawl additional pages (up to your depth limit)
5. Saves all extracted data to the dataset for easy access

# Actor input Schema

## `startUrls` (type: `array`):

One or more URLs of the pages where the crawler will start. The crawler will additionally only crawl sub-pages of these URLs.

## `crawlerType` (type: `string`):

Select the crawling engine. Playwright browsers can handle JavaScript-heavy sites but are slower. Cheerio is a fast HTTP client but cannot execute JavaScript.

## `maxCrawlDepth` (type: `integer`):

The maximum number of links starting from the start URL that the crawler will recursively descend. Start URLs have a depth of 0, pages linked from start URLs have a depth of 1, etc. By setting this to 0, the crawler will only crawl start URLs.

## `maxCrawlPages` (type: `integer`):

The maximum number of pages to crawl. The crawler will stop after reaching this number. This includes the start URLs.

## `maxRequestsPerMinute` (type: `integer`):

Maximum number of pages to crawl per minute. Set to 0 for unlimited.

## `readableTextCharThreshold` (type: `integer`):

Minimum number of readable characters required for a page to be included in results. Pages with less text will be skipped.

## `removeCookieWarnings` (type: `boolean`):

Remove common cookie consent dialogs and banners from pages before extracting content.

## `clickElementsCssSelector` (type: `string`):

CSS selector for elements to click on each page (e.g., to expand content). The crawler will click all matching elements before extracting content.

## `htmlTransformer` (type: `string`):

How to transform HTML before text extraction.

## `removeElementsCssSelector` (type: `string`):

CSS selector for elements to remove from pages before extracting content (e.g., 'nav, footer, .ads').

## `keepElementsCssSelector` (type: `string`):

CSS selector for elements to keep. All other elements will be removed. Takes precedence over 'Remove elements CSS selector'.

## `renderingTypeDetectionPercentage` (type: `integer`):

Percentage of pages to automatically check if they need JavaScript rendering. If most pages work without JS, the crawler will switch to faster HTTP mode.

## `saveHtml` (type: `boolean`):

Save the raw HTML of each page to the key-value store.

## `saveMarkdown` (type: `boolean`):

Convert HTML to Markdown and save it in the output.

## `saveFiles` (type: `boolean`):

Download and save linked files (PDFs, documents, etc.) from crawled pages.

## `saveScreenshots` (type: `boolean`):

Save screenshots of each page (only available with Playwright crawlers).

## `maxScrollHeightPixels` (type: `integer`):

Maximum height to scroll on pages with infinite scroll. Set to 0 to disable scrolling.

## `proxyConfiguration` (type: `object`):

Proxy settings for the crawler.

## `maxRequestRetries` (type: `integer`):

Maximum number of times to retry failed requests.

## `debugMode` (type: `boolean`):

Enable detailed debug logging.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://apify.com/"
    }
  ],
  "crawlerType": "cheerio",
  "maxCrawlDepth": 1,
  "maxCrawlPages": 100,
  "maxRequestsPerMinute": 0,
  "readableTextCharThreshold": 100,
  "removeCookieWarnings": true,
  "htmlTransformer": "readableText",
  "renderingTypeDetectionPercentage": 10,
  "saveHtml": false,
  "saveMarkdown": true,
  "saveFiles": false,
  "saveScreenshots": false,
  "maxScrollHeightPixels": 0,
  "proxyConfiguration": {
    "useApifyProxy": true
  },
  "maxRequestRetries": 3,
  "debugMode": false
}
```

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://apify.com/"
        }
    ],
    "proxyConfiguration": {
        "useApifyProxy": true
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("jasondev/website-content-crawler").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [{ "url": "https://apify.com/" }],
    "proxyConfiguration": { "useApifyProxy": True },
}

# Run the Actor and wait for it to finish
run = client.actor("jasondev/website-content-crawler").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://apify.com/"
    }
  ],
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}' |
apify call jasondev/website-content-crawler --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,jasondev/website-content-crawler"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/uiIidGSZLzKrUL1Ie/builds/Uy0RAOT2ez9Zv4tkS/openapi.json