# Website Content Crawler (`parseforge/website-content-crawler`) Actor

Crawls websites from starting URLs and returns each page's full text content as Markdown and cleaned HTML, ready for AI and LLM pipelines.

- **URL**: https://apify.com/parseforge/website-content-crawler.md
- **Developed by:** [ParseForge](https://apify.com/parseforge) (community)
- **Categories:** AI, Developer tools, Automation
- **Stats:** 131 total users, 19 monthly users, 100.0% runs succeeded, 2 bookmarks
- **User rating**: No ratings yet

## Pricing

from $20.00 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

[![ParseForge](https://raw.githubusercontent.com/ParseForge/apify-assets/main/banner.jpg)](https://apify.com/parseforge?fpr=vmoqkp)

### Website Content Crawler

**Crawl any website and extract clean text content for your AI models, LLM pipelines, and vector databases.** Every page returns its full Markdown body, cleaned HTML, and metadata. No sitemap needed. Export to CSV, JSON, Excel, or XML.

Building a RAG pipeline or fine-tuning an LLM requires clean, structured text from websites, but writing a custom crawler for each site is slow and brittle. The Website Content Crawler reads public pages from any starting URL, follows links up to a set depth, and returns the full text content of each page in a consistent format. It handles HTML cleaning, Markdown conversion, and domain filtering so your data is ready for embeddings or training.

| Who uses it | What they scrape Website Content Crawler for |
|---|---|
| AI engineers | Building a knowledge base for a RAG application or fine-tuning dataset. |
| Data scientists | Collecting a corpus of text from a specific domain for NLP analysis. |
| SEO specialists | Extracting all body text from a competitor's site to audit content strategy. |
| Developers | Feeding a vector database with Markdown-formatted pages from documentation sites. |

### What it does

This Actor crawls websites from a list of starting URLs and returns each page's full text content, cleaned HTML, and metadata as a flat row.

- 📝 **Markdown output:** every page is converted to clean Markdown, ready for LLM context windows.
- 🧹 **HTML cleaning:** navigation, scripts, and boilerplate are stripped, leaving only the main content.
- 🔗 **Link crawling:** follows internal links up to a configurable depth to discover all pages on a site.
- 🌐 **Domain control:** restrict the crawl to the same domain, or optionally include subdomains.
- 🎯 **URL filtering:** include or exclude pages with regex patterns to target only the content you need.

Results export to CSV, JSON, Excel, or XML, or straight from the API.

### What you can do with Website Content Crawler data

**🤖 Build a RAG knowledge base.**

An AI engineer crawls a product documentation site to extract every page as Markdown, then loads the dataset into a vector store for a support chatbot.

**📊 Audit a competitor's content.**

An SEO specialist crawls a competitor's blog, filtering URLs to only article pages, and extracts the full body text to analyze topic coverage and keyword density.

**🧠 Create a fine-tuning dataset.**

A data scientist crawls a public forum, limits the depth to thread pages, and collects a corpus of question-answer pairs for instruction-tuning a model.

**📚 Archive documentation for offline use.**

A developer crawls an open-source library's docs site, stays on the same domain, and exports the full text to a single JSON file for local search.

### Why choose this scraper

|  | What you get |
|---|---|
| **Clean text extraction** | Full page body in Markdown and cleaned HTML, stripped of navigation and scripts. |
| **Configurable crawl scope** | Set max depth, page limit, and domain restrictions to control the crawl size. |
| **Regex URL filtering** | Include or exclude specific URL patterns to target only relevant sections of a site. |
| **AI-ready output** | Output format is designed for direct ingestion into LLMs, vector databases, and LangChain. |

### How it compares

This Actor focuses on extracting clean, AI-ready text from any website, while the other Website Content Crawler listing offers similar Markdown extraction and LangChain integration.

| Feature | ParseForge | Website Content Crawler |
|---|---|---|
| Full page text in Markdown | Yes | Yes |
| Cleaned HTML output | Yes | Yes |
| Regex URL filtering | Yes | Not listed |
| Subdomain inclusion control | Yes | Not listed |
| LangChain integration | Not listed | Yes |

### Configure the run

Drive the Actor from one or more starting URLs, set a crawl depth and page limit, and apply domain and regex filters so only matching pages reach your dataset. The Input tab lists every parameter.

A first run with the defaults:

```json
{
  "startUrls": [
    {
      "url": "https://apify.com/docs"
    }
  ],
  "maxDepth": 2,
  "maxItems": 10,
  "sameDomain": true,
  "includeSubdomains": true
}
```

A larger pull:

```json
{
  "startUrls": [
    {
      "url": "https://apify.com/docs"
    }
  ],
  "maxDepth": 2,
  "maxItems": 200,
  "sameDomain": true,
  "includeSubdomains": true
}
```

### Pricing

Pay-per-result: **$0.025 per result** collected. You pay only for the results written to your dataset.

| Results collected | Approximate cost |
|---|---|
| 100 results | $2.50 |
| 1,000 results | $25.00 |
| 10,000 results | $250.00 |

New Apify accounts start with $5 in free credit.

### Free users

Free-plan runs return up to 10 results as a preview. [Upgrade your Apify plan](https://console.apify.com/sign-up?fpr=vmoqkp) to collect up to 1,000,000 results per run.

### Run it

1. [Create a free Apify account with $5 in credit](https://console.apify.com/sign-up?fpr=vmoqkp).
2. Open the [Website Content Crawler](https://apify.com/parseforge/website-content-crawler?fpr=vmoqkp).
3. Set your inputs and any filters, then click **Start**.
4. Export the results as CSV, Excel, JSON, or XML from the **Dataset** tab.

Run it programmatically through the [Apify API](https://docs.apify.com/api/v2) (`run-sync-get-dataset-items`) or the [ApifyClient](https://docs.apify.com/api/client/js) for JavaScript and Python.

### Use with AI agents (MCP)

Give an AI agent live access to Website Content Crawler through the Model Context Protocol. Add the Actor to Claude, Cursor, or any MCP client:

```bash
claude mcp add --transport http apify "https://mcp.apify.com/?tools=parseforge/website-content-crawler"
```

Then prompt it in plain language to run the scraper and read back the results.

### Troubleshooting

**Why am I getting no results?**

Check your Include URL Patterns regex. If it is too restrictive, no discovered URLs will match. Try removing the pattern or testing it against a known page URL. Also verify the starting URL is accessible and returns a 200 status.

**The crawl is taking too long.**

Reduce the maxDepth or maxItems to limit the crawl scope. You can also add Exclude URL Patterns to skip large sections like archives or tag pages that generate many low-value URLs.

**The extracted text contains navigation menus and footer links.**

The Actor cleans common HTML boilerplate, but some sites use unusual markup. The cleaned HTML field removes scripts and styles, but complex layouts may still include some chrome. The Markdown field focuses on the main content area.

**I am getting pages from other domains even with 'Stay on Same Domain' enabled.**

Check if you have 'Include Subdomains' enabled. A subdomain like blog.example.com is treated as a different domain from www.example.com. Disable subdomain inclusion to restrict to the exact starting domain.

**The Markdown output is missing formatting like headings or lists.**

The Markdown conversion preserves semantic HTML elements like h1-h6, ul, ol, and p tags. If a site uses divs with CSS classes instead of semantic tags, the structure may be lost. The cleaned HTML field will still contain the original structure.

### FAQ

| Question | Answer |
|---|---|
| What format is the extracted text in? | Each page returns a Markdown version of the main content and a cleaned HTML version. The Markdown is ready for direct use with LLMs and vector databases. |
| Can I limit the crawl to only one section of a website? | Yes. Use the Include URL Patterns field with a regex like `/docs/.*` to crawl only pages under the /docs/ path, or use Exclude URL Patterns to skip sections like /blog/. |
| Does it handle JavaScript-rendered content? | This Actor fetches the initial HTML of each page. For sites that load content dynamically with JavaScript, you may need a headless browser crawler instead. |
| How many pages can I crawl in one run? | You can set the maximum pages up to 1,000,000 per run. The actual number depends on your max depth, domain restrictions, and URL filters. |
| Will it follow links to external websites? | Not by default. The 'Stay on Same Domain' option is on by default, which restricts the crawl to the starting domain. You can disable it, but the Actor will still only follow links it discovers from the pages it crawls. |
| Can I use this to scrape a single page? | Yes. Set maxDepth to 0 and maxItems to 1, and it will extract only the content from the starting URL without following any links. |
| What is the output schema? | Each row in the dataset includes the page URL, the full Markdown body, the cleaned HTML, the page title, and metadata like the crawl depth and load timestamp. |
| Does it download images or PDFs? | No, this Actor extracts text content only. It does not download binary files, images, or PDFs. |
| Can I crawl multiple websites in one run? | Yes. Add multiple starting URLs from different domains. If 'Stay on Same Domain' is on, each starting URL will spawn a separate crawl restricted to its own domain. |
| Is this suitable for a production RAG pipeline? | Yes. The Markdown output is designed for direct ingestion into vector databases like Pinecone or Weaviate, and the Actor integrates with LangChain for loading documents. |

### Related actors

Browse the full [ParseForge collection](https://apify.com/parseforge?fpr=vmoqkp) for more scrapers.

🆘 **Need help?** Email parseforge@protonmail.com with your run ID, your input, and what you expected.

⚠️ **Disclaimer.** This Actor is unofficial and is not affiliated with, endorsed by, or sponsored by Apify. It collects only publicly available data. You are responsible for using the collected data in compliance with the source's terms of service and applicable data-protection laws, including GDPR, CCPA, and PIPL. Do not use it to collect personal data unlawfully.

# Actor input Schema

## `startUrls` (type: `array`):

Starting URLs to crawl.

## `maxDepth` (type: `integer`):

How many link hops from start URLs.

## `maxItems` (type: `integer`):

How many pages to collect per run.

## `sameDomain` (type: `boolean`):

Restrict crawl to the starting domain

## `includeSubdomains` (type: `boolean`):

Allow subdomains when crawling

## `includeUrlPatterns` (type: `array`):

Regex patterns to include URLs

## `excludeUrlPatterns` (type: `array`):

Regex patterns to exclude URLs

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://apify.com/docs"
    }
  ],
  "maxDepth": 2,
  "maxItems": 10,
  "sameDomain": true,
  "includeSubdomains": true
}
```

# Actor output Schema

## `results` (type: `string`):

Complete dataset

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://apify.com/docs"
        }
    ],
    "maxDepth": 2,
    "maxItems": 10,
    "sameDomain": true,
    "includeSubdomains": true
};

// Run the Actor and wait for it to finish
const run = await client.actor("parseforge/website-content-crawler").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [{ "url": "https://apify.com/docs" }],
    "maxDepth": 2,
    "maxItems": 10,
    "sameDomain": True,
    "includeSubdomains": True,
}

# Run the Actor and wait for it to finish
run = client.actor("parseforge/website-content-crawler").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://apify.com/docs"
    }
  ],
  "maxDepth": 2,
  "maxItems": 10,
  "sameDomain": true,
  "includeSubdomains": true
}' |
apify call parseforge/website-content-crawler --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,parseforge/website-content-crawler"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/XvOSmmcClKGzwlQOJ/builds/xeJ1rRkgOdgz1IHHg/openapi.json