# AI / RAG Web Crawler (`groupoject/ai-rag-web-crawler`) Actor

Crawl any website and extract clean, LLM-ready Markdown chunks to feed AI agents, chatbots, and RAG pipelines. One row per embeddable chunk.

- **URL**: https://apify.com/groupoject/ai-rag-web-crawler.md
- **Developed by:** [Group Oject](https://apify.com/groupoject) (community)
- **Categories:** AI, Automation, Agents
- **Stats:** 1 total users, 0 monthly users, 100.0% runs succeeded, 1 bookmarks
- **User rating**: No ratings yet

## Pricing

from $2.00 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## AI / RAG Web Crawler

**Crawl any website and get clean, LLM-ready Markdown chunks — ready to feed AI agents, chatbots, and RAG pipelines.**

Point it at a docs site, knowledge base, or blog. It crawls the pages, strips the navigation/ads/boilerplate, converts the main content to clean Markdown, and (optionally) splits it into overlapping chunks. **One dataset row per chunk** — pipe it straight into a vector database.

> ⚡ Fast HTTP crawler (no headless browser). No API key required.

***

### What it does

1. **Crawls** from your start URLs, following links up to a depth/page limit you set (same-domain by default).
2. **Extracts** the main content — removes nav, header, footer, sidebars, scripts, ads.
3. **Converts** it to clean Markdown (headings, lists, links, code preserved).
4. **Chunks** it into overlapping, embeddings-sized pieces for RAG.

Output is **one row per chunk**, each tagged with its source URL, title, and chunk position — exactly the shape you want for an embeddings/vector pipeline.

***

### Who it's for

- **AI/RAG builders** — turn a docs site or knowledge base into a clean corpus for retrieval.
- **Chatbot makers** — feed your support docs into a customer-facing assistant.
- **Agent developers** — give an agent a fresh, structured snapshot of a site.
- **Data teams** — bulk-convert web content to Markdown without writing a parser.

***

### Popular use cases

- **Docs to RAG dataset** - crawl product documentation into LLM-ready Markdown chunks for embeddings.
- **Help center chatbot data** - turn support articles, FAQs, and knowledge bases into clean chatbot context.
- **Website to Markdown export** - convert public pages into structured Markdown for analysis or archiving.
- **AI agent knowledge refresh** - schedule repeat crawls so agents work from current website content.
- **Competitor docs monitoring** - snapshot competitor documentation, pricing pages, or changelogs.
- **Blog corpus builder** - collect editorial content into chunked rows for semantic search and content analysis.

***

### Input

| Field | Type | Default | Description |
|---|---|---|---|
| `startUrls` | array | — | URLs to crawl (plain strings or `{ "url": "..." }`) |
| `maxCrawlPages` | integer | `50` | Total page cap |
| `maxCrawlDepth` | integer | `1` | Link-hops from start URLs (0 = start URLs only) |
| `sameDomainOnly` | boolean | `true` | Only follow links on the start domain(s) |
| `includeUrlGlobs` | array | — | Only crawl URLs matching these globs (e.g. `https://site.com/docs/*`) |
| `excludeUrlGlobs` | array | — | Skip URLs matching these globs (e.g. `*.pdf`) |
| `chunkContent` | boolean | `true` | Split pages into RAG chunks (one row each) |
| `chunkSize` | integer | `1000` | Target characters per chunk |
| `chunkOverlap` | integer | `100` | Overlap chars between chunks |
| `minChunkChars` | integer | `50` | Drop chunks smaller than this |
| `saveHtml` | boolean | `false` | Also include cleaned HTML |
| `maxConcurrency` | integer | `10` | Pages crawled in parallel |
| `proxyConfiguration` | object | — | Optional Apify Proxy |

#### Example input

```json
{
  "startUrls": [{ "url": "https://docs.apify.com/" }],
  "maxCrawlPages": 30,
  "maxCrawlDepth": 2,
  "includeUrlGlobs": ["https://docs.apify.com/*"],
  "chunkContent": true,
  "chunkSize": 1000,
  "chunkOverlap": 100
}
```

More in [`examples/`](examples/).

***

### Output

One dataset row per chunk:

```json
{
  "url": "https://docs.apify.com/platform/actors",
  "title": "Actors | Apify Docs",
  "description": "Learn how Apify Actors work.",
  "chunkIndex": 0,
  "chunkCount": 4,
  "content": "# Actors\n\nActors are serverless programs...",
  "contentChars": 980,
  "depth": 1,
  "crawledAt": "2026-06-15T12:00:00.000Z"
}
```

To build a vector index: embed the `content` field, store `url` + `title` + `chunkIndex` as metadata. Done.

#### Key-value store outputs

- `SUMMARY` — pages crawled/failed, total chunks, average chunk size, settings

***

### Tips for clean RAG data

- **Use `includeUrlGlobs`** to stay inside the section you care about (e.g. `.../docs/*`) and skip marketing pages.
- **`chunkSize` 800–1200 chars** suits most embedding models; bump `chunkOverlap` to 150–200 for prose-heavy sites.
- **Turn off `chunkContent`** if you want whole pages (one row each) and prefer to chunk in your own pipeline.
- **Exclude noise** with `excludeUrlGlobs` (`*.pdf`, `*/tag/*`, `*/author/*`).

***

### Limitations & compliance

- HTTP crawler — it reads server-rendered HTML. Pages that render content purely client-side (heavy SPA) may yield little; those need a browser-based crawler.
- Main-content extraction is heuristic (prefers `<article>`/`<main>`, strips common boilerplate). Unusual layouts may include or drop some content.
- **You choose the targets.** Crawl only sites you're permitted to, respect each site's terms and robots policy, and don't collect private or paywalled data. This Actor accesses publicly reachable pages only.

***

### Changelog

See [CHANGELOG.md](CHANGELOG.md).

### Run with the Apify API

```bash
curl -X POST \
  "https://api.apify.com/v2/acts/groupoject~ai-rag-web-crawler/run-sync-get-dataset-items?token=YOUR_APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "startUrls":[{"url":"https://docs.example.com/"}],
    "maxCrawlPages":50,
    "maxCrawlDepth":2,
    "chunkContent":true,
    "chunkSize":1000,
    "chunkOverlap":100
  }'
```

### RAG ingestion workflow

1. Restrict the crawl to the relevant documentation section with include globs.
2. Exclude search, tag, login, asset, and duplicate navigation URLs.
3. Choose a chunk size appropriate for the embedding model and retrieval strategy.
4. Embed `content` and retain URL, title, and chunk index as metadata.
5. Schedule refresh crawls and replace or version records by source URL.

### Choosing chunk settings

- `800-1200` characters works well for compact documentation passages.
- Larger chunks preserve more context but can reduce retrieval precision.
- Overlap helps retain meaning across boundaries; start around 10-15% of chunk size.
- Disable chunking when the downstream system already handles semantic splitting.

### FAQ

#### Does it create embeddings?

No. It creates clean Markdown chunks ready for your embedding provider or vector database, avoiding a required third-party AI key.

#### Does it execute browser JavaScript?

No. It is an efficient HTTP crawler for server-rendered pages. JavaScript-only application content may require a browser crawler.

#### Can it crawl several domains?

Yes. Add multiple start URLs. `sameDomainOnly` keeps discovered links associated with the supplied start domains.

#### How do I avoid duplicate or irrelevant pages?

Use strict include globs, exclusion globs, normalized source URLs, depth limits, and a conservative initial page cap.

# Actor input Schema

## `startUrls` (type: `array`):

URLs to start crawling from. Accepts plain URLs or { "url": "..." } objects.

## `maxCrawlPages` (type: `integer`):

Maximum number of pages to crawl in total.

## `maxCrawlDepth` (type: `integer`):

How many link-hops to follow from the start URLs. 0 = only the start URLs.

## `sameDomainOnly` (type: `boolean`):

Only follow links on the same domain(s) as the start URLs.

## `includeUrlGlobs` (type: `array`):

Only crawl URLs matching these glob patterns (e.g. https://site.com/docs/\*). Empty = no include filter.

## `excludeUrlGlobs` (type: `array`):

Skip URLs matching these glob patterns (e.g. *.pdf, https://site.com/blog/*).

## `chunkContent` (type: `boolean`):

Split each page into overlapping chunks (one dataset row per chunk) — ready for embeddings. Off = one row per page.

## `chunkSize` (type: `integer`):

Target characters per chunk.

## `chunkOverlap` (type: `integer`):

Characters of context carried between consecutive chunks.

## `minChunkChars` (type: `integer`):

Drop chunks shorter than this many characters.

## `saveHtml` (type: `boolean`):

Also include the cleaned HTML of the main content in each row.

## `maxConcurrency` (type: `integer`):

How many pages to crawl in parallel.

## `proxyConfiguration` (type: `object`):

Optional. Use Apify Proxy to crawl sites that block datacenter IPs.

## `debugMode` (type: `boolean`):

Verbose logging.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://docs.apify.com/"
    }
  ],
  "maxCrawlPages": 50,
  "maxCrawlDepth": 1,
  "sameDomainOnly": true,
  "chunkContent": true,
  "chunkSize": 1000,
  "chunkOverlap": 100,
  "minChunkChars": 50,
  "saveHtml": false,
  "maxConcurrency": 10,
  "proxyConfiguration": {
    "useApifyProxy": true
  },
  "debugMode": false
}
```

# Actor output Schema

## `chunks` (type: `string`):

One row per chunk: url, title, chunk index, and LLM-ready Markdown content.

## `summary` (type: `string`):

Pages crawled/failed, total chunks, average chunk size, settings.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://docs.apify.com/"
        }
    ],
    "proxyConfiguration": {
        "useApifyProxy": true
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("groupoject/ai-rag-web-crawler").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [{ "url": "https://docs.apify.com/" }],
    "proxyConfiguration": { "useApifyProxy": True },
}

# Run the Actor and wait for it to finish
run = client.actor("groupoject/ai-rag-web-crawler").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://docs.apify.com/"
    }
  ],
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}' |
apify call groupoject/ai-rag-web-crawler --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,groupoject/ai-rag-web-crawler"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/LoEsRxGcgYYlgMVzX/builds/ypBpd2Ubo2heNKlTc/openapi.json