# Tech Docs to LLM-Ready Markdown (`hedelka/tech-docs-scraper`) Actor

Scrapes technical documentation sites (Docusaurus, GitBook, MkDocs, ReadTheDocs) and converts them to clean, structured Markdown for RAG pipelines, LLM training, and AI assistants. Automatically detects documentation framework and removes navigation elements.

- **URL**: https://apify.com/hedelka/tech-docs-scraper.md
- **Developed by:** [Dmitry Goncharov](https://apify.com/hedelka) (community)
- **Categories:** Developer tools
- **Stats:** 26 total users, 0 monthly users, 100.0% runs succeeded, 1 bookmarks
- **User rating**: No ratings yet

## Pricing

$0.50 / 1,000 pages

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Tech Docs to LLM-Ready Markdown Scraper

🚀 **Convert any technical documentation site to clean, structured Markdown** — ready for RAG pipelines, LLM training, and AI assistants.

### Why This Actor?

While generic web scrapers dump raw HTML, this Actor is **specifically designed for technical documentation**:

| Feature | Generic Scrapers | This Actor |
|---------|-----------------|------------|
| Code block preservation | ❌ Lost or broken | ✅ With language tags |
| Framework-aware extraction | ❌ One-size-fits-all | ✅ Docusaurus, GitBook, MkDocs |
| Navigation removal | ❌ Mixed with content | ✅ Clean content only |
| RAG-ready output | ❌ Needs post-processing | ✅ `doc_id`, `section_path`, chunking |

#### 🔄 Before / After

<details>
<summary><b>❌ Generic Scraper Output</b> (messy HTML noise)</summary>

```
Skip to main content | Docs | Community | Blog | GitHub | 
Search docs... | Introduction | Quick Start | Guides |
← Previous | Next → | Edit this page | 
Introduction Crawlee covers your crawling...
Last updated 2 days ago | Was this page helpful? Yes No
```

</details>

<details open>
<summary><b>✅ This Actor Output</b> (clean, structured Markdown)</summary>

````markdown
## Introduction

Crawlee covers your crawling and scraping end-to-end and helps you 
**build reliable scrapers. Fast.**

### 🛠 Features

- Single interface for **HTTP and headless browser** crawling
- Persistent **queue** for URLs to crawl
- Automatic **scaling** with available system resources

```javascript
import { PlaywrightCrawler } from 'crawlee';
````

````
</details>

📚 **[More real examples →](EXAMPLES.md)** (Docusaurus, MkDocs, ReadTheDocs)

### 🎯 RAG-First Output

Every result includes fields optimized for vector databases and LLM loaders:

```json
{
    "doc_id": "acdb145c14f4310b",
    "url": "https://crawlee.dev/docs/introduction",
    "title": "Introduction | Crawlee",
    "section_path": "Guides > Quick Start > Introduction",
    "content": "# Introduction\n\nCrawlee covers your crawling...",
    "framework": "docusaurus",
    "chunk_index": 0,
    "total_chunks": 1,
    "metadata": {
        "crawledAt": "2025-12-12T03:34:46.151Z",
        "depth": 0,
        "wordCount": 358,
        "charCount": 2475
    }
}
````

### Supported Documentation Frameworks

| Framework | Status | Example |
|-----------|--------|---------|
| **Docusaurus** | ✅ Verified | React, Crawlee, Playwright docs |
| **GitBook** | ✅ Verified | Many SaaS products |
| **MkDocs Material** | ✅ Verified | Python projects |
| **ReadTheDocs** | ✅ Verified | Sphinx documentation |
| **VuePress** | ✅ Supported | Vue.js ecosystem |
| **Nextra** | ✅ Supported | Next.js docs |
| **Generic** | ✅ Fallback | Any HTML docs |

### Input Example

```json
{
    "startUrls": [{"url": "https://crawlee.dev/docs/introduction"}],
    "maxPages": 100,
    "maxDepth": 10,
    "enableChunking": true,
    "chunkSize": 2000,
    "outputFormat": "markdown"
}
```

### 🔗 LangChain Integration (Python)

```python
from langchain.document_loaders import ApifyDatasetLoader
from langchain.docstore.document import Document

loader = ApifyDatasetLoader(
    dataset_id="YOUR_DATASET_ID",
    dataset_mapping_function=lambda item: Document(
        page_content=item["content"],
        metadata={
            "source": item["url"],
            "title": item["title"],
            "doc_id": item["doc_id"],
            "section": item["section_path"]
        }
    ),
)
docs = loader.load()

## Ready for vectorstore!
from langchain.vectorstores import Chroma
vectorstore = Chroma.from_documents(docs, embeddings)
```

### 🦙 LlamaIndex Integration

```python
from llama_index.readers.apify import ApifyActor

reader = ApifyActor("hedelka/tech-docs-scraper")
documents = reader.load_data(
    run_input={"startUrls": [{"url": "https://docs.example.com/"}], "maxPages": 50}
)

## Build index directly
index = VectorStoreIndex.from_documents(documents)
```

### 📡 API Call

```bash
curl -X POST "https://api.apify.com/v2/acts/hedelka~tech-docs-scraper/runs?token=YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"startUrls": [{"url": "https://docs.example.com/"}], "maxPages": 50}'
```

### Use Cases

1. **RAG Pipelines**: Feed documentation to LangChain/LlamaIndex for "Chat with Docs"
2. **LLM Fine-tuning**: Create high-quality datasets from official docs
3. **Knowledge Bases**: Build searchable documentation archives
4. **AI Assistants**: Power coding assistants with up-to-date API references
5. **Scheduled Updates**: Keep your RAG knowledge base in sync with docs

#### 📅 Scheduled Docs Updates

Use Apify Scheduler to automatically re-scrape documentation and update your vector store:

1. **Create a Schedule** in Apify Console → Schedules
2. **Set cron**: `0 0 * * 0` (weekly) or `0 0 1 * *` (monthly)
3. **Use a Webhook** to trigger re-indexing in your RAG pipeline

```json
{
    "startUrls": [{"url": "https://docs.example.com/"}],
    "maxPages": 500,
    "preset": "large-docs",
    "exportJsonl": true
}
```

Your vector store always has the latest documentation!

### Pricing

**Pay per Result**: $0.50 per 1,000 pages

| Pages | Cost |
|-------|------|
| 100 | $0.05 |
| 1,000 | $0.50 |
| 10,000 | $5.00 |

### Author

Built with ❤️ by [HEDELKA](https://apify.com/hedelka) for the LLM/RAG community.

Questions? Issues? Open a [GitHub issue](https://github.com/HEDELKA/tech-docs-scraper) or contact on Apify.

# Actor input Schema

## `preset` (type: `string`):

Pre-configured settings for common documentation frameworks. Select a preset or use 'Custom' for manual configuration.

## `startUrls` (type: `array`):

URLs of documentation sites to scrape (e.g., https://docs.example.com)

## `maxPages` (type: `integer`):

Maximum number of pages to scrape

## `maxDepth` (type: `integer`):

Maximum link depth to follow from start URLs

## `outputFormat` (type: `string`):

Format of the output content

## `enableChunking` (type: `boolean`):

Split content into chunks for RAG embeddings. Creates multiple dataset items per page.

## `chunkSize` (type: `integer`):

Maximum characters per chunk. Only used when chunking is enabled.

## `exportJsonl` (type: `boolean`):

Also save output as JSONL file (1 document per line) for direct import to vector stores.

## `includeMetadata` (type: `boolean`):

Include page metadata (crawl date, word count, char count) in output

## `removeNavigation` (type: `boolean`):

Remove navigation elements (sidebars, headers, footers)

## `preserveCodeBlocks` (type: `boolean`):

Preserve code blocks with language syntax highlighting

## Actor input object example

```json
{
  "preset": "custom",
  "startUrls": [
    {
      "url": "https://crawlee.dev/docs/introduction"
    }
  ],
  "maxPages": 100,
  "maxDepth": 10,
  "outputFormat": "markdown",
  "enableChunking": false,
  "chunkSize": 2000,
  "exportJsonl": false,
  "includeMetadata": true,
  "removeNavigation": true,
  "preserveCodeBlocks": true
}
```

# Actor output Schema

## `documentation` (type: `string`):

Clean Markdown pages ready for LLM ingestion

## `jsonl_export` (type: `string`):

Download all pages as JSONL file for direct import into vector databases

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://crawlee.dev/docs/introduction"
        }
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("hedelka/tech-docs-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "startUrls": [{ "url": "https://crawlee.dev/docs/introduction" }] }

# Run the Actor and wait for it to finish
run = client.actor("hedelka/tech-docs-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://crawlee.dev/docs/introduction"
    }
  ]
}' |
apify call hedelka/tech-docs-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,hedelka/tech-docs-scraper"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/sgjYDvPoTyrRR3Yrn/builds/iZiHRgtnCDiLwlSrx/openapi.json