# Context Layer (`evertools/context-layer`) Actor

Transforms documentation sites into a clean, structured context layer for AI systems—handling crawling, extraction, intelligent chunking, and optional enrichment for RAG, fine-tuning, and semantic search.

- **URL**: https://apify.com/evertools/context-layer.md
- **Developed by:** [Mike](https://apify.com/evertools) (community)
- **Categories:** AI, Developer tools, MCP servers
- **Stats:** 1 total users, 0 monthly users, 0.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.50 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Context Layer

Transform documentation sites into a clean, structured **context layer** for AI systems — optimized for RAG, fine-tuning, embeddings, and semantic search.

Context Layer is an end-to-end pipeline that **scrapes and extracts** documentation, help centers, and knowledge bases, then converts them into AI-ready data in minutes instead of days. No custom code required.

***

### 🚀 What This Actor Does

Context Layer automates the hardest part of AI knowledge engineering: preparing high-quality context from real documentation.

It performs the full pipeline:

1. **Crawls** documentation sites and knowledge bases
2. **Extracts** clean content (removes navigation, footers, ads, UI noise)
3. **Chunks** content intelligently using semantic boundaries and token-aware sizing
4. **Enriches** content with AI-generated summaries and Q\&A pairs (optional)
5. **Embeds** chunks with vector embeddings for semantic search (optional)
6. **Exports** data in formats ready for RAG systems, fine-tuning, or markdown

This Actor is designed for **AI systems**, not raw scraping.

***

### ⚡ Quick Start

**1. Enter a documentation URL**

```json
{
  "startUrl": "https://docs.example.com/"
}
```

**2. Run the Actor**

Click "Start" and wait for the crawl to complete.

**3. Download your data**

- Go to the **Dataset** tab for structured JSON chunks
- Or download `context_layer.md` from the **Key-value store** for markdown output

That's it — your documentation is now AI-ready.

***

### 🎯 When to Use Context Layer

Use this Actor when you want to:

- Build a **RAG chatbot** from your documentation
- Prepare clean datasets for **LLM fine-tuning**
- Generate **semantic embeddings** for vector databases
- Convert docs into a **portable markdown knowledge base**
- Power **semantic search** over documentation
- **Extract and scrape API documentation** for AI processing

You likely don't need this Actor if you only want raw HTML or screenshots.

***

### 📦 Output Formats

#### RAG Format (Default)

Optimized for vector databases such as Pinecone, Weaviate, Qdrant, or Chroma.

```json
{
  "id": "chunk-0001",
  "content": "The actual chunk text...",
  "metadata": {
    "source_url": "https://docs.example.com/getting-started",
    "title": "Getting Started",
    "section": "Installation",
    "chunk_index": 0,
    "total_chunks": 5
  },
  "enrichment": {
    "summary": "This section explains how to install...",
    "questions": [
      "How do I install the software?",
      "What are the system requirements?"
    ]
  },
  "embedding": [0.123, -0.456, "..."]
}
```

***

#### Fine-tuning Format (OpenAI)

Ready for the OpenAI fine-tuning API (JSONL).

```json
{
  "messages": [
    { "role": "system", "content": "You are a helpful assistant." },
    { "role": "user", "content": "How do I install the software?" },
    { "role": "assistant", "content": "To install, follow these steps..." }
  ]
}
```

***

#### Fine-tuning Format (Alpaca)

Instruction-tuning format for open-source models.

```json
{
  "instruction": "How do I reset my password?",
  "input": "",
  "output": "To reset your password, go to Settings..."
}
```

***

#### Markdown Format

Exports a clean `context_layer.md` file containing all processed documentation, organized by source page.

***

### ⚙️ Input Options

#### Crawling

| Parameter         | Description                                  | Default                                        |
| ----------------- | -------------------------------------------- | ---------------------------------------------- |
| `startUrl`        | URL of the documentation or knowledge base   | **Required**                                   |
| `maxPages`        | Maximum pages to crawl (0 = unlimited)       | 50                                             |
| `crawlDepth`      | Link depth from the start URL                | 3                                              |
| `urlPatterns`     | Only crawl URLs matching these glob patterns | `[]`                                           |
| `excludePatterns` | Skip URLs matching these patterns            | `["**/changelog**", "**/blog**", "**/news**"]` |

***

#### Chunking

| Parameter      | Description                            | Default |
| -------------- | -------------------------------------- | ------- |
| `chunkSize`    | Target chunk size in tokens (0 = auto) | 0       |
| `chunkOverlap` | Overlapping tokens between chunks      | 50      |

**Auto chunk sizes:**

- RAG: ~500 tokens
- Fine-tuning: ~1000 tokens
- Markdown: ~2000 tokens

***

#### Output

| Parameter      | Description                                                | Default |
| -------------- | ---------------------------------------------------------- | ------- |
| `outputFormat` | `rag`, `finetune-openai`, `finetune-alpaca`, or `markdown` | `rag`   |

***

#### 🤖 AI Enrichment (Optional)

| Parameter           | Description                                            | Default  |
| ------------------- | ------------------------------------------------------ | -------- |
| `generateQA`        | Generate Q\&A pairs for each chunk                      | false    |
| `generateSummary`   | Generate summaries for each chunk                      | false    |
| `questionsPerChunk` | Number of Q\&A pairs per chunk                          | 3        |
| `llmProvider`       | `openai` (GPT-4o-mini) or `anthropic` (Claude 3 Haiku) | `openai` |
| `llmApiKey`         | API key for selected LLM provider                      | —        |

***

#### 🔢 Vector Embeddings (Optional)

| Parameter            | Description                                        | Default                  |
| -------------------- | -------------------------------------------------- | ------------------------ |
| `generateEmbeddings` | Generate vector embeddings                         | false                    |
| `embeddingModel`     | `text-embedding-3-small`, `text-embedding-3-large` | `text-embedding-3-small` |
| `embeddingApiKey`    | OpenAI API key for embeddings                      | —                        |

***

### 📝 Example Usage

#### Basic RAG Export

```json
{
  "startUrl": "https://docs.example.com/",
  "outputFormat": "rag"
}
```

***

#### RAG with Embeddings

```json
{
  "startUrl": "https://docs.example.com/",
  "generateEmbeddings": true,
  "embeddingModel": "text-embedding-3-small",
  "embeddingApiKey": "sk-..."
}
```

***

#### Fine-tuning with AI-Generated Q\&A

```json
{
  "startUrl": "https://help.example.com/",
  "outputFormat": "finetune-openai",
  "generateQA": true,
  "questionsPerChunk": 5,
  "llmProvider": "openai",
  "llmApiKey": "sk-..."
}
```

***

### 📂 Output Files

- **Default dataset** — all processed context chunks
- **training\_data.jsonl** — for fine-tuning formats
- **context\_layer.md** — markdown export (if selected)
- **report.json** — crawl and processing statistics

***

### 💰 Pricing

Context Layer uses **Pay-Per-Event** pricing:

| Event                        | Price   | Description                        |
| ---------------------------- | ------- | ---------------------------------- |
| `apify-actor-start`          | $0.02   | Charged once when the Actor starts |
| `apify-default-dataset-item` | $0.0015 | Charged per context chunk produced |

**Example cost:** Processing 100 pages producing 500 chunks ≈ $0.77

This pricing is designed to be fair, predictable, and scalable.

***

### 🌐 Why Use Apify?

Running Context Layer on Apify gives you:

- **Scheduled runs** — Keep your AI context fresh with automatic updates
- **REST API access** — Trigger runs programmatically from your app
- **Monitoring & alerts** — Get notified if something fails
- **Integrations** — Connect to Zapier, Make, Google Sheets, and more
- **No infrastructure** — No servers to manage or scale

***

### 🔧 Supported Documentation Platforms

Works with most public documentation sites, including:

- GitBook
- ReadTheDocs
- Docusaurus
- MkDocs
- Zendesk Help Centers
- Intercom Articles
- Notion (public pages)
- Confluence (public pages)
- Custom documentation sites

***

### ❓ FAQ

**Do I need an LLM API key?**
Only if you enable Q\&A or summary generation.

**Do I need an embedding API key?**
Only if you enable embeddings.

**Can this crawl private or authenticated sites?**
No. Only publicly accessible content is supported.

**What makes this different from a scraper?**
Scrapers extract text. Context Layer produces **structured, semantic context** designed for AI systems.

**How do I handle large documentation sites?**
Increase `maxPages` and use `urlPatterns` to focus on specific sections.

***

### 💬 Support & Feedback

- **Issues or bugs?** Open an issue on the Actor's Issues tab
- **Feature requests?** We'd love to hear from you — drop a message in Issues
- **Custom solutions?** Contact us for enterprise or custom integration needs

***

### 📚 About

Context Layer is built for teams who want AI-ready knowledge without building and maintaining custom ingestion pipelines.

It fits naturally into modern AI stacks alongside vector databases, RAG frameworks, and agent systems — and serves as a foundational **context ingestion layer** for larger knowledge systems.

# Actor input Schema

## `startUrl` (type: `string`):

The URL of the knowledge base or documentation site to process

## `maxPages` (type: `integer`):

Maximum number of pages to crawl (0 = unlimited)

## `crawlDepth` (type: `integer`):

How many links deep to follow from the start URL

## `urlPatterns` (type: `array`):

Only crawl URLs matching these patterns (glob format). Leave empty to crawl all.

## `excludePatterns` (type: `array`):

Skip URLs matching these patterns

## `chunkSize` (type: `integer`):

Target size for each chunk in tokens. Set to 0 for auto-detection based on output format.

## `chunkOverlap` (type: `integer`):

Number of overlapping tokens between chunks for context preservation

## `outputFormat` (type: `string`):

Format for the output data

## `generateQA` (type: `boolean`):

Generate question-answer pairs from each chunk. Great for fine-tuning datasets and improved RAG matching.

## `generateSummary` (type: `boolean`):

Generate a brief summary of each chunk.

## `questionsPerChunk` (type: `integer`):

How many Q\&A pairs to generate per chunk.

## `llmProvider` (type: `string`):

Which AI provider to use for Q\&A and summary generation.

## `llmApiKey` (type: `string`):

Your API key for the selected LLM provider.

## `generateEmbeddings` (type: `boolean`):

Generate vector embeddings for each chunk.

## `embeddingModel` (type: `string`):

Which OpenAI embedding model to use.

## `embeddingApiKey` (type: `string`):

Your OpenAI API key for generating embeddings.

## Actor input object example

```json
{
  "startUrl": "https://docs.apify.com/",
  "maxPages": 50,
  "crawlDepth": 3,
  "urlPatterns": [],
  "excludePatterns": [
    "**/changelog**",
    "**/blog**",
    "**/news**"
  ],
  "chunkSize": 0,
  "chunkOverlap": 50,
  "outputFormat": "rag",
  "generateQA": false,
  "generateSummary": false,
  "questionsPerChunk": 3,
  "llmProvider": "openai",
  "generateEmbeddings": false,
  "embeddingModel": "text-embedding-3-small"
}
```

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrl": "https://docs.apify.com/"
};

// Run the Actor and wait for it to finish
const run = await client.actor("evertools/context-layer").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "startUrl": "https://docs.apify.com/" }

# Run the Actor and wait for it to finish
run = client.actor("evertools/context-layer").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrl": "https://docs.apify.com/"
}' |
apify call evertools/context-layer --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,evertools/context-layer"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/iyHOfw9LIvm7RwsmF/builds/D5gbo1yh0VhKSrS45/openapi.json