# Ai Training Data Curator (`omarchydev/ai-training-data-curator`) Actor

Crawl websites and curate high-quality training data for LLM fine-tuning. Automatic deduplication, quality scoring, and language detection. Export to JSONL, Parquet, or CSV formats ready for OpenAI, Claude, or Llama training.

- **URL**: https://apify.com/omarchydev/ai-training-data-curator.md
- **Developed by:** [Omarchy Dev](https://apify.com/omarchydev) (community)
- **Categories:** AI, Agents, Other
- **Stats:** 7 total users, 0 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.01 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## AI Training Data Curator

Curate high-quality, deduplicated training data for LLM fine-tuning. Extract clean text from any website OR process your own documents with automatic quality scoring, deduplication, and format conversion.

### Features

- **Smart Content Extraction**: Automatically detects and extracts main content, filtering out navigation, ads, and boilerplate
- **Bring Your Own Data (BYOD)**: Process your own text documents without crawling - perfect for existing datasets
- **Quality Scoring**: Scores each document based on vocabulary diversity, sentence structure, and content density
- **Deduplication**: Uses MinHash/Jaccard similarity to remove near-duplicate content
- **Flexible Crawling**: Single page, same domain, same subdomain, or follow all links
- **Document Chunking**: Split long documents into training-ready chunks with configurable overlap
- **Multiple Output Formats**: JSONL (OpenAI compatible), JSON, Parquet, CSV, or HuggingFace Datasets format
- **Language Filtering**: Filter content by language (ISO 639-1 codes)
- **Privacy Features**: Optionally remove emails and URLs from extracted text

### Use Cases

- **LLM Fine-tuning**: Collect domain-specific training data for fine-tuning language models
- **RAG Systems**: Build high-quality document collections for retrieval-augmented generation
- **Knowledge Bases**: Create clean text corpora from documentation sites
- **Research**: Gather datasets from academic or technical resources
- **Data Cleaning**: Clean and deduplicate existing text datasets for ML training

### Input Configuration

#### Mode Selection

The actor supports two modes - provide **either** `start_urls` (for crawling) **or** `documents` (for BYOD):

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `start_urls` | array | - | URLs to start crawling from (Crawl mode) |
| `documents` | array | - | Your own documents to process (BYOD mode) |

#### BYOD (Bring Your Own Data) Settings

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `documents` | array | - | Array of text strings or objects with `text` field |
| `byod_text_field` | string | `text` | Field name containing text in document objects |
| `max_byod_documents` | integer | 500 | Maximum documents to process (hard limit) |

#### Crawl Settings

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `start_urls` | array | - | URLs to start crawling from |
| `crawl_mode` | string | `same_domain` | `single_page`, `same_domain`, `same_subdomain`, or `all_links` |
| `max_pages` | integer | 100 | Maximum pages to crawl |
| `max_depth` | integer | 3 | Maximum link depth from start URLs |

#### Content Extraction

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `content_selectors` | array | `["article", "main", ".content"]` | CSS selectors for main content |
| `exclude_selectors` | array | `["nav", "header", "footer", ".sidebar"]` | CSS selectors to exclude |
| `min_word_count` | integer | 100 | Minimum words per document |
| `max_word_count` | integer | 50000 | Maximum words per document |

#### Quality & Deduplication

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `deduplicate` | boolean | true | Remove duplicate/near-duplicate content |
| `dedup_threshold` | number | 0.85 | Similarity threshold (0.5-1.0) |
| `quality_filter` | boolean | true | Filter low-quality content |
| `min_quality_score` | number | 0.5 | Minimum quality score (0.0-1.0) |
| `language_filter` | array | `["en"]` | Languages to include (ISO codes) |

#### Output Settings

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `output_format` | string | `jsonl` | `jsonl`, `json`, `parquet`, `csv`, or `huggingface` |
| `text_field_name` | string | `text` | Name of the text field in output |
| `include_metadata` | boolean | true | Include URL, title, date metadata |
| `include_raw_html` | boolean | false | Also save original HTML |

#### Chunking

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `chunk_documents` | boolean | false | Split documents into chunks |
| `chunk_size` | integer | 512 | Target chunk size in tokens |
| `chunk_overlap` | integer | 64 | Overlap between chunks |

#### Text Cleaning

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `clean_html` | boolean | true | Remove HTML tags |
| `normalize_whitespace` | boolean | true | Collapse multiple spaces/newlines |
| `remove_urls` | boolean | false | Strip embedded URLs |
| `remove_emails` | boolean | true | Strip email addresses |

#### Performance

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `use_proxies` | boolean | false | Use residential proxies |
| `max_concurrency` | integer | 10 | Parallel requests |
| `request_delay_ms` | integer | 500 | Delay between requests |
| `respect_robots_txt` | boolean | true | Follow robots.txt rules |

### Output Format

Each document in the output contains:

```json
{
  "text": "The cleaned document text content...",
  "doc_id": "abc123def456",
  "source_url": "https://example.com/page",
  "word_count": 1523,
  "quality_score": 0.847,
  "language": "en",
  "title": "Page Title",
  "description": "Meta description",
  "content_type": "documentation",
  "scraped_at": "2024-01-15T10:30:00Z"
}
```

If chunking is enabled, additional fields are included:

```json
{
  "chunk_index": 0,
  "total_chunks": 5,
  "parent_doc_id": "abc123def456"
}
```

### Quality Metrics

The quality scorer evaluates documents based on:

- **Word count**: Penalizes very short documents
- **Sentence length**: Flags very short (fragments) or very long sentences
- **Vocabulary diversity**: Ratio of unique words to total words
- **Boilerplate ratio**: Detection of common web boilerplate patterns
- **Character composition**: Penalizes excessive uppercase, digits, or special characters

Documents with scores below `min_quality_score` are automatically filtered out.

### Example Input

#### Crawl Python Documentation

```json
{
  "start_urls": [
    { "url": "https://docs.python.org/3/tutorial/" }
  ],
  "crawl_mode": "same_subdomain",
  "max_pages": 500,
  "content_selectors": [".document", ".body"],
  "exclude_selectors": [".sphinxsidebar", ".related", "footer"],
  "output_format": "jsonl",
  "chunk_documents": true,
  "chunk_size": 1024
}
```

#### Build Knowledge Base from Blog

```json
{
  "start_urls": [
    { "url": "https://example.com/blog/" }
  ],
  "crawl_mode": "same_domain",
  "max_pages": 100,
  "content_selectors": ["article", ".post-content"],
  "quality_filter": true,
  "min_quality_score": 0.6,
  "deduplicate": true,
  "output_format": "parquet"
}
```

#### BYOD: Process Your Own Documents

```json
{
  "documents": [
    "This is a plain text document that will be processed...",
    {
      "text": "This document has metadata attached to it...",
      "source_id": "doc_001",
      "metadata": {
        "title": "My Document",
        "author": "John Doe",
        "language": "en"
      }
    }
  ],
  "deduplicate": true,
  "quality_filter": true,
  "min_quality_score": 0.5,
  "output_format": "jsonl"
}
```

#### BYOD: Clean Existing Dataset

```json
{
  "documents": [
    {"text": "First document from your dataset..."},
    {"text": "Second document from your dataset..."},
    {"text": "Third document from your dataset..."}
  ],
  "byod_text_field": "text",
  "deduplicate": true,
  "dedup_threshold": 0.85,
  "chunk_documents": true,
  "chunk_size": 512,
  "output_format": "jsonl"
}
```

### Tips for Best Results

1. **Use specific content selectors**: Better extraction with precise CSS selectors for your target site
2. **Set appropriate word counts**: Filter out navigation pages and indexes with `min_word_count`
3. **Enable deduplication**: Prevents training on repetitive content (common on content farms)
4. **Adjust quality threshold**: Lower for technical content, higher for prose
5. **Use chunking for long documents**: Better for training context windows
6. **Start small**: Test with `max_pages: 20` before large crawls

### Pricing

- **$0.01 per document** - charged for each cleaned document (both crawled and BYOD)

Additional costs:

- **Proxy**: ~$0.001-0.005 per request (if enabled)
- **Storage**: ~$0.0001 per document

### Support

- [Apify Documentation](https://docs.apify.com)
- [Report Issues](https://github.com/your-repo/issues)
- [Crawlee Documentation](https://crawlee.dev/python/docs)

# Actor input Schema

## `start_urls` (type: `array`):

URLs to start crawling from. Leave empty if using BYOD mode.

## `documents` (type: `array`):

Your own documents to process. Provide text strings or objects with 'text' field. Use this instead of Start URLs for BYOD mode.

## `byod_text_field` (type: `string`):

Field name containing text in BYOD documents (default: 'text')

## `max_byod_documents` (type: `integer`):

Maximum documents to process in BYOD mode (prevents memory issues)

## `crawl_mode` (type: `string`):

How extensively to crawl: single\_page, same\_domain, same\_subdomain, all\_links

## `max_pages` (type: `integer`):

Maximum number of pages to crawl

## `max_depth` (type: `integer`):

Maximum link depth from start URLs

## `content_selectors` (type: `array`):

CSS selectors for main content elements (e.g., article, main, .content)

## `exclude_selectors` (type: `array`):

CSS selectors for elements to exclude (e.g., nav, footer, .sidebar)

## `min_word_count` (type: `integer`):

Minimum words per document

## `max_word_count` (type: `integer`):

Maximum words per document (longer documents will be truncated)

## `deduplicate` (type: `boolean`):

Remove near-duplicate documents using fuzzy matching

## `dedup_threshold_percent` (type: `integer`):

Similarity threshold for deduplication (50-100%, higher = stricter)

## `quality_filter` (type: `boolean`):

Filter out low-quality documents based on quality score

## `min_quality_score_percent` (type: `integer`):

Minimum quality score to include document (0-100%)

## `language_filter` (type: `array`):

Language codes to include (e.g., en, es, fr). Empty = all languages

## `chunk_documents` (type: `boolean`):

Split documents into smaller chunks for training

## `chunk_size` (type: `integer`):

Target tokens per chunk when chunking is enabled

## `chunk_overlap` (type: `integer`):

Overlap tokens between consecutive chunks

## `output_format` (type: `string`):

Format for output data

## `text_field_name` (type: `string`):

Field name for text content in output

## `include_metadata` (type: `boolean`):

Include document metadata (title, author, URL, etc.)

## `include_raw_html` (type: `boolean`):

Include raw HTML in output (increases data size)

## `remove_urls` (type: `boolean`):

Remove URLs from extracted text

## `remove_emails` (type: `boolean`):

Remove email addresses from extracted text (recommended for privacy)

## `normalize_whitespace` (type: `boolean`):

Normalize whitespace and remove excessive blank lines

## `use_proxies` (type: `boolean`):

Use Apify proxy for requests

## `max_concurrency` (type: `integer`):

Maximum concurrent page requests

## `request_delay_ms` (type: `integer`):

Delay between requests in milliseconds

## Actor input object example

```json
{
  "start_urls": [
    {
      "url": "https://docs.python.org/3/"
    }
  ],
  "documents": [
    {
      "text": "Your document text here...",
      "source_id": "doc_001"
    }
  ],
  "byod_text_field": "text",
  "max_byod_documents": 500,
  "crawl_mode": "same_domain",
  "max_pages": 100,
  "max_depth": 3,
  "content_selectors": [
    "article",
    "main",
    ".content"
  ],
  "exclude_selectors": [
    "nav",
    "footer",
    "aside",
    ".sidebar"
  ],
  "min_word_count": 100,
  "max_word_count": 50000,
  "deduplicate": true,
  "dedup_threshold_percent": 85,
  "quality_filter": true,
  "min_quality_score_percent": 50,
  "language_filter": [
    "en"
  ],
  "chunk_documents": false,
  "chunk_size": 512,
  "chunk_overlap": 64,
  "output_format": "jsonl",
  "text_field_name": "text",
  "include_metadata": true,
  "include_raw_html": false,
  "remove_urls": false,
  "remove_emails": true,
  "normalize_whitespace": true,
  "use_proxies": false,
  "max_concurrency": 10,
  "request_delay_ms": 500
}
```

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "start_urls": [
        {
            "url": "https://docs.python.org/3/"
        }
    ],
    "documents": [
        {
            "text": "Your document text here...",
            "source_id": "doc_001"
        }
    ],
    "content_selectors": [
        "article",
        "main",
        ".content"
    ],
    "exclude_selectors": [
        "nav",
        "footer",
        "aside",
        ".sidebar"
    ],
    "language_filter": [
        "en"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("omarchydev/ai-training-data-curator").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "start_urls": [{ "url": "https://docs.python.org/3/" }],
    "documents": [{
            "text": "Your document text here...",
            "source_id": "doc_001",
        }],
    "content_selectors": [
        "article",
        "main",
        ".content",
    ],
    "exclude_selectors": [
        "nav",
        "footer",
        "aside",
        ".sidebar",
    ],
    "language_filter": ["en"],
}

# Run the Actor and wait for it to finish
run = client.actor("omarchydev/ai-training-data-curator").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "start_urls": [
    {
      "url": "https://docs.python.org/3/"
    }
  ],
  "documents": [
    {
      "text": "Your document text here...",
      "source_id": "doc_001"
    }
  ],
  "content_selectors": [
    "article",
    "main",
    ".content"
  ],
  "exclude_selectors": [
    "nav",
    "footer",
    "aside",
    ".sidebar"
  ],
  "language_filter": [
    "en"
  ]
}' |
apify call omarchydev/ai-training-data-curator --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,omarchydev/ai-training-data-curator"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/jKpCBEHK0ULZKV3da/builds/5uAj1uayEGy3AmThH/openapi.json