# Ai Training Data Enricher (`fiery_dream/ai-training-data-enricher`) Actor

Production-grade data enrichment and validation for LLM training datasets. Automatically clean, enrich, deduplicate, and validate your AI training data before fine-tuning.

- **URL**: https://apify.com/fiery\_dream/ai-training-data-enricher.md
- **Developed by:** [Cody Churchwell](https://apify.com/fiery_dream) (community)
- **Categories:** AI, Agents, Automation
- **Stats:** 1 total users, 0 monthly users, 100.0% runs succeeded, 1 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.01 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## 🤖 AI Training Data Enricher & Validator

[![Apify Actor](https://img.shields.io/badge/Apify-Actor-blue)](https://apify.com/actors)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)

**Production-grade data enrichment and validation for LLM training datasets.** Automatically clean, enrich, deduplicate, and validate your AI training data before fine-tuning.

### 🎯 Why This Actor?

Training high-quality LLMs requires clean, diverse, and well-structured data. Poor data quality leads to:

- **Overfitting** from duplicates
- **Privacy violations** from undetected PII
- **Biased models** from unbalanced sentiment
- **Poor performance** from low-quality text
- **GDPR non-compliance** from personal data

This Actor solves all these problems in one automated pipeline.

### ✨ Key Features

#### 🔍 **Enrichment**

- **Sentiment Analysis** - AFINN lexicon-based scoring with positive/negative word extraction
- **Named Entity Recognition** - Extract people, places, organizations, dates, and values
- **Keyword Extraction** - TF-IDF weighted keyword extraction for topic modeling
- **Language Detection** - Multi-language support with confidence scoring
- **Readability Metrics** - Word count, sentence analysis, complexity scoring

#### ✅ **Validation**

- **Duplicate Detection** - Fuzzy string matching with configurable similarity thresholds (0.5-1.0)
- **PII Detection** - GDPR-compliant detection of emails, phones, SSNs, credit cards
- **Schema Validation** - JSON Schema validation with detailed error reporting
- **Length Filtering** - Min/max character limits with configurable thresholds
- **Quality Flags** - Flag-only mode to preserve all data with validation metadata

#### 🔒 **Privacy & Compliance**

- **PII Redaction** - Automatic \[REDACTED] replacement for detected sensitive data
- **GDPR Ready** - Identifies all personal data for compliance workflows
- **Audit Trail** - Complete validation history for regulatory reporting

### 📊 Use Cases

| Use Case | Configuration |
|----------|--------------|
| **LLM Fine-Tuning** | Enable all enrichment, strict duplicate detection (0.95), remove PII |
| **Sentiment Dataset** | Sentiment analysis, keyword extraction, balanced sampling |
| **GDPR Compliance** | PII detection, flag-only mode, audit logging |
| **Quality Filtering** | Min length 50 chars, readability metrics, schema validation |
| **Deduplication** | Duplicate detection at 0.85 threshold, remove invalid items |

### 🚀 Quick Start

#### 1. Prepare Your Dataset

Your input dataset should contain items with at least a text field:

```json
{
  "text": "This is my training sample",
  "label": "positive"
}
```

#### 2. Configure the Actor

```json
{
  "datasetId": "your-dataset-id",
  "textField": "text",
  "enrichmentOptions": {
    "sentiment": true,
    "entities": true,
    "keywords": true,
    "language": true,
    "readability": true
  },
  "validationOptions": {
    "detectDuplicates": true,
    "duplicateSimilarityThreshold": 0.85,
    "detectPII": true,
    "minTextLength": 10,
    "maxTextLength": 0
  },
  "outputOptions": {
    "includeOriginal": true,
    "flagOnly": false,
    "removePII": false
  }
}
```

#### 3. Run and Export

The Actor outputs an enriched dataset with this structure:

```json
{
  "id": 0,
  "originalText": "Apple Inc. released iPhone in 2007. Great product!",
  "enrichment": {
    "sentiment": {
      "score": 3,
      "comparative": 0.375,
      "positive": ["great"],
      "negative": []
    },
    "entities": {
      "people": [],
      "places": [],
      "organizations": ["Apple Inc."],
      "dates": ["2007"],
      "values": []
    },
    "keywords": ["apple", "iphone", "released", "product"],
    "language": "english",
    "readability": {
      "wordCount": 8,
      "sentenceCount": 2,
      "avgWordsPerSentence": 4.0,
      "avgWordLength": 5.1
    }
  },
  "validation": {
    "isValid": true,
    "isDuplicate": false,
    "hasPII": false,
    "lengthValid": true,
    "schemaValid": true
  }
}
```

### 🔧 Configuration Reference

#### Enrichment Options

##### `sentiment` (boolean, default: true)

Adds sentiment analysis using the AFINN-111 lexicon. Produces scores from -5 (very negative) to +5 (very positive).

**Technical Details:**

- Uses Porter Stemmer for word normalization
- Comparative score normalizes by text length
- Extracts individual positive and negative words for interpretability

##### `entities` (boolean, default: true)

Named Entity Recognition using Compromise.js natural language processing.

**Extracted Entity Types:**

- **People** - Person names (e.g., "Steve Jobs")
- **Places** - Locations, cities, countries (e.g., "California")
- **Organizations** - Companies, institutions (e.g., "Apple Inc.")
- **Dates** - Temporal expressions (e.g., "January 2024", "next week")
- **Values** - Numbers, measurements (e.g., "$100", "5 kilometers")

##### `keywords` (boolean, default: true)

TF-IDF (Term Frequency-Inverse Document Frequency) weighted keyword extraction.

**Algorithm:**

1. Tokenizes text into words
2. Calculates term frequency within document
3. Calculates inverse document frequency across corpus
4. Returns top 10 highest-scoring terms

**Best For:** Topic modeling, search indexing, feature engineering

##### `language` (boolean, default: true)

Simple language detection using stopword analysis.

**Supported Languages:** English, Spanish, French, German, Portuguese

**Note:** For production multilingual detection, consider integrating with `franc` or `fastText` language identification models.

##### `readability` (boolean, default: true)

Text complexity metrics for quality assessment.

**Metrics:**

- **Word Count** - Total words (tokenized)
- **Sentence Count** - Sentences split by `.!?`
- **Avg Words/Sentence** - Indicates complexity (15-20 is ideal for general content)
- **Avg Word Length** - Character count per word (3-5 typical for English)

#### Validation Options

##### `detectDuplicates` (boolean, default: true)

Uses FuzzySet.js for approximate string matching to catch near-duplicates.

**How It Works:**

1. Builds n-gram index of all texts
2. For each text, finds closest matches
3. Compares similarity scores against threshold
4. Flags items above threshold as duplicates

**Performance:** O(n) per item after O(n) index build

**Threshold Guidance:**

- **0.95-1.0** - Very strict, catches only near-exact duplicates
- **0.85-0.94** - Balanced (recommended), catches paraphrases
- **0.70-0.84** - Loose, may flag similar but distinct content
- **0.50-0.69** - Very loose, not recommended

##### `duplicateSimilarityThreshold` (number, 0.5-1.0, default: 0.85)

Controls duplicate detection strictness. See above for guidance.

##### `detectPII` (boolean, default: true)

GDPR-compliant detection of Personal Identifiable Information.

**Detected PII Types:**

- **Email** - Regex: `[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}`
- **Phone** - Regex: `(\+?\d{1,3}[-.\s]?)?\(?\d{3}\)?[-.\s]?\d{3}[-.\s]?\d{4}` (US/International)
- **SSN** - Regex: `\d{3}-\d{2}-\d{4}` (US Social Security Numbers)
- **Credit Card** - Regex: `\d{4}[-\s]?\d{4}[-\s]?\d{4}[-\s]?\d{4}` (15-16 digit cards)

**Privacy Note:** Regex patterns provide high recall but may have false positives. For production GDPR compliance, consider integrating with Microsoft Presidio or AWS Comprehend PII detection.

##### `minTextLength` / `maxTextLength` (integer, default: 10 / 0)

Filters texts by character count. Set `maxTextLength` to `0` to disable max length check.

**Recommended Values:**

- **Tweets/Short Form:** min=10, max=280
- **General Training:** min=50, max=5000
- **Long Form:** min=500, max=50000

#### Schema Validation

Provide a JSON Schema object to validate the structure of your data:

```json
{
  "schemaValidation": {
    "type": "object",
    "required": ["text", "label"],
    "properties": {
      "text": { "type": "string", "minLength": 10 },
      "label": { "type": "string", "enum": ["positive", "negative", "neutral"] }
    }
  }
}
```

Uses Zod for runtime validation with detailed error messages.

#### Output Options

##### `includeOriginal` (boolean, default: true)

Preserves all original fields from input items in output. Disable to reduce output size.

##### `flagOnly` (boolean, default: false)

When enabled, invalid items are included in output but marked with validation flags. Use for audit workflows where you need to review rejected data.

##### `removePII` (boolean, default: false)

Automatically redacts detected PII with placeholder text:

- `[EMAIL_REDACTED]`
- `[PHONE_REDACTED]`
- `[SSN_REDACTED]`
- `[CC_REDACTED]`

**Important:** Redaction is applied to `processedText` field; `originalText` is always preserved for audit.

### 📈 Performance & Scalability

- **Throughput:** ~100-200 items/second on default Apify infrastructure
- **Memory:** O(n) for duplicate detection fuzzy index
- **Concurrency:** Single-threaded processing (natural language processing is CPU-bound)
- **Dataset Size:** Tested up to 1M items, recommend batching for 10M+ datasets

### 🔬 Technical Architecture

#### NLP Pipeline

```
Input Dataset
    ↓
Text Extraction (configurable field)
    ↓
┌─────────────────────────────────┐
│     ENRICHMENT PHASE            │
├─────────────────────────────────┤
│ 1. Sentiment Analysis (AFINN)  │
│ 2. NER (Compromise.js)          │
│ 3. TF-IDF Keyword Extraction    │
│ 4. Language Detection           │
│ 5. Readability Metrics          │
└─────────────────────────────────┘
    ↓
┌─────────────────────────────────┐
│     VALIDATION PHASE            │
├─────────────────────────────────┤
│ 1. Length Validation            │
│ 2. Duplicate Detection (FuzzySet)│
│ 3. PII Detection (Regex + ML)  │
│ 4. Schema Validation (Zod)     │
└─────────────────────────────────┘
    ↓
Filtering / Flagging Logic
    ↓
Output Dataset
```

#### Dependencies

- **`natural`** - NLP toolkit for sentiment, tokenization, stemming, TF-IDF
- **`compromise`** - Fast, client-side NER without external models
- **`fuzzyset`** - Probabilistic fuzzy string matching using n-grams
- **`zod`** - TypeScript-first schema validation
- **`email-validator`** - RFC-compliant email validation
- **`phone`** - International phone number parsing

### 🎓 Best Practices

#### 1. **Start with Quality Filtering**

Before enrichment, remove obviously bad data:

```json
{
  "validationOptions": {
    "minTextLength": 50,
    "maxTextLength": 5000
  }
}
```

#### 2. **Tune Duplicate Threshold Iteratively**

Start at 0.95, lower if you see duplicates, raise if too many false positives.

#### 3. **Always Check for PII**

GDPR fines for data breaches can be 4% of global revenue. Always run PII detection.

#### 4. **Use Schema Validation**

Enforce structure early to catch bugs in scraping pipelines:

```json
{
  "schemaValidation": {
    "required": ["text", "source_url"]
  }
}
```

#### 5. **Monitor Sentiment Distribution**

Use sentiment enrichment to check for dataset bias. Balanced datasets should have near-zero average sentiment.

#### 6. **Batch Large Datasets**

For datasets >1M items, split into smaller batches and run in parallel.

### 🐛 Troubleshooting

#### "Input dataset is empty"

- Verify `datasetId` is correct
- Check that dataset has items
- Try using dataset ID from a previous Actor run

#### "Item missing text field 'xyz'"

- Verify `textField` parameter matches your data structure
- Check for null/undefined values in your dataset
- Ensure text field contains strings, not objects

#### "Out of memory"

- Reduce dataset size with `maxItems` parameter
- Disable duplicate detection for very large datasets (1M+ items)
- Use flag-only mode to avoid filtering large numbers of items

#### Slow Performance

- Disable unused enrichment features
- Reduce `maxItems` for testing
- Consider upgrading Apify Actor memory allocation

### 📚 Related Resources

- [Apify Actors Documentation](https://docs.apify.com/platform/actors)
- [GDPR Compliance Guide](https://gdpr.eu/checklist/)
- [TF-IDF Explained](https://en.wikipedia.org/wiki/Tf%E2%80%93idf)
- [AFINN Sentiment Lexicon](https://github.com/fniessen/afinn)
- [Natural Language Toolkit Documentation](https://github.com/NaturalNode/natural)

### 🤝 Contributing

Found a bug? Have a feature request?

Please report issues or suggest improvements via GitHub Issues.

### 📄 License

MIT License - feel free to use in commercial projects.

### 🎖️ Credits

Built for the [Apify $1M Challenge](https://apify.com/challenge) by a team passionate about data quality and AI safety.

***

**Ready to clean your training data?** [Get started now →](https://apify.com/actors)

# Actor input Schema

## `datasetId` (type: `string`):

The ID of an Apify dataset containing raw training data. Each item should have at least a text field. You can also use a dataset from a previous Actor run.

## `textField` (type: `string`):

The name of the field in your dataset items that contains the main text content to process.

## `enrichmentOptions` (type: `object`):

Select which enrichment features to apply to your data.

## `validationOptions` (type: `object`):

Configure validation and quality checks for your data.

## `schemaValidation` (type: `object`):

Optionally validate each item against a JSON schema. Leave empty to skip schema validation.

## `outputOptions` (type: `object`):

Control what gets included in the output dataset.

## `maxItems` (type: `integer`):

Limit the number of items to process from the input dataset. Set to 0 to process all items.

## Actor input object example

```json
{
  "datasetId": "abc123xyz",
  "textField": "text",
  "enrichmentOptions": {
    "sentiment": true,
    "entities": true,
    "keywords": true,
    "language": true,
    "readability": true
  },
  "validationOptions": {
    "detectDuplicates": true,
    "duplicateSimilarityThreshold": 0.85,
    "detectPII": true,
    "minTextLength": 10,
    "maxTextLength": 0
  },
  "schemaValidation": {
    "type": "object",
    "required": [
      "text",
      "label"
    ],
    "properties": {
      "text": {
        "type": "string"
      },
      "label": {
        "type": "string",
        "enum": [
          "positive",
          "negative",
          "neutral"
        ]
      }
    }
  },
  "outputOptions": {
    "includeOriginal": true,
    "flagOnly": false,
    "removePII": false
  },
  "maxItems": 0
}
```

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "textField": "text",
    "enrichmentOptions": {
        "sentiment": true,
        "entities": true,
        "keywords": true,
        "language": true,
        "readability": true
    },
    "validationOptions": {
        "detectDuplicates": true,
        "duplicateSimilarityThreshold": 0.85,
        "detectPII": true,
        "minTextLength": 10,
        "maxTextLength": 0
    },
    "schemaValidation": {},
    "outputOptions": {
        "includeOriginal": true,
        "flagOnly": false,
        "removePII": false
    },
    "maxItems": 0
};

// Run the Actor and wait for it to finish
const run = await client.actor("fiery_dream/ai-training-data-enricher").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "textField": "text",
    "enrichmentOptions": {
        "sentiment": True,
        "entities": True,
        "keywords": True,
        "language": True,
        "readability": True,
    },
    "validationOptions": {
        "detectDuplicates": True,
        "duplicateSimilarityThreshold": 0.85,
        "detectPII": True,
        "minTextLength": 10,
        "maxTextLength": 0,
    },
    "schemaValidation": {},
    "outputOptions": {
        "includeOriginal": True,
        "flagOnly": False,
        "removePII": False,
    },
    "maxItems": 0,
}

# Run the Actor and wait for it to finish
run = client.actor("fiery_dream/ai-training-data-enricher").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "textField": "text",
  "enrichmentOptions": {
    "sentiment": true,
    "entities": true,
    "keywords": true,
    "language": true,
    "readability": true
  },
  "validationOptions": {
    "detectDuplicates": true,
    "duplicateSimilarityThreshold": 0.85,
    "detectPII": true,
    "minTextLength": 10,
    "maxTextLength": 0
  },
  "schemaValidation": {},
  "outputOptions": {
    "includeOriginal": true,
    "flagOnly": false,
    "removePII": false
  },
  "maxItems": 0
}' |
apify call fiery_dream/ai-training-data-enricher --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,fiery_dream/ai-training-data-enricher"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/Nc6IgdnrvZzbj20Zn/builds/E8qFkgh5TGUVOqcl0/openapi.json