AI Training Data Collector — Structured Web Datasets for LLMs

Crawl websites and extract structured, clean text datasets perfect for fine-tuning LLMs and RAG pipelines. This AI training data collector removes boilerplate, deduplicates content, and scores quality for AI training.

How It Works

The actor launches a Cheerio crawler from your start URLs and follows internal links up to the configured crawl depth, skipping any URL that matches an exclude pattern. For each page it strips nav, header, footer, sidebar, ads, and script elements, then isolates the main content region and converts the remaining HTML into markdown, plain text, or a structured JSON object. It counts words, headings, paragraphs, links, and images, computes a 0-100 quality score from length, vocabulary diversity, sentence count, and structure, and deduplicates via an MD5 hash of the first 2,000 characters before pushing the record to the dataset.

Features

• Smart content extraction: Removes navigation, ads, footers, and boilerplate
• Multi-format output: Markdown, plain text, or JSON-Lines
• Quality scoring: Each page scored 0-100 for training suitability
• Deduplication: Content hash-based deduplication across pages
• Configurable depth: 0-3 levels of crawl depth from start URLs
• Pattern exclusion: Skip unwanted URL patterns (tags, categories, etc.)

Input Parameters

Parameter	Type	Default	Description
urls	array	Wikipedia AI page	Start URLs to crawl
crawlDepth	integer	1	Link follow depth (0-3)
maxPages	integer	5	Max pages to process
outputFormat	string	markdown	Content format
excludePatterns	array	tags, categories	URL patterns to skip
minWordCount	integer	100	Skip short pages
proxyConfiguration	object	Apify Proxy	Proxy for reliable scraping

Output Example

{
  "url": "https://en.wikipedia.org/wiki/Artificial_intelligence",
  "title": "Artificial intelligence - Wikipedia",
  "wordCount": 4128,
  "qualityScore": 87,
  "sourceDomain": "en.wikipedia.org",
  "contentType": "start",
  "language": "en",
  "crawlDepth": 0,
  "headingCount": 56,
  "paragraphCount": 142,
  "linkCount": 311,
  "images": 14,
  "contentHash": "a3f9c1e8b2d4",
  "extractionMethod": "cheerio-html2text",
  "scrapedAt": "2026-07-27T16:00:00.000Z",
  "cleanText": "# Artificial intelligence\n\nArtificial intelligence (AI) is the intelligence of machines..."
}

Use Cases

• LLM fine-tuning: Build custom training datasets from any website
• RAG pipelines: Create knowledge base documents for retrieval-augmented generation
• Research datasets: Collect structured content for academic research
• Competitive analysis: Extract and analyze competitor website content

Cost Estimate

• 5 pages: ~$0.25
• 100 pages: ~$2.00
• 1000 pages: ~$15.00