# Wikipedia Scraper (`automation-lab/wikipedia-scraper`) Actor

Search and extract Wikipedia articles — titles, summaries, full content, categories, and images. Uses the free MediaWiki API.

- **URL**: https://apify.com/automation-lab/wikipedia-scraper.md
- **Developed by:** [Stas Persiianenko](https://apify.com/automation-lab) (community)
- **Categories:** Education, AI
- **Stats:** 43 total users, 9 monthly users, 100.0% runs succeeded, 1 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.60 / 1,000 article extracteds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Wikipedia Scraper

Extract Wikipedia articles by keyword search. Get titles, full summaries, URLs, word counts, thumbnails, and last edit dates from any of Wikipedia's 300+ language editions.

### What does Wikipedia Scraper do?

Wikipedia Scraper searches Wikipedia using the official MediaWiki API and extracts structured data from matching articles. For each search keyword, it returns article metadata including the introductory `extract` (summary), word count, page size, thumbnail image, and direct URL. Enable `includeFullContent` to also receive the full article as plain text in `fullContent`, ready for RAG and knowledge-base datasets.

The scraper uses Wikipedia's built-in search API, so results match what you'd find searching on Wikipedia itself — ranked by relevance with support for all Wikipedia languages.

### Who is it for?

- 🎓 **Academic researchers** — extracting structured knowledge from Wikipedia articles at scale
- 🤖 **NLP engineers** — building training datasets from Wikipedia text and metadata
- 📊 **Data analysts** — collecting factual data and statistics from Wikipedia pages
- 💻 **App developers** — enriching applications with Wikipedia content and summaries
- 📝 **Content creators** — gathering reference material and structured facts for writing

### Why scrape Wikipedia?

Wikipedia is the world's largest free encyclopedia with over 60 million articles across 300+ languages. It's a primary source for:

- **Knowledge base construction** — build reference datasets for AI training, chatbots, or research databases
- **LLM and RAG pipelines** — feed clean, structured article text into retrieval-augmented generation systems, fine-tuning datasets, or AI agent knowledge bases
- **Content enrichment** — add Wikipedia summaries to product catalogs, educational platforms, or content management systems
- **Research and analysis** — analyze article coverage, word counts, and edit patterns across topics
- **Multilingual data** — gather information in any language Wikipedia supports
- **SEO and content strategy** — understand topic coverage and find content gaps

### How much does it cost to scrape Wikipedia?

Wikipedia Scraper uses pay-per-event pricing:

| Event | Price |
|-------|-------|
| Run started | $0.001 |
| Article extracted | $0.001 per article |

**Example costs:**

- 10 articles on "machine learning": ~$0.011
- 100 articles on "history": ~$0.101
- 500 articles across 5 keywords: ~$0.506

Platform costs are minimal — a typical run uses under $0.001 in compute. Wikipedia's API is fast and does not require proxies.

### Input parameters

| Parameter | Type | Description | Default |
|-----------|------|-------------|---------|
| `searchQueries` | string\[] | Keywords to search on Wikipedia. Each keyword runs a separate search. | Required |
| `language` | string | Wikipedia language code (e.g., `en`, `de`, `fr`, `es`, `ja`, `zh`) | `"en"` |
| `maxResultsPerSearch` | integer | Maximum articles per keyword (1–500) | `50` |
| `includeFullContent` | boolean | Add the full article plaintext in `fullContent`; `extract` remains the introductory summary | `false` |

#### Input example

```json
{
  "searchQueries": ["artificial intelligence", "quantum computing"],
  "language": "en",
  "maxResultsPerSearch": 20,
  "includeFullContent": true
}
```

### Output example

Each article is returned as a JSON object:

```json
{
  "pageId": 1164,
  "title": "Artificial intelligence",
  "extract": "Artificial intelligence (AI) is the capability of computational systems to perform tasks typically associated with human intelligence...",
  "fullContent": "Artificial intelligence (AI), in its broadest sense, is intelligence exhibited by machines...",
  "url": "https://en.wikipedia.org/wiki/Artificial_intelligence",
  "wordCount": 26473,
  "size": 266568,
  "lastEdited": "2026-03-02T11:28:15Z",
  "thumbnail": "https://upload.wikimedia.org/wikipedia/commons/thumb/...",
  "scrapedAt": "2026-03-03T04:08:23.785Z"
}
```

#### Output fields

| Field | Type | Description |
|-------|------|-------------|
| `pageId` | number | Wikipedia internal page identifier |
| `title` | string | Article title |
| `extract` | string | Introductory summary (plain text, no HTML) |
| `fullContent` | string | Full article plaintext when `includeFullContent` is enabled; empty if Wikipedia has no full extract |
| `url` | string | Direct link to the Wikipedia article |
| `wordCount` | number | Total word count of the article |
| `size` | number | Article size in bytes |
| `lastEdited` | string | ISO timestamp of the last edit |
| `thumbnail` | string | URL to article thumbnail image (if available) |
| `scrapedAt` | string | ISO timestamp when the data was extracted |

### Supported languages

Wikipedia Scraper supports all 300+ Wikipedia language editions. Use the standard language code:

| Code | Language | Articles |
|------|----------|----------|
| `en` | English | 6.9M+ |
| `de` | German | 2.9M+ |
| `fr` | French | 2.6M+ |
| `es` | Spanish | 2.0M+ |
| `ja` | Japanese | 1.4M+ |
| `ru` | Russian | 1.9M+ |
| `zh` | Chinese | 1.4M+ |
| `pt` | Portuguese | 1.1M+ |
| `it` | Italian | 1.8M+ |
| `ar` | Arabic | 1.2M+ |

Any valid Wikipedia language code works — see the [full list](https://meta.wikimedia.org/wiki/List_of_Wikipedias).

### How to scrape Wikipedia articles

1. Open [Wikipedia Scraper](https://apify.com/automation-lab/wikipedia-scraper) on Apify.
2. Enter one or more search keywords in the `searchQueries` field.
3. Set the `language` code (e.g., `en`, `de`, `fr`) for the Wikipedia edition you want.
4. Adjust `maxResultsPerSearch` to control how many articles per keyword (default: 50).
5. Optionally enable `includeFullContent` to add each full article in the `fullContent` field.
6. Click **Start** and wait for the scrape to finish.
7. Download articles as JSON, CSV, or Excel from the Dataset tab.

### API usage

#### Python

```python
from apify_client import ApifyClient

client = ApifyClient("YOUR_API_TOKEN")

run = client.actor("automation-lab/wikipedia-scraper").call(run_input={
    "searchQueries": ["climate change", "renewable energy"],
    "language": "en",
    "maxResultsPerSearch": 20,
})

for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(f"{item['title']} — {item['wordCount']} words")
    print(f"  {item['url']}")
    print(f"  {item['extract'][:200]}...")
```

#### Node.js

```javascript
import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: 'YOUR_API_TOKEN' });

const run = await client.actor('automation-lab/wikipedia-scraper').call({
    searchQueries: ['climate change', 'renewable energy'],
    language: 'en',
    maxResultsPerSearch: 20,
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach(item => {
    console.log(`${item.title} — ${item.wordCount} words`);
    console.log(`  ${item.url}`);
});
```

#### REST API

```bash
curl -X POST "https://api.apify.com/v2/acts/automation-lab/wikipedia-scraper/runs?token=YOUR_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "searchQueries": ["artificial intelligence"],
    "language": "en",
    "maxResultsPerSearch": 10
  }'
```

### Integrations

Connect Wikipedia Scraper to hundreds of apps using built-in integrations:

- **Google Sheets** — export article data to spreadsheets
- **Slack / Microsoft Teams** — get notifications when scraping completes
- **Zapier / Make** — trigger workflows with scraped Wikipedia data
- **Amazon S3 / Google Cloud Storage** — store large datasets in cloud storage
- **Webhook** — send results to your own API endpoint

### Tips and best practices

1. **Use specific keywords** — more specific searches return more relevant results. "Quantum entanglement" is better than "quantum".
2. **Batch keywords efficiently** — combine related keywords in one run to save on startup costs.
3. **Language parameter** — set the language code to search non-English Wikipedias. Results, summaries, and URLs will all be in the selected language.
4. **Word count filtering** — use the `wordCount` field to filter out stub articles (typically < 500 words).
5. **Rate limits** — Wikipedia's API is generous but has rate limits. The scraper handles pagination and batching automatically.
6. **Choose the content depth** — the `extract` field always remains the short introductory summary. Enable `includeFullContent` when your workflow needs full article plaintext in `fullContent`; this opt-in mode makes one additional batched Wikipedia API request.
7. **Full-content scope** — `fullContent` contains plaintext only. It does not add categories, external links, coordinates, Wikidata IDs, or HTML.
8. **Max 500 results per keyword** — this is a Wikipedia API limit. For broader coverage, use multiple related keywords.

### Legality

Scraping publicly available data is generally legal according to the [US Court of Appeals ruling](https://en.wikipedia.org/wiki/HiQ_Labs_v._LinkedIn) (HiQ Labs v. LinkedIn). This actor only accesses publicly available information and does not require authentication. Always review and comply with the target website's Terms of Service before scraping. For personal data, ensure compliance with GDPR, CCPA, and other applicable privacy regulations.

### FAQ

**Q: Does this scraper get the full article text?**
A: Yes, as an opt-in feature. Set `includeFullContent` to `true` and read the full article plaintext from `fullContent`. The existing `extract` field still contains only the introductory section. If Wikipedia returns no full extract for an article, `fullContent` is an empty string and the base article record is retained.

**Q: How fast is it?**
A: Very fast. Wikipedia's API is highly optimized. A typical run extracting 50 articles completes in under 5 seconds.

**Q: Does it need proxies?**
A: No. Wikipedia's API is open and does not block automated requests. The scraper identifies itself with a proper User-Agent header.

**Q: Can I search in multiple languages at once?**
A: Each run uses one language. To search multiple languages, run the scraper once per language.

### Use with Claude AI (MCP)

This actor is available as a tool in Claude AI through the Model Context Protocol (MCP). Add it to Claude Desktop, Cursor, Windsurf, or any MCP-compatible client.

#### Setup for Claude Code

```bash
claude mcp add --transport http apify "https://mcp.apify.com/?tools=automation-lab/wikipedia-scraper"
```

#### Setup for Claude Desktop, Cursor, or VS Code

Add this to your MCP config file:

```json
{
    "mcpServers": {
        "apify": {
            "url": "https://mcp.apify.com/?tools=automation-lab/wikipedia-scraper"
        }
    }
}
```

#### Example prompts

- "Search Wikipedia for articles about quantum computing and give me the summaries"
- "Fetch Wikipedia articles on these 5 historical events and compare their word counts"
- "Look up Wikipedia articles on machine learning in both English and German and extract the introductions"

Learn more in the [Apify MCP documentation](https://docs.apify.com/platform/integrations/mcp).

**The extract is truncated or too short.**
The `extract` field intentionally contains only the article's introductory section. Set `includeFullContent` to `true` to receive the full article plaintext in `fullContent` while preserving the concise introductory `extract`.

**I'm getting irrelevant results for my search query.**
Wikipedia's search API ranks by relevance, which may include loosely related articles. Use more specific keywords (e.g., "quantum entanglement" instead of "quantum") and reduce `maxResultsPerSearch` to get only the top matches.

### Other research and news scrapers on Apify

- [ArXiv Scraper](https://apify.com/automation-lab/arxiv-scraper) -- search and extract academic papers from ArXiv
- [CrossRef Scraper](https://apify.com/automation-lab/crossref-scraper) -- extract scholarly article metadata from CrossRef
- [OpenAlex Scraper](https://apify.com/automation-lab/openalex-scraper) -- search and extract academic research data from OpenAlex

# Actor input Schema

## `searchQueries` (type: `array`):

Keywords to search on Wikipedia. Each keyword runs a separate search.

## `language` (type: `string`):

Wikipedia language code (e.g. en, de, fr, es, ja).

## `maxResultsPerSearch` (type: `integer`):

Maximum articles per keyword (max 500).

## `includeFullContent` (type: `boolean`):

Fetch the full article as plain text in the fullContent field. The extract field remains the introductory summary.

## Actor input object example

```json
{
  "searchQueries": [
    "artificial intelligence"
  ],
  "language": "en",
  "maxResultsPerSearch": 20,
  "includeFullContent": false
}
```

# Actor output Schema

## `overview` (type: `string`):

Dataset containing the extracted Wikipedia articles.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "searchQueries": [
        "artificial intelligence"
    ],
    "language": "en",
    "maxResultsPerSearch": 20
};

// Run the Actor and wait for it to finish
const run = await client.actor("automation-lab/wikipedia-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "searchQueries": ["artificial intelligence"],
    "language": "en",
    "maxResultsPerSearch": 20,
}

# Run the Actor and wait for it to finish
run = client.actor("automation-lab/wikipedia-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "searchQueries": [
    "artificial intelligence"
  ],
  "language": "en",
  "maxResultsPerSearch": 20
}' |
apify call automation-lab/wikipedia-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,automation-lab/wikipedia-scraper"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/Nka97qJs0YHXosATX/builds/C5uweXbULgOcTKnaY/openapi.json