# Sitemap Content Extractor (`darknezz/sitemap-content-extractor`) Actor

Crawl any website sitemap.xml and extract structured content from each page. Full-text extraction, metadata, headings, and word counts for SEO audits and content inventories.

- **URL**: https://apify.com/darknezz/sitemap-content-extractor.md
- **Developed by:** [Oaida Adrian](https://apify.com/darknezz) (community)
- **Categories:** SEO tools, AI, Developer tools
- **Stats:** 2 total users, 0 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

Pay per usage

This Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage, which gets cheaper the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-usage

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Sitemap Content Extractor — Crawl a Sitemap, Extract Clean Content

Point this Actor at any website's `sitemap.xml` and it crawls every listed URL and returns **clean full-text content plus metadata** for each page — title, meta description, keywords, H1 headings, word count, and last-modified date. Perfect for SEO audits, content inventories, site migrations, and building AI training corpora from documentation sites.

No browser automation to configure, no page-by-page URL lists to maintain — the sitemap *is* the input.

**Free to use** — no per-page charges, no start fee. You only pay Apify's standard platform usage for the compute your run consumes.

### Why this Actor

- **Sitemap index aware** — handles both plain sitemaps (`<urlset>`) and sitemap indexes (`<sitemapindex>`), recursively following every child sitemap.
- **Gzip support** — reads `.xml.gz` sitemaps transparently.
- **Clean extraction** — uses trafilatura to strip nav/ads/boilerplate and return just the article text.
- **Precise scoping** — include/exclude URL regex patterns so you crawl only `/blog/` or skip `/tag/` pages.
- **Structured output** — one dataset item per page, ready for search indexing, embeddings, or a content spreadsheet.

### How it works

Give it a `sitemapUrl`. The Actor fetches and parses the sitemap (following index files and gzip automatically), applies your include/exclude filters, then visits up to `maxUrls` pages and extracts clean content and metadata from each. Set `extractContent: false` to inventory URLs and metadata only (faster, no page fetches).

**Budget semantics:** `maxUrls` is a hard cap on pages *processed*, applied *after* filtering — the Actor keeps scanning child sitemaps until it has collected `maxUrls` URLs that match your `includePatterns` (or the sitemap index is exhausted), so a broad pattern can never push the crawl over your budget, a narrow pattern still fills it where matches exist, and duplicate URLs across child sitemaps are collected once.

### Input

```json
{
  "sitemapUrl": "https://apify.com/sitemap.xml",
  "maxUrls": 50,
  "extractContent": true,
  "includePatterns": ["/blog/"],
  "excludePatterns": ["/tag/", "/author/"],
  "proxyConfiguration": { "useApifyProxy": true }
}
```

| Field | Type | Required | Default | Description |
|---|---|---|---|---|
| `sitemapUrl` | string | Yes | — | URL to `sitemap.xml` or a sitemap index |
| `maxUrls` | integer | No | 50 | Maximum URLs to process |
| `extractContent` | boolean | No | true | Fetch each page and extract full text |
| `includePatterns` | array | No | \[] | Only process URLs matching these regex patterns |
| `excludePatterns` | array | No | \[] | Skip URLs matching these regex patterns |
| `proxyConfiguration` | object | No | Apify Proxy | Proxy settings for page fetches |

### Output

One item per page:

```json
{
  "url": "https://apify.com/blog/web-scraping-guide",
  "title": "The Complete Web Scraping Guide",
  "content": "Web scraping is the process of ...",
  "wordCount": 2184,
  "metaDescription": "Learn web scraping from scratch...",
  "metaKeywords": "web scraping, crawling",
  "h1Headings": ["The Complete Web Scraping Guide"],
  "lastmod": "2026-06-30",
  "extractedAt": "2026-07-18T09:12:44Z"
}
```

### Use cases

- 🔍 **SEO audits** — inventory every indexable page, spot missing titles/meta descriptions, and measure content depth by word count.
- 🚚 **Site migrations** — pull all content from a legacy site into a structured dataset before rebuilding.
- 🤖 **AI training data** — collect clean text from documentation and blog sitemaps to feed LLM fine-tuning or RAG.
- 📚 **Documentation indexing** — build a searchable index of a docs site from its sitemap in one run.
- 🕵️ **Competitor analysis** — map a competitor's content coverage and structure across their whole site.

### Run it from your code

**Python (Apify SDK):**

```python
from apify_client import ApifyClient

client = ApifyClient("YOUR-APIFY-TOKEN")

run_input = {
    "sitemapUrl": "https://apify.com/sitemap.xml",
    "maxUrls": 100,
    "includePatterns": ["/blog/"],
}

run = client.actor("darknezz/sitemap-content-extractor").call(run_input=run_input)
for page in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(page["url"], page["wordCount"], page["title"])
```

**cURL (one-liner, sync):**

```bash
curl -X POST "https://api.apify.com/v2/acts/darknezz~sitemap-content-extractor/run-sync-get-dataset-items?token=YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"sitemapUrl":"https://apify.com/sitemap.xml","maxUrls":100,"includePatterns":["/blog/"]}'
```

**Worked example** — pointing the actor at a documentation site's sitemap with `includePatterns: ["/docs/"]` returns every docs URL with its clean article text and word count; the `lastmod` field shows which pages changed since your last crawl, so a scheduled daily run doubles as a change detector.

**Scheduling:** attach an Apify Schedule (e.g. daily) to re-crawl a sitemap and keep a content inventory or search index continuously fresh — the `lastmod` field lets you detect which pages changed.

### Pricing

**Free.** This Actor carries no per-page charges — no `page-extracted` fee, no start fee. You pay only Apify's standard platform usage (compute + storage) for your runs.

### Companion tools

No sitemap, or need **LLM-ready text from sites that don't publish one**? **[AI Web Content Crawler](https://apify.com/darknezz/ai-web-crawler)** (pay-per-event) crawls arbitrary websites and returns clean, token-estimated chunks with heading structure, tables and JSON-LD — one dataset item per page, no post-processing.

### FAQ

**Does it follow sitemap index files?** Yes — nested `<sitemapindex>` files are followed recursively, so a single index URL crawls the whole site.

**Can I crawl only part of a site?** Yes — use `includePatterns` / `excludePatterns` with regex to scope to specific sections (e.g. only `/blog/`, skip `/tag/`).

**What if a site has no sitemap?** This Actor requires a sitemap URL. For arbitrary link-following crawls, use a general web crawler instead.

**Do I need a proxy?** Most sitemaps and pages fetch fine over Apify Proxy (the default). Sites with heavy anti-bot protection may need residential proxies.

**How do I get metadata without full text?** Set `extractContent: false` — you still get title, meta tags, and `lastmod` per URL, much faster.

**How is the "clean content" extracted?** The actor runs trafilatura, which scores HTML blocks and keeps only the main article content — nav, sidebars, cookie banners and boilerplate are dropped before you see the text.

### Limitations

- **Requires a real sitemap** — sites without `sitemap.xml` (or a sitemap index) cannot be crawled by this actor.
- **`maxUrls` is a hard cap** — very large sites (100k+ URLs) need multiple runs or a raised `maxUrls`; the actor stops at the cap.
- **JS-rendered content** — pages that render their text with client-side JavaScript may yield little or no text without a browser; plain HTML pages extract best.
- **Proxy-dependent pages** — heavily protected sites may need `proxyConfiguration` with residential proxies (extra cost applies on Apify's side, not ours).

Enjoying the extractor? A quick review on the Apify Store helps others find it.

# Actor input Schema

## `sitemapUrl` (type: `string`):

URL to sitemap.xml or sitemap index file.

## `maxUrls` (type: `integer`):

Maximum URLs to process.

## `extractContent` (type: `boolean`):

Visit each page and extract full text content.

## `includePatterns` (type: `array`):

Only process URLs matching these regex patterns.

## `excludePatterns` (type: `array`):

Skip URLs matching these regex patterns.

## `proxyConfiguration` (type: `object`):

Proxy settings for scraping.

## Actor input object example

```json
{
  "sitemapUrl": "https://apify.com/sitemap.xml",
  "maxUrls": 50,
  "extractContent": true,
  "includePatterns": [],
  "excludePatterns": [],
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}
```

# Actor output Schema

## `results` (type: `string`):

No description

## `url` (type: `string`):

No description

## `lastmod` (type: `string`):

No description

## `title` (type: `string`):

No description

## `metaDescription` (type: `string`):

No description

## `content` (type: `string`):

No description

## `wordCount` (type: `string`):

No description

## `extractedAt` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "sitemapUrl": "https://apify.com/sitemap.xml",
    "proxyConfiguration": {
        "useApifyProxy": true
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("darknezz/sitemap-content-extractor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "sitemapUrl": "https://apify.com/sitemap.xml",
    "proxyConfiguration": { "useApifyProxy": True },
}

# Run the Actor and wait for it to finish
run = client.actor("darknezz/sitemap-content-extractor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "sitemapUrl": "https://apify.com/sitemap.xml",
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}' |
apify call darknezz/sitemap-content-extractor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,darknezz/sitemap-content-extractor"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/hj9j22bTb99RFXhvu/builds/3F0sE0C5n2XbiPvLU/openapi.json