# Sitemap URL Extractor: Every URL, Recursive (`thoob/sitemap-extractor`) Actor

Reads sitemap.xml, sitemap index files, .gz compressed sitemaps, and robots.txt Sitemap directives, and returns one clean row per URL with lastmod, changefreq, and priority. Runs without a proxy. Billed only per delivered URL.

- **URL**: https://apify.com/thoob/sitemap-extractor.md
- **Developed by:** [Pono Data](https://apify.com/thoob) (community)
- **Categories:** Developer tools, SEO tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

$1.00 / 1,000 extracted urls

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Sitemap Extractor

Give it a sitemap URL, a robots.txt URL, or a site root. It returns one clean row
per URL, following sitemap index files, `.gz` compressed sitemaps, and the
`Sitemap:` directives in robots.txt. Every row carries the sitemap it came from,
so any row is verifiable at its source.

### What it does that other actors split across three

- **Finds the sitemap for you.** Give a bare site root and it probes the common sitemap locations, robots.txt, and the homepage's `<link rel="sitemap">`, then extracts what it finds. Other actors either find the sitemap or extract one, not both.
- **Validates each URL (optional).** Turn on status checking and every row gets an `httpStatus` and a `live` flag, checked concurrently. Filter to live URLs only for a list with no dead links.
- **Reads the sitemap extensions.** Image, news, video, and hreflang-alternate data are pulled out per URL when the sitemap carries them, not dropped.

Plus the basics done right: index recursion, `.gz`, lastmod filtering, dedupe, and a `sourceUrl` on every row.

### Input

- **Start URLs**: sitemap URLs, robots.txt URLs, or site roots. A site root is resolved by probing the common locations, robots.txt, and the homepage link.
- **Max delivered URLs**: cap on delivered rows (0 means no cap).
- **Max sitemap-index depth**: how deep to follow nested index files.
- **Only URLs modified on/after**: optional `YYYY-MM-DD` lastmod filter (entries with no lastmod are kept).
- **Include changefreq and priority**: toggle the optional hint fields.
- **Validate URL status codes**: add `httpStatus` and `live` to every row.
- **Deliver only live URLs**: with validation on, drop dead URLs to the free `rejected` dataset.

### Output

One row per URL: `loc`, `lastmod`, `changefreq`, `priority`, `host`, `depth`, the
optional `httpStatus` and `live`, the image / news / video / hreflang extension
fields when present, `sourceUrl` (the sitemap it came from), `retrievedAt`,
`confidence`, `dataSource`.

### How it works

Sitemaps are published by sites for machines to read. The actor fetches them with
a declared User-Agent, decompresses `.gz`, follows index files up to your depth
limit, dedupes by URL, and never invents a field: a value is emitted only if it is
present in the sitemap. A supplied sitemap that fails to fetch or parse is recorded
in the free `rejected` dataset; a probe of a guessed location that misses is
silent. Status checks, when on, run concurrently so validation stays fast.

### Billing

Pay per delivered URL row. Failed or unparseable sitemaps, and dead URLs when you
filter to live only, cost nothing. Status validation is included at the same
per-URL price.

### Coverage

Global. Targets in any country are processed. The one exclusion is jurisdictions
under US sanctions (Cuba, Iran, North Korea, Syria, Russia, Belarus, Venezuela,
Myanmar, matched by country-code TLD), which are written to the free `rejected`
dataset and never billed.

### Opt out

A domain owner can ask us to skip their domain at https://ponodata.com/opt-out .
Suppressed domains are returned by nothing and never billed.

### Sample output

A real run extracting URLs from a sitemap, with optional live-status validation:

| URL | HTTP | live | source |
| --- | --- | --- | --- |
| https://developer.mozilla.org/en-US/ | 200 | True | developer.mozilla… |
| https://developer.mozilla.org/en-US/404 | 200 | True | developer.mozilla… |
| https://developer.mozilla.org/en-US/about | 200 | True | developer.mozilla… |
| https://developer.mozilla.org/en-US/advertising | 200 | True | developer.mozilla… |

Every URL carries its `sourceUrl` (the sitemap it came from), for example `https://developer.mozilla.org/sitemaps/en-us/sitemap.xml.gz`.

### Use cases

- Seed a crawler or content audit with a site's full URL list, one clean row per URL with lastmod and changefreq when present.
- Run an SEO coverage check: compare what a site publishes in its sitemaps against what is indexed, and spot stale lastmod dates.
- Plan a content migration or archive: enumerate every page across a sitemap index, optionally filtered to URLs modified since a date.
- Get a dead-link-free list: turn on status validation and deliver only live URLs, each with its httpStatus.

### FAQ

- Do I need to find the sitemap first? No. Give a bare site root and it probes the common locations, robots.txt, and the homepage link, then extracts what it finds.
- Does it follow index files and gzipped sitemaps? Yes, it walks nested index files up to your depth limit and decompresses .gz sitemaps.
- Does it crawl the pages themselves? No. It reads the sitemaps and returns the URLs they list; optional status validation only checks each URL's response code.
- How am I billed? Per delivered URL row; failed or unparseable sitemaps, and dead URLs when you filter to live only, cost nothing.

### See also

More clean, pay-only-for-results data tools from Pono Data:

- [URL Metadata & OpenGraph Extractor](https://apify.com/thoob/url-metadata-extractor) - title, OpenGraph, and meta tags per page
- [Bulk DNS Lookup](https://apify.com/thoob/dns-bulk-lookup) - DNS records plus SPF, DMARC, and CAA
- [Domain WHOIS via RDAP](https://apify.com/thoob/rdap-domain-lookup) - registration data, structured from RDAP

Full catalog: https://apify.com/thoob

# Actor input Schema

## `startUrls` (type: `array`):

Sitemap URLs (https://site.com/sitemap.xml), robots.txt URLs, or site roots (https://site.com). For a site root the actor probes the common sitemap locations, robots.txt, and the homepage's <link rel="sitemap">. Sitemap index files and .gz compressed sitemaps are followed automatically.

## `maxUrls` (type: `integer`):

Safety cap on delivered and billed URL rows. 0 means no cap. The platform spend cap (ACTOR\_MAX\_TOTAL\_CHARGE\_USD) is honored regardless.

## `maxDepth` (type: `integer`):

How deep to follow nested sitemap index files. 0 means only the sitemaps you supply.

## `lastmodAfter` (type: `string`):

Optional. Keep only entries whose lastmod is on or after this date. Entries with no lastmod are kept.

## `includeSitemapMeta` (type: `boolean`):

Include the optional changefreq and priority fields from each entry.

## `checkStatus` (type: `boolean`):

Check each extracted URL's HTTP status (HEAD with a GET fallback) and add httpStatus and live to every row. Adds time on large sitemaps; runs concurrently.

## `onlyLive` (type: `boolean`):

When validating status, deliver and bill only URLs that respond 2xx or 3xx. Dead URLs go to the free rejected dataset. Ignored when status validation is off.

## Actor input object example

```json
{
  "startUrls": [
    "https://www.cloudflare.com/sitemap.xml"
  ],
  "maxUrls": 0,
  "maxDepth": 5,
  "includeSitemapMeta": true,
  "checkStatus": false,
  "onlyLive": false
}
```

# Actor output Schema

## `urls` (type: `string`):

One row per URL discovered across the supplied sitemaps, with optional status and extension data.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        "https://www.cloudflare.com/sitemap.xml"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("thoob/sitemap-extractor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "startUrls": ["https://www.cloudflare.com/sitemap.xml"] }

# Run the Actor and wait for it to finish
run = client.actor("thoob/sitemap-extractor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    "https://www.cloudflare.com/sitemap.xml"
  ]
}' |
apify call thoob/sitemap-extractor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,thoob/sitemap-extractor"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/DlODTqS0C96xapdOp/builds/P4gkIf0E84wtFFlyy/openapi.json