# Sitemap URL Extractor – Robots.txt, Index, Gzip & Lastmod (`rowfeed/sitemap-url-extractor`) Actor

Extract every URL from any site's sitemaps: robots.txt discovery, sitemap-index recursion, gzip, plain-text and RSS/Atom sitemaps, lastmod/regex filters. Plain HTTP, no browser, no proxies.

- **URL**: https://apify.com/rowfeed/sitemap-url-extractor.md
- **Developed by:** [Rowfeed](https://apify.com/rowfeed) (community)
- **Categories:** SEO tools, Developer tools, AI
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.10 / 1,000 urls

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Sitemap URL Extractor – Robots.txt, Index, Gzip & Lastmod

Pull **every URL out of a site's sitemaps** as clean JSON rows: robots.txt discovery, sitemap-index recursion, gzip and plain-text sitemaps, and a `lastmod` date filter, without a browser, a proxy or a login.
Built for **SEO teams** auditing site structure, **migration projects** diffing old vs. new URL sets, **RAG/LLM ingestion pipelines** that need a full, deduplicated crawl list before fetching content, and **monitoring** jobs that watch a sitemap for newly-published pages.
Plain HTTPS calls with retries and a silent-failure check, so a scheduled run keeps working when a site serves a broken or half-empty sitemap.

### What you get

- **Every URL as one row** – `site`, `url`, `lastmod`, `changefreq`, `priority`, the `sitemap_url` it came from and `scraped_at`.
- **The formats that break other sitemap tools** – robots.txt `Sitemap:` discovery with fallback path guessing, nested sitemap indexes (recursed, not just one level), `.gz` files whether or not the server tags them `Content-Encoding: gzip`, plain-text sitemaps (one URL per line), and basic RSS/Atom feeds used as sitemaps.
- **Filters that keep a run's cost predictable** – a `lastmod` cutoff date, an include/exclude regex, per-site caps on URLs and sitemap files, and optional dedupe, so one run never explodes into an unbounded crawl.

### Sample rows

A run against `https://apify.com` (its `sitemap.xml` is itself a 14-file sitemap index):

```json
{
  "type": "url",
  "site": "https://apify.com/",
  "url": "https://apify.com/01010101/mcp-scout",
  "lastmod": null,
  "changefreq": null,
  "priority": null,
  "sitemap_url": "https://apify.com/sitemap/actors1.xml",
  "scraped_at": "2026-09-09T18:20:11+00:00"
}
```

With `outputSitemapRows: true`, one extra row per sitemap file parsed (`url_count` is the number of URLs kept from that file after your filters and `maxUrlsPerSite` cap; apify.com's actor sitemaps hold up to 50,000 URLs each):

```json
{ "type": "sitemap", "site": "https://apify.com/", "sitemap_url": "https://apify.com/sitemap/actors1.xml", "url_count": 5000, "status": "ok" }
```

A site with no reachable sitemap produces an error row instead of failing the whole run:

```json
{ "type": "error", "site": "https://bogus-domain-example.invalid/", "url": null, "error": "network", "errorMessage": "..." }
```

### What it handles that others break on

Apify's own sitemap Actor and most third-party ones report success rates in the 55–80% range. The usual failure points, all handled here:

- **Missing or blocked robots.txt** – a 404 or 403 on `/robots.txt` is not treated as a failure; the Actor falls back to `/sitemap.xml`, `/sitemap_index.xml`, `/sitemap-index.xml` and `/sitemap/sitemap.xml`.
- **Sitemap indexes nested more than one level deep** – recursed until `maxSitemapsPerSite` is hit, not just the first `<sitemap>` list.
- **Gzip that lies about itself** – `.gz` files served with `Content-Encoding: gzip` (auto-decoded), served with the raw gzip bytes untagged, or served already decompressed under a `.gz` URL — all three are detected correctly.
- **Huge sitemaps** – parsed with `xml.etree.iterparse`, clearing each `<url>` as it's read, so memory stays flat instead of building one giant DOM.
- **A 200 response that isn't actually a sitemap** (an HTML error page, an empty body) – counted as a failed fetch, not a silent zero-row success.
- **One bad site in a batch** – never aborts the run; it becomes an `error` row and the rest of `startUrls` still gets processed.

### Filters

| Input | Default | What it does |
|---|---|---|
| `startUrls` | `[{"url": "https://apify.com/"}]` | Sites to extract from. A direct sitemap URL (`.xml`, `.xml.gz`, `.txt`) is used as-is instead of triggering discovery. |
| `maxUrlsPerSite` | `5000` | Stop after this many URL rows per site. |
| `maxSitemapsPerSite` | `100` | Stop after this many sitemap files (including nested index entries) per site. |
| `modifiedAfter` | `""` | Keep only URLs with `lastmod` on or after this ISO date. URLs with no `lastmod` are dropped when this is set. |
| `urlPattern` | `""` | Regex a URL must match to be kept. |
| `excludePattern` | `""` | Regex that drops a URL if it matches. |
| `dedupe` | `true` | Skip a URL already seen for the same site. |
| `outputSitemapRows` | `false` | Also push one row per sitemap file parsed. |

### Pricing

Pay per event, no subscription: **$0.10 per 1,000 URLs** and **$1 per 1,000 sitemap files parsed**. A default run against `https://apify.com` (a 14-file index, a few thousand URLs) costs a few cents. Set a maximum charge on the run and the Actor stops cleanly when it is reached.

### Use it from your tools

- **API and SDKs** – call it via the Apify API or the official Python/JavaScript clients: one call to start the run, one to fetch the URL list as JSON or CSV.
- **Schedules** – run it daily inside Apify and push newly-published URLs to Google Sheets, a webhook or your own storage automatically.
- **n8n, Make and Zapier** – trigger runs and pipe extracted URLs into a workflow through Apify's integration for each.
- **AI agents and MCP** – this Actor is eligible for agentic use via Apify's MCP server and supports pay-per-event pricing, so an agent can call it mid-task and pay only per URL it actually extracts.
- **Webhooks** – fire on run finished to kick off crawling or ingestion as soon as the URL list is ready.

### Details

- **Source**: robots.txt plus standard sitemap fallback paths for whatever site you give it. No authentication, no proxies, no browser, no personal data collected.
- **Reliability**: 429 and 5xx responses are retried with exponential backoff (5 tries), a 200 that isn't a real sitemap counts as a failure, and one broken site never stops the run — it becomes an `error` row and the rest continues. A run fails only when nothing was extracted anywhere *and* every request failed; a site with genuinely no matching URLs (after filters) is a normal, successful run.
- **Run stats**: the `STATS` record in the run's key-value store holds URL/sitemap/error counts and per-category request-error counts (`network`, `rate_limit`, `blocked`, `not_found`, `other`).
- **Politeness**: 0.1 s between requests, 30 s timeout, redirects followed, a User-Agent that names the Actor.
- **Output**: one row per URL with the Overview table showing site, URL, last-modified date and source sitemap. Export as JSON, CSV or Excel, fetch through the Apify API, or schedule runs to catch newly-published pages.

# Actor input Schema

## `startUrls` (type: `array`):

Sites to extract sitemap URLs from. For each entry the Actor reads robots.txt for Sitemap: lines and also tries /sitemap.xml, /sitemap\_index.xml, /sitemap-index.xml and /sitemap/sitemap.xml. You can also point directly at a sitemap file (.xml, .xml.gz, .txt) and it will be used as-is.

## `maxUrlsPerSite` (type: `integer`):

Stop once this many URL rows have been kept for one site (after filters and dedupe). Each row is one `url` event ($0.10 per 1,000).

## `maxSitemapsPerSite` (type: `integer`):

Stop fetching more sitemap files (including nested index entries) for one site once this many have been parsed.

## `modifiedAfter` (type: `string`):

Keep only URLs whose sitemap <lastmod> (or RSS/Atom date) is on or after this ISO date, e.g. 2024-01-01 or 2024-01-01T00:00:00Z. URLs with no lastmod are dropped when this is set. Leave empty to keep everything.

## `urlPattern` (type: `string`):

Keep only URLs whose text matches this regular expression, e.g. /blog/ to keep only blog posts. Leave empty to keep every URL.

## `excludePattern` (type: `string`):

Drop URLs whose text matches this regular expression, e.g. .pdf$ to skip PDFs. Leave empty to exclude nothing.

## `dedupe` (type: `boolean`):

Skip a URL that already appeared for the same site (some sites list the same page in more than one sitemap).

## `outputSitemapRows` (type: `boolean`):

In addition to URL rows, push one row per sitemap file parsed (site, sitemap\_url, url\_count, status). Useful to see the sitemap structure itself.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://apify.com/"
    }
  ],
  "maxUrlsPerSite": 5000,
  "maxSitemapsPerSite": 100,
  "modifiedAfter": "",
  "urlPattern": "",
  "excludePattern": "",
  "dedupe": true,
  "outputSitemapRows": false
}
```

# Actor output Schema

## `results` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://apify.com/"
        }
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("rowfeed/sitemap-url-extractor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "startUrls": [{ "url": "https://apify.com/" }] }

# Run the Actor and wait for it to finish
run = client.actor("rowfeed/sitemap-url-extractor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://apify.com/"
    }
  ]
}' |
apify call rowfeed/sitemap-url-extractor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,rowfeed/sitemap-url-extractor"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/4dIXkpA4g8ZzaPxWD/builds/8bD6InW3X6s49NE3X/openapi.json