Sitemap URL Extractor – Robots.txt, Index, Gzip & Lastmod

Pull every URL out of a site's sitemaps as clean JSON rows: robots.txt discovery, sitemap-index recursion, gzip and plain-text sitemaps, and a lastmod date filter, without a browser, a proxy or a login. Built for SEO teams auditing site structure, migration projects diffing old vs. new URL sets, RAG/LLM ingestion pipelines that need a full, deduplicated crawl list before fetching content, and monitoring jobs that watch a sitemap for newly-published pages. Plain HTTPS calls with retries and a silent-failure check, so a scheduled run keeps working when a site serves a broken or half-empty sitemap.

What you get

• Every URL as one row – site, url, lastmod, changefreq, priority, the sitemap_url it came from and scraped_at.
• The formats that break other sitemap tools – robots.txt Sitemap: discovery with fallback path guessing, nested sitemap indexes (recursed, not just one level), .gz files whether or not the server tags them Content-Encoding: gzip, plain-text sitemaps (one URL per line), and basic RSS/Atom feeds used as sitemaps.
• Filters that keep a run's cost predictable – a lastmod cutoff date, an include/exclude regex, per-site caps on URLs and sitemap files, and optional dedupe, so one run never explodes into an unbounded crawl.

Sample rows

A run against https://apify.com (its sitemap.xml is itself a 14-file sitemap index):

{
  "type": "url",
  "site": "https://apify.com",
  "url": "https://apify.com/01010101/mcp-scout",
  "lastmod": null,
  "changefreq": null,
  "priority": null,
  "sitemap_url": "https://apify.com/sitemap/actors1.xml",
  "scraped_at": "2026-09-09T18:20:11+00:00"
}

With outputSitemapRows: true, one extra row per sitemap file parsed (url_count is the number of URLs kept from that file after your filters and maxUrlsPerSite cap; apify.com's actor sitemaps hold up to 50,000 URLs each):

{ "type": "sitemap", "site": "https://apify.com", "sitemap_url": "https://apify.com/sitemap/actors1.xml", "url_count": 5000, "status": "ok" }

A site with no reachable sitemap produces an error row instead of failing the whole run:

{ "type": "error", "site": "https://bogus-domain-example.invalid", "url": null, "error": "network", "errorMessage": "..." }

What it handles that others break on

Apify's own sitemap Actor and most third-party ones report success rates in the 55–80% range. The usual failure points, all handled here:

• Missing or blocked robots.txt – a 404 or 403 on /robots.txt is not treated as a failure; the Actor falls back to /sitemap.xml, /sitemap_index.xml, /sitemap-index.xml and /sitemap/sitemap.xml.
• Sitemap indexes nested more than one level deep – recursed until maxSitemapsPerSite is hit, not just the first <sitemap> list.
• Gzip that lies about itself – .gz files served with Content-Encoding: gzip (auto-decoded), served with the raw gzip bytes untagged, or served already decompressed under a .gz URL — all three are detected correctly.
• Huge sitemaps – parsed with xml.etree.iterparse, clearing each <url> as it's read, so memory stays flat instead of building one giant DOM.
• A 200 response that isn't actually a sitemap (an HTML error page, an empty body) – counted as a failed fetch, not a silent zero-row success.
• One bad site in a batch – never aborts the run; it becomes an error row and the rest of startUrls still gets processed.

Filters

Input	Default	What it does
startUrls	[{"url": "https://apify.com"}]	Sites to extract from. A direct sitemap URL (.xml, .xml.gz, .txt) is used as-is instead of triggering discovery.
maxUrlsPerSite	5000	Stop after this many URL rows per site.
maxSitemapsPerSite	100	Stop after this many sitemap files (including nested index entries) per site.
modifiedAfter	""	Keep only URLs with lastmod on or after this ISO date. URLs with no lastmod are dropped when this is set.
urlPattern	""	Regex a URL must match to be kept.
excludePattern	""	Regex that drops a URL if it matches.
dedupe	true	Skip a URL already seen for the same site.
outputSitemapRows	false	Also push one row per sitemap file parsed.

Pricing

Pay per event, no subscription: $0.10 per 1,000 URLs and $1 per 1,000 sitemap files parsed. A default run against https://apify.com (a 14-file index, a few thousand URLs) costs a few cents. Set a maximum charge on the run and the Actor stops cleanly when it is reached.

Use it from your tools

• API and SDKs – call it via the Apify API or the official Python/JavaScript clients: one call to start the run, one to fetch the URL list as JSON or CSV.
• Schedules – run it daily inside Apify and push newly-published URLs to Google Sheets, a webhook or your own storage automatically.
• n8n, Make and Zapier – trigger runs and pipe extracted URLs into a workflow through Apify's integration for each.
• AI agents and MCP – this Actor is eligible for agentic use via Apify's MCP server and supports pay-per-event pricing, so an agent can call it mid-task and pay only per URL it actually extracts.
• Webhooks – fire on run finished to kick off crawling or ingestion as soon as the URL list is ready.

Details

• Source: robots.txt plus standard sitemap fallback paths for whatever site you give it. No authentication, no proxies, no browser, no personal data collected.
• Reliability: 429 and 5xx responses are retried with exponential backoff (5 tries), a 200 that isn't a real sitemap counts as a failure, and one broken site never stops the run — it becomes an error row and the rest continues. A run fails only when nothing was extracted anywhere and every request failed; a site with genuinely no matching URLs (after filters) is a normal, successful run.
• Run stats: the STATS record in the run's key-value store holds URL/sitemap/error counts and per-category request-error counts (network, rate_limit, blocked, not_found, other).
• Politeness: 0.1 s between requests, 30 s timeout, redirects followed, a User-Agent that names the Actor.
• Output: one row per URL with the Overview table showing site, URL, last-modified date and source sitemap. Export as JSON, CSV or Excel, fetch through the Apify API, or schedule runs to catch newly-published pages.