# Sitemap Inventory & Diff - URL Extractor with Change Detection (`gratifying_graph/sitemap-diff`) Actor

Extract every URL from a site's sitemaps, then diff against the previous run: pages added, removed, or updated since last check. Built for SEO monitoring, RAG freshness, and competitor watching.

- **URL**: https://apify.com/gratifying\_graph/sitemap-diff.md
- **Developed by:** [Jimmy A](https://apify.com/gratifying_graph) (community)
- **Categories:** Developer tools, SEO tools
- **Stats:** 2 total users, 1 monthly users, 0.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $10.00 / 1,000 1,000 urls processeds

This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Sitemap Inventory & Diff - URL extractor with change detection

Extract **every URL from a website's sitemaps** and find out **what changed since your last check**: pages added, pages removed, pages updated. One run gives you the full URL inventory. Scheduled runs give you a change feed for any site on the internet.

No browser, no proxies, no login. It reads the same sitemap.xml files sites publish for Google.

### What it does

1. Discovers sitemaps from `robots.txt` (falls back to `/sitemap.xml` and common index paths)
2. Follows sitemap index files recursively, including gzipped (`.xml.gz`) sitemaps
3. Extracts every URL with its `lastmod` date
4. Saves a snapshot per domain, then on the next run reports **added / removed / changed** URLs
5. Outputs a clean summary to the dataset; optionally the full URL inventory

### Use cases

- **SEO monitoring**: catch when a competitor publishes new landing pages, kills old ones, or refreshes content
- **RAG and AI pipelines**: keep a vector index fresh by re-crawling only the URLs that changed instead of the whole site
- **Content watch**: see when a publisher, government site, or documentation portal adds pages on a topic
- **Site audits**: instant URL inventory for any domain, exportable as JSON or CSV
- **Index bloat checks**: compare what a site publishes in sitemaps over time

### Input

```json
{
  "domains": ["competitor.com", "docs.example.com"],
  "computeDiff": true,
  "outputInventory": false
}
```

You can also pass exact sitemap URLs via `sitemapUrls` if a site keeps them in a non-standard place.

### Output

One summary item per domain:

```json
{
  "type": "summary",
  "domain": "competitor.com",
  "sitemapFiles": 7,
  "urlCount": 3741,
  "diff": {
    "previousRunFound": true,
    "added": 12,
    "removed": 3,
    "changed": 41,
    "addedUrls": ["https://competitor.com/new-feature", "..."],
    "removedUrls": ["..."],
    "changedUrls": ["..."]
  }
}
```

Set `outputInventory: true` to also get one item per URL (`url`, `lastmod`, `domain`).

The first run for a domain saves the baseline snapshot; diffs start with the second run. Snapshots persist between runs, so a weekly schedule gives you a weekly change report.

### Scheduling

Pair this actor with an Apify Schedule (for example weekly per domain). Each scheduled run compares against the previous snapshot automatically. Use the `snapshotGroup` input to track the same domain on two independent schedules without the snapshots interfering.

### API / Standby mode for AI agents

The actor also runs as an HTTP endpoint (Standby). Agents and integrations can call:

```
GET /?domain=example.com&diff=true
```

and receive the summary JSON synchronously. Works as an MCP-style tool for agent frameworks that support Apify actors.

### Pricing

Pay per event - you only pay for what the run actually does:

| Event | Price |
|---|---|
| Actor start | $0.0001 |
| Per 1,000 URLs extracted | $0.01 |
| Diff computed (per domain) | $0.02 |
| API call (standby mode) | $0.01 |

A weekly check of a 10,000-URL site costs about $0.12/month.

### FAQ

**How is this different from a sitemap URL extractor?**
Extractors give you the URL list. This actor also remembers the last run and tells you what changed: that is the part you actually want on a schedule.

**Does it work on sites without robots.txt?**
Yes. It falls back to `/sitemap.xml`, `/sitemap_index.xml`, and `/sitemap-index.xml`, and you can pass exact sitemap URLs.

**Does it handle huge sites?**
Yes. Sitemap indexes are followed recursively up to 500 sitemap files per domain, with a configurable URL cap (default 100,000).

**Does it crawl pages?**
No. It only reads sitemap files, which makes it fast, cheap, and gentle on the target site. If a URL is not in the sitemaps, it will not appear.

**Can I get the result as CSV?**
Yes, every Apify dataset exports as CSV, JSON, Excel, or via API.

# Actor input Schema

## `domains` (type: `array`):

Domains to inventory (e.g. example.com). Sitemaps are discovered via robots.txt, then /sitemap.xml fallbacks.

## `sitemapUrls` (type: `array`):

Exact sitemap URLs to fetch, if you already know them. Used in addition to discovered ones.

## `computeDiff` (type: `boolean`):

Compare against the snapshot saved by the last run for each domain and report added / removed / changed URLs.

## `outputInventory` (type: `boolean`):

Push every URL to the dataset, not just the diff summary. Large sites produce large datasets.

## `maxUrlsPerDomain` (type: `integer`):

Safety cap on URLs collected per domain.

## `snapshotGroup` (type: `string`):

Advanced: namespace for stored snapshots. Use different groups to track the same domain on independent schedules.

## Actor input object example

```json
{
  "domains": [
    "apify.com"
  ],
  "sitemapUrls": [],
  "computeDiff": true,
  "outputInventory": false,
  "maxUrlsPerDomain": 100000,
  "snapshotGroup": "default"
}
```

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "domains": [
        "apify.com"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("gratifying_graph/sitemap-diff").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "domains": ["apify.com"] }

# Run the Actor and wait for it to finish
run = client.actor("gratifying_graph/sitemap-diff").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "domains": [
    "apify.com"
  ]
}' |
apify call gratifying_graph/sitemap-diff --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,gratifying_graph/sitemap-diff"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/dPVCoKJ43W9pCF3wN/builds/730enSLxeWpeVFhm4/openapi.json