Wayback Machine Toolkit — Archived URLs, Snapshots & Diffs

Scrape the Wayback Machine three ways from one Actor: list every archived URL for a page, path or whole domain, read what a page said on a given date, and diff one page between two dates.

Everything comes from the Internet Archive's own public CDX and replay endpoints — no API key, no login, no proxies — and a CDX query returns in seconds even for thousands of rows. Index rows cost $0.0002 each ($0.20 per 1,000), and a URL that was never archived is a free diagnostic row.

The third mode is the point. Other archive tools list snapshots or fetch them. This one answers the question people actually have — what changed on this page between January and June? — in a single row, with the added and removed lines, a similarity score, and the headings and prices that moved.

Try it in 30 seconds

Every archived capture of a page:

{ "mode": "cdx", "urls": ["example.com"], "limit": 20 }

What changed on a pricing page over a decade:

{ "mode": "diff", "urls": ["example.com/pricing"], "dates": ["earliest", "latest"], "diffGranularity": "line" }

What a page said on a given date:

{ "mode": "snapshot", "urls": ["example.com/pricing"], "dates": ["20200105"], "snapshotFormat": "markdown" }

Three modes: archived URL index, page snapshot, page diff

Mode	Give it	Get back	Charged as
cdx	a URL or domain, optional date range and filters	One row per archived capture: timestamp, ISO date, archived status, MIME type, digest, size, Wayback URL	snapshot-index-row
snapshot	a URL and one or more dates	One row per capture, with the archived page as text, Markdown or HTML, plus title, headings, links and word count	snapshot-fetched
diff	a URL and exactly two dates	One row: added and removed lines, similarity ratio, word counts, heading changes, price changes, summary	diff-report

A few more things worth knowing up front:

• Archived pages come back clean. Fetched through the id_ raw endpoint, so no Wayback toolbar is injected into your data.
• Only content changes are billed by default. The index collapses on content digest, so a page that sat unchanged for two years is one row, not four hundred.
• Never an empty dataset. A URL that was never archived produces a diagnostic row explaining what to try instead — and is not charged.
• Honest failures. If archive.org is unreachable and the run produces no real rows, the run finishes FAILED, diagnostic row still in the dataset, nothing charged. A run that legitimately found nothing finishes successfully.

Use cases

• Get all archived URLs for a domain — matchType: "domain" gives you the URL inventory the archive holds, in one request.
• Compare two versions of a website — diff a competitor's pricing page between any two dates and read removedPrices / addedPrices. On a schedule, that is a price-change timeline nobody is publishing.
• SEO recovery after a site migration — matchType: "prefix" gives you the URL inventory a site had before a redesign, which is the list you need to build redirects from.
• Domain vetting before a purchase — a gap in the index, a run of 404s, or a sudden switch to a different language is how you find out a domain was a spam farm before you pay for it.
• Compliance and evidence capture — fetch what a page said on the day a claim was made, with the archive's own timestamp, digest and a citable Wayback URL on the row.
• Content-change monitoring — diff with dates: ["20240101", "latest"] on terms of service, policy pages or documentation, scheduled monthly.
• Recover lost content — snapshot in markdown turns an archived page back into something you can paste into a CMS.

How it compares

• Diff is the product, and nothing else in this category has it. Listing snapshots is easy. The hard part is that a 2015 page and a 2024 page of the same site share almost no HTML — table layouts became flexbox, class names churned, analytics tags came and went — so diffing raw archived markup returns thousands of meaningless changes and buries the one that matters.
• $0.20 per 1,000 index rows, where comparable archive listers run to several dollars per 1,000 items — and digest collapsing means you are not billed for years of stasis.
• Three tools in one Actor rather than one Actor per job, on a single input schema.

What the diff actually does:

1. Fetches raw bytes, not the replay page — always via https://web.archive.org/web/{timestamp}id_/{url}. Without the id_ suffix, archive.org injects its own toolbar and rewrites every link, and that chrome lands in your diff.
2. Decodes the archived charset. A 2003 capture is frequently windows-1252, not UTF-8. The archived Content-Type is honoured, with a <meta charset> sniff and a UTF-8 fallback, so accented text survives instead of becoming mojibake that reads as a change.
3. Reduces both captures to block-level text — one line per heading, paragraph, list item or table cell, with script, style, nav and footer removed, so whitespace and markup churn register as no change at all.
4. Compares at line and word level, with a similarity ratio of 2 × matching words / total words — the same measure Python's difflib.SequenceMatcher reports. 1 means identical text even when the HTML is not; 0 means nothing survived.
5. Overlays what people actually look for — headings that appeared or disappeared, and currency amounts that changed ($, £, €, ¥, ₹, and ISO codes either side of the number). Percentages and bare numbers are deliberately not treated as prices: a false positive in pricesChanged is worse than a miss.
6. Always compares oldest → newest, whichever order you write the two dates in, so "added" means "added over time".

Input reference

Field	Type	Default	Notes
mode	cdx | snapshot | diff	cdx	Required
urls	array of strings	—	URLs or domains. Scheme optional. A single URL may also be given as a plain url string
matchType	exact | prefix | host | domain	exact	cdx mode only. snapshot and diff always match exactly — they compare a page against itself
from / to	date	—	YYYYMMDD, an ISO date, or a partial date such as 2019 or 201907
statusFilter	array of strings	["200"]	Archived HTTP statuses to keep. Empty = all, including archived redirects and error pages
mimeFilter	array of strings	["text/html"]	Archived Content-Types to keep. Empty = all, including images and PDFs
collapse	digest | timestamp:8 | timestamp:6 | urlkey | none	digest	cdx mode. digest = one row per content change; timestamp:8 = daily; timestamp:6 = monthly
limit	integer 1–10000	200	Hard cap on index rows per URL, enforced by the archive before anything is billed
dates	array of strings	see notes	earliest, latest, or a date; the nearest capture is used. snapshot: any number, empty means the latest capture. diff: exactly two, empty means earliest vs latest
snapshotFormat	text | markdown | html	text	snapshot mode
diffGranularity	line | word	line	word additionally returns every changed word run
maxSnapshotKb	integer 16–8192	2048	Per-document read budget. Anything longer is cut and flagged truncated: true
maxRunSecs	integer 30–3600	240	Wall-clock budget. On reaching it the run stops requesting more, keeps every row written, and says so

Output reference

Every row, in every mode, carries scrapedAt, source and sourceUrl — the exact archive.org URL it was built from.

Index rows (cdx) — ok, mode, queryUrl, originalUrl, timestamp, isoDate, statusCode, mimeType, digest, length, archiveUrl, rawArchiveUrl:

{
  "ok": true,
  "originalUrl": "https://example.com/pricing",
  "timestamp": "20240620093000",
  "isoDate": "2024-06-20T09:30:00.000Z",
  "statusCode": "200", "mimeType": "text/html",
  "digest": "PRICING2024PRICING2024PRICING202", "length": 3120,
  "archiveUrl": "https://web.archive.org/web/20240620093000/https://example.com/pricing"
}

Snapshot rows (snapshot) — all of the above plus requestedDate, title, content, contentFormat, wordCount, headings[], links[], contentBytes, contentCharset, truncated, indexUrl.

Diff rows (diff) — url, fromTimestamp, toTimestamp, fromIsoDate, toIsoDate, fromArchiveUrl, toArchiveUrl, fromStatusCode, toStatusCode, fromDigest, toDigest, granularity, identical, similarityRatio, addedLines[], removedLines[], addedLineCount, removedLineCount, unchangedLineCount, changedCount, addedLinesTruncated, removedLinesTruncated, fromLineCount, toLineCount, addedWords[], removedWords[], addedWordCount, removedWordCount, fromWordCount, toWordCount, fromTitle, toTitle, titleChanged, addedHeadings[], removedHeadings[], headingsChanged, fromPrices[], toPrices[], addedPrices[], removedPrices[], pricesChanged, fromTruncated, toTruncated, summary:

{
  "fromIsoDate": "2015-06-12T09:00:00.000Z",
  "toIsoDate": "2024-06-20T09:30:00.000Z",
  "similarityRatio": 0.7379,
  "changedCount": 14,
  "addedLines": ["$15 per month", "$39 per month", "Enterprise", "…"],
  "removedLines": ["$9 per month", "$29 per month", "…"],
  "addedHeadings": ["Enterprise", "Enterprise add-ons"],
  "removedPrices": ["$9", "$29", "$99"],
  "addedPrices": ["$15", "$39", "$149"],
  "pricesChanged": true,
  "summary": "73.8% of the text is unchanged; 8 line(s) added and 6 line(s) removed; new price(s): \"$15\", \"$39\", \"$149\"; prices on this page changed."
}

Diagnostic rows — ok: false, plus error, errorType, checkedAt. Never charged.

Pricing

You pay per result, not per minute. There is no subscription and no platform-usage surcharge.

Event	What triggers it	Free	Bronze	Silver	Gold
actor-start	Once per run	$0.0005	$0.0005	$0.0005	$0.0005
snapshot-index-row	Each archive index row returned	$0.0002	$0.0002	$0.0002	$0.0002
snapshot-fetched	Each archived page fetched and parsed	$0.002	$0.002	$0.0015	$0.001
diff-report	Each two-date comparison returned	$0.01	$0.01	$0.008	$0.006

Bronze is the Starter plan, Silver is Scale, Gold is Business. Higher plans pay less per result.

What that costs in practice

You ask for	You get	You pay
Every capture of one page, limit: 20	20 index rows	$0.0005 + 20 × $0.0002 = $0.0045
Every archived URL under a domain, limit: 5000	5,000 index rows	$0.0005 + 5,000 × $0.0002 = $1.00
One archived page read at three dates	3 snapshots	$0.0005 + 3 × $0.002 = $0.0065
A competitor's pricing page compared across 10 years	1 diff report	$0.0005 + $0.01 = $0.0105
50 competitor pages diffed year-over-year	50 diff reports	$0.0005 + 50 × $0.01 = $0.50

You are never charged for: diagnostic rows — a URL that was never archived, a filter matching no captures, a page with only one capture to diff; anything at all on a run that never got an answer out of archive.org — the actor-start fee is billed once, and only after your input has validated and an archive.org request has come back successfully, so an outage that starts before we reach the archive costs nothing and finishes FAILED; captures the index collapsed as unchanged; rows the Actor could not write because your run or time budget was exhausted; anything beyond the four events above — no platform-usage pass-through.

Performance notes

archive.org is slow, and it is a donation-funded public archive. This Actor treats it accordingly: at most two concurrent requests with a minimum gap between request starts (not configurable upward); Retry-After obeyed exactly on 429 and 503, otherwise exponential backoff with jitter, up to three retries; a 45-second per-request timeout, because a cold capture can take that long to come out of storage; and a per-document read budget (maxSnapshotKb, default 2 MB) so an enormous archived document is truncated with truncated: true rather than killing the run.

Index first, then fetch only the captures you want — a cdx query is one request per URL and returns in seconds even for thousands of rows, while a snapshot costs one index request plus one fetch per date and a diff costs one index request plus two fetches. A matchType: "domain" query on a large site can match millions of captures, and limit is a hard cap applied by the archive, so raise it deliberately. Batching many URLs into one run beats many single-URL runs — but raise maxRunSecs when you do, since the default 240 s is sized for a handful of URLs, not a hundred.

Integrations and use from code

One JSON object in, one flat array out. The Actor runs with limited permissions, uses pay-per-event pricing and never enters Standby, so it works over the Apify MCP server and with x402 agentic payments. Apify Schedules run it unattended — a monthly diff on a competitor's pricing page with dates: ["20240101", "latest"] is the common pattern.

curl -X POST "https://api.apify.com/v2/acts/insight.solutions~wayback-toolkit/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"mode":"diff","urls":["example.com/pricing"],"dates":["earliest","latest"]}'

# pip install apify-client
from apify_client import ApifyClient

client = ApifyClient("<APIFY_TOKEN>")
run = client.actor("insight.solutions/wayback-toolkit").call(run_input={
    "mode": "diff",
    "urls": ["example.com/pricing", "example.com/terms"],
    "dates": ["20230601", "20240601"],
})

for row in client.dataset(run["defaultDatasetId"]).iterate_items():
    if row.get("ok"):
        print(row["url"], row["similarityRatio"], row["addedPrices"], row["summary"], sep=" | ")

FAQ

Why is a capture missing that I can see in the Wayback Machine? Almost always the default filters. statusFilter: ["200"] drops archived redirects and error pages, and mimeFilter: ["text/html"] drops everything that is not a page. Empty both to see the raw index. collapse: "digest" also hides captures byte-identical to the one before — set it to none to see every crawl.

Why do http:// and https:// versions show up as one list? The archive indexes both under a single canonical key, so a query for example.com returns captures of both. The originalUrl field on each row tells you which it was.

What does similarityRatio actually measure? 2 × matching words / total words across both captures, over the extracted visible text. It ignores HTML entirely. 1 means identical text; 0 means nothing shared.

Why does a diff show changes when the page looks the same? Check addedLines and removedLines. Common culprits are dates, view counters, "last updated" strings and cookie banners — real text changes a human eye skips over. Anything inside script, style, nav and footer is already excluded.

Can it diff more than two dates? One diff row compares exactly two captures. For a timeline, run snapshot across many dates and compare the rows yourself, or run several diffs with different date pairs in one run.

What if a page was only archived once? The run finishes successfully with a diagnostic row saying so, and you are not charged for the attempt.

How fresh is the archive? That is the Internet Archive's crawl schedule, not ours. A popular page may be captured daily; an obscure one may have a five-year gap. cdx mode shows you the real cadence for any URL.

Legal

• Not affiliated with, endorsed by, or sponsored by the Internet Archive. "Wayback Machine" and "Internet Archive" are used descriptively to identify the public data source this Actor reads from.
• All data comes from the Internet Archive's public, unauthenticated endpoints, which exist to be queried programmatically. No login, no authentication, no paywall is bypassed, and no content is fetched from the live web.
• Archived pages remain the copyright of their original publishers. This Actor retrieves and compares them; what you do with the text is your responsibility, and normal copyright and fair-use rules apply.
• The Actor holds itself to at most two concurrent requests with backoff and Retry-After compliance, so that using it does not degrade a free public service for anyone else.
• Archived pages can contain personal data that their publishers have since removed. This Actor does not extract, enrich, or infer personal data; it returns page text as archived. If you are processing personal data out of it, that is your obligation as controller, not ours.

Our other Actors

Public-data Actors on the same principles: official or public endpoints only, no login, no personal data, pay only for results.

Actor	What it does
Career Site Jobs API	Every open role from Greenhouse, Lever, Ashby, Workable and 10+ other ATS platforms, normalized
New Job Postings Monitor	Only the roles that opened, closed or changed since your last run — hiring signals for sales and recruiting
Website Contact Extractor	Business emails, phone numbers and social profiles from a list of website domains
SEC EDGAR API	SEC filings, normalized XBRL financials and full-text search, by ticker or CIK