Wayback Machine URL Archive Scraper

Search and retrieve archived snapshots of any URL from the Internet Archive's Wayback Machine via its CDX API. Batch multiple URLs, filter by date range or HTTP status, deduplicate results, and optionally fetch the archived HTML content.

---

What It Does

This Actor queries the CDX Server API — a fast index of the Wayback Machine — to retrieve metadata and optionally content for every archived snapshot of one or more URLs.

Each output item represents a single archived snapshot including its timestamp, status code, MIME type, content digest, and a direct playback URL.

---

Features

• Batch URL support — process multiple URLs in a single run with controlled concurrency
• Date range filtering — restrict results to a specific time window (YYYYMMDD format)
• Deduplication — collapse by content digest, day, or month to avoid redundant snapshots
• HTTP status filtering — limit results to 200 OK, 301 redirects, or any combination
• Latest-only mode — return only the most recent snapshot per URL
• Optional HTML content fetch — retrieve the full archived HTML for each snapshot
• PPE pricing — pay only for what you use ($0.001 per snapshot returned)

---

Input Parameters

Parameter	Type	Default	Description
urls	array of strings	required	URLs to search in the Wayback Machine
dateFrom	string	—	Start date YYYYMMDD (e.g. 20200101). No lower bound if omitted.
dateTo	string	—	End date YYYYMMDD (e.g. 20241231). No upper bound if omitted.
limit	integer	100	Max snapshots per URL (1–1000)
collapseBy	enum	digest	Deduplication strategy: digest, timestamp:8 (daily), timestamp:6 (monthly), or "" (none)
statusFilter	string	200	HTTP status codes to include. "200", "200,301", or "" for all
includeLatestOnly	boolean	false	Return only the most recent snapshot per URL
fetchContent	boolean	false	Fetch archived HTML content for each snapshot (increases runtime and cost)

Example Input

{
  "urls": [
    "https://example.com",
    "https://github.com/openai"
  ],
  "dateFrom": "20200101",
  "dateTo": "20241231",
  "limit": 100,
  "collapseBy": "digest",
  "statusFilter": "200",
  "includeLatestOnly": false,
  "fetchContent": false
}

---

Output Fields

Each item in the dataset represents one archived snapshot.

Field	Type	Description
original_url	string	The original URL that was archived
timestamp	string	CDX timestamp in YYYYMMDDHHmmSS format
datetime	string	ISO 8601 UTC datetime (e.g. 2024-01-15T12:34:56Z)
snapshot_url	string	Direct Wayback Machine playback URL
status_code	string	HTTP status code of the archived response
mime_type	string	MIME type of the archived content
digest	string	SHA1 content digest (prefixed with SHA1:)
length	integer	Size of the archived content in bytes
content	string or null	Full archived HTML content (only when fetchContent: true)

Example Output Item

{
  "original_url": "https://example.com",
  "timestamp": "20240115123456",
  "datetime": "2024-01-15T12:34:56Z",
  "snapshot_url": "https://web.archive.org/web/20240115123456/https://example.com",
  "status_code": "200",
  "mime_type": "text/html",
  "digest": "SHA1:ABCDEF1234567890ABCDEF1234567890ABCDEF12",
  "length": 12345,
  "content": null
}

---

Use Cases

Web Archive Research

Track how a website evolved over time. Useful for journalism, academic research, and competitive intelligence.

Deleted Content Recovery

Find previously published pages, articles, or product listings that have since been removed from the live web.

SEO History Analysis

Audit historical title tags, meta descriptions, and page structures to understand past SEO strategies and content changes.

Competitive Monitoring

Build a timeline of a competitor's landing pages, pricing pages, or product launches using archived snapshots.

Legal & Compliance Evidence

Retrieve timestamped proof of past web content for legal documentation or intellectual property disputes.

---

Pricing

This Actor uses Pay-Per-Event (PPE) pricing:

Event	Price
Actor start	$0.005 (one-time per run)
Per snapshot returned	$0.001

Example: Searching 5 URLs and retrieving 200 snapshots total costs approximately $0.205.

---

Technical Details

• Built with Python + httpx (async HTTP client)
• Queries the Internet Archive CDX API
• Processes up to 3 URLs concurrently (configurable via semaphore)
• No external proxy or third-party service required — all data comes directly from web.archive.org

---

Related actors

• Wikipedia Article Scraper — Wikipedia revision history as an alternative archive source for content drift analysis
• Domain DNS Checker — DNS records for archived domains to verify current registration status
• Open Library Search — Internet Archive's book catalog alongside web archive snapshots for full research workflows

Feedback

If this actor helps your web archival research, a review helps others find it: Leave a review on Apify Store