Internet Archive Scraper

Search archive.org — books, films, audio, software, images, web captures — or list every Wayback Machine snapshot of a URL. Both run on archive.org's public APIs with no API key and return flat, spreadsheet-ready rows.

What does Internet Archive Scraper do?

Pick a mode:

• search — run an archive.org advanced-search query (Lucene syntax) over the whole archive and get one row per item: identifier, title, creator, media type, download counts, URLs, and (optionally) full metadata.
• wayback — give it a waybackUrl and get every archived snapshot of that URL from the Wayback Machine CDX index: timestamp, a direct snapshot URL, MIME type, HTTP status, content digest and size — perfect for tracking how a page changed over years.

Why use it?

• Research & archiving — pull large, filtered slices of the world's biggest free library as JSON/CSV/Excel.
• Change tracking / OSINT — see the full history of any web page: pricing pages, ToS, product listings, news articles, deleted content.
• No key, no login — archive.org publishes these endpoints for exactly this.
• Runs on the Apify platform: schedule it, hit it via API, pipe into integrations, monitor runs.

How to use it

1. Choose Mode — Search archive.org or Wayback Machine snapshots of a URL.
2. Fill the fields for that mode (a query, or a URL).
3. Set Max items.
4. Start — results land in the dataset; download as JSON, CSV, Excel or XML.

Input

mode = search

field	notes
query	advanced-search query, Lucene syntax. e.g. subject:"machine learning" AND mediatype:texts
mediatype	optional filter: texts, movies, audio, image, software, web, …
sort	downloads desc (default), this week / month, newest, recently added, title, rating
startPage	first result page (100 / page)
fetchMetadata	also call /metadata/<id> per item for description, subjects, license, file / review counts (slower)

mode = wayback

field	notes
waybackUrl	the URL to look up, e.g. https://apify.com/pricing
waybackMatchType	exact (default), prefix (path and below), host, domain (+ subdomains)
waybackFrom / waybackTo	date bounds — YYYY, YYYYMM or YYYYMMDD
waybackCollapse	skip consecutive snapshots with an unchanged content digest (default on)
waybackOnlyOk	keep only HTTP 200 snapshots (default on)

Shared: maxItems (default 100).

Examples

{ "mode": "search", "query": "collection:nasa", "mediatype": "image", "maxItems": 500 }

{ "mode": "wayback", "waybackUrl": "https://apify.com/", "waybackMatchType": "prefix",
  "waybackFrom": "2016", "waybackTo": "2020", "maxItems": 1000 }

Output

Search rows: identifier, title, creator, date, publicdate, mediatype, collection (fav-* stripped), language, subjects, downloads, downloads_last_week, downloads_last_month, num_reviews, avg_rating, item_size_bytes, details_url, download_url, thumbnail_url — plus with fetchMetadata: description, licenseurl, uploader, files_count, num_files, review_count.

Wayback rows: query_url, timestamp, archived_at (ISO), snapshot_url, original_url, url_key, mimetype, status_code, digest, length_bytes.

A run summary is written to the key-value store as OUTPUT.

Pricing

The Actor only makes lightweight JSON requests — a typical run is a few hundred kB and well within the Apify free tier. Wayback CDX responses for a busy site can be large; use waybackCollapse, the date bounds, and maxItems to keep runs small.

Tips

• archive.org caps search page * rows at ~10 000. For bigger pulls, split the query by date range / collection / media type and run slices.
• CDX matchType=domain on a large site can return hundreds of thousands of rows — always pair it with a date range.
• Query syntax: https://archive.org/advancedsearch.php · CDX docs: https://github.com/internetarchive/wayback/tree/master/wayback-cdx-server

FAQ & disclaimers

archive.org's search and CDX endpoints are public and intended for programmatic use; this Actor adds a descriptive User-Agent and paces its requests. It stores no personal data. Found a bug or want a field added? Open an issue on the Actor's Issues tab.