Internet Archive Scraper (archive.org Items & Metadata)

Search the Internet Archive catalogue — 40M+ texts, audio, movies, images, software, data and web collections. Keyless, no login.

• search — a Solr query and/or structured filters (media type, collection, creator, subject, language, year range), sorted. Up to 10,000 results per query (archive.org's advancedsearch.php ceiling).
• items — a list of archive.org identifiers (or details / metadata URLs) → the full /metadata record each, optionally with the file list (name, format, size, length, md5, download URL).

Record type	One per	Carries
SEARCH_SUMMARY	search run	composed query, upstreamCount, resultsReturned, pagesFetched, resultCeilingHit, duplicatesSkipped
ITEM	archive.org item	identifier, title, creator, mediaType, description, year + yearInt, subjects, primaryCollection + collections, favoritedByCount, licenseUrl / rights, downloads, avgRating, detailsUrl — plus filesCount, files, runtime, server when fetched via items
ERROR	bad input / missing item	_error + _errorDetail

Every ITEM row carries the verbatim payload in raw (drop with slimOutput).

Things archive.org will mislead you about

Each is measured, and each has a scenario in tests/smoke/internet-archive-scraper_traps.sh (10/10 passing).

advancedsearch.php silently caps at 10,000 results (page × rows). Past it the response is an empty {} — not an error, not the last real page. resultCeilingHit on the summary flags it. To go deeper, slice the query (a fromYear/toYear window, a collection) and run the slices.

/metadata/<bad-id> returns HTTP 200 with {} — an empty object, not a 404. A missing identifier is reported as item_not_found.

A + in a URL-encoded value is taken LITERALLY by Solr. A range query written year:[1940+TO+1955] is malformed and silently dropped — you get the unfiltered set. Every value is %-encoded here (spaces → %20).

A downloads desc sort tie reshuffles items across pages. Two items with the same download count can both appear on page 2 and page 3. A seen set dedupes; duplicatesSkipped reports how many.

collection is a long array polluted with fav-<username> entries (a favourite is modelled as a collection). collections keeps only the real ones; primaryCollection is the first; favoritedByCount is how many fav- entries there were. subject is a string or a list (and may be ;/,-delimited inside the string) — normalised to a list.

downloads / favoritedByCount / avgRating are search-index fields — present on search rows, absent (null/0) on items rows (the /metadata endpoint doesn't carry them).

Not the scrape API. /services/search/v1/scrape (cursor, no ceiling) exists but holds one crawl per client IP — after you hit it once, every later request replays that first result set. This Actor does not use it.

Notes on cost

search: one request per 200 results (+ 1 for a numFound-only page). items: one request per identifier. No proxy needed.