# Internet Archive Scraper (archive.org Items & Metadata) (`scrapyx/internet-archive-scraper`) Actor

Search the archive.org catalogue -- 40M+ texts, audio, movies, images, software and data. Solr query plus filters (media type, collection, creator, subject, year), sorted, up to 10,000 per query. Or fetch full metadata (with the file list) for specific identifiers. Keyless, no login.

- **URL**: https://apify.com/scrapyx/internet-archive-scraper.md
- **Developed by:** [Ibnu Adzim](https://apify.com/scrapyx) (community)
- **Categories:** Developer tools, AI
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.40 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Internet Archive Scraper (archive.org Items & Metadata)

Search the **[Internet Archive](https://archive.org)** catalogue — 40M+
texts, audio, movies, images, software, data and web collections. Keyless, no
login.

- **`search`** — a Solr `query` and/or structured filters (media type,
  collection, creator, subject, language, year range), sorted. Up to **10,000
  results per query** (archive.org's `advancedsearch.php` ceiling).
- **`items`** — a list of archive.org identifiers (or `details` / `metadata`
  URLs) → the full `/metadata` record each, optionally with the file list
  (name, format, size, length, md5, download URL).

| Record type | One per | Carries |
| --- | --- | --- |
| `SEARCH_SUMMARY` | search run | composed `query`, `upstreamCount`, `resultsReturned`, `pagesFetched`, `resultCeilingHit`, `duplicatesSkipped` |
| `ITEM` | archive.org item | `identifier`, `title`, `creator`, `mediaType`, `description`, `year` + `yearInt`, `subjects`, `primaryCollection` + `collections`, `favoritedByCount`, `licenseUrl` / `rights`, `downloads`, `avgRating`, `detailsUrl` — plus `filesCount`, `files`, `runtime`, `server` when fetched via `items` |
| `ERROR` | bad input / missing item | `_error` + `_errorDetail` |

Every `ITEM` row carries the verbatim payload in `raw` (drop with
`slimOutput`).

### Things archive.org will mislead you about

Each is measured, and each has a scenario in
`tests/smoke/internet-archive-scraper_traps.sh` (10/10 passing).

**`advancedsearch.php` silently caps at 10,000 results** (`page × rows`).
Past it the response is an empty `{}` — not an error, not the last real page.
`resultCeilingHit` on the summary flags it. To go deeper, slice the query
(a `fromYear`/`toYear` window, a `collection`) and run the slices.

**`/metadata/<bad-id>` returns HTTP 200 with `{}`** — an empty object, not a
404\. A missing identifier is reported as `item_not_found`.

**A `+` in a URL-encoded value is taken LITERALLY by Solr.** A range query
written `year:[1940+TO+1955]` is malformed and *silently dropped* — you get
the unfiltered set. Every value is `%`-encoded here (spaces → `%20`).

**A `downloads desc` sort tie reshuffles items across pages.** Two items with
the same download count can both appear on page 2 and page 3. A `seen` set
dedupes; `duplicatesSkipped` reports how many.

**`collection` is a long array polluted with `fav-<username>` entries** (a
favourite is modelled as a collection). `collections` keeps only the real
ones; `primaryCollection` is the first; `favoritedByCount` is how many
`fav-` entries there were. `subject` is a string *or* a list (and may be
`;`/`,`-delimited inside the string) — normalised to a list.

**`downloads` / `favoritedByCount` / `avgRating` are search-index fields** —
present on `search` rows, absent (null/0) on `items` rows (the `/metadata`
endpoint doesn't carry them).

**Not the scrape API.** `/services/search/v1/scrape` (cursor, no ceiling)
exists but holds one crawl per client IP — after you hit it once, every later
request replays that first result set. This Actor does not use it.

### Notes on cost

`search`: one request per 200 results (+ 1 for a `numFound`-only page).
`items`: one request per identifier. No proxy needed.

# Actor input Schema

## `mode` (type: `string`):

`search` runs a query / filters against the archive.org catalogue (via the cursor scrape API — no 10k ceiling). `items` fetches full `/metadata` for specific identifiers.

## `query` (type: `string`):

`search` mode. Solr / Lucene syntax against all fields — `apollo 11`, `"grateful dead" AND year:[1977 TO 1977]`, `creator:(Charlie Chaplin)`. Optional if you supply filters instead. A malformed query does not error (Solr is lenient) — it just returns fewer results.

## `identifiers` (type: `array`):

`items` mode. One per line: an archive.org identifier (`nasa`, `gov.uscourts.dcd.178502`) or a `https://archive.org/details/...` / `/metadata/...` URL.

## `mediaType` (type: `string`):

`search` filter. One of: `texts`, `audio`, `movies`, `image`, `software`, `data`, `web`, `collection`. An unknown value is rejected.

## `collection` (type: `string`):

`search` filter. An archive.org collection identifier, e.g. `prelinger`, `librivoxaudio`, `opensource_movies`, `internetarchivebooks`, `nasa`.

## `creator` (type: `string`):

`search` filter. Matched against the `creator` field.

## `subject` (type: `string`):

`search` filter. Matched against the `subject` field.

## `language` (type: `string`):

`search` filter. e.g. `English`, `eng`, `French`.

## `uploader` (type: `string`):

`search` filter. The uploading account (usually an email).

## `fromYear` (type: `integer`):

`search` filter. Keep items with `year` >= this.

## `toYear` (type: `integer`):

`search` filter. Keep items with `year` <= this.

## `sort` (type: `string`):

archive.org sort clause, e.g. `downloads desc`, `date desc`, `addeddate desc`, `avg_rating desc`, `titleSorter asc`. Default `downloads desc`.

## `rawQuery` (type: `string`):

Appended verbatim (AND-joined) to the composed query. Full Solr syntax.

## `includeFiles` (type: `boolean`):

`items` mode. Attach the item's full file list (name, format, size, length, md5, download URL) and `fileFormats`. Off by default — a large item can have hundreds of files.

## `slimOutput` (type: `boolean`):

By default every ITEM row carries the verbatim payload in `raw`. Turn on for the normalised fields only.

## `maxResults` (type: `integer`):

`search` mode cap. Cursor pagination is used, so this can exceed 10,000 (`advancedsearch.php` would silently cap there).

## `maxConcurrency` (type: `integer`):

Parallel in-flight requests (matters for `items` mode).

## `minRequestInterval` (type: `number`):

Politeness pacing for archive.org's infrastructure.

## `proxyConfiguration` (type: `object`):

Optional. No anti-bot layer on the allowed endpoints, so a proxy is OFF by default.

## Actor input object example

```json
{
  "mode": "search",
  "query": "old time radio",
  "mediaType": "",
  "sort": "downloads desc",
  "includeFiles": false,
  "slimOutput": false,
  "maxResults": 500,
  "maxConcurrency": 3,
  "minRequestInterval": 0.5,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}
```

# Actor output Schema

## `items` (type: `string`):

One row per scraped record. See the dataset's default view for field definitions.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "query": "old time radio"
};

// Run the Actor and wait for it to finish
const run = await client.actor("scrapyx/internet-archive-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "query": "old time radio" }

# Run the Actor and wait for it to finish
run = client.actor("scrapyx/internet-archive-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "query": "old time radio"
}' |
apify call scrapyx/internet-archive-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,scrapyx/internet-archive-scraper"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/lfJeVhHgEfX1Piudu/builds/oh5XnBHH18Fd9czoH/openapi.json