# Research Repository Harvester (Dataverse / DSpace / InvenioRDM) (`datamule/repository-platform-extractor`) Actor

Point at ANY Dataverse, DSpace 7/8, or InvenioRDM/Zenodo install and harvest every dataset's metadata via one unified API — title, authors, DOI/handle, subjects, date, type, license, files, landing page + raw. One actor spans thousands of institutional repositories. Pay per record.

- **URL**: https://apify.com/datamule/repository-platform-extractor.md
- **Developed by:** [Datamule](https://apify.com/datamule) (community)
- **Categories:** Developer tools
- **Stats:** 3 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.35 / 1,000 record harvesteds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Research Repository Harvester — Dataverse / DSpace / InvenioRDM

Point at **any** Dataverse, DSpace 7/8, or InvenioRDM (Zenodo) install and harvest
every dataset's metadata through **one unified API**. Three of the world's most
widely-deployed research-data / institutional-repository software platforms each
expose a single uniform REST API that every institution's deployment answers
identically — so **one actor spans thousands of independent repositories**, and you
get the same normalized output no matter which platform or install you point it at.

- **Dataverse** — the Harvard/IQSS software behind 150+ registered installs
  (Harvard, QDR, DaRUS Stuttgart, Borealis Canada, DataverseNL, …).
- **DSpace 7/8** — the most-deployed institutional-repository software (theses,
  ETDs, gray literature) with 2,600+ installs worldwide.
- **InvenioRDM / Zenodo** — the CERN/Zenodo platform (Zenodo, Caltech Data, CERN).

### What you get

Every dataset / item is normalized into **one flat, consistent row** — the same
columns whether it came from Dataverse, DSpace or Invenio — plus a lossless `raw`
copy of the entire original object so nothing is ever dropped:

| Field | Description |
|---|---|
| `repoId` | The record's native id (Dataverse global id, DSpace UUID, Invenio record id) |
| `platform` | `dataverse` · `dspace` · `invenio` |
| `baseUrl` | The install the record came from |
| `persistentId` | DOI (Dataverse / Invenio) or Handle (DSpace) |
| `title` | Dataset / item title |
| `authors` | List of author / creator names |
| `description` | Abstract / description |
| `subjects` | Subjects + keywords |
| `publicationDate` | Publication / issued date |
| `type` | Resource type (dataset, thesis, publication, …) |
| `license` | Licence / rights |
| `fileCount` | Number of files (where the API exposes it) |
| `landingPage` | Canonical landing / DOI URL |
| `raw` | The complete original platform object (lossless) |

Fields that a given platform or record doesn't provide come back as `null` — the
schema is fully nullable, so a partial record never breaks a run.

### Input

| Field | Required | Description |
|---|---|---|
| `platform` | ✅ | `dataverse`, `dspace`, or `invenio` |
| `baseUrl` | ✅ | Install root — e.g. `https://dataverse.harvard.edu`, `https://dspace.mit.edu`, `https://zenodo.org` |
| `query` | | Free-text search (empty = browse everything) |
| `maxRecords` | | Max records to return (default: one page) |
| `pageSize` | | Records per API page, 1–100 (default 25) |
| `bearerToken` | | Optional token for a gated/private install (never required for public search, never logged) |
| `extraHeaders` | | Optional extra HTTP headers as a JSON object |

#### Examples

Zenodo (Invenio):

```json
{ "platform": "invenio", "baseUrl": "https://zenodo.org/", "query": "climate", "maxRecords": 100 }
```

Harvard Dataverse:

```json
{ "platform": "dataverse", "baseUrl": "https://dataverse.harvard.edu/", "query": "climate", "maxRecords": 100 }
```

MIT DSpace (point at the site root — the actor appends `/server/api`):

```json
{ "platform": "dspace", "baseUrl": "https://dspace.mit.edu/", "query": "climate", "maxRecords": 100 }
```

### How it works

For each platform the actor calls its uniform search endpoint, pages through the
result set, and maps each record into the unified schema:

- **Dataverse** → `GET {base}/api/search?q=…&type=dataset&per_page=N&start=M`
  (paginated by record offset), with a `{base}/api/info/version` handshake.
- **DSpace** → `GET {base}/server/api/discover/search/objects?query=…&page=N&size=M`
  (HAL/JSON envelope; Dublin-Core `dc.*` metadata flattened), with a
  `{base}/server/api` handshake that reports `dspaceName` / `dspaceVersion`.
- **Invenio** → `GET {base}/api/records?q=…&size=M&page=N`; handles **both** the
  legacy Zenodo record shape and the modern InvenioRDM shape transparently.

It returns catalog **metadata only** (what datasets exist and where to get them),
never the underlying data files. A valid install with no matches returns 0 records
and exits cleanly; a non-repository host (an HTML anti-bot page, a wrong URL, or a
`status:ERROR` envelope) fails fast with a clear error rather than a silent empty
run. No authentication is required for public repository search.

### Pricing

Pay-per-event: you are charged per repository record returned. No monthly fee — you
only pay for the records you harvest.

# Actor input Schema

## `platform` (type: `string`):

Which repository software the target install runs. dataverse = Harvard/IQSS Dataverse (/api/search). dspace = DSpace 7/8 REST API (/server/api). invenio = InvenioRDM or Zenodo (/api/records). All three expose one uniform API across their entire installed base, so ONE actor spans thousands of independent institutional repositories.

## `baseUrl` (type: `string`):

The install root of the repository. Dataverse example: https://dataverse.harvard.edu (or data.qdr.syr.edu, darus.uni-stuttgart.de, borealisdata.ca). DSpace example: https://dspace.mit.edu (the site root — the actor appends /server/api). Invenio example: https://zenodo.org (or data.caltech.edu). A trailing slash is normalized away.

## `query` (type: `string`):

Free-text search passed to the platform's search API. Leave empty to browse everything (Dataverse maps empty to q=\*). Example: climate, genomics, "machine learning".

## `maxRecords` (type: `integer`):

Maximum number of dataset / item records to return. The actor pages the search API until this cap is reached or the result set is exhausted. Each returned record is one billable event. Default: one page (see Page size).

## `pageSize` (type: `integer`):

How many records to request per API page (1-100). Larger pages = fewer round-trips; the total is still capped by Max records. Default 25.

## `bearerToken` (type: `string`):

Optional API token for a gated / private institutional install. Sent as Authorization: Bearer \*\*\*. NOT required for public repository search — leave empty. Never logged.

## `extraHeaders` (type: `object`):

Optional extra HTTP headers to send with every request, as a JSON object, e.g. {"X-Api-Key": "..."} for a gated deployment. Leave empty for public installs.

## Actor input object example

```json
{
  "platform": "invenio",
  "baseUrl": "https://zenodo.org/",
  "query": "climate",
  "maxRecords": 25,
  "pageSize": 25
}
```

# Actor output Schema

## `results` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "baseUrl": "https://zenodo.org/",
    "query": "climate",
    "maxRecords": 25
};

// Run the Actor and wait for it to finish
const run = await client.actor("datamule/repository-platform-extractor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "baseUrl": "https://zenodo.org/",
    "query": "climate",
    "maxRecords": 25,
}

# Run the Actor and wait for it to finish
run = client.actor("datamule/repository-platform-extractor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "baseUrl": "https://zenodo.org/",
  "query": "climate",
  "maxRecords": 25
}' |
apify call datamule/repository-platform-extractor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,datamule/repository-platform-extractor"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/bs7CitwdfyEgBwLlU/builds/p1QWEQtjomvneCUyd/openapi.json