# Semantic Scholar Scraper (`parseforge/semantic-scholar-scraper`) Actor

Scrapes academic papers from Semantic Scholar search results. Returns each paper as a flat row with title, authors, year, citations, venue, and abstract. Supports year and PDF filters.

- **URL**: https://apify.com/parseforge/semantic-scholar-scraper.md
- **Developed by:** [ParseForge](https://apify.com/parseforge) (community)
- **Categories:** Automation, Developer tools, Other
- **Stats:** 51 total users, 3 monthly users, 85.0% runs succeeded, 2 bookmarks
- **User rating**: 1.08 out of 5 stars

## Pricing

from $8.00 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

[![ParseForge](https://raw.githubusercontent.com/ParseForge/apify-assets/main/banner.jpg)](https://apify.com/parseforge?fpr=vmoqkp)

### Semantic Scholar Scraper

**Search Semantic Scholar for papers by title or keywords, look up authors with their h-index and papers, or fetch papers by DOI, arXiv id or Semantic Scholar id.** Every paper comes with its title, authors, year, venue, journal, abstract, citation counts, fields of study, DOI/arXiv/PubMed ids and the open-access PDF link. No API key required. Export to CSV, JSON, Excel, or XML.

This Actor drives Semantic Scholar's official Graph API — clean JSON, no browser, no proxy. A title search such as "attention is all you need" returns the paper you mean first, because the query is matched as a phrase and ranked by citations; a keyword search such as "graph neural networks drug discovery" returns the most-cited matches with server-side filters for year, field of study, publication type, venue, minimum citations and PDF availability.

| Who uses it | What they scrape Semantic Scholar for |
|---|---|
| Academic researchers | Building a corpus of papers for a systematic literature review. |
| PhD students | Recent publications in a field, filtered by year and venue, to find research gaps. |
| Data scientists | Author profiles (h-index, citation totals) and paper metadata for bibliometrics. |
| Librarians and research offices | Publication lists per author, with DOIs and open-access links. |
| AI agents | Resolving a paper from its title, DOI or arXiv id in one call. |

### What it does

Three search modes, one flat row per paper or author:

- **Papers:** a title or keyword query. Phrase match first (the exact paper for a title query), most-cited first by default, or newest/oldest first. Filters: year or year range, fields of study, publication types, venue, minimum citations, open-access PDF only.
- **Authors:** an author name. Each author row carries affiliations, homepage, paper count, citation count, h-index, ORCID and DBLP ids. Optionally list each author's papers (newest first) after the author row.
- **Paper IDs:** a list of ids — 40-character Semantic Scholar ids, `DOI:10.…`, `ArXiv:1706.03762`, `PMID:…`, `PMCID:…`, `CorpusId:…`. Plain DOIs and arXiv numbers are recognised without a prefix.

A semanticscholar.org URL works too: a search URL (query, year range and PDF filter are read from it), a paper URL or an author URL.

Every paper row contains: `paperId`, `corpusId`, `title`, `authors`, `authorIds`, `firstAuthor`, `year`, `venue`, `journalName`, `journalVolume`, `journalPages`, `publicationDate`, `publicationTypes`, `abstract`, `url`, `citationCount`, `referenceCount`, `influentialCitationCount`, `fieldsOfStudy`, `s2FieldsOfStudy`, `doi`, `arxivId`, `pubmedId`, `isOpenAccess`, `openAccessPdfUrl`, `source`, `query`, `scrapedAt`.

Every author row contains: `authorId`, `name`, `affiliations`, `homepage`, `paperCount`, `citationCount`, `hIndex`, `orcid`, `dblp`, `url`, `query`, `scrapedAt`.

Results export to CSV, JSON, Excel, or XML, or straight from the API.

### What you can do with Semantic Scholar data

**Find the paper behind a title.**

An agent sends "deep residual learning for image recognition" and gets He et al. 2015 back as the first row, with its DOI, arXiv id and 237,000 citations.

**Build a literature review corpus.**

A PhD candidate searches "graph neural networks drug discovery" with year `2022-`, field of study `Computer Science` and at least 20 citations, and exports 500 abstracts to screen for relevance.

**Profile an author.**

A research office pulls "Yoshua Bengio" with *Include each author's papers* on and gets the author row (813 papers, h-index 212) followed by the publication list, newest first.

**Resolve a reading list.**

A lab manager pastes 200 DOIs and arXiv ids and gets one row per paper with the open-access PDF link where one exists.

### Why choose this scraper

|  | What you get |
|---|---|
| **Title search that works** | Phrase match plus citation ranking returns the well-known paper first instead of thousands of loosely related ones. |
| **Three modes** | Papers, authors and direct id lookup in one Actor. |
| **Server-side filters** | Year, field of study, publication type, venue, minimum citations and PDF availability narrow the result count rather than costing rows. |
| **No API key required** | Runs on the public pool at about one request per second; add a free key for relevance ranking and higher limits. |
| **Scales to a million items** | Paid users can pull entire research fields in one run. |

### How it compares

| Feature | ParseForge | Semantic Scholar Search Scraper |
|---|---|---|
| Title search returns the exact paper first | Yes | Not listed |
| Author search with h-index and papers | Yes | Not listed |
| Lookup by DOI, arXiv, PubMed or Corpus id | Yes | Not listed |
| Year, field of study, publication type, venue and citation filters | Yes | Not listed |
| Start from a full Semantic Scholar URL | Yes | Not listed |
| Optional API key for higher rate limits | Yes | Not listed |

### Configure the run

Pick the mode, give a query (or a Semantic Scholar URL, or a list of ids) and set the maximum rows. The Input tab lists every filter.

A first run with the defaults:

```json
{
  "searchType": "papers",
  "query": "attention is all you need",
  "maxItems": 10
}
```

A filtered keyword pull:

```json
{
  "searchType": "papers",
  "query": "graph neural networks drug discovery",
  "year": "2022-",
  "fieldsOfStudy": ["Computer Science"],
  "minCitationCount": 20,
  "sort": "newest",
  "maxItems": 200
}
```

An author with their papers:

```json
{
  "searchType": "authors",
  "query": "Yoshua Bengio",
  "includeAuthorPapers": true,
  "maxItems": 100
}
```

Papers by id:

```json
{
  "searchType": "ids",
  "ids": ["ArXiv:1706.03762", "DOI:10.1145/3065386", "204e3073870fae3d05bcbc2f6a8e263d9b72e776"]
}
```

### Limits

- Without an API key, Semantic Scholar's public pool allows roughly one request per second, and its relevance-ranked search endpoint is closed to anonymous traffic. The Actor therefore matches your query as a phrase and ranks by citations, which surfaces the paper you mean for a title search. A free key from [semanticscholar.org/product/api](https://www.semanticscholar.org/product/api) switches *Relevance* to true relevance ranking and lifts the rate limit.
- When the public pool is saturated the Actor backs off and retries; if it is still refused, the run writes one diagnostic row (`type: "error"`), which is not charged.
- Paper ids that Semantic Scholar does not know return a diagnostic row instead of a paper row.

### Pricing

Pay-per-event: **$0.16 per run start** plus **$12 per 1,000 results** (`result-item`, one per paper or author row). Diagnostic rows are free.

| Results collected | Approximate cost |
|---|---|
| 100 results | $1.36 |
| 1,000 results | $12.16 |
| 10,000 results | $120.16 |

New Apify accounts start with $5 in free credit.

### Free users

Free-plan runs return up to 10 results as a preview. [Upgrade your Apify plan](https://console.apify.com/sign-up?fpr=vmoqkp) to collect up to 1,000,000 results per run.

### Run it

1. [Create a free Apify account with $5 in credit](https://console.apify.com/sign-up?fpr=vmoqkp).
2. Open the [Semantic Scholar Scraper](https://apify.com/parseforge/semantic-scholar-scraper?fpr=vmoqkp).
3. Pick a mode, enter a query and any filters, then click **Start**.
4. Export the results as CSV, Excel, JSON, or XML from the **Dataset** tab.

Run it programmatically through the [Apify API](https://docs.apify.com/api/v2) (`run-sync-get-dataset-items`) or the [ApifyClient](https://docs.apify.com/api/client/js) for JavaScript and Python.

### Use with AI agents (MCP)

Give an AI agent live access to Semantic Scholar through the Model Context Protocol. Add the Actor to Claude, Cursor, or any MCP client:

```bash
claude mcp add --transport http apify "https://mcp.apify.com/?tools=parseforge/semantic-scholar-scraper"
```

Then ask: *"Find the paper 'Attention is All You Need' and give me its DOI and citation count."*

### FAQ

**Why did the old version return unrelated papers for a title?** The public bulk endpoint orders results by internal id unless told otherwise, so a five-word title matched thousands of papers containing those words. The Actor now matches the words as a phrase and ranks by citations, and falls back to the bare keywords only when the phrase matches nothing.

**Can I get relevance ranking like the website?** Yes, with a free Semantic Scholar API key in the *apiKey* input.

**Is the data public?** Yes. Semantic Scholar publishes this metadata through its open Graph API; the Actor reads only public records.

# Actor input Schema

## `searchType` (type: `string`):

Papers: keyword or title search (phrase match first, most-cited first). Authors: author name search, optionally with each author's papers. Paper IDs: look up specific papers by Semantic Scholar id, DOI, arXiv id, PubMed id or Corpus id.

## `query` (type: `string`):

Paper title, keywords, or an author name. Examples: "attention is all you need", "graph neural networks drug discovery", "Yoshua Bengio".

## `startUrl` (type: `string`):

Alternative to the query: a semanticscholar.org search URL (query, year range and PDF filter are read from it), a paper URL or an author URL.

## `ids` (type: `array`):

One id per line: a 40-character Semantic Scholar id, DOI:10.xxxx/…, ArXiv:1706.03762, PMID:…, PMCID:… or CorpusId:…. Plain DOIs and arXiv numbers are recognised without a prefix.

## `maxItems` (type: `integer`):

Stop after this many rows (papers, or authors plus their papers). Free-plan runs return up to 10 rows as a preview; paid plans up to 1,000,000.

## `sort` (type: `string`):

Relevance needs an API key (Semantic Scholar's relevance endpoint is closed to anonymous traffic); without one, relevance falls back to most-cited first, which surfaces the well-known paper for a title search.

## `year` (type: `string`):

A single year (2019), a range (2016-2020), or an open-ended span (2020- or -2015).

## `fieldsOfStudy` (type: `array`):

Restrict to these fields. Valid values: Computer Science, Medicine, Chemistry, Biology, Materials Science, Physics, Geology, Psychology, Art, History, Geography, Sociology, Business, Political Science, Economics, Philosophy, Mathematics, Engineering, Environmental Science, Agricultural and Food Sciences, Education, Law, Linguistics.

## `publicationTypes` (type: `array`):

Restrict to these types: Review, JournalArticle, CaseReport, ClinicalTrial, Conference, Dataset, Editorial, LettersAndComments, MetaAnalysis, News, Study, Book, BookSection.

## `venue` (type: `string`):

Journal or conference name, e.g. Nature, NeurIPS, ICML. Comma-separate several venues.

## `minCitationCount` (type: `integer`):

Only papers cited at least this many times.

## `hasPdf` (type: `boolean`):

Return only papers with an open-access PDF link.

## `includeAuthorPapers` (type: `boolean`):

After each author row, list that author's papers (most recent first) until Maximum rows is reached. Author search stops at 25 authors when this is on.

## `apiKey` (type: `string`):

Unlocks relevance-ranked search and higher rate limits. Free at semanticscholar.org/product/api. Without a key the actor uses the public pool at about one request per second.

## Actor input object example

```json
{
  "searchType": "papers",
  "query": "attention is all you need",
  "maxItems": 10,
  "sort": "relevance",
  "hasPdf": false,
  "includeAuthorPapers": false
}
```

# Actor output Schema

## `results` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "searchType": "papers",
    "query": "attention is all you need",
    "maxItems": 10
};

// Run the Actor and wait for it to finish
const run = await client.actor("parseforge/semantic-scholar-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "searchType": "papers",
    "query": "attention is all you need",
    "maxItems": 10,
}

# Run the Actor and wait for it to finish
run = client.actor("parseforge/semantic-scholar-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "searchType": "papers",
  "query": "attention is all you need",
  "maxItems": 10
}' |
apify call parseforge/semantic-scholar-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,parseforge/semantic-scholar-scraper"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/CPUhBJbKQsxvtXRgX/builds/yqdxRqZd8S4Lq5KjD/openapi.json