Semantic Scholar Academic Paper Scraper

Search and extract academic papers, citations, and authors from Semantic Scholar. 200M+ papers with citation graphs and impact metrics. Search papers, get detailed paper info, or find researchers. API key optional. For research and AI.

Use cases

• Rank papers by citation count in a research area
• Build a citation-aware literature map
• Find the most-cited work at a venue or in a year
• Feed a research assistant with paper metadata
• Track a researcher's citation growth

Input

You need your own free Semantic Scholar key. Get one at https://www.semanticscholar.org/product/api#api-key-form — it takes a minute and costs nothing. Without it the Actor cannot fetch data.

Parameter	Type	Required	Default	Description
mode	string	Yes	"searchPapers"	Operation mode: search papers, get single paper details, or search authors Allowed: searchPapers, getPaper, getAuthor.
query	string	No	"machine learning"	Search query for papers or authors
paperId	string	No	""	Paper ID, DOI, or ArXiv ID (for getPaper mode). Examples: '649def34f8be52c8b66281af98ae884c09aef38b', 'DOI:10.1145/3292500.3330919', 'ARXIV:2005.14165'
apiKey	string	No	""	Semantic Scholar API key (optional, increases rate limit from 100 req/5min to 1 req/sec). Get one at semanticscholar.org/product/api
maxResults	integer	No	50	Maximum number of results to return (1-1000)

Example input

{
  "mode": "searchPapers",
  "query": "machine learning",
  "paperId": "",
  "apiKey": "YOUR_API_KEY",
  "maxResults": 10
}

Output

Each dataset item has the following shape (fields vary by mode):

Field	Type
paperId	string
title	string
year	number
citationCount	number
venue	string
url	string
publicationDate	string

The dataset also ships a preset table view (Papers), so the key columns are readable straight away in Apify Console, and exportable to JSON, CSV, Excel, or XML.

How to run it

In Apify Console — open the Actor, fill in the input form, click Start, then download the results from the Dataset tab.

With the JavaScript client

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: 'YOUR_APIFY_TOKEN' });

const run = await client.actor('cloud9_ai/semantic-scholar-scraper').call({
  "mode": "searchPapers",
  "query": "machine learning",
  "paperId": "",
  "apiKey": "YOUR_API_KEY",
  "maxResults": 10
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items);

With the Python client

from apify_client import ApifyClient

client = ApifyClient('YOUR_APIFY_TOKEN')

run = client.actor('cloud9_ai/semantic-scholar-scraper').call(run_input={
  "mode": "searchPapers",
  "query": "machine learning",
  "paperId": "",
  "apiKey": "YOUR_API_KEY",
  "maxResults": 10
})

for item in client.dataset(run['defaultDatasetId']).iterate_items():
    print(item)

With the API — POST https://api.apify.com/v2/acts/cloud9_ai~semantic-scholar-scraper/run-sync-get-dataset-items?token=YOUR_APIFY_TOKEN with the input JSON as the body.

Notes and limits

• Bring your own Semantic Scholar key (free — https://www.semanticscholar.org/product/api#api-key-form). It is a secret input, so it is not written to the dataset or the log.
• maxResults caps how much a single run collects, which is also what caps the run's cost.
• Requests are paced and failed requests are retried automatically, so runs stay inside the source's rate limits.
• Only publicly available data is collected. How you use the output is your responsibility, including the source's terms of use and any applicable data-protection law.

Support

Found a bug, or need a field that isn't in the output? Open an issue on the Issues tab of this Actor in Apify Console. Issues there are read and answered.

License

Apache-2.0