# ArXiv Paper Scraper (`sheshinmcfly/arxiv-paper-scraper`) Actor

Search and extract scientific papers from ArXiv.org. Returns title, authors, abstract, categories, submission date, and PDF link. Ideal for AI/ML research, RAG pipelines, academic trend monitoring, and systematic literature reviews. No API key required.

- **URL**: https://apify.com/sheshinmcfly/arxiv-paper-scraper.md
- **Developed by:** [Sheshinmcfly](https://apify.com/sheshinmcfly) (community)
- **Categories:** News, AI, Automation
- **Stats:** 3 total users, 0 monthly users, 100.0% runs succeeded, 1 bookmarks
- **User rating**: No ratings yet

## Pricing

from $2.00 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## ArXiv Paper Scraper

Search and extract **scientific papers from ArXiv.org** — the largest open-access repository of preprints in physics, mathematics, computer science, AI, and more.

Returns full metadata including title, authors, abstract, categories, submission date, and PDF link. Perfect for AI research pipelines, RAG systems, and academic trend monitoring.

***

### What data does it extract?

| Field | Description | Example |
|---|---|---|
| `arxivId` | ArXiv paper ID | `"2604.18584"` |
| `title` | Full paper title | `"MathNet: a Global Multimodal Benchmark..."` |
| `authors` | List of authors | `["Shaden Alshammari", "Kevin Wen"]` |
| `abstract` | Full abstract text | `"Mathematical problem solving remains..."` |
| `categories` | ArXiv subject tags | `["cs.AI", "cs.LG", "cs.IR"]` |
| `primaryCategory` | Primary category | `"cs.AI"` |
| `submittedDate` | Submission date | `"2026-04-20T00:00:00Z"` |
| `updatedDate` | Last update date | `"2026-04-21T00:00:00Z"` |
| `publishedDate` | Publication date | `"2026-04-20T00:00:00Z"` |
| `comments` | Author comments | `"ICLR 2026; 30 pages"` |
| `journalRef` | Journal reference | `"Proceedings of ICLR, 2026"` |
| `doi` | DOI if available | `"10.48550/arXiv.2604.18584"` |
| `pdfUrl` | Direct PDF link | `"https://arxiv.org/pdf/2604.18584"` |
| `url` | ArXiv abstract page | `"https://arxiv.org/abs/2604.18584"` |
| `citationCount` | Total citations (Semantic Scholar) | `142` |
| `influentialCitationCount` | Influential citations | `18` |
| `tldr` | AI-generated 1-sentence summary | `"A new benchmark for math reasoning..."` |
| `relevanceScore` | Computed relevance 0–100 | `72` |
| `query` | Search query used | `"large language models"` |
| `extractedAt` | Extraction timestamp | `"2026-04-21T12:00:00Z"` |

***

### Use cases

- **RAG pipelines**: Feed domain-specific papers into retrieval-augmented AI systems
- **AI research monitoring**: Track the latest publications in LLMs, computer vision, NLP
- **Academic trend analysis**: Identify hot topics and emerging research areas
- **Literature review automation**: Collect papers for a specific topic at scale
- **LLM fine-tuning data**: High-quality scientific text for model training
- **Competitive intelligence**: Monitor what research competitors are publishing

***

### How to use

1. Open the actor and configure:
   - **Mode**: `search` (keyword), `category` (e.g. `cs.AI`), or `id` (specific paper IDs)
   - **Search queries**: One or more search terms (e.g. `"diffusion models"`, `"reinforcement learning"`)
   - **Search field**: All fields, title only, abstract only, or author
   - **Sort by**: Newest first or by relevance
   - **Max results**: Number of papers to return (up to 500 per run)
   - **Semantic Scholar enrichment**: Enable to add citation counts and AI-generated TLDRs
2. Click **Start**
3. Download results as JSON, CSV, or Excel

> **Agent-ready via x402:** AI agents can run this actor directly with USDC on Base — no Apify account needed. See [x402 protocol docs](https://docs.apify.com/platform/integrations/x402).

***

### Input parameters

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `mode` | string | `"search"` | `search`, `category`, or `id` |
| `queries` | array | `["large language models"]` | Search terms (mode=search) |
| `searchField` | string | `"all"` | `all`, `title`, `abstract`, or `author` |
| `categories` | array | `["cs.AI"]` | ArXiv category codes (mode=category) |
| `idList` | array | `[]` | ArXiv IDs to fetch (mode=id) |
| `sortBy` | string | `"submittedDate"` | `submittedDate`, `relevance`, or `lastUpdatedDate` |
| `sortOrder` | string | `"descending"` | `descending` or `ascending` |
| `maxResults` | integer | `50` | Max papers to return (up to 500) |
| `minYear` | integer | `0` | Filter: exclude papers before this year |
| `maxYear` | integer | `0` | Filter: exclude papers after this year |
| `includeSemanticScholar` | boolean | `true` | Enrich with citation counts and TLDRs |

***

### Example output (JSON)

```json
{
  "arxivId": "2604.18584",
  "title": "MathNet: a Global Multimodal Benchmark for Mathematical Reasoning and Retrieval",
  "authors": ["Shaden Alshammari", "Kevin Wen", "Antonio Torralba"],
  "abstract": "Mathematical problem solving remains a challenging test of reasoning...",
  "categories": ["cs.AI", "cs.DL", "cs.IR", "cs.LG"],
  "primaryCategory": "cs.AI",
  "submittedDate": "2026-04-20T00:00:00Z",
  "publishedDate": "2026-04-20T00:00:00Z",
  "comments": "ICLR 2026; Website: http://mathnet.mit.edu",
  "journalRef": "Proceedings of ICLR, 2026",
  "doi": "10.48550/arXiv.2604.18584",
  "pdfUrl": "https://arxiv.org/pdf/2604.18584",
  "url": "https://arxiv.org/abs/2604.18584",
  "citationCount": 142,
  "influentialCitationCount": 18,
  "tldr": "A new multimodal benchmark for mathematical reasoning and retrieval across diverse problem types.",
  "relevanceScore": 72,
  "query": "large language models",
  "extractedAt": "2026-04-21T12:00:00.000Z"
}
```

***

### Pricing

This actor charges **$0.002 USD per paper extracted**. Extracting 100 papers costs approximately $0.20 USD.

***

### FAQ

**What is the difference between search, category, and id mode?**
`search` finds papers by keywords across all fields (or title/abstract/author). `category` returns the latest papers in a specific ArXiv subject (e.g. `cs.AI`, `math.ST`). `id` fetches exact papers by their ArXiv ID (e.g. `2312.00752`).

**How many papers can I extract per run?**
Up to 500 papers per run. For larger batches, run multiple times with different date ranges using `minYear`/`maxYear`.

**What are citation counts and TLDRs?**
When `includeSemanticScholar` is enabled, each paper is enriched with citation counts and an AI-generated one-sentence summary (TLDR) from the Semantic Scholar API. This adds ~0.4s per paper.

**Does it require an API key or login?**
No. ArXiv's public API (`export.arxiv.org`) is free and unauthenticated. Semantic Scholar's free tier is used for enrichment.

**What ArXiv categories are supported?**
All ArXiv categories: `cs.*` (Computer Science), `math.*`, `physics.*`, `stat.*`, `q-bio.*`, `econ.*`, and more. Full list at [arxiv.org/category\_taxonomy](https://arxiv.org/category_taxonomy).

***

### Other actors you may like

- **[StackOverflow Scraper](https://apify.com/sheshinmcfly/stackoverflow-scraper)** — search developer Q\&A by keyword or tag.
- **[Trustpilot Reviews Scraper](https://apify.com/sheshinmcfly/trustpilot-reviews-scraper)** — extract reviews and ratings from Trustpilot.
- **[SEC EDGAR Scraper](https://apify.com/sheshinmcfly/sec-edgar-scraper)** — company filings and financial disclosures from the SEC.
- **[FinViz Stock Screener](https://apify.com/sheshinmcfly/finviz-stock-screener)** — stock screener with gainers, losers, and sector filters.

***

### Keywords

arxiv scraper, scientific paper extractor, research paper scraper, arxiv API, AI paper scraper, academic data extractor, preprint scraper, NLP research data, LLM training data, arxiv search scraper

***

### Legal Disclaimer

This actor extracts **publicly available open-access data only** from ArXiv.org, in compliance with Chilean Law 19.628 on the Protection of Private Life (*Ley 19.628 sobre Protección de la Vida Privada*).

ArXiv is an open-access repository operated by Cornell University. All papers and metadata extracted are freely and publicly accessible without authentication.

**What this actor does NOT collect:**

- Names or personal data of any private individuals
- User accounts, submissions portals, or private information
- Any data not freely visible to anonymous visitors

**What this actor collects:**

- Paper titles, abstracts, and author names (public academic data)
- Subject categories and submission dates
- Public URLs and PDF links

Users are solely responsible for ensuring their use of this data complies with applicable laws and ArXiv's terms of use.

# Actor input Schema

## `mode` (type: `string`):

How to search: by keyword (search), by category (category), or by specific paper IDs (id).

## `queries` (type: `array`):

Search terms for mode=search (e.g. 'large language models', 'diffusion models').

## `searchField` (type: `string`):

Which field to search in (mode=search only).

## `categories` (type: `array`):

Category codes for mode=category (e.g. 'cs.AI', 'math.ST', 'physics.hep-th').

## `idList` (type: `array`):

ArXiv paper IDs to fetch for mode=id (e.g. '2312.00752', '2404.01234').

## `sortBy` (type: `string`):

How to order results.

## `sortOrder` (type: `string`):

Ascending or descending.

## `maxResults` (type: `integer`):

Maximum number of papers to return.

## `minYear` (type: `integer`):

Exclude papers published before this year (0 = no filter).

## `maxYear` (type: `integer`):

Exclude papers published after this year (0 = no filter).

## `includeSemanticScholar` (type: `boolean`):

Add citation counts and AI-generated TLDRs from Semantic Scholar (adds ~0.4s per paper).

## Actor input object example

```json
{
  "mode": "search",
  "queries": [
    "large language models"
  ],
  "searchField": "all",
  "categories": [
    "cs.AI"
  ],
  "idList": [],
  "sortBy": "submittedDate",
  "sortOrder": "descending",
  "maxResults": 50,
  "minYear": 0,
  "maxYear": 0,
  "includeSemanticScholar": true
}
```

# Actor output Schema

## `dataset` (type: `string`):

All scraped research papers stored in the default dataset.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {};

// Run the Actor and wait for it to finish
const run = await client.actor("sheshinmcfly/arxiv-paper-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {}

# Run the Actor and wait for it to finish
run = client.actor("sheshinmcfly/arxiv-paper-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{}' |
apify call sheshinmcfly/arxiv-paper-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,sheshinmcfly/arxiv-paper-scraper"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/S0VKm9yIKb1mBE3cd/builds/4RatpPq5qzcbSFbtL/openapi.json