# ArXiv Research Paper Scraper (`datapilot/arxiv-research-paper-scraper`) Actor

It extracts titles, abstracts, authors with affiliations, DOI, categories, submission dates, and PDF links. Supports proxy usage and outputs structured JSON results for research and data analysis.

- **URL**: https://apify.com/datapilot/arxiv-research-paper-scraper.md
- **Developed by:** [Data Pilot](https://apify.com/datapilot) (community)
- **Categories:** Videos, Integrations, Social media
- **Stats:** 3 total users, 0 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $2.00 / 1,000 scraped-results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## ArXiv Research Paper Scraper

🚀 **ArXiv Research Paper Scraper** is a powerful Apify Actor designed to retrieve academic paper metadata from the arXiv API based on specific keywords. This tool provides comprehensive **ArXiv Research information**, including titles, abstracts, authors with affiliations, DOI, categories, submission dates, and PDF links for any search query. Whether you're conducting **ArXiv Research analysis**, literature reviews, or data mining, the ArXiv Research Paper Scraper delivers accurate **ArXiv Research data** efficiently.

With API-based scraping and proxy support, the ArXiv Research Paper Scraper ensures reliable extraction of **ArXiv Research metadata** that may be subject to rate limits. It focuses on key **ArXiv Research metrics** like author affiliations and submission dates, making it an essential tool for **ArXiv Research analysis** and academic data collection.

### 🔥 Features

- **Comprehensive ArXiv Research Extraction** – Scrapes detailed **ArXiv Research data**, including titles, abstracts, authors, and PDF links for any keyword.
- **Metadata Enrichment** – Provides **ArXiv Research metadata** like DOI, categories, and submission dates for in-depth analysis.
- **Author Affiliation Tracking** – Extracts author names and affiliations for **ArXiv Research networking** and citation tracking.
- **Proxy Support** – Utilizes Apify's residential proxies to bypass restrictions and ensure high success rates for **ArXiv Research scraping**.
- **Structured JSON Output** – Returns structured **ArXiv Research data** for easy integration into research databases.
- **Batch Processing** – Processes multiple **ArXiv Research keywords** in a single run for efficient data collection.
- **Error Handling** – Robust logging and fallback mechanisms for failed **ArXiv Research extractions**.
- **Dataset Integration** – Automatically uploads **ArXiv Research data** to your Apify dataset for easy export and analysis.

***

### ⚙️ How It Works

The ArXiv Research Paper Scraper takes a search keyword as input and queries the arXiv API to retrieve **ArXiv Research papers**. It parses the API response to extract metadata such as titles, abstracts, and author details. The scraper returns structured **ArXiv Research data** on success or error details on failure, providing a reliable way to gather **ArXiv Research information** for academic and data analysis purposes.

**Key Processing Steps:**

1. **Keyword Input** – Parse and validate search keywords
2. **API Query** – Query arXiv API with search parameters
3. **Response Parsing** – Parse XML API response
4. **Metadata Extraction** – Extract title, abstract, authors
5. **Author Affiliation Tracking** – Extract author details and affiliations
6. **DOI & Category Extraction** – Extract DOI and research categories
7. **PDF Link Generation** – Generate PDF download links
8. **Export** – Push results to dataset in JSON format

**Key benefits for ArXiv Research analysis:**

- Track **ArXiv Research trends** and publication dates.
- Analyze **ArXiv Research author networks** and affiliations.
- Build **ArXiv Research databases** for literature reviews.
- Research emerging topics and trends.
- Find related papers and authors.

***

### 📥 Input

The scraper accepts the following input parameters:

| Field              | Type    | Default           | Description |
|--------------------|---------|-------------------|-------------|
| `keyword`          | string  | required          | The search keyword to find ArXiv Research papers (e.g., `"machine learning"`, `"quantum physics"`). |
| `useApifyProxy`    | boolean | `true`            | Enable residential proxies for ArXiv Research scraping. |
| `apifyProxyGroups` | array   | `["RESIDENTIAL"]` | Proxy groups to use (e.g., `["RESIDENTIAL"]`). |

**Example input JSON:**

```json
{
  "keyword": "artificial intelligence",
  "useApifyProxy": true,
  "apifyProxyGroups": ["RESIDENTIAL"]
}
```

**Example Search Keywords:**

- `"machine learning"` – ML papers
- `"quantum computing"` – Quantum research
- `"neural networks"` – Deep learning papers
- `"natural language processing"` – NLP research
- `"computer vision"` – CV papers

***

### 📤 Output

The scraper outputs detailed **ArXiv Research data** in JSON format for each paper. Each record includes:

| Field            | Type   | Description |
|------------------|--------|-------------|
| `title`          | string | Title of the ArXiv Research paper. |
| `abstract`       | string | Abstract of the ArXiv Research paper. |
| `authors`        | array  | List of authors with affiliations. |
| `doi`            | string | DOI of the ArXiv Research paper. |
| `categories`     | array  | Categories of the ArXiv Research paper. |
| `submissionDate` | string | Submission date of the paper. |
| `pdfLink`        | string | Direct link to the PDF. |
| `url`            | string | URL to the ArXiv paper page. |

**Example output for ArXiv Research data:**

```json
{
  "title": "Example ArXiv Research Paper",
  "abstract": "This is an example abstract for the arXiv research paper...",
  "authors": [
    {
      "name": "John Doe",
      "affiliation": "University of Example"
    },
    {
      "name": "Jane Smith",
      "affiliation": "Institute of Technology"
    }
  ],
  "doi": "10.48550/arXiv.1234.5678",
  "categories": ["cs.AI", "stat.ML"],
  "submissionDate": "2025-02-14",
  "pdfLink": "https://arxiv.org/pdf/1234.5678.pdf",
  "url": "https://arxiv.org/abs/1234.5678"
}
```

**Example summary record:**

```json
{
  "summary": true,
  "keyword": "artificial intelligence",
  "total_papers": 100,
  "papers_returned": 50,
  "date_range": "2023-2025",
  "categories_found": 12,
  "authors_found": 250,
  "completed_at": "2025-02-14T12:35:00Z"
}
```

***

### 🧰 Technical Stack

- **API Integration:** arXiv API – Official academic paper repository
- **HTTP Client:** requests – API calls and data fetching
- **Data Parsing:** XML parsing for API responses
- **JSON Processing:** Structured data formatting
- **Proxy Support:** Apify Proxy with RESIDENTIAL support
- **Platform:** Apify Actor – serverless, scalable, integrated with Dataset
- **Deployment:** One‑click run on Apify Console or via REST API

***

### 🎯 Use Cases

- **Literature Reviews** – Find papers for comprehensive literature review.
- **Research Trend Analysis** – Identify emerging trends in research fields.
- **Author Network Analysis** – Map author networks and collaborations.
- **Citation Tracking** – Track papers and their impact.
- **Dataset Creation for ML** – Create datasets for machine learning research.
- **Academic Research** – Conduct academic and meta-research studies.
- **Topic Modeling** – Analyze research topics and trends.
- **Affiliation Analysis** – Analyze research by institution and affiliation.
- **Time Series Analysis** – Track research publication trends over time.
- **Field Analysis** – Analyze specific research fields comprehensively.
- **Researcher Profiling** – Build profiles of researchers and their work.
- **Collaboration Analysis** – Identify collaboration patterns.
- **Emerging Technology Research** – Track new technologies and methodologies.
- **Academic Benchmarking** – Compare research output across institutions.

***

### 🚀 Quick Start

1. **Open in Apify Console** – visit the Actor page and click Try for free.
2. **Enter search keyword** – provide a research topic (e.g., "artificial intelligence").
3. **Set proxy option** – enabled by default for reliable access.
4. **Click Start** – the Actor will query the arXiv API.
5. **View Results** – check the dataset for extracted paper metadata.
6. **Review Papers** – examine titles, abstracts, authors, and affiliations.
7. **Download PDFs** – use provided PDF links to access full papers.
8. **Export** – download the results as JSON, CSV, or Excel for analysis.

You can also call this Actor programmatically via Apify SDK or REST API – ideal for automated literature review and academic research pipelines.

***

### 💎 Why This Scraper?

| Feature | Benefit |
|---------|---------|
| ✅ Official API | Direct access to arXiv's official API. |
| ✅ Complete metadata | Get titles, abstracts, authors, DOI, PDFs. |
| ✅ Author affiliations | Track institution affiliations. |
| ✅ Research categories | Identify research field/category. |
| ✅ Structured output | JSON format ready for databases. |
| ✅ Proxy support | Reliable access with fallback. |
| ✅ Error handling | Robust error handling. |
| ✅ Apify ecosystem | Seamless integration with other Actors, triggers, and webhooks. |

### 📦 Changelog

- Initial release of ArXiv Research Paper Scraper
- arXiv API integration
- Keyword-based paper search
- Title and abstract extraction
- Author and affiliation extraction
- DOI and category extraction
- Submission date parsing
- PDF link generation
- Structured JSON output
- Batch processing for multiple keywords
- Error handling with fallback mechanisms
- Proxy support for reliability
- Summary statistics and reporting
- Automatic dataset integration
- Full Apify Actor integration

***

### 🧑‍💻 Support & Feedback

- **Issues & Ideas:** Open a ticket on the Apify Actor issue tracker
- **Contributions:** Pull requests are welcome via the GitHub repository
- **Documentation:** Visit Apify Docs for comprehensive platform guides
- **Community:** Join the Apify community forum for discussions and support
- **Bug Reports:** Submit detailed bug reports through the issue tracker
- **Feature Requests:** Suggest new features to improve the scraper

***

### 💰 Pricing

- **Free** for basic usage on Apify platform
- **Paid plans** available for higher limits and priority support

***

**Disclaimer:**
ArXiv Research Paper Scraper is provided as-is for research and academic purposes. Users are responsible for ensuring their usage complies with arXiv's policies and applicable laws. Always attribute papers appropriately and respect academic integrity standards.

***

### 🎉 Get Started Today

**Begin researching papers now!**

Use ArXiv Research Paper Scraper for:

- 📚 Literature Reviews
- 📊 Research Analysis
- 🔍 Topic Research
- 💡 Trend Analysis
- 📈 Academic Research

**Perfect for:**

- Researchers
- PhD Students
- Academics
- Data Scientists
- Students

***

**Last Updated:** February 2025\
**Version:** 1.0.0\
**Status:** Active Development\
**Support:** 24/7 Customer Support Available\
**Platform:** Apify

***

### 📚 Related Tools

For comprehensive academic research and paper analysis, explore our full suite of tools:

- Smart Article Extractor
- AI Blog Dataset Creator
- Fast News Content Scraper
- RAG Web Scraper
- All-in-One Media Downloader

# Actor input Schema

## `keyword` (type: `string`):

The topic you want to search for.

## `max_results` (type: `integer`):

Number of papers to fetch.

## `proxyConfiguration` (type: `object`):

Recommended to avoid rate limits.

## Actor input object example

```json
{
  "keyword": "machine learning",
  "max_results": 10,
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}
```

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "keyword": "machine learning"
};

// Run the Actor and wait for it to finish
const run = await client.actor("datapilot/arxiv-research-paper-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "keyword": "machine learning" }

# Run the Actor and wait for it to finish
run = client.actor("datapilot/arxiv-research-paper-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "keyword": "machine learning"
}' |
apify call datapilot/arxiv-research-paper-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,datapilot/arxiv-research-paper-scraper"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/FymOmuhglv3RrVAE8/builds/1gmmz6P7AxPzTogLy/openapi.json