# arXiv Scraper for RAG: Papers as Chunked JSON (`getascraper/arxiv-rag-extractor`) Actor Scrape arXiv papers by date and category. Strips LaTeX and returns RAG-ready JSON with tokenizer-aware chunks (cl100k\_base, 512/50). Drop-in for LangChain, LlamaIndex, Qdrant, Pinecone, Weaviate, pgvector, Chroma. Skip GROBID / Nougat / pandoc. $0.015 per paper. - **URL**: https://apify.com/getascraper/arxiv-rag-extractor.md - **Developed by:** [GetAScraper](https://apify.com/getascraper) (community) - **Categories:** AI, Agents, Developer tools - **Stats:** 2 total users, 1 monthly users, 93.1% runs succeeded, 0 bookmarks - **User rating**: 5.00 out of 5 stars ## Pricing from $0.67 / 1,000 dossier records This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have. Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event ## What's an Apify Actor? Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. - **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use). - **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`. - **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`. - **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`. - **Any other language** — the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## 📄 arXiv scraper for RAG: papers as chunked JSON
Turn arXiv preprints into RAG-ready chunks in one API call. Pulls papers by date and category, strips LaTeX source to clean text, and returns 512-token chunks with full metadata ready for your vector database.
RESEARCH AND REGULATORY RAG EXTRACTORS   •  Turns arXiv, bioRxiv/medRxiv, PubMed, CourtListener case law, and SEC EDGAR filings into RAG-ready, chunked JSON for grounded LLM retrieval.
 arXiv RAG
➤ You are here 		 bioRxiv/medRxiv
Preprints, chunked JSON 		 CourtListener
Case law, chunked 		 PubMed RAG
PubMed literature, chunked
arXiv papers, delivered as clean chunked JSON ready for embedding
Strips LaTeX source into plain text and splits it into tokenizer-aware chunks, no manual parsing or extraction servers needed.
📄 LaTeX stripped clean
Plain text output, no math markup to parse 		🧩 Tokenizer-aware chunks
512 tokens, 50 overlap, cl100k_base 		🗄️ Vector-DB neutral
Drops into Qdrant, Pinecone, Weaviate, Chroma 		📅 Date and category filters
Query arXiv metadata directly, no listing scrape
**Scrape arXiv research papers into RAG-ready JSON in one call.** Pulls papers by date range and category, strips the LaTeX source to clean text, and returns fixed-token chunks (512 tokens, 50 overlap) with full metadata, ready to drop into LangChain, LlamaIndex, Qdrant, Pinecone, Weaviate, pgvector, or Chroma. Built for AI training data teams and anyone who has fought LaTeX compiling trying to get arXiv into an embedding pipeline. ### 🔍 What does arXiv scraper for RAG do? This Apify Actor scrapes **[arXiv](https://arxiv.org/)** papers within a date range and category filter, fetches the LaTeX source for each paper, strips the markup, and splits the resulting plain text into **tokenizer-aware chunks (512 tokens, 50-token overlap, tiktoken cl100k\_base)** ready to embed or feed into a RAG index. Each output record contains clean metadata (title, authors, categories, DOI, PDF URL, published/updated dates) and a `chunks` array of `{ idx, text, tokens }` ready for direct ingestion into a vector database. **Try it in the Apify Console.** Fill in a category (e.g. `cs.LG`), a date range, a paper cap, and hit Start. Download the results as JSON, CSV, or Excel. Built on the Apify platform, you also get: scheduled runs, HTTP API access, integrations with Zapier, Make, and Zapier, proxy rotation when needed, monitoring, and alerts. No infrastructure to run yourself. ### 💡 Why use arXiv scraper for RAG? - **Skip the LaTeX parsing hell**: Papers are delivered as plain text, not mathematical syntax soup. No complex extraction servers or custom macro handlers needed. - **Pre-chunked for RAG**: `tiktoken cl100k_base` tokenization, compatible with OpenAI `text-embedding-3`, Claude, Cohere, and most BGE/E5/nomic embedding models. - **Vector-DB-neutral**: Drops straight into Qdrant, Pinecone, Weaviate, pgvector (Supabase / Neon), Chroma, and Milvus without reformatting. - **Framework-ready**: Works out of the box with LangChain, LlamaIndex, Haystack, and LangGraph pipelines. - **Dates + categories**: We query the repository metadata directly, no need to scrape arXiv listing pages yourself. - **Respects arXiv's rate limits**: 1 request per 3 seconds, handled server-side, so your runs stay stable and unblocked. - **Cheap**: $0.015 per paper. A week of machine learning submissions (~500 papers) costs under $8. ### 🚀 How to use arXiv RAG extractor
STEP 1
Set your filter
Pick `categoriesFilter` tags and a `dateFrom`/`dateTo` range. 		STEP 2
Run the extractor
Fetches LaTeX source per paper and strips it to plain text. 		STEP 3
Download chunked JSON
Get token-aware chunks ready to embed, from the Storage tab.
### ⚙️ Input | Field | Type | Required | Description | |---|---|---|---| | `dateFrom` | string | Yes | Inclusive lower bound of paper submission date in YYYY-MM-DD format. Default: `"2024-01-01"`. | | `dateTo` | string | Yes | Inclusive upper bound of paper submission date in YYYY-MM-DD format. Default: `"2024-01-02"`. | | `categoriesFilter` | array of strings | No | arXiv category tags (e.g. `["cs.LG", "cs.AI"]`). Empty matches all categories. | | `maxPapers` | integer | No | Hard cap on papers returned (1 to 100000). Default: `10`. | **Example input:** ```json { "categoriesFilter": ["cs.LG", "cs.AI"], "dateFrom": "2024-01-01", "dateTo": "2024-01-02", "maxPapers": 10 } ``` ### 📦 Output Each paper becomes one dataset item. You can download the dataset in JSON, HTML, CSV, or Excel. ```json { "arxiv_id": "2401.12345", "title": "Attention Is All You Need", "abstract": "The dominant sequence transduction models...", "authors": ["Ashish Vaswani", "Noam Shazeer", "..."], "categories": ["cs.LG", "cs.CL"], "published": "2024-01-15T00:00:00Z", "updated": "2024-01-18T00:00:00Z", "doi": "10.48550/arXiv.2401.12345", "pdf_url": "https://arxiv.org/pdf/2401.12345v1", "source": "latex", "chunks": [ { "idx": 0, "text": "...", "tokens": 512 }, { "idx": 1, "text": "...", "tokens": 512 }, { "idx": 2, "text": "...", "tokens": 487 } ] } ``` #### Data table | Field | Type | Description | |---|---|---| | `arxiv_id` | string | arXiv identifier (e.g. `2401.12345`) | | `title` | string | Paper title (whitespace-normalized) | | `abstract` | string | Abstract as returned by arXiv | | `authors` | string\[] | Author display names, order preserved | | `categories` | string\[] | arXiv category tags | | `published` | ISO date | First submission datetime | | `updated` | ISO date | Latest update datetime | | `doi` | string? | DOI (when provided by arXiv) | | `pdf_url` | string | Direct PDF link | | `source` | `"latex"` | `"abstract"` | Text origin: `latex` if source was extractable, `abstract` as fallback | | `chunks` | Chunk\[] | Fixed-token chunks ready for embedding | | `chunks[].idx` | number | 0-indexed position | | `chunks[].text` | string | Chunk text | | `chunks[].tokens` | number | Token count under cl100k\_base (≤ 512) | ### 💰 Pricing **$0.015 per paper** (PPR, pay per result). #### How much does it cost to scrape arXiv? | Volume | Estimated cost | |---|---| | 10 papers | **~$0.15** | | 100 papers | **~$1.50** | | 1,000 papers | **~$15.00** | | 10,000 papers | **~$150.00** | | 100,000 papers | **~$1,500.00** | No subscription. No minimum. You pay only for successful records. ### ⭐ Enjoying arXiv scraper for RAG?
⭐ ⭐ ⭐ ⭐ ⭐
Pre-chunked, tokenizer-aware arXiv text ready for your vector database, without wrestling LaTeX yourself.
A 5-star rating takes 10 seconds and helps other AI training data teams and RAG pipeline builders find it. Your feedback also tells us what to build next.
★  Rate this Actor on Apify
### ⚠️ Limits you should know before you run - **Only the latest version** of each preprint is returned. Version history is a v2 feature. - **No figure or table extraction**: Captions stay inline as text inside body chunks. Figure and table content is dropped during the JATS strip pass. - **No citation graph**: Reference lists are stripped from body text to keep chunks dense. Reference extraction is a v2 feature. - **No section-aware chunking**: Chunks are fixed-token (512 with 50 overlap). Section-level splitting (Abstract / Introduction / Methods / Results / Discussion) is deferred. ### 📌 Tips - **Narrow the category filter.** Running on all of arXiv will hit your `maxPapers` cap fast. Use specific categories like `cs.LG`, `stat.ML`, `q-bio.QM`. - **Short date windows for testing.** 1-day windows are a good smoke test. - **`source: "abstract"` means no LaTeX source was available.** Either arXiv only hosts a PDF, or the tarball was unreadable. About 5 to 10% of papers fall into this category depending on era. - **Schedule weekly runs** to keep an embedding index fresh on newly-published research. - **For large backfills**, split into month-sized windows and run in parallel (separate Actor runs) to stay under the OAI rate limit cleanly. ### ❓ FAQ and limitations #### Is scraping arXiv legal? arXiv offers an open API. This Actor respects arXiv's stated rate limit of 1 request every 3 seconds and fetches only publicly-available content. #### What's not in the current release? - Multi-source support (PubMed, bioRxiv, Semantic Scholar). - Entity extraction (datasets, tasks, models). - Section-aware splitting (we chunk by fixed tokens across the whole paper). - Equation, figure, and table preservation. These are dropped by the LaTeX stripper. - Semantic chunking (fixed-token only). - Custom chunk sizes (hardcoded at 512/50). Many of these are deferred. Open an issue (Issues tab) if one is blocking for you. #### Rate limits - **arXiv API:** 1 request per 3 seconds (enforced). ~1000 papers minimum run time ≈ 50 minutes. - **arXiv e-print:** Same rate limit, handled independently. #### Support Found a bug or want a feature? Use the **Issues** tab on the Actor's page. Custom requirements (non-arXiv sources, different chunking, section-aware splitting)? Reach out via the Actor's Support link. Custom solutions available. #### Disclaimer Output metadata is from arXiv's public API. Full text, when available, is from arXiv's public e-print archive. All content remains under the license specified by the paper's authors on arXiv. Check `arxiv.org/abs/` for each paper's license before downstream use (CC-BY, CC-0, arXiv non-exclusive, etc. Some papers do not permit commercial redistribution). ### 🔗 Other actors - [bioRxiv and medRxiv scraper for RAG](https://apify.com/getascraper/biorxiv-medrxiv-rag-extractor) ↗ - extracts preprint papers as chunked JSON for RAG pipelines. - [PubMed Scraper for RAG](https://apify.com/getascraper/pubmed-rag-extractor) ↗ - pulls biomedical literature as chunked JSON for embeddings. - [CourtListener RAG Extractor](https://apify.com/getascraper/courtlistener-rag-extractor) ↗ - extracts legal opinions and case law as chunked JSON. - [SEC EDGAR Scraper for RAG](https://apify.com/getascraper/sec-edgar-rag-extractor) ↗ - extracts 10-K, 10-Q and 8-K filings as chunked JSON. # Actor input Schema ## `categoriesFilter` (type: `array`): Select subject areas (such as cs.LG for Machine Learning, or cs.AI for Artificial Intelligence). Leave empty to collect all. ## `dateFrom` (type: `string`): Collect papers published on or after this date. ## `dateTo` (type: `string`): Collect papers published on or before this date. ## `maxPapers` (type: `integer`): Set the maximum number of papers you want to save for this run. ## Actor input object example ```json { "categoriesFilter": [ "cs.LG" ], "dateFrom": "2024-01-01", "dateTo": "2024-01-02", "maxPapers": 10 } ``` # Actor output Schema ## `results` (type: `string`): No description # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "categoriesFilter": [ "cs.LG" ], "dateFrom": "2024-01-01", "dateTo": "2024-01-02", "maxPapers": 10 }; // Run the Actor and wait for it to finish const run = await client.actor("getascraper/arxiv-rag-extractor").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "categoriesFilter": ["cs.LG"], "dateFrom": "2024-01-01", "dateTo": "2024-01-02", "maxPapers": 10, } # Run the Actor and wait for it to finish run = client.actor("getascraper/arxiv-rag-extractor").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}") for item in client.dataset(run.default_dataset_id).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "categoriesFilter": [ "cs.LG" ], "dateFrom": "2024-01-01", "dateTo": "2024-01-02", "maxPapers": 10 }' | apify call getascraper/arxiv-rag-extractor --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "type": "http", "url": "https://mcp.apify.com/?tools=fetch-actor-details,getascraper/arxiv-rag-extractor" } } } ``` The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer ` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations). ## OpenAPI specification Download the OpenAPI definition: https://api.apify.com/v2/actors/fTIeAjdm2wmoKI6l5/builds/9tC3wqRX2LcKC5uJJ/openapi.json