arXiv Scraper — papers by search, category or ID

Turn arXiv into clean, structured, agent-ready data via the official open API — no key, no login. Built for AI / RAG pipelines and research monitoring: answer "find recent papers on X" in one call. Three modes:

Mode	What you get	Use it for
Search	Papers matching a query (title, abstract, author, category), newest first	Literature review, RAG ingestion, topic monitoring
Category latest	The newest papers in a subject (cs.AI, cs.LG, cs.CL, stat.ML, …)	Daily research digests, staying current in a field
By ID	Specific papers by arXiv ID	Enriching a reading list or citation graph

Built for AI agents

Exposed as an MCP tool, an agent can pull the latest relevant papers, with abstracts and PDF links, straight into a RAG pipeline — clean JSON, no PDF-parsing or scraping fragility.

Output

Flat JSON rows: id, title, summary (abstract), authors, primary_category, categories, published, updated, doi, abs_url, pdf_url.

Input examples

{ "mode": "search", "query": "all:retrieval augmented generation", "maxItems": 100 }

{ "mode": "category", "category": "cs.AI", "sortBy": "submittedDate", "maxItems": 50 }

{ "mode": "id", "ids": ["2308.08155", "1706.03762"] }

Use cases

• Research monitoring — schedule daily pulls of new papers in your categories (e.g. cs.AI, cs.LG) for a personal or team digest.
• LLM/RAG pipelines — feed fresh paper metadata and abstracts into embeddings and retrieval systems.
• Literature reviews — bulk-collect metadata for a topic into CSV for screening.
• Trend analysis — track publication volume by topic to watch fields heat up.
• AI agents — as an MCP tool: "what was published on diffusion models this week?"

FAQ

How do I download arXiv paper metadata in bulk? Give the actor a search query, category or list of IDs; it uses arXiv's public API politely and returns title, authors, abstract, categories, dates and PDF links.

Can I get new AI papers from arXiv every day automatically? Yes — schedule the actor with a category like cs.AI sorted by date; each run's dataset holds the newest papers, exportable or webhook-delivered.

Does it export to CSV for literature reviews? Every dataset downloads as CSV, Excel, JSON or XML — ready for screening spreadsheets or Zotero import pipelines.

Is scraping arXiv allowed? The actor uses arXiv's official public API with polite rate limits — the access arXiv provides exactly for this purpose.

Legal

Uses arXiv's official, documented API for public paper metadata. No login, no circumvention.

Pricing

Pay-per-event: a small actor-start fee plus a per-paper charge — you pay for exactly the papers you pull.