PubMed Articles Scraper

Full biomedical literature records from PubMed, via NCBI's own E-utilities API — abstracts, authors with affiliations and ORCIDs, MeSH terms, grant funding, DOIs and free-full-text links.

No API key needed. No login. Nothing to bypass.

What makes this different

PubMed serves at most 9,999 records per query, and the workaround everyone cites does not work. Every E-utilities tutorial says to pass usehistory=y and page through WebEnv for large result sets. Measured against cancer (5,687,758 matches), the history server returns the same refusal: 'retstart' cannot be larger than 9998.

Slicing the query by publication date does not rescue it either: a record published "2020 Jan" is indexed as 2020/01/01, so 75% of a month lands on its first day — 40,541 records for cancer, four times the window, at the finest granularity the API has.

This actor slices on Entrez date instead, which always carries a real day and distributes evenly, bisecting until every slice fits. Verified: 10,050 unique records from a query that stops at 9,999 any other way, zero duplicates, 24 requests.

Modes

Mode	What you give it	What you get
Search PubMed	a query — full PubMed syntax works	matching articles
Specific PMIDs	PMIDs or pubmed.ncbi.nlm.nih.gov URLs	those articles
Articles citing a PMID	seed PMIDs	every article citing them, each tagged with the seed it cites

Output

One SEARCH_SUMMARY row per run, one ARTICLE row per record, and an ERROR row for any input that failed — so every input maps to at least one row.

Each article carries, among ~50 fields:

• pmid, doi, pmcId, articleUrl, doiUrl, pmcUrl, isFreeFullText
• title, abstract, abstractSections (labelled sections preserved when the abstract is structured — 41% of clinical trials are), hasAbstract
• journal, journalIsoAbbrev, issn, volume, issue, pages
• pubDate, pubDatePrecision, pubDateEarliest, pubDateRaw, entrezDate, receivedDate, acceptedDate
• authors (last/fore name, initials, ORCID, affiliations), authorCount, firstAuthor, lastAuthor, affiliationCount
• meshTerms (descriptor, UI, major-topic flag, qualifiers), keywords
• publicationTypes, languages, grants (id, agency, country), journalCountry, citationStatus
• citedByCount when you ask for it; citesPmid in citedby mode

Dates are never invented

pubDate is an ISO 8601 reduced-precision string — 2026-09-21, 2026-09 or 2026, whichever the record actually supports — and pubDatePrecision says which. MEDLINE publishes seasons (Nov-Dec) and date ranges (1985 Dec 18-31) that are not dates; filling the gaps with 01 would produce a specific, wrong, entirely real-looking day. pubDateEarliest gives you a full date when you need one, and its name says it is a floor.

Honesty flags on the summary row

Because almost every PubMed failure arrives as HTTP 200 with the problem buried in the body, the summary row publishes what actually happened:

• queryTranslation — the query PubMed really ran. A plain phrase is silently expanded through MeSH: heart attack runs as a five-clause boolean.
• upstreamCount vs articlesReturned — matched vs reachable
• resultWindow, resultWindowHit, dateSlicesUsed, unsliceableDays
• maxDateDefaulted — PubMed ignores a from-date with no to-date entirely, returning the unfiltered set under a 200. The actor completes the range and tells you it did.
• pmidsNotFound, duplicatePmidsDropped, bookRecordsSkipped
• rateLimitHits, rateLimitPerSecond, ncbiApiKeyUsed

An unknown sort is refused, not passed through: PubMed ignores one without error and silently returns a different ordering.

Speed and the rate limit

NCBI allows 3 requests/second per IP, enforced with a real HTTP 429 (12 concurrent calls → 9 of them 429). A free NCBI API key raises it to 10/s; supply it as ncbiApiKey and the pace tightens automatically.

minRequestInterval, not maxConcurrency, is the honest speed control — concurrency buys nothing once the cap binds.

The proxy is off by default, deliberately. The limit is keyed on your IP, so routing through a rotating residential pool would not fix a 429; it would evade a published limit by making one client look like many.

Known limits

• 9,999 records per single query. Enabling the default date slicing goes past it; a single Entrez day holding more than 9,999 records cannot be cut finer, and those days are listed in unsliceableDays rather than silently truncated.
• esummary is deliberately not used — its JSON is a subset of the XML this actor already fetches.
• Book chapters (PubmedBookArticle) are counted but not parsed.