Medium Scraper

Extract writer profiles, article listings, reading history, and topic/tag discovery from Medium.com. Built for content marketers, thought-leadership researchers, AI training-data teams, and publication scouts.

What does Medium Scraper do?

Medium Scraper collects structured data from Medium using RSS feeds (for article listings) and Cheerio scraping of public pages — reading each page's embedded JSON (?format=json) with an HTML fallback (for detailed profiles and article metadata). It extracts:

• Writer profiles: name, username, bio, follower/following count, profile image, verification status, join date, post count, top tags, publications
• Auto-discovered publication feeds: when a user owns a publication (e.g. @dailynewsletter owns The Medium Blog), its article feed is pulled automatically for more results
• Full user history: fetchUserHistory paginates a user's complete article timeline (up to maxItemsPerUserHistory, default 10,000) with claps, responses, reading time and tags — RSS only ever lists ~10
• Article listings: title, description, URL, author, publication, tags, clap count, response count, reading time, publish/update dates
• Topic/tag discovery: tag names, slugs, post counts from tag feeds
• Publication feeds: articles from specific publications

All data is delivered as clean JSON/CSV/Excel via Apify dataset.

Why use Medium Scraper?

Use Case	Value
Content marketers	Discover trending topics, top writers, and competitive content in your niche
Thought-leadership researchers	Track influential voices, their publishing frequency, and engagement metrics
AI training-data teams	Harvest high-quality, structured article corpora with metadata for LLM fine-tuning
Publication scouts	Identify rising authors, analyze publication performance, find syndication partners

How to use Medium Scraper

1. Add input – Specify usernames, tags, and/or publications to scrape
2. Configure limits – Set max items per feed and total request cap
3. Enable details – Toggle user profiles and/or full article details (requires Apify Proxy)
4. Run – Click Start and wait for results
5. Export – Download data as JSON, CSV, Excel, or hook into your pipeline via API

Input

Field	Type	Required	Description
usernames	string[]	No	Medium usernames (without @) to scrape profiles and articles from. Example: ["medium", "towardsdatascience"]
tags	string[]	No	Medium tags to scrape articles from. Example: ["technology", "artificial-intelligence", "programming"]
publications	string[]	No	Medium publication slugs. Example: ["towardsdatascience", "betterhumans", "uxdesign"]
maxItemsPerFeed	integer	No (default: 50)	Max articles to fetch per RSS feed (1–1000)
maxItemsPerUserHistory	integer	No (default: 10000)	Max articles to fetch from each user's full timeline stream (1–100000). Set high (e.g. 10000) to pull essentially every post a user has published
fetchUserProfiles	boolean	No (default: true)	Fetch detailed profiles (bio, followers, post count, top tags) from the page's embedded JSON, with HTML fallback
fetchUserHistory	boolean	No (default: true)	Paginate each user's full article history (up to maxItemsPerUserHistory) via the timeline stream, instead of only the ~10 latest from RSS
fetchArticleDetails	boolean	No (default: false)	Fetch full article metadata (clap count, responses, reading time) by scraping the public article page
proxyConfiguration	object	No	Proxy settings (recommended for profile/article pages to avoid IP-based blocks). The history stream and embedded JSON fetch via direct connection to avoid Medium's datacenter-proxy blocking

At least one of usernames, tags, or publications is required.

Example Input

{
  "usernames": ["medium", "towardsdatascience"],
  "tags": ["technology", "artificial-intelligence", "programming"],
  "publications": ["betterhumans", "uxdesign"],
  "maxItemsPerFeed": 30,
  "fetchUserProfiles": true,
  "fetchUserHistory": true,
  "fetchArticleDetails": false,
  "maxRequestsPerCrawl": 300,
  "proxyConfiguration": {
    "useApifyProxy": true,
    "apifyProxyGroups": ["BUYPROXIES94952"]
  }
}

Output

Results are stored in the default dataset. Each item has a type field: profile, article, or tag.

User Profile (type: "profile")

{
  "type": "profile",
  "username": "towardsdatascience",
  "name": "Towards Data Science",
  "bio": "Your home for data science. A Medium publication sharing concepts, ideas, and codes.",
  "followerCount": 1250000,
  "followingCount": 42,
  "profileImageUrl": "https://cdn-images-1.medium.com/fit/c/200/200/1*abc123.png",
  "userId": "1234567890abcdef",
  "createdAt": "2016-03-15T10:30:00.000Z",
  "isVerified": true,
  "numberOfPostsPublished": 5230,
  "mediumMemberSince": "2016-03-15T10:30:00.000Z",
  "twitterScreenName": "TDataScience",
  "topTags": [{ "slug": "data-science", "name": "Data Science" }],
  "publications": [{ "slug": "towardsdatascience", "name": "Towards Data Science" }],
  "sourceFeed": "user:@towardsdatascience"
}

Article (type: "article")

{
  "type": "article",
  "title": "The Difference Between Python That Works and Python That Lasts",
  "description": "Python that works is easy to write but hard to maintain...",
  "url": "https://blog.stackademic.com/the-difference-between-python-that-works-and-python-that-lasts-e1796054bda6",
  "author": "Abdur Rahman",
  "authorUsername": "abdur-rahman",
  "authorProfileUrl": "https://medium.com/@abdur-rahman",
  "publication": "Stackademic",
  "publicationUrl": "https://blog.stackademic.com",
  "tags": ["technology", "python", "data-science", "artificial-intelligence", "programming"],
  "clapCount": 1240,
  "responseCount": 42,
  "readingTimeMinutes": 8,
  "publishedAt": "2026-09-01T12:36:26.000Z",
  "updatedAt": "2026-09-01T12:36:26.000Z",
  "postId": "e1796054bda6",
  "sourceFeed": "tag:technology",
  "sourceFeedType": "tag"
}

Tag (type: "tag")

{
  "type": "tag",
  "tagName": "Technology",
  "tagSlug": "technology",
  "tagPostCount": 2500000,
  "sourceFeed": "tag:technology"
}

Data table

Field	Profile	Article	Tag	Description
type	✓	✓	✓	Item type: profile, article, or tag
username	✓			Writer username
name	✓			Writer display name
bio	✓			Writer bio
followerCount	✓			Number of followers
followingCount	✓			Number of users followed
profileImageUrl	✓			Profile avatar URL
userId	✓			Medium internal user ID
createdAt	✓			Account creation date
isVerified	✓			Verified author badge
numberOfPostsPublished	✓			Total posts published
mediumMemberSince	✓			Membership start date
twitterScreenName	✓			Twitter/X handle
topTags	✓			Top author topic tags
publications	✓			Publications owned by the user
title		✓		Article title
description		✓		Article snippet/description
url		✓		Canonical article URL
author		✓		Author display name
authorUsername		✓		Author username
authorProfileUrl		✓		Link to author profile
publication		✓		Publication name
publicationUrl		✓		Publication homepage
tags		✓		Article tags/categories
clapCount		✓		Number of claps (requires fetchArticleDetails)
responseCount		✓		Number of responses (requires fetchArticleDetails)
readingTimeMinutes		✓		Estimated reading time (requires fetchArticleDetails)
publishedAt		✓		First publication date
updatedAt		✓		Last update date
postId		✓		Medium internal post ID
tagName			✓	Tag display name
tagSlug			✓	Tag URL slug
tagPostCount			✓	Total posts in tag
sourceFeed	✓	✓	✓	Origin feed identifier
sourceFeedType		✓		user, tag, or publication

Pricing / Cost estimation

Tier	Cost	Limits
Free (Apify Free plan)	$0	20 items total per run
Paid (Apify Starter+)	$49/mo+	Unlimited (compute units apply)

Compute unit estimates (approximate):

• RSS feed requests: ~0.01 CU per feed (very cheap)
• Profile page requests: ~0.05 CU per profile
• Article page requests: ~0.1 CU per article

A typical run scraping 10 users × 30 articles + 5 tags × 30 articles + 3 publications × 30 articles = ~540 articles + 10 profiles ≈ 0.5–1 CU without article details, ~5–10 CU with full article details.

Tip: Disable fetchArticleDetails for large-scale tag/publication scraping. Enable only for targeted deep-dives.

Tips & Advanced options

• Proxy recommended for article pages: fetchArticleDetails scrapes Medium's public article HTML, which can block datacenter IPs. Use Apify Proxy group (BUYPROXIES94952).
• Profiles are largely block-resistant: Profiles are read from the page's embedded JSON (?format=json); if that gets blocked the actor automatically retries the plain HTML page, and falls back gracefully instead of failing the run.
• RSS feeds are free & fast: Article listings from RSS work without proxy and cost almost nothing.
• Combine feeds: Mix usernames, tags, and publications in one run for comprehensive coverage.
• Deduplicate: Use postId field to deduplicate articles appearing in multiple feeds.
• Schedule: Set up daily/weekly runs on Apify to track trends over time.

FAQ, disclaimers & support

Is scraping Medium legal?

This Actor accesses publicly available RSS feeds and public HTML pages. Always review Medium's Terms of Service and Robots.txt. Use responsibly and respect rate limits.

Unknown yet

• clapCount, responseCount, readingTimeMinutes are populated from the timeline stream (with fetchUserHistory) or from article-page scraping (with fetchArticleDetails)
• followerCount, bio, profileImageUrl require fetchUserProfiles: true (profile embedded JSON)
• RSS feeds only provide basic article metadata

Known limitations

• Reading history (personal) requires authentication — not supported
• Private/unlisted articles not accessible
• Publication feeds may 404 if publication uses custom domain
• Public page structure (HTML/embedded JSON) may change without notice

Support

• Found a bug? Open an issue
• Need custom extraction? Contact Apify for enterprise solutions

---

Built with Crawlee + Apify SDK • Strategy: RSS feeds (Cheerio) → embedded page JSON + timeline stream pagination + HTML scraping (Cheerio) • No browser automation, no internal GraphQL API