📰 Substack Newsletter & Post Scraper

Extract posts, author bios, likes, comments, and engagement metrics from any Substack publication — fast, via direct JSON API with zero headless browser overhead.

---

✨ Features

• ⚡ Direct API Access: Hits Substack's internal /api/v1/archive and /api/v1/publication endpoints directly.
• 🚀 Zero Browser Overhead: Pure async HTTP via httpx; runs efficiently in ~128 MB RAM.
• 🌐 Multi-Publication Batching: Scrape as many Substack slugs or URLs as needed in a single run.
• 📄 Automatic Pagination: Pages through full publication archives automatically.
• 📊 Standardized Pydantic Output: Clean, validated output dataset ready for downstream AI pipelines, analytics, or CSV/JSON exports.

---

📥 Input Parameters

The Actor accepts the following input options in JSON format:

Field	Type	Default	Description
subdomains	array	["lennysnewsletter", "thealgorithmicbridge"]	List of Substack subdomains/slugs or full URLs to scrape.
maxPostsPerSubdomain	integer	50	Maximum posts to scrape per publication (set 0 for unlimited).

Example Input JSON

{
  "subdomains": [
    "lennysnewsletter",
    "thealgorithmicbridge",
    "https://pragmaticengineer.substack.com"
  ],
  "maxPostsPerSubdomain": 100
}

---

📤 Output Format

Each dataset record represents an individual Substack post:

{
  "id": 123456789,
  "publication_name": "Lenny's Newsletter",
  "publication_subdomain": "lennysnewsletter",
  "publication_url": "https://lennysnewsletter.substack.com",
  "title": "How to build a product people love",
  "subtitle": "Lessons from 50 top product leaders",
  "slug": "how-to-build-a-product-people-love",
  "post_url": "https://lennysnewsletter.substack.com/p/how-to-build-a-product-people-love",
  "published_at": "2024-03-15T09:00:00.000Z",
  "canonical_url": "https://lennysnewsletter.substack.com/p/how-to-build-a-product-people-love",
  "audience": "everyone",
  "cover_image_url": "https://substackcdn.com/image/fetch/...",
  "likes_count": 1842,
  "comments_count": 94,
  "word_count": 2300,
  "reading_time_minutes": 10,
  "authors": ["Lenny Rachitsky"],
  "body_excerpt": "After interviewing 50 of the world's best product leaders..."
}

---

🚀 Running Locally

Step 1: Install Dependencies

$pip install -r requirements.txt

Step 2: Run the Actor

$python -m src.main

---

🐳 Docker Support

To build and run containerized:

docker build -t substack-scraper .
docker run -it substack-scraper