# Substack Publication Scraper (`parseforge/substack-publication-scraper`) Actor

Scrapes Substack publications for posts, authors, and engagement data. Returns each post as a flat row with title, date, likes, comments, and full text. No login required.

- **URL**: https://apify.com/parseforge/substack-publication-scraper.md
- **Developed by:** [ParseForge](https://apify.com/parseforge) (community)
- **Categories:** News, Marketing, Business
- **Stats:** 1 total users, 1 monthly users, 90.6% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $8.25 / 1,000 items

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

[![ParseForge](https://raw.githubusercontent.com/ParseForge/apify-assets/main/banner.jpg)](https://apify.com/parseforge?fpr=vmoqkp)

### Substack Publication Scraper

**Scrape Substack publications for posts, authors, and engagement data, up to a million per run.** Each post comes with its title, subtitle, publication date, likes, comments, and full text. No login or API key. Export to CSV, JSON, Excel, or XML.

Substack's official API requires an account and limits access to your own publication. This actor reads public publication pages directly, so you can collect posts from any Substack you specify, filtered by post type and sorted by recency, popularity, or community picks. It returns each post as a flat row with consistent fields.

| Who uses it | What they scrape Substack for |
|---|---|
| Content marketers | Monitor competitor newsletters for topic ideas and publishing frequency. |
| Journalists | Track what writers on a beat are publishing and how readers respond. |
| Researchers | Build a dataset of Substack posts for analysis of trends or discourse. |
| Newsletter operators | Benchmark their own engagement against similar publications. |

### What it does

This actor collects Substack posts from specified publications and returns each post as a flat row with metadata and content.

- 📰 **Publication targeting:** Provide one or more Substack subdomains or full URLs, like `lex` or `https://www.platformer.news`.
- 🔀 **Post type filter:** Choose `all`, `newsletter`, `podcast`, `thread`, or `discussion` to narrow results.
- 📊 **Sorting options:** Order posts by `new` (newest first), `top` (most liked), or `community` (community picks).
- 📦 **Bulk collection:** Set a maximum number of publications to scrape per run, up to 1,000,000.

Results export to CSV, JSON, Excel, or XML, or straight from the API.

### What you can do with Substack data

**📈 Track competitor newsletters.**

A content marketer enters a list of rival Substack subdomains, scrapes all newsletter posts sorted by new, and reviews titles and engagement to spot trending topics.

**📰 Monitor a beat.**

A journalist scrapes posts from several writers covering a niche, filters to newsletter type, and checks recent posts daily to stay current.

**🔍 Research public discourse.**

An academic researcher collects posts from a set of publications, exports the dataset, and analyzes text and engagement metrics for a study.

**📊 Benchmark engagement.**

A newsletter operator scrapes their own publication and a few peers, compares likes and comments per post, and adjusts content strategy.

### Why choose this scraper

|  | What you get |
|---|---|
| **No API key needed** | Scrape public Substack pages without registration or authentication. |
| **Flexible input** | Use subdomains or full URLs, and mix multiple publications in one run. |
| **Structured output** | Get consistent fields for every post, ready for analysis or export. |
| **Scalable** | Collect up to a million publications per run, limited only by your plan. |

### How it compares

No other Store actor targets Substack the same way, so the honest comparison is with the alternatives teams actually weigh.

| | Substack Publication Scraper | Build it in-house | By hand |
|---|---|---|---|
| Setup | Run it now, zero config | Days of engineering | None, but hours per pull |
| When Substack changes | Maintained for you | You fix it | You re-learn the page |
| Proxies, retries, anti-bot | Built in | Your problem | Browser only |
| Output | Fixed JSON schema, CSV/Excel export | Whatever you build | Copy-paste |
| Cost | Pay per result | Engineering time | Analyst hours |

### Configure the run

Drive the actor from a list of Substack publications, then filter by post type and sort order to control which posts are returned. The Input tab lists every parameter.

A first run with the defaults:

```json
{
  "maxItems": 10,
  "publications": [
    "lex",
    "stratechery"
  ],
  "sortBy": "new",
  "postType": "all"
}
```

A larger pull:

```json
{
  "maxItems": 200,
  "publications": [
    "lex",
    "stratechery"
  ],
  "sortBy": "new",
  "postType": "all"
}
```

### Pricing

Pay-per-result: **$0.011 per result** collected. You pay only for the results written to your dataset.

| Results collected | Approximate cost |
|---|---|
| 100 results | $1.10 |
| 1,000 results | $11.00 |
| 10,000 results | $110.00 |

New Apify accounts start with $5 in free credit.

### Free users

Free-plan runs return up to 10 results as a preview. [Upgrade your Apify plan](https://console.apify.com/sign-up?fpr=vmoqkp) to collect up to 1,000,000 results per run.

### Run it

1. [Create a free Apify account with $5 in credit](https://console.apify.com/sign-up?fpr=vmoqkp).
2. Open the [Substack Publication Scraper](https://apify.com/parseforge/substack-publication-scraper?fpr=vmoqkp).
3. Set your inputs and any filters, then click **Start**.
4. Export the results as CSV, Excel, JSON, or XML from the **Dataset** tab.

Run it programmatically through the [Apify API](https://docs.apify.com/api/v2) (`run-sync-get-dataset-items`) or the [ApifyClient](https://docs.apify.com/api/client/js) for JavaScript and Python.

### Use with AI agents (MCP)

Give an AI agent live access to Substack through the Model Context Protocol. Add the Actor to Claude, Cursor, or any MCP client:

```bash
claude mcp add --transport http apify "https://mcp.apify.com/?tools=parseforge/substack-publication-scraper"
```

Then prompt it in plain language to run the scraper and read back the results.

### Troubleshooting

**Why am I getting no results?**

Check that the publication subdomain or URL is correct and publicly accessible. Also ensure your postType filter is not too restrictive; try `all` first.

**The actor is slow or times out.**

Reduce maxItems or split large lists into multiple runs. Substack pages can be slow to load, so consider increasing the run timeout in Apify settings.

**Some posts are missing from the output.**

The actor only scrapes public posts. If a post is behind a paywall or deleted, it will not appear. Also verify the sort order; `new` may not include older posts if pagination is limited.

**I get an error about invalid input.**

Ensure publications are provided as subdomains (e.g., `lex`) or full URLs (e.g., `https://www.platformer.news`). Remove any trailing slashes or extra characters.

### FAQ

| Question | Answer |
|---|---|
| Do I need a Substack account or API key? | No. This actor scrapes public Substack pages directly, so no login or key is required. |
| Can I scrape multiple publications in one run? | Yes. Provide a list of subdomains or URLs in the input, and the actor will process each one. |
| What post types can I filter by? | You can choose `all`, `newsletter`, `podcast`, `thread`, or `discussion` using the postType input. |
| How do I sort the posts? | Use the sortBy input: `new` for newest first, `top` for most liked, or `community` for community picks. |
| What data does each post include? | Each post returns fields like title, subtitle, publication date, likes, comments, and full text. See the sample output for the complete list. |
| Can I export the results? | Yes. You can export to CSV, JSON, Excel, or XML from the Apify dataset. |
| Is there a limit on how many posts I can scrape? | You can set maxItems up to 1,000,000 publications per run, subject to your Apify plan limits. |
| Does this actor handle pagination? | Yes, it automatically paginates through the publication's post list to collect all available posts up to your maxItems. |
| Can I scrape a publication that requires a paid subscription? | Only public posts are accessible. Paid or subscriber-only content will not be returned. |
| What if a publication URL is invalid? | The actor will skip invalid entries and continue with the rest. Check the run log for details. |

### Related actors

Browse the full [ParseForge collection](https://apify.com/parseforge?fpr=vmoqkp) for more scrapers.

🆘 **Need help?** Email parseforge@protonmail.com with your run ID, your input, and what you expected.

⚠️ **Disclaimer.** This Actor is unofficial and is not affiliated with, endorsed by, or sponsored by Substack, Inc. It collects only publicly available data. You are responsible for using the collected data in compliance with the source's terms of service and applicable data-protection laws, including GDPR, CCPA, and PIPL. Do not use it to collect personal data unlawfully.

# Actor input Schema

## `maxItems` (type: `integer`):

How many publications to collect per run.

## `publications` (type: `array`):

Substack publication subdomains or full URLs. Examples: `lex`, `stratechery`, `https://www.platformer.news`.

## `sortBy` (type: `string`):

How to order posts: `new` (newest first), `top` (most liked), or `community` (community picks).

## `postType` (type: `string`):

Filter by post type: `all`, `newsletter`, `podcast`, `thread`, or `discussion`.

## Actor input object example

```json
{
  "maxItems": 10,
  "publications": [
    "lex",
    "stratechery"
  ],
  "sortBy": "new",
  "postType": "all"
}
```

# Actor output Schema

## `overview` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "maxItems": 10,
    "publications": [
        "lex",
        "stratechery"
    ],
    "sortBy": "new",
    "postType": "all"
};

// Run the Actor and wait for it to finish
const run = await client.actor("parseforge/substack-publication-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "maxItems": 10,
    "publications": [
        "lex",
        "stratechery",
    ],
    "sortBy": "new",
    "postType": "all",
}

# Run the Actor and wait for it to finish
run = client.actor("parseforge/substack-publication-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "maxItems": 10,
  "publications": [
    "lex",
    "stratechery"
  ],
  "sortBy": "new",
  "postType": "all"
}' |
apify call parseforge/substack-publication-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,parseforge/substack-publication-scraper"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/GA4uXzmiLCQFPJKOa/builds/Ihh1sxvp9WBV9vR1V/openapi.json