# Hacker News Watchlist and Story Monitor (`skootle/hackernews-watchlist`) Actor

Monitor Hacker News stories and keywords with scores, comments, URLs, authors, and discussion metadata for startup research, tech trend monitoring, and AI agents.

- **URL**: https://apify.com/skootle/hackernews-watchlist.md
- **Developed by:** [Skootle](https://apify.com/skootle) (community)
- **Categories:** News, Developer tools, AI
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $2.00 / 1,000 hacker news records

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

### Fast answer: what this Actor is for

Track Hacker News stories and keywords for startup, AI, and developer trend signals with clean records and discussion links.

- Run it from the Apify UI for one-off exports.
- Schedule it or call it by API for recurring monitoring.
- Use the dataset output directly in spreadsheets, automations, and AI agents.

![Hacker News Watchlist hero](https://raw.githubusercontent.com/kesjam/skootle-actors-assets/main/heroes/hackernews.png)

### TL;DR

Monitor Hacker News stories and comments across top, new, best, ask, show, and jobs streams. Returns clean structured JSON with story-type enum, ISO timestamps, author karma + account age, and a 300-500 character markdown summary per story. Watchlist mode emits only NEW records since the previous run. Built on HN's official Firebase API. Zero authentication, zero anti-bot, no rate-limit issues in practice.

***

> Try it on a small dataset, then let us know what you think in a [review](https://apify.com/skootle/hackernews-watchlist/reviews).

***

### What does Hacker News Watchlist do?

Hacker News Watchlist extracts stories and comments from any Hacker News stream (`top`, `new`, `best`, `ask`, `show`, `jobs`). For each story you get: title, URL, external domain, score, comment count, author, author's karma + account age, rank in the stream, story-type enum (`story`, `ask_hn`, `show_hn`, `job`, `poll`), and ISO 8601 timestamps.

With `fetchComments: true`, the actor walks the comment tree per story (configurable cap, max 500 per story) and emits one record per comment with depth, parent ID, body, score, and author info.

Watchlist mode (`watchlistMode: true`) makes this scraper schedulable. State persists across runs in the actor's key-value store, so a daily cron only emits stories and comments NEW since the last run.

### Why scrape Hacker News?

HN moves fast, important threads age out in 12 hours. Watch top + new + best + Show HN + Ask HN + jobs across the day without 30 tabs open. Useful for founders watching for product mentions, VC scouts watching Show HN for early signal, journalists watching for breaking tech news, and recruiters scanning the monthly Who Is Hiring thread.

Daily AI-driven HN summary digests, brand-mention alerts on competitor domains, and labeled tech-discourse corpora for LLM training all run off one watchlist feed.

### Who needs this?

- **DevRel teams** monitoring HN for new posts about competitor or category technologies
- **Recruiters** scanning the monthly `Who Is Hiring?` thread plus the `jobs` stream
- **Brand and product teams** watching for unexpected HN posts about their tools
- **VC and tech-scouting analysts** filtering `Show HN` for new product launches
- **AI / LLM teams** building training corpora from high-quality tech discourse
- **AI agents** consuming a daily filtered HN digest as a topic-of-interest feed

### How to use Hacker News Watchlist

1. Open the **Input** tab on the actor page
2. Pick streams in the `streams` field (`top`, `new`, `best`, `ask`, `show`, `jobs`). One run handles many.
3. Set `storiesPerStream` (default 20)
4. Optionally enable `fetchAuthorProfile` (default `true`) for author karma + account age
5. Optionally enable `fetchComments` and set `commentsPerStory` (max 500 per story tree)
6. Optionally set `domainAllowlist` to filter to specific external domains
7. Optionally set `minScore` to ignore low-engagement posts
8. Optionally enable `watchlistMode` for daily diffs
9. Click **Start**

### How much will scraping Hacker News cost?

This actor is priced per event:

- **Actor Start**: $0.002 once per run
- **Hacker News record (story or comment)**: tiered, charged per record written

| Apify plan | $/1000 records |
|---|---|
| FREE | $2.00 |
| BRONZE | $2.50 |
| SILVER | $2.80 |
| GOLD | $3.00 |
| PLATINUM | $3.00 |
| DIAMOND | $3.00 |

A daily watchlist on `top` with `storiesPerStream: 30` and `fetchAuthorProfile: true` is charged for the records it writes. With `watchlistMode` enabled, repeat runs can emit fewer records because already-seen stories are not written again.

### Is it legal to scrape Hacker News?

Yes. Hacker News's Firebase API is explicitly published as a public read-only API for developers (`hacker-news.firebaseio.com/v0/`). HN encourages programmatic access. There is no authentication, no terms-of-service block on commercial use, and the data (titles, URLs, author handles, scores, public comments) is freely visible to anyone in a browser.

Use the data for research, AI training, brand monitoring, recruiting, internal analytics. Standard practice is to attribute HN as a source if you republish content, but the API itself is unrestricted.

### Maintenance status

Verified 2026-07-29: the published pricing explanation was reconciled to the current per-event price configuration, and the maintained source is checked with its declared TypeScript build and bounded Cloud canary before deployment. A canary is considered successful only when it returns one or more structured story records with no reported run errors.

### Examples

#### Example 1: Daily top-30 digest

```json
{
  "streams": ["top"],
  "storiesPerStream": 30,
  "fetchAuthorProfile": true,
  "watchlistMode": true,
  "maxItems": 30
}
```

#### Example 2: New posts above 100 score, last 24h

```json
{
  "streams": ["new"],
  "storiesPerStream": 100,
  "minScore": 100,
  "fetchAuthorProfile": true,
  "maxItems": 30
}
```

#### Example 3: Show HN product-launch tracker

```json
{
  "streams": ["show"],
  "storiesPerStream": 50,
  "watchlistMode": true,
  "fetchAuthorProfile": true,
  "maxItems": 50
}
```

#### Example 4: Ask HN community question feed

```json
{
  "streams": ["ask"],
  "storiesPerStream": 30,
  "fetchComments": true,
  "commentsPerStory": 30,
  "watchlistMode": true,
  "maxItems": 1000
}
```

#### Example 5: Job-listing watchlist

```json
{
  "streams": ["jobs"],
  "storiesPerStream": 100,
  "watchlistMode": true,
  "maxItems": 100
}
```

#### Example 6: Brand monitoring (filter by domain)

```json
{
  "streams": ["top", "new"],
  "storiesPerStream": 100,
  "domainAllowlist": ["yourcompany.com", "competitor1.com", "competitor2.com"],
  "watchlistMode": true,
  "maxItems": 50
}
```

#### Example 7: AI / LLM corpus build

```json
{
  "streams": ["top"],
  "storiesPerStream": 500,
  "fetchComments": true,
  "commentsPerStory": 100,
  "fetchAuthorProfile": true,
  "maxItems": 50000
}
```

Run weekly to accumulate a labeled tech-discourse dataset for fine-tuning.

#### Example 8: Author-trust filter

```json
{
  "streams": ["new"],
  "storiesPerStream": 200,
  "fetchAuthorProfile": true,
  "minScore": 5,
  "maxItems": 100
}
```

Filter the output downstream for `authorAccountAge != 'today'` and `authorKarma > 100` to skip brand-new spam accounts.

### Input parameters

| Field | Type | Default | Description |
|---|---|---|---|
| `streams` | enum\[] | `["top"]` | `top`, `new`, `best`, `ask`, `show`, `jobs`. One run handles many. |
| `storiesPerStream` | int | `20` | 1-500 |
| `fetchAuthorProfile` | bool | `true` | Adds author karma + account age. One extra API call per unique author, cached. |
| `fetchComments` | bool | `false` | Walks the comment tree per story |
| `commentsPerStory` | int | `0` | Max 500 |
| `domainAllowlist` | string\[] | `[]` | Only emit stories whose external URL matches |
| `minScore` | int | `0` | Score threshold |
| `watchlistMode` | bool | `false` | Idempotent diff against KV-stored seen IDs |
| `maxItems` | int | `50` | Hard cap on records (stories + comments) |

### Story-type enum

| Value | Meaning |
|---|---|
| `story` | Standard linked story |
| `ask_hn` | Ask HN: question to the community |
| `show_hn` | Show HN: project/product launch |
| `job` | YC company job posting |
| `poll` | HN poll |

### Hacker News output format

The dataset has two record types. Filter by `recordType`.

#### `hn_story`

| Field | Type | Description |
|---|---|---|
| `outputSchemaVersion`, `recordType`, `recordId` | string | Discriminated identity |
| `itemId`, `url`, `hnUrl` | int/string | HN ID + external URL + HN comment-page URL |
| `storyType` | enum | See enum table |
| `title`, `text`, `textPlain` | string | Title + body (HTML + stripped) |
| `externalUrl`, `domain` | string | External link + parsed domain |
| `author`, `authorKarma`, `authorAccountAge` | string/int/string | Author profile (when `fetchAuthorProfile: true`); accountAge as `'12y'`, `'5mo'`, etc. |
| `score`, `descendants`, `rank` | int | Score, comment count, position in stream |
| `stream` | enum | Source stream |
| `createdAt`, `scrapedAt` | ISO 8601 | |
| `fieldCompletenessScore`, `agentMarkdown` | int / string | Quality + LLM-ready summary |

#### `hn_comment`

| Field | Type | Description |
|---|---|---|
| `outputSchemaVersion`, `recordType`, `recordId` | string | Discriminated identity |
| `itemId`, `url` | int/string | Comment ID + URL |
| `storyId`, `parentId`, `depth` | int | Tree linkage |
| `text`, `textPlain` | string | Body (HTML + stripped) |
| `author`, `createdAt`, `scrapedAt` | string / ISO 8601 | |
| `fieldCompletenessScore`, `agentMarkdown` | int / string | Quality + LLM-ready summary |

#### Hacker News scraper output example (story)

```json
{
  "outputSchemaVersion": "2026-05-08",
  "recordType": "hn_story",
  "recordId": "hn:story:48067119",
  "itemId": 48067119,
  "stream": "top",
  "rank": 1,
  "storyType": "story",
  "title": "Google broke reCAPTCHA for de-googled Android users",
  "url": "https://reclaimthenet.org/google-broke-recaptcha-for-de-googled-android-users",
  "domain": "reclaimthenet.org",
  "score": 656,
  "descendants": 234,
  "author": "anonymousiam",
  "authorKarma": 5099,
  "authorAccountAge": "9y",
  "createdAt": "2026-05-08T14:22:53.000Z",
  "fieldCompletenessScore": 100,
  "agentMarkdown": "**📰 HN · Google broke reCAPTCHA for de-googled Android users**\n- ⬆ 656 · 💬 234 · #1\n- 👤 u/anonymousiam · 5099 karma · 9y\n- 🌐 reclaimthenet.org\n- 🔗 https://reclaimthenet.org/..."
}
```

### During the Actor run

The actor pulls stories, comments, and author profiles from HN's official Firebase API with respectful pacing. No authentication required, no rate-limit issues in practice; author lookups are cached per run so a busy thread doesn't multiply API calls. Alongside the dataset, three artifacts land in the actor's key-value store: **`OUTPUT`** (run summary), **`AGENT_BRIEFING`** (markdown digest with top stories by score), and **`WATCHLIST_STATE`** (seen story + comment IDs, when `watchlistMode: true`).

### FAQ

#### Is there a rate limit?

HN's Firebase API doesn't publish a hard rate limit and is generous in practice. The actor paces requests respectfully so a daily run never trips a soft cap.

#### Can I monitor for new stories only?

Yes. Set `watchlistMode: true`. The first run captures everything; subsequent runs only emit records new since the previous run.

#### Can I get author karma and account age?

Yes, that's the default (`fetchAuthorProfile: true`). Adds one HN API call per unique author, cached within the run.

#### Can I get comments along with stories?

Yes. Set `fetchComments: true` and `commentsPerStory` to your cap.

#### Can I filter by domain?

Yes. Set `domainAllowlist` to a list of allowed domains (e.g., `["yourcompany.com"]`). Only stories whose external URL matches will be emitted.

#### Can I filter by score?

Yes. Set `minScore` to your threshold. Stories below it are skipped.

#### Can I use this with the Apify API?

Yes. POST to `https://api.apify.com/v2/acts/skootle~hackernews-watchlist/runs`.

#### Can I integrate with Make / Zapier / n8n / Slack?

Yes. Click **Integrations** on the actor page.

#### Why use this when HN's API is free?

The free API requires per-item fetches and gives you no schema. This actor handles all the orchestration (stream → IDs → items → authors → comments → watchlist diff), normalizes into a versioned typed schema, joins author profile data per story, computes ranks, and ships agent-ready markdown summaries. If you're feeding this into a daily Slack digest or an AI agent, it pays back the per-record cost in saved engineering time.

#### Your feedback

Hit a bug or want a feature? Open an issue on the [Issues tab](https://apify.com/skootle/hackernews-watchlist/issues/open) rather than the reviews page, and we'll fix it fast (typically within 48 hours).

### Why choose Hacker News Watchlist

- **Monitor mode emits only what's new since last run**, so a daily Slack digest or AI agent feed never replays yesterday's stories
- **All 6 streams in one run**, `top`, `new`, `best`, `ask`, `show`, `jobs`, instead of refreshing HN tab by tab through the day
- **Author profile join**, `authorKarma` and `authorAccountAge` per story, so brand monitors and VC scouts can filter spam vs trusted contributors immediately
- **Story-type filter without grep**, typed enum (`story`, `ask_hn`, `show_hn`, `job`, `poll`) means Show HN trackers and job watchlists work in one downstream query
- **Sub-minute typical runtime**, built on HN's official public Firebase API, no anti-bot, no auth, no rate-limit issues in practice
- **Agent-ready markdown** per record drops straight into an LLM context window
- **Stories and comments in one dataset**, filterable by `recordType`
- **Re-runs are safe to dedupe by ID**, stable `hn:story:<id>` and `hn:comment:<id>` keys
- **Schema doesn't break your pipeline**, versioned and bumped on breaking change

### Other Skootle actors you might want to check

- **[Reddit Subreddit Scraper](https://apify.com/skootle/reddit-subreddit-monitor)**, same pattern for subreddit monitoring
- **[GitHub Trending Repos](https://apify.com/skootle/github-trending)**, daily trending dev repos with API enrichment
- **[SEC EDGAR Filings Monitor](https://apify.com/skootle/sec-edgar-filings)**, public-company filings stream
- **[Apple App Store Reviews Monitor](https://apify.com/skootle/app-store-reviews)**, App Store reviews + metadata
- **[Shopify App Store Scraper](https://apify.com/skootle/shopify-app-store-scraper)**, Shopify app listings + pricing tiers

### Support and contact

File issues on this actor's page, replies within 48 hours. Feature requests welcome, tag with `enhancement`.

# Actor input Schema

## `streams` (type: `array`):

Which HN streams to pull. One run handles many.

## `storiesPerStream` (type: `integer`):

How many stories to fetch from each stream (1 to 500).

## `fetchAuthorProfile` (type: `boolean`):

Adds one HN API call per unique author. Cached within a single run.

## `fetchComments` (type: `boolean`):

Walks the comment tree per story. Adds latency and records.

## `commentsPerStory` (type: `integer`):

Hard cap on comments per story (0 to 500).

## `domainAllowlist` (type: `array`):

Only emit stories whose external URL domain is in this list. Empty = all.

## `minScore` (type: `integer`):

Only emit stories with score >= N.

## `watchlistMode` (type: `boolean`):

When true, only emit records NEW since the previous run. State persists in the key-value store.

## `maxItems` (type: `integer`):

Hard cap on total records (stories + comments). Conservative default for the 5-minute auto-test.

## Actor input object example

```json
{
  "streams": [
    "top"
  ],
  "storiesPerStream": 20,
  "fetchAuthorProfile": true,
  "fetchComments": false,
  "commentsPerStory": 0,
  "domainAllowlist": [],
  "minScore": 0,
  "watchlistMode": false,
  "maxItems": 50
}
```

# Actor output Schema

## `datasetItems` (type: `string`):

Stories and comments. Discriminate by recordType.

## `agentBriefing` (type: `string`):

Markdown digest with top 10 stories by score.

## `runSummary` (type: `string`):

Compact OUTPUT object with row counts and per-stage error counts.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "streams": [
        "top"
    ],
    "domainAllowlist": []
};

// Run the Actor and wait for it to finish
const run = await client.actor("skootle/hackernews-watchlist").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "streams": ["top"],
    "domainAllowlist": [],
}

# Run the Actor and wait for it to finish
run = client.actor("skootle/hackernews-watchlist").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "streams": [
    "top"
  ],
  "domainAllowlist": []
}' |
apify call skootle/hackernews-watchlist --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,skootle/hackernews-watchlist"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/YsGLYKHJHj4mVM2lv/builds/SfVLEx04uOHcaJeiz/openapi.json