# Hacker News Scraper - Stories, Comments, Polls & Users (`solidscrape/hacker-news-scraper`) Actor

Scrape Hacker News without an API key: full-text search, stories, comment trees, polls, and user profiles via the official Algolia HN Search and Firebase APIs. Export JSON/CSV/Excel.

- **URL**: https://apify.com/solidscrape/hacker-news-scraper.md
- **Developed by:** [Shahryar](https://apify.com/solidscrape) (community)
- **Categories:** News, Developer tools
- **Stats:** 2 total users, 2 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.00 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Hacker News Scraper – Stories, Comments, Polls & User Profiles (No API Key)

Scrape **Hacker News** at scale without an API key or login. This **Hacker News scraper** runs **full-text searches**, pulls **stories**, flattens entire **comment trees**, collects **polls**, and fetches **user profiles**, then lets you **export to JSON, CSV, or Excel**. Use it as a no-code **Hacker News API** alternative for data, research, and monitoring.

Built on Hacker News' two official, key-free JSON APIs (the **Algolia HN Search API** and the **Firebase HN API**). Made for **developers, researchers, data teams, and trend watchers** who want clean, structured HN data without parsing HTML or hitting rate limits.

> Pairs well with the **Reddit Scraper** and **Google News Scraper** for end-to-end tech and news monitoring across communities.

### What it does

- 🔎 **Full-text search** – query HN via Algolia by relevance or newest-first, filtered by tags (story, comment, poll, Ask HN, Show HN, front page) and author.
- 📰 **Built-in lists** – grab the official Top, New, Best, Ask HN, Show HN, or Jobs front-page lists.
- 🆔 **Direct items by ID** – fetch any story, poll, or comment by its Hacker News item ID.
- 💬 **Comment trees** – optionally pull and flatten the full nested comment tree for every story/poll, with per-comment `depth` and `parentId`.
- 👤 **User profiles** – karma, about text, account creation date, and submission count by username.
- 🎯 **Filtering** – minimum points, minimum comments, and start/end date windows.
- 🧹 **Clean text** – comment, story, and bio HTML is stripped and entity-decoded to plain UTF-8 text (no `'` mojibake).
- 📤 **Export anywhere** – download results as JSON, CSV, or Excel, or pull them from the Apify API.

### Why this scraper

- **No API key, no login.** Uses Hacker News' official, public Algolia Search and Firebase endpoints — no credentials, no account, no cookies.
- **Fast and reliable.** These are open JSON APIs, so there's no anti-bot or HTML markup to fight; runs are quick and stable.
- **One pass, fully populated.** Algolia search hits already carry the core story fields, so search results come back complete without extra per-item requests.
- **Goes past Algolia's 1000-result cap.** With `sortBy: "date"` and a `startDate`, the Actor automatically pages backward through creation-date windows to fetch far more than 1000 results per query.
- **Resilient HTTP.** Automatic retries with exponential backoff on `429`/`5xx`/network errors.

### Example input

```json
{
  "searchQueries": ["large language models"],
  "tags": "story",
  "sortBy": "date",
  "startDate": "2024-01-01",
  "endDate": "2024-12-31",
  "minPoints": 50,
  "minComments": 10,
  "includeComments": true,
  "maxCommentDepth": 3,
  "storyList": "top",
  "itemIds": ["8863"],
  "usernames": ["pg"],
  "maxItems": 200,
  "maxItemsPerQuery": 100,
  "proxyConfiguration": { "useApifyProxy": false }
}
```

Provide any combination of sources — search queries, item IDs, usernames, and/or a built-in list. Leave a field empty to skip it.

### Example output

Every item carries a `type` field (`story`, `comment`, `poll`, or `user`) so you can split the dataset by output kind. Every item also carries a `scrapedAt` ISO 8601 timestamp.

**Story** (`type: "story"`)

```json
{
  "type": "story",
  "id": 8863,
  "objectID": "8863",
  "title": "My YC app: Dropbox - Throw away your USB drive",
  "url": "http://www.getdropbox.com/u/2/screencast.html",
  "author": "dhouston",
  "points": 111,
  "numComments": 71,
  "createdAt": "2007-04-04T19:16:40.000Z",
  "createdAtTimestamp": 1175714200,
  "text": null,
  "tags": ["story", "author_dhouston", "story_8863"],
  "isAskHn": false,
  "isShowHn": false,
  "hnUrl": "https://news.ycombinator.com/item?id=8863",
  "scrapedAt": "2026-06-26T12:00:00.000Z"
}
```

**Comment** (`type: "comment"`)

```json
{
  "type": "comment",
  "id": 8952,
  "objectID": "8952",
  "storyId": 8863,
  "storyTitle": "My YC app: Dropbox - Throw away your USB drive",
  "storyUrl": "http://www.getdropbox.com/u/2/screencast.html",
  "parentId": 8863,
  "author": "BrandonM",
  "text": "I have a few qualms with this app...",
  "createdAt": "2007-04-04T20:12:00.000Z",
  "createdAtTimestamp": 1175717520,
  "depth": 1,
  "hnUrl": "https://news.ycombinator.com/item?id=8952",
  "scrapedAt": "2026-06-26T12:00:00.000Z"
}
```

**Poll** (`type: "poll"`)

```json
{
  "type": "poll",
  "id": 126809,
  "objectID": "126809",
  "title": "Poll: What's your favorite programming language?",
  "url": null,
  "author": "pg",
  "points": 99,
  "numComments": 116,
  "createdAt": "2008-04-29T05:54:00.000Z",
  "createdAtTimestamp": 1209448440,
  "text": null,
  "tags": ["poll", "author_pg", "story_126809"],
  "parts": [126810, 126811, 126812],
  "hnUrl": "https://news.ycombinator.com/item?id=126809",
  "scrapedAt": "2026-06-26T12:00:00.000Z"
}
```

**User** (`type: "user"`)

```json
{
  "type": "user",
  "id": "pg",
  "about": "Bug fixer.",
  "karma": 157000,
  "created": 1160418092,
  "createdAt": "2006-10-09T18:21:32.000Z",
  "submittedCount": 8950,
  "submitted": [1, 2, 3],
  "hnUrl": "https://news.ycombinator.com/user?id=pg",
  "scrapedAt": "2026-06-26T12:00:00.000Z"
}
```

### Output fields

#### Story (`type: "story"`)

| Field | Type | Description |
|---|---|---|
| `type` | string | Always `"story"`. |
| `id` | number | HN item ID. |
| `objectID` | string | Same ID as a string (Algolia's identifier). |
| `title` | string | Story title. |
| `url` | string / null | Linked URL (null for text / Ask HN posts). |
| `author` | string | Submitter username. |
| `points` | number / null | Score / upvotes. |
| `numComments` | number / null | Number of comments. |
| `createdAt` | string | ISO 8601 creation time. |
| `createdAtTimestamp` | number / null | Unix creation time (seconds). |
| `text` | string / null | Post body as decoded plain text (Ask HN / text posts). |
| `tags` | array / null | Algolia tags array; `null` when sourced from Firebase (item IDs / story lists). |
| `isAskHn` | boolean | Whether it's an Ask HN post. |
| `isShowHn` | boolean | Whether it's a Show HN post. |
| `hnUrl` | string | Link to the item on news.ycombinator.com. |
| `scrapedAt` | string | ISO 8601 timestamp of when the item was scraped. |

#### Comment (`type: "comment"`)

| Field | Type | Description |
|---|---|---|
| `type` | string | Always `"comment"`. |
| `id` | number | Comment ID. |
| `objectID` | string | Same ID as a string. |
| `storyId` | number / null | ID of the story/poll the comment belongs to (null for a directly-fetched comment ID). |
| `storyTitle` | string / null | Title of the parent story (when known). |
| `storyUrl` | string / null | URL of the parent story (when known). |
| `parentId` | number / null | ID of the direct parent (story or comment). |
| `author` | string / null | Commenter username. |
| `text` | string / null | Comment body as decoded plain text. |
| `createdAt` | string / null | ISO 8601 creation time. |
| `createdAtTimestamp` | number / null | Unix creation time (seconds). |
| `depth` | number / null | Nesting depth from comment-tree flattening (top-level = 1); `null` for a directly-fetched comment ID. |
| `hnUrl` | string | Link to the comment on news.ycombinator.com. |
| `scrapedAt` | string | ISO 8601 timestamp. |

#### Poll (`type: "poll"`)

Same fields as a story (`type`, `id`, `objectID`, `title`, `url`, `author`, `points`, `numComments`, `createdAt`, `createdAtTimestamp`, `text`, `tags`, `hnUrl`, `scrapedAt`), plus:

| Field | Type | Description |
|---|---|---|
| `parts` | array / null | Poll-option item IDs. |

Polls do **not** include `isAskHn` / `isShowHn`.

#### User (`type: "user"`)

| Field | Type | Description |
|---|---|---|
| `type` | string | Always `"user"`. |
| `id` | string | Username. |
| `about` | string / null | Profile bio as decoded plain text. |
| `karma` | number / null | Karma points. |
| `created` | number / null | Account creation time (Unix seconds). |
| `createdAt` | string / null | Account creation time (ISO 8601). |
| `submittedCount` | number | Number of items the user has submitted. |
| `submitted` | array | IDs of items the user has submitted. |
| `hnUrl` | string | Link to the user profile. |
| `scrapedAt` | string | ISO 8601 timestamp. |

### Input reference

| Field | Type | Default | Description |
|---|---|---|---|
| `searchQueries` | array | `[]` | Free-text queries for Algolia HN Search (e.g. `"large language models"`, `"rust"`). |
| `itemIds` | array | `[]` | Specific HN item IDs to fetch directly (story, poll, or comment). |
| `usernames` | array | `[]` | HN usernames to fetch as user profiles. |
| `storyList` | string | `""` | Built-in list: `top`, `new`, `best`, `ask`, `show`, or `job`. |
| `tags` | string | `story` | Search filter: `story`, `comment`, `poll`, `show_hn`, `ask_hn`, or `front_page`. Only applies to search queries. |
| `author` | string | `""` | Restrict search results to one author (Algolia `author_{name}` tag). |
| `sortBy` | string | `relevance` | `relevance` or `date` (newest first). |
| `startDate` | string | `""` | Only items created on/after this date (`YYYY-MM-DD` or any value `Date` can parse). |
| `endDate` | string | `""` | Only items created on/before this date. |
| `minPoints` | integer | `0` | Drop stories/polls below this score (client-side). |
| `minComments` | integer | `0` | Drop stories/polls below this comment count (client-side). |
| `includeComments` | boolean | `false` | Also fetch and flatten each story/poll's comment tree. |
| `maxCommentDepth` | integer | `0` | Limit comment recursion depth (`0` = unlimited). |
| `maxItems` | integer | `200` | Max items pushed in total (`0` = no limit). |
| `maxItemsPerQuery` | integer | `100` | Cap on stories/polls per search query / list (`0` = no limit). Comments do not count toward this cap. |
| `proxyConfiguration` | object | disabled | Optional Apify proxy; HN's open APIs rarely need it. |

### Common use cases

- **Tech trend & topic research** – search a keyword over a date window and export the matching stories to JSON/CSV/Excel for analysis.
- **Comment & sentiment analysis** – pull entire discussion threads as flat, depth-tagged comments for NLP or LLM pipelines.
- **Front-page monitoring** – snapshot the Top / Best / New / Ask HN / Show HN / Jobs lists on a schedule.
- **Product & launch tracking** – follow Show HN and Ask HN posts mentioning your product, competitor, or keyword.
- **Hiring & jobs intel** – scrape the Jobs list and "Who is hiring?" threads.
- **User research** – look up karma, account age, and submission counts in bulk.
- **Dataset building** – assemble a clean Hacker News dataset for machine learning, dashboards, or archival.

### FAQ

**Do I need a Hacker News or Algolia API key?**
No. This Hacker News scraper uses the public Algolia HN Search API and Firebase HN API — no API key, no login, no account required.

**How do I find an item ID?**
It's the number in the URL, e.g. `news.ycombinator.com/item?id=8863` → `8863`.

**Can I filter by points or comments?**
Yes, via `minPoints` / `minComments`. These are applied client-side — Algolia only supports server-side numeric filtering on creation date, so point/comment thresholds run after fetching.

**How many search results can I get per query?**
Algolia caps reachable results at 1000 per query. For larger crawls, set `sortBy: "date"` plus a `startDate`; the Actor automatically pages backward through creation-date windows to fetch well beyond 1000.

**Does it support pagination?**
Yes. Search results are paged automatically (100 hits per page), and the date-window strategy above transparently continues past the 1000-result cap.

**Which countries or regions does it cover?**
Hacker News is a single global site with no regional editions, so results are worldwide. The APIs are not geo-restricted, so no country selection is needed.

**Do I need a proxy?**
Usually not — HN's APIs are open and tolerant. Enable Apify Proxy only if you hit rate limits on very large runs.

**Why are some stories missing comments even with `includeComments` on?**
Deleted, dead, and empty comment nodes are skipped, though their replies are still traversed. Also, `maxCommentDepth` may be limiting how deep the tree is flattened.

**What export formats are supported?**
Results land in an Apify dataset you can download as JSON, CSV, or Excel, or fetch via the Apify API. Clean table views are provided for stories/polls, comments, and users.

**Can I run it on a schedule?**
Yes. Use Apify Schedules to run front-page or keyword monitoring automatically (hourly, daily, etc.).

# Actor input Schema

## `searchQueries` (type: `array`):

Free-text queries to run against Algolia HN Search, e.g. "large language models", "rust". Leave empty if you only want item IDs, usernames, or a built-in story list.

## `itemIds` (type: `array`):

Specific Hacker News item IDs to fetch directly (the number in news.ycombinator.com/item?id=8863). Works for stories, comments, and polls.

## `usernames` (type: `array`):

Hacker News usernames to fetch as user profiles (karma, about, account age, submission count).

## `storyList` (type: `string`):

Fetch one of Hacker News' official front-page lists. Each list returns up to ~500 item IDs which are then fetched individually.

## `tags` (type: `string`):

Restrict search results by Algolia tag. Only applies to search queries.

## `author` (type: `string`):

Restrict search results to a single author (Algolia tag author\_{name}). Leave empty for all authors.

## `sortBy` (type: `string`):

How to sort search results: by Algolia relevance, or strictly newest-first by date.

## `startDate` (type: `string`):

Only return items created on or after this date (YYYY-MM-DD or any value Date can parse). Applied server-side via created\_at\_i for search; client-side for other sources.

## `endDate` (type: `string`):

Only return items created on or before this date (YYYY-MM-DD or any value Date can parse).

## `minPoints` (type: `integer`):

Drop stories/polls with fewer than this many points. Applied client-side (Algolia no longer supports server-side points filtering).

## `minComments` (type: `integer`):

Drop stories/polls with fewer than this many comments. Applied client-side.

## `includeComments` (type: `boolean`):

For every story/poll collected, also fetch and flatten its full comment tree via Algolia /items/{id}. Each comment becomes a separate output item.

## `maxCommentDepth` (type: `integer`):

Only used when 'Include comments' is on. Limit how deep the comment tree is flattened (top-level comments are depth 1). Set 0 for unlimited depth.

## `maxItems` (type: `integer`):

Maximum number of items to push in total across all sources and types. Set 0 for no limit.

## `maxItemsPerQuery` (type: `integer`):

Cap on stories/polls collected per individual search query, per built-in list, etc. Comments fetched via 'Include comments' do not count toward this cap.

## `proxyConfiguration` (type: `object`):

The Hacker News Algolia and Firebase APIs are open and rarely block, so a proxy is optional. Leave disabled unless you hit rate limits.

## Actor input object example

```json
{
  "searchQueries": [
    "large language models",
    "rust"
  ],
  "itemIds": [
    "8863",
    "1"
  ],
  "usernames": [
    "pg",
    "dang"
  ],
  "storyList": "",
  "tags": "story",
  "author": "pg",
  "sortBy": "relevance",
  "startDate": "2024-01-01",
  "endDate": "2024-12-31",
  "minPoints": 50,
  "minComments": 10,
  "includeComments": false,
  "maxCommentDepth": 0,
  "maxItems": 200,
  "maxItemsPerQuery": 100,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}
```

# Actor output Schema

## `stories` (type: `string`):

No description

## `comments` (type: `string`):

No description

## `users` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "searchQueries": [
        "large language models"
    ],
    "maxItems": 200,
    "proxyConfiguration": {
        "useApifyProxy": false
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("solidscrape/hacker-news-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "searchQueries": ["large language models"],
    "maxItems": 200,
    "proxyConfiguration": { "useApifyProxy": False },
}

# Run the Actor and wait for it to finish
run = client.actor("solidscrape/hacker-news-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "searchQueries": [
    "large language models"
  ],
  "maxItems": 200,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}' |
apify call solidscrape/hacker-news-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,solidscrape/hacker-news-scraper"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/Oppp8G1h2ooDZvsFP/builds/WxvXkfhhh5Yry8tta/openapi.json