# Hacker News Scraper (`devilscrapes/hacker-news-scraper`) Actor

Scrape Hacker News stories (top, new, best, ask, show, jobs) plus per-story metadata in one call — title, URL, score, author, comment count, posted-at — export to JSON or CSV. A Hacker News API wrapper that handles pagination, fan-out, retries, and rate-limit pacing.

- **URL**: https://apify.com/devilscrapes/hacker-news-scraper.md
- **Developed by:** [DevilScrapes](https://apify.com/devilscrapes) (community)
- **Categories:** News, Developer tools
- **Stats:** 1 total users, 0 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

Pay per event

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

<div align="center">
  <img src="https://apify.com/.actor/icon.svg" width="160" alt="Devil Scrapes mark" />

## Hacker News Scraper

▶️ **[Watch the 45-second demo on YouTube](https://youtube.com/shorts/i0Eiww6qpDs)**

**$2.20 / 1 000 results**  ·  pay only for results  ·  no credit card to try

*We do the dirty work so your dataset stays clean.* 😈

Extract Hacker News stories from top, new, best, ask, show, and jobs feeds with full per-story enrichment — score, author, comment count, ISO timestamps, feed rank, and HN permalink — delivered as typed dataset rows. We handle the fan-out, retries, proxy rotation, and rate-limit pacing so you get clean data on a schedule.

</div>

***

### 🎯 What this scrapes

This Actor fetches Hacker News story lists from any of the six available feeds — `top`, `new`, `best`, `ask`, `show`, or `jobs` — fans out to each individual story record, and writes one typed dataset row per story. The underlying Firebase API returns only item IDs at the feed level; we perform the full N+1 enrichment call per story and assemble the complete record before it hits your dataset.

You pick the feed and the result cap; we deliver clean, schema-validated rows on a schedule in JSON, CSV, or Excel — ready to pipe into Google Sheets, S3, a data warehouse, a webhook, or a RAG pipeline.

### 🔥 Features

- 🛡️ **Browser fingerprint rotation** — `curl-cffi` impersonates real Chrome, Firefox, and Safari TLS handshakes so we look like a browser, not a Python script.
- 🌐 **Residential proxy rotation** via Apify Proxy — fresh session and exit IP on every block or rate-limit response.
- 🔁 **Retries with exponential backoff** on `408 / 429 / 5xx` — up to 5 attempts per request, `Retry-After` header honoured.
- 🧱 **Rate-limit-aware pacing** — when the target pushes back we slow down and surface a clear status message instead of silently returning an empty dataset.
- 🧊 **Clean, typed dataset rows** — Pydantic-validated, ISO-8601 timestamps, stable IDs, feed rank included.
- 💰 **Pay-Per-Event pricing** — you pay only for results that land in your dataset. No data, no charge.

### 💡 Use cases

- **Trend monitoring** — diff top stories hourly to see which posts gain traction fastest.
- **Comment-volume alerts** — pipe rows into Slack when a story passes 100 comments.
- **Lead gen for dev tools** — surface Show HN launches that mention your stack and reach out early.
- **Newsletter curation** — feed the top 10 stories from the `best` feed into a weekly digest.
- **ML training data** — historical top-story metadata for score-prediction or topic-classification models.
- **Show HN tracker** — schedule a daily run against the `show` feed to watch new product launches.

### ⚙️ How to use it

1. Click **Try for free** at the top of the Store page.
2. Fill in the input form — most fields have sensible defaults (feed: `top`, max results: 100).
3. Click **Start**. Output streams into the run's dataset in real time.
4. Export from **Storage → Dataset** as JSON, CSV, or Excel — or pull via the Apify REST API.

### 📥 Input

| Field | Type | Required | Default | Notes |
|---|---|:--:|---|---|
| `feed` | `string` | no | `top` | Which story feed to pull: `top` (front page), `new` (most recent), `best` (time-decayed best), `ask`, `show`, or `jobs`. |
| `maxResults` | `integer` | no | `100` | Total dataset rows to produce. Each feed exposes up to 500 items; set to `0` for the full feed length. |
| `includeText` | `boolean` | no | `true` | Fetch the full self-post body for Ask HN and Show HN entries. Has no effect on regular link stories. |
| `concurrency` | `integer` | no | `8` | How many story records to fetch in parallel (1–32). |
| `proxyConfiguration` | `object` | no | `{"useApifyProxy": false}` | Apify Proxy configuration. Enable residential proxies if you need to route traffic through Apify for compliance or high-volume runs. |

#### Example input

```json
{
  "feed": "top",
  "maxResults": 3,
  "includeText": false,
  "concurrency": 4,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}
```

### 📤 Output

Every row is one dataset item.

| Field | Type | Notes |
|---|---|---|
| `id` | `integer` | Hacker News story ID (stable, monotonically increasing). |
| `type` | `string` | HN item type — `story`, `job`, `ask`, `show`, `comment`, `poll`. |
| `title` | `string` | Story headline. |
| `url` | `string \| null` | Outbound link (null for self-posts). |
| `permalink` | `string` | Hacker News permalink (`news.ycombinator.com/item?id=...`). |
| `by` | `string` | Author username on Hacker News. |
| `score` | `integer \| null` | Upvotes — null for jobs and dead items. |
| `descendants` | `integer \| null` | Total comment count, including replies. |
| `text` | `string \| null` | Self-post body (Ask HN / Show HN). HTML; only present when `includeText` is `true`. |
| `time` | `integer` | Unix epoch seconds — when the story was posted. |
| `posted_at` | `string` | ISO-8601 UTC timestamp derived from `time`. |
| `scraped_at` | `string` | ISO-8601 UTC timestamp of when this row was recorded. |
| `rank` | `integer` | Position of this story in the feed at scrape time (1-indexed). |

#### Example output

```json
{
  "id": 39000000,
  "type": "story",
  "title": "Show HN: Devil Scrapes — public-data Apify Actors with honest pricing",
  "url": "https://apify.com/DevilScrapes",
  "permalink": "https://news.ycombinator.com/item?id=39000000",
  "by": "devilscrapes",
  "score": 142,
  "descendants": 33,
  "text": null,
  "time": 1747353600,
  "posted_at": "2026-05-15T20:00:00+00:00",
  "scraped_at": "2026-05-15T20:05:00+00:00",
  "rank": 1
}
```

### 💰 Pricing

Pay-Per-Event — you pay only when these events fire:

| Event | USD | What it is |
|---|---:|---|
| `actor-start` | $0.20 | One-off warm-up charge per run |
| `result` | $0.002 | Per dataset item written |

Example: 1,000 results at the rates above ≈ **$2.00**. No subscription, no minimum, no card to start — Apify gives every new account $5 of free credit.

### 🚧 Limitations

Comment threads are not expanded — we return `descendants` (the count) but not the full tree. Dead and deleted stories are skipped automatically. The `text` field for self-posts is raw HTML, not Markdown — run it through your own sanitiser before display. Each feed is capped at 500 items by the upstream; we cannot exceed that without supplementing via search.

### ❓ FAQ

**Is scraping Hacker News legal?**

Yes — Y Combinator makes Hacker News data available through a documented, open API at `github.com/HackerNews/API`. We fetch only what that API surfaces, pace requests responsibly, and surface every call in the run log.

**Why use this instead of calling the API myself?**

The raw API returns an array of item IDs at the feed level — you need a separate round-trip per story to get title, score, and comment count. At 500 stories that is 501 HTTP calls to coordinate, de-duplicate, and fan out concurrently. We do that work, add ISO timestamps, attach the feed `rank` column (which the API does not expose), and deliver structured rows you can export or schedule without writing a line of code.

**What about the hacker news show HN tracker use case?**

Set `feed` to `show` and schedule your run on a cron. Each run captures the Show HN feed at that point in time with title, score, comment count, and author — ready for a Slack alert or spreadsheet diff without any glue code.

**Can I export Hacker News data to a spreadsheet?**

Yes — finish a run, open **Storage → Dataset**, and click **Export as CSV** or **Export as Excel**. Every field in the output table maps cleanly to a spreadsheet column. You can also connect the dataset URL directly to a Google Sheets `IMPORTDATA` formula.

**Can I scrape comments too?**

Not in this Actor — comment trees fan out 10-100x per story and would multiply cost significantly. A sibling `hacker-news-comments-scraper` will follow if there is enough demand.

**How fresh is the data?**

The upstream API reflects changes in near-real time. Your run captures whatever the feed contained the moment each story record was fetched.

### 💬 Your feedback

Spotted a bug, hit a weird edge case, or need a new field? Open an
issue on the Actor's **Issues** tab on Apify Console — we ship
fixes weekly and we read every report.

***

<div align="center">

Built by **[Devil Scrapes](https://apify.com/DevilScrapes)** 😈 — a small fleet of
opinionated public-data Actors. Honest pricing, real engineering, zero fine print.

</div>

# Actor input Schema

## `feed` (type: `string`):

Which Hacker News story feed to pull from. <code>top</code> mirrors the front page; <code>best</code> is the time-decayed best-of; <code>ask</code> / <code>show</code> / <code>jobs</code> are the dedicated sub-feeds.

## `maxResults` (type: `integer`):

Total dataset items to keep. The feeds expose up to 500 items each; pulling all of them costs ~500 results. Set to <code>0</code> for the feed's full length (up to 500).

## `includeText` (type: `boolean`):

Fetch the full self-post body for Ask HN / Show HN entries. Has no effect on regular link stories.

## `concurrency` (type: `integer`):

How many story records to fetch in parallel. The Firebase endpoint is generous; 8 is comfortable.

## `proxyConfiguration` (type: `object`):

Apify Proxy is optional here — the Firebase API is happy to be hit directly. Enable proxy only if you're routing all traffic through Apify for compliance reasons.

## Actor input object example

```json
{
  "feed": "top",
  "maxResults": 100,
  "includeText": true,
  "concurrency": 8,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}
```

# Actor output Schema

## `datasetItems` (type: `string`):

All dataset items as JSON.

## `datasetItemsCsv` (type: `string`):

Same data exported to CSV.

## `datasetView` (type: `string`):

Open the run dataset in the Console.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "proxyConfiguration": {
        "useApifyProxy": false
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("devilscrapes/hacker-news-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "proxyConfiguration": { "useApifyProxy": False } }

# Run the Actor and wait for it to finish
run = client.actor("devilscrapes/hacker-news-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}' |
apify call devilscrapes/hacker-news-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,devilscrapes/hacker-news-scraper"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/IhhkquOLSLgir6Mbq/builds/xaew2kFQNOQk4McJq/openapi.json