# Startpage News Scraper (`searchapi/startpage-news-scraper`) Actor

Scrapes Startpage News for any query. Extracts the full canonical news-vertical schema: title, source, snippet, dates, thumbnail, category, language, region, authors, anonymous-view URL, and more.

- **URL**: https://apify.com/searchapi/startpage-news-scraper.md
- **Developed by:** [Search API](https://apify.com/searchapi) (community)
- **Categories:** News, Developer tools, Automation
- **Stats:** 2 total users, 1 monthly users, 89.7% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.99 / 1,000 search results

This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Startpage News Scraper

Collect dedicated Startpage News results as normalized, API-ready records. The Actor accepts one search or a batch of searches, follows native news pagination, applies optional date/publisher/domain/image filters, and selects records fairly across batch queries.

The Actor extracts Startpage's structured news payload first. A DOM fallback is retained for compatible page changes. Ordinary web-search results, advertisements, error pages, and challenge pages are never stored as news records.

### Input

Provide either `query` or `queries`.

```json
{
  "queries": ["world news", "technology news"],
  "maxItems": 50,
  "maxPages": 5,
  "dateFilter": "week",
  "sortBy": "newest",
  "requireImage": false,
  "language": "english",
  "maxConcurrency": 3,
  "proxyConfiguration": {
    "useApifyProxy": true,
    "apifyProxyGroups": ["RESIDENTIAL"],
    "apifyProxyCountry": "US"
  }
}
```

Important options:

| Field | Purpose |
| --- | --- |
| `query` / `queries` | One search phrase or up to 20 phrases. |
| `maxItems` | Global output limit, 1–500. |
| `maxPages` | Native news pages per query, 1–10. |
| `dateFilter` | `anytime`, `day`, `week`, or `month`. |
| `publishedAfter` / `publishedBefore` | Optional client-side `YYYY-MM-DD` limits. |
| `sortBy` | `relevance`, `newest`, or `oldest`. |
| `includeDomains` / `excludeDomains` | Article-domain filters. |
| `includePublishers` / `excludePublishers` | Case-insensitive publisher filters. |
| `requireImage` | Keep only records with a source-provided image. |
| `language` | Startpage result-language preference. |
| `region` | Locale provenance attached to output. |
| `maxConcurrency` | Parallel query limit, 1–8 and memory-capped. |
| `proxyConfiguration` | No proxy, Apify Residential, or custom proxy URLs. |

`GOOGLE_SERP` is not supported. It is an HTTP proxy for supported Google Search/Shopping requests, while this Actor visits Startpage through an HTTPS browser session. Use Apify Residential or custom proxies when direct access is unreliable.

### Output

Records contain only non-empty values. Core fields include:

- stable `id`, global `position`, source `page`, and `sourcePosition`
- `title`, canonical article `url`, `domain`, and `description`
- `publisher`, `publisherDomain`, `publishedAt`, and `publishedAtRaw`
- source-provided `thumbnailUrl`, `imageUrl`, `sourceLogoUrl`, `authors`, `tags`, and `category` when available
- `query`, `queryPosition`, `matchedQueries`, and sanitized `searchUrl`
- extraction provenance, locale, date filter, and `scrapedAt`

Example:

```json
{
  "id": "news_...",
  "position": 1,
  "queryPosition": 1,
  "sourcePosition": 1,
  "page": 1,
  "title": "Example headline",
  "url": "https://publisher.example/article",
  "domain": "publisher.example",
  "description": "Example summary",
  "publisher": "Example Publisher",
  "publishedAt": "2026-08-30T00:00:00.000Z",
  "resultType": "news-search-result",
  "resultSource": "startpage-news-bing",
  "extractionMethod": "embedded-structured-data",
  "query": "world news",
  "matchedQueries": ["world news"],
  "language": "english",
  "country": "global",
  "sourceSearchDomain": "startpage.com",
  "scrapedAt": "2026-08-30T00:00:05.000Z"
}
```

Optional fields are omitted when the source does not provide them; the Actor does not emit `null`, empty strings, empty arrays, placeholders, raw responses, or signed proxy URLs.

### Runtime behavior

- Sessions, stable Crawlee browser fingerprints, bounded retries, and persistent cookies are enabled.
- Images, media, and fonts are blocked only after verifying structured news extraction remains complete.
- All records are normalized, filtered, deduplicated, fairly selected, and written atomically after crawling.
- A genuine no-result search completes successfully with an empty dataset.
- CAPTCHA or unusual-traffic pages fail the run and are never stored. The Actor does not solve or bypass CAPTCHAs.

### Local development

```bash
npm ci
npx playwright install firefox
npm test
apify run --purge --input-file qa-inputs/local-single.json
node validate-datasets.js
```

The `qa-inputs` directory also contains batch, pagination, filtered, empty-result, invalid-input, and cloud Residential-proxy cases.

# Actor input Schema

## `query` (type: `string`):

One phrase to enter in Startpage News.

## `queries` (type: `array`):

Up to 20 searches; output is selected fairly across queries.

## `maxItems` (type: `integer`):

Overall output limit across all searches.

## `maxPages` (type: `integer`):

Maximum native Startpage News pages inspected per search.

## `dateFilter` (type: `string`):

Age filter sent to the Startpage News form.

## `publishedAfter` (type: `string`):

Optional YYYY-MM-DD client-side filter.

## `publishedBefore` (type: `string`):

Optional YYYY-MM-DD client-side filter.

## `sortBy` (type: `string`):

Order each query bucket before fair selection.

## `includeDomains` (type: `array`):

Optional article-domain allowlist.

## `excludeDomains` (type: `array`):

Optional article-domain blocklist.

## `includePublishers` (type: `array`):

Keep publisher names containing one of these values.

## `excludePublishers` (type: `array`):

Remove publisher names containing one of these values.

## `requireImage` (type: `boolean`):

Keep only records with a source-provided image.

## `language` (type: `string`):

Startpage result-language preference.

## `region` (type: `string`):

Locale provenance such as us, gb, de, or in.

## `maxConcurrency` (type: `integer`):

Maximum searches processed in parallel, capped to run memory.

## `maxRequestRetries` (type: `integer`):

Bounded retries for temporary browser, proxy, or target failures.

## `navigationTimeoutSecs` (type: `integer`):

Maximum seconds per browser navigation.

## `requestHandlerTimeoutSecs` (type: `integer`):

Maximum seconds for one query handler.

## `proxyConfiguration` (type: `object`):

Optional Apify Residential or custom proxy. GOOGLE\_SERP is incompatible with Startpage HTTPS browser traffic.

## `debug` (type: `boolean`):

Log field names and structural counts only; never raw HTML or response bodies.

## Actor input object example

```json
{
  "query": "world news",
  "queries": [],
  "maxItems": 50,
  "maxPages": 5,
  "dateFilter": "anytime",
  "sortBy": "relevance",
  "includeDomains": [],
  "excludeDomains": [],
  "includePublishers": [],
  "excludePublishers": [],
  "requireImage": false,
  "language": "english",
  "region": "",
  "maxConcurrency": 3,
  "maxRequestRetries": 2,
  "navigationTimeoutSecs": 45,
  "requestHandlerTimeoutSecs": 180,
  "proxyConfiguration": {
    "useApifyProxy": false
  },
  "debug": false
}
```

# Actor output Schema

## `dataset` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "query": "world news",
    "proxyConfiguration": {
        "useApifyProxy": false
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("searchapi/startpage-news-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "query": "world news",
    "proxyConfiguration": { "useApifyProxy": False },
}

# Run the Actor and wait for it to finish
run = client.actor("searchapi/startpage-news-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "query": "world news",
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}' |
apify call searchapi/startpage-news-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,searchapi/startpage-news-scraper"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/dHgi5k2pqFYbNlnoV/builds/i0ReVcqEC5t7szPWH/openapi.json