# 🔎 Yahoo Scraper (`scraper-engine/yahoo-scraper`) Actor

Yahoo Scraper: Extract Yahoo search results, news, finance data, images, and other public content with titles, URLs, descriptions, prices, timestamps, and metadata. Build structured datasets for research, market analysis, content discovery, monitoring, and data intelligence.

- **URL**: https://apify.com/scraper-engine/yahoo-scraper.md
- **Developed by:** [Scraper Engine](https://apify.com/scraper-engine) (community)
- **Categories:** Automation, Developer tools, SEO tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $4.99 / 1,000 results

This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## 🔎 Yahoo Search Scraper

Scrape **Yahoo Search** results at scale — titles, URLs, snippets, favicons, in-article sub-links, and a clean Markdown excerpt for every result. Bulk queries, time-window filtering and smart proxy auto-escalation (direct → datacenter → residential) keep your runs fast and unblocked.

***

### ⭐ Why Choose Us?

- **Bulk-first** — paste dozens of queries (or full Yahoo URLs) and walk every result page until the cap.
- **Smart proxy ladder** — starts direct, only escalates if Yahoo blocks. You don't pay for residential traffic you didn't need.
- **Rich back-fill** — when Yahoo's snippet is thin, the actor visits the result page and harvests in-article sub-links + a Markdown summary.
- **Live results** — rows stream to the dataset as they're scraped, so a mid-run interruption never loses your data.
- **Production-grade error handling** — 3-tier proxy retries, graceful PPE limit handling, exponential cool-downs.

***

### 🔑 Key Features

- 🌐 Bulk queries — plain keywords **or** full Yahoo SERP URLs, mixed freely.
- 📅 Time-window filter — Anytime / Past day / Past week / Past month.
- 🛡️ Auto-escalating proxy: **direct → Apify Datacenter → Apify Residential (3 retries)**, then sticky.
- 🧩 Optional second-pass back-fill of sub-links + Markdown excerpts.
- 📋 Per-section dataset views: **Overview**, **Snippet**, **Sub-links**.
- 🔄 Custom proxy URLs supported — they go first, then the smart ladder.

***

### 🧾 Input

```json
{
  "queries": [
    "java developer",
    "https://search.yahoo.com/search?p=python+jobs"
  ],
  "maxItems": 10,
  "timePeriod": "Anytime",
  "backfillEmptyResults": true,
  "backfillConcurrency": 8,
  "backfillMaxLinks": 10,
  "proxyConfiguration": { "useApifyProxy": false }
}
```

| Field | Type | Description |
|--|--|--|
| `queries` | `string[]` | One or more search terms **or** Yahoo SERP URLs. |
| `maxItems` | `integer` | Hard cap on unique results per query (1–500). |
| `timePeriod` | `string` | `Anytime` / `Past day` / `Past week` / `Past month`. |
| `backfillEmptyResults` | `boolean` | Visit each result page to harvest sub-links + Markdown excerpt. |
| `backfillConcurrency` | `integer` | Parallelism for back-fill (1–32). |
| `backfillMaxLinks` | `integer` | Max in-article sub-links per result page (1–50). |
| `proxyConfiguration` | `object` | Apify proxy config. Defaults to **direct** (no proxy). |

***

### 📤 Output

Each row matches the per-section views in the dataset.

```json
{
  "query": "java developer",
  "title": "How to become a Java Developer? - GeeksforGeeks",
  "url": "https://www.geeksforgeeks.org/gfg-academy/how-to-become-a-java-developer/",
  "description": "A Java developer is a software engineer who builds...",
  "text": "  * Core Java\n\nCore Fundamentals: Learn concepts and practice DSA...\n",
  "logo_url": "https://s.yimg.com/pv/.../32x32_7eae5aac8b7f7402.png",
  "links": [
    "https://www.geeksforgeeks.org/java/java",
    "https://www.geeksforgeeks.org/advance-java/spring"
  ],
  "domain": "www.geeksforgeeks.org"
}
```

| Field | Description |
|--|--|
| `query` | The query (or URL) the row was scraped under. |
| `title` | The result's headline. |
| `url` | The clean target URL (Yahoo's tracker is stripped). |
| `description` | Yahoo's SERP snippet, rendered as Markdown. |
| `text` | Markdown excerpt — either Yahoo's list block or, after back-fill, an in-article summary. |
| `logo_url` | The result's favicon. |
| `links` | Up to N harvested in-article sub-links (after back-fill). |
| `domain` | The host portion of `url`. |

***

### 🚀 How to Use (Apify Console)

1. Open **[Apify Console → Actors](https://console.apify.com/actors)**.
2. Find this actor and open it.
3. Paste your queries (one per line) into **🌐 Search Queries / URLs**.
4. Pick a **🎁 Maximum results** cap and a **📅 Time window**.
5. (Optional) Leave proxy on **direct** — the actor will auto-escalate only when needed.
6. Click **Start**.
7. Watch live logs — rows appear in the **Output** tab as they're scraped.
8. Export results as JSON / CSV / XLSX.

***

### 🤖 Use via API

```bash
curl -X POST "https://api.apify.com/v2/acts/%3CACTOR_ID%3E/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
     -H "Content-Type: application/json" \
     -d '{
       "queries": ["java developer"],
       "maxItems": 10,
       "timePeriod": "Anytime"
     }'
```

***

### 💼 Best Use Cases

- SEO & SERP monitoring on Yahoo.
- Competitive intelligence — track who appears for a query over time.
- Lead generation — feed result URLs into your own enrichment pipeline.
- Content discovery — harvest in-article sub-links for further crawling.

***

### 💳 Pricing

This actor uses Apify's **Pay-per-event** model. The primary event is **`result-item`** — one charge per result row pushed to the dataset. You pay only for the rows you actually receive; back-fill, retries and failed attempts are **not** billed.

You also pay the underlying Apify platform usage (compute units, proxy traffic when used). Direct (no-proxy) requests cost no proxy traffic at all — which is why the actor stays on **direct** until Yahoo forces it to escalate.

***

### ❓ Frequently Asked Questions

**Does it work when Yahoo blocks me?**
Yes. The default no-proxy run is the fastest, but the moment Yahoo returns a block (HTTP 429/503 or a captcha page), the actor auto-escalates to the Apify Datacenter pool, then to Residential with up to 3 retries. Once a tier works, it's locked in for the rest of the run.

**Can I bring my own proxies?**
Yes — paste them into the proxy field's **Custom proxy URLs**. Your URLs are tried first (3 retries), then the datacenter → residential fallback ladder kicks in.

**Does it follow pagination?**
Yes. Yahoo returns ~7 results per page; the actor walks pages until your `maxItems` cap is hit or 3 consecutive pages return nothing.

**What about non-Latin queries?**
Yahoo handles UTF-8 queries natively — paste them as-is.

**Why is my back-filled `text` empty for some rows?**
Some sites block all bots (or render with JS only). In that case the actor falls back to a minimal block built from Yahoo's own title + description so the field is never blank.

***

### 📨 Support & Feedback

- Issues / feature requests → please open a thread on the actor's detail page.
- Custom solutions → **dev.scraperengine@gmail.com**.

# Actor input Schema

## `queries` (type: `array`):

Paste one or more **search queries** (e.g. `java developer`) **OR** ready-made **Yahoo Search URLs** (e.g. `https://search.yahoo.com/search?p=java+developer`). You can mix both freely — the actor auto-detects each entry. One entry per line.

## `maxItems` (type: `integer`):

Hard cap on the number of unique Yahoo result rows pushed for each query. Yahoo paginates ~7 results per page, so e.g. `maxItems: 30` walks ~5 pages.

## `timePeriod` (type: `string`):

Restrict results to a specific freshness window. `Anytime` keeps Yahoo's default ranking.

## `proxyConfiguration` (type: `object`):

Smart auto-escalation is built in. The actor starts **direct (no proxy)**. If Yahoo blocks, it switches to **Apify Datacenter**, then to **Apify Residential** with up to 3 retries — and **sticks with the working tier** for the rest of the run. If you provide **custom proxy URLs** here, those are tried first (3 retries) before the datacenter/residential fallback ladder.

## Actor input object example

```json
{
  "queries": [
    "java developer"
  ],
  "maxItems": 10,
  "timePeriod": "Anytime",
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}
```

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "queries": [
        "java developer"
    ],
    "proxyConfiguration": {
        "useApifyProxy": false
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("scraper-engine/yahoo-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "queries": ["java developer"],
    "proxyConfiguration": { "useApifyProxy": False },
}

# Run the Actor and wait for it to finish
run = client.actor("scraper-engine/yahoo-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "queries": [
    "java developer"
  ],
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}' |
apify call scraper-engine/yahoo-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,scraper-engine/yahoo-scraper"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/EnZ65OIvZqoX6MiHJ/builds/AdMkFIpLT0A7DEEat/openapi.json