# Product Finder Plus: Crawler & Extractor (`datavault/product-finder-plus-crawler-extractor`) Actor

Product Finder Plus is a high-end e-commerce crawler built for websites where standard scraping tools fall short. It is designed to extract structured product data from complex, dynamic e-commerce stores and platforms.

- **URL**: https://apify.com/datavault/product-finder-plus-crawler-extractor.md
- **Developed by:** [Datavault](https://apify.com/datavault) (community)
- **Categories:** Automation, E-commerce
- **Stats:** 11 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.00 / 1,000 product details

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Product Finder Plus - Crawler & Extractor

> **Recommendation**: For simpler sites, we highly recommend trying the [Product Finder Crawler & Extractor](https://apify.com/datavault/product-finder-crawler-extractor) as a first step. It is generally faster and more cost-effective. This "Plus" version is designed for sites that require more complex solutions, specifically those with dynamic content or advanced anti-bot protections.

The Product Finder Crawler & Extractor Plus is an enhanced, high-performance implementation of our versatile e-commerce scraper. It is designed to extract product information from virtually any website, including modern Single Page Applications (SPAs) and PWA-based stores. It leverages multi-threaded concurrency and sophisticated parsing strategies (JSON-LD, Microdata, and JS-Global objects) to ensure maximum data yield with minimal overhead.

### Features

- **High-Performance Concurrency**: Uses a worker pool to crawl multiple pages in parallel, significantly reducing total execution time.
- **State Persistence & Resume**: Automatically saves crawl progress (visited URLs and queue) to the Apify Key-Value Store. If the run is interrupted, it resumes exactly where it left off.
- **Comprehensive Product Discovery**: Automatically identifies and extracts products using Schema.org (JSON-LD, Microdata), Meta Tags, and Next.js `__NEXT_DATA__`.
- **Dynamic JS-Object Extraction**: Specifically tuned for ScandiPWA and React stores by extracting data directly from `window.actionName` and other global JavaScript objects.
- **Multi-Country Proxy Support**: Fully integrated with Apify Proxy to bypass geo-blocks and analyze price differences across regions.
- **Pay-per-event (PPE) Integration**: Fully compatible with Apify's PPE model, charging only for successful page loads and products found.
- **Configurable Limits**: Control `maxPagesPerCrawl`, `maxConcurrency`, and `maxRetries` to manage depth and operational costs.

### Input Parameters

- `startUrls`: An array of URLs to start the crawl.
- `crawlSubpages`: If checked (default: true), the crawler will follow links found on the pages.
- `maxPagesPerCrawl`: The maximum number of pages to visit in a single run.
- `maxConcurrency`: How many pages to process in parallel (Default: 5).
- `maxRetries`: Number of times to retry a failed page fetch (Default: 3).
- `minRequestDelay`: Minimum time in milliseconds to wait between requests.
- `proxyConfiguration`: Apify Proxy configuration. Recommended for residential proxies on protected sites.

### Output

The scraper outputs a dataset where each item represents a found product. Fields include:

- `url`: The product page URL.
- `name`: Product name.
- `description`: Product description.
- `sku`: Stock Keeping Unit.
- `brand`: Brand name.
- `price`: Product price.
- `currency`: Currency code (e.g., USD, NOK).
- `image`: URL of the product image.
- `availability`: Availability status (e.g., InStock).
- `gtin`: Global Trade Item Number (GTIN) such as EAN, UPC, ISBN.
- `rawSchema`: The full extracted object for debugging or extra fields.

#### Sample Input

```json
{
    "startUrls": [
        { "url": "https://www.example-store.com/" }
    ],
    "crawlSubpages": true,
    "maxPagesPerCrawl": 200,
    "maxConcurrency": 5,
    "proxyConfiguration": {
        "useApifyProxy": true,
        "apifyProxyGroups": ["RESIDENTIAL"]
    }
}
```

### How it works

1. **Initialization**: The crawler loads any existing state and charges the `apify-actor-start` event.
2. **Concurrent Fetching**: Workers pick URLs from the queue and fetch them using a persistent HTTP client.
3. **Advanced Parsing**: It parses the page content using various strategies:
   - Schema.org (JSON-LD, Microdata)
   - Next.js and ScandiPWA data structures
   - Global JavaScript objects and Meta Tags
4. **Resilient Storage**: Products are pushed to the Apify Dataset, and the crawl state is periodically saved to the Key-Value Store.
5. **Smart Discovery**: New links are identified from both HTML anchors and dynamic JavaScript content to ensure deep coverage.

### Common issue when there is no result

- **Blocking**: Some sites might require Residential Proxies or specific User-Agent headers.
- **Non-Standard Structures**: If a site doesn't use standard markup or common HTML patterns, generic extraction might fail.

### Tip

Try setting just one URL of your site in the list of `startUrls` and set `crawlSubpages` to `false`. See if you get any result before scaling up the crawl.

***

**Feedback & Improvements**
If the results don't align with your goals, please reach out and leave us a message. We use your feedback to continuously update and refine our extraction engine, helping us make the Product Finder better for everyone.

# Actor input Schema

## `startUrls` (type: `array`):

List of URLs to start crawling from.

## `crawlSubpages` (type: `boolean`):

If checked, the crawler will follow links found on the pages. If unchecked, only the Start URLs will be scraped.

## `maxPagesPerCrawl` (type: `integer`):

Maximum number of pages that the crawler will open. The crawl will stop when this limit is reached.

## `maxConcurrency` (type: `integer`):

How many pages to process in parallel. Higher values are faster but risk blocking.

## `maxRetries` (type: `integer`):

Number of times to retry a failed page fetch.

## `minRequestDelay` (type: `integer`):

Wait at least this many milliseconds between requests.

## `enableSkipping` (type: `boolean`):

If checked, pages matching the 'Skip Patterns' will be ignored.

## `skipPatterns` (type: `array`):

List of strings or regex patterns to exclude from crawling (checks URL and Page Title).

## `roam` (type: `boolean`):

If checked, the crawler will follow links to other domains.

## `allowSubdomains` (type: `boolean`):

If checked, the crawler will follow links to subdomains of the start URLs.

## `proxyConfiguration` (type: `object`):

Use Apify Proxy (recommended for e-commerce sites).

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://www.greats.com/"
    }
  ],
  "crawlSubpages": true,
  "maxPagesPerCrawl": 20,
  "maxConcurrency": 5,
  "maxRetries": 3,
  "minRequestDelay": 500,
  "enableSkipping": true,
  "skipPatterns": [
    "contact",
    "about",
    "policy",
    "terms",
    "faq",
    "login",
    "signup",
    "cart",
    "checkout",
    "wishlist",
    "blog"
  ],
  "roam": false,
  "allowSubdomains": false
}
```

# Actor output Schema

## `results` (type: `string`):

The scraped products.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://www.greats.com/"
        }
    ],
    "skipPatterns": [
        "contact",
        "about",
        "policy",
        "terms",
        "faq",
        "login",
        "signup",
        "cart",
        "checkout",
        "wishlist",
        "blog"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("datavault/product-finder-plus-crawler-extractor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [{ "url": "https://www.greats.com/" }],
    "skipPatterns": [
        "contact",
        "about",
        "policy",
        "terms",
        "faq",
        "login",
        "signup",
        "cart",
        "checkout",
        "wishlist",
        "blog",
    ],
}

# Run the Actor and wait for it to finish
run = client.actor("datavault/product-finder-plus-crawler-extractor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://www.greats.com/"
    }
  ],
  "skipPatterns": [
    "contact",
    "about",
    "policy",
    "terms",
    "faq",
    "login",
    "signup",
    "cart",
    "checkout",
    "wishlist",
    "blog"
  ]
}' |
apify call datavault/product-finder-plus-crawler-extractor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,datavault/product-finder-plus-crawler-extractor"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/rCQx4afhiezfXX3e9/builds/fD83CzbsYCPeYgb40/openapi.json