# Product Finder: Crawler & Extractor (`datavault/product-finder-crawler-extractor`) Actor

The Product Finder Crawler & Extractor is a versatile e-commerce scraper designed to extract product information from virtually any website but with a focus on e-commerce. Comprehensive Product Discovery, Up-to-Date Pricing, Multi-Country Price Comparison

- **URL**: https://apify.com/datavault/product-finder-crawler-extractor.md
- **Developed by:** [Datavault](https://apify.com/datavault) (community)
- **Categories:** E-commerce
- **Stats:** 25 total users, 0 monthly users, 100.0% runs succeeded, 1 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.50 / 1,000 product details

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Product Finder Crawler & Extractor

The Product Finder Crawler & Extractor is a versatile e-commerce scraper designed to extract product information from virtually any website with a focus on e-commerce. It leverages structured data formats such as Schema.org (JSON-LD, Microdata), and can work with some data embedded directly in HTML scripts. This crawler prioritizes speed and efficiency by not rendering full JavaScript, making it ideal for sites where product data is available in the initial HTML response.

### Features

- **Comprehensive Product Discovery**: Automatically identifies and extracts all products available on a target website.
- **Up-to-Date Pricing**: Tracks and retrieves the latest price updates for products.
- **Multi-Country Price Comparison**: Use proxy configuration to analyze product and price differences across countries.
- **Generic Extraction Engine**: Compatible with any site using standard structured data (JSON-LD, Microdata) or basic HTML patterns.
- **Deep Crawling Capabilities**: Optionally follows internal links to uncover additional products across the domain.
- **Configurable Crawl Limits**: Control the maximum number of pages to manage depth and operational costs.
- **Request Delay Management**: Adjust crawling speed to minimize server load and prevent throttling.

### Input Parameters

- `startUrls`: An array of URLs to start the crawl.
- `crawlSubpages`: If checked (default: true), the crawler will follow links found on the pages. If unchecked, only the Start URLs will be scraped.
- `maxPagesPerCrawl`: The maximum number of pages to visit in a single run. Default is 100.
- `minRequestDelay`: Minimum time in milliseconds to wait between requests (rate limiting). Default is 1000ms.
- `roam`: If checked (default: false), the crawler will follow links to other domains.
- `allowSubdomains`: If checked (default: false), the crawler will follow links to subdomains of the start URLs (e.g., blog.example.com).
- `proxyConfiguration`: Apify Proxy configuration. Recommended for most e-commerce sites and crucial for avoiding blocking on sites like Amazon.

### Output

The scraper outputs a dataset where each item represents a found product. Fields include:

- `url`: The product page URL.
- `name`: Product name.
- `description`: Product description.
- `sku`: Stock Keeping Unit.
- `brand`: Brand name.
- `price`: Product price.
- `currency`: Currency code (e.g., USD, NOK).
- `image`: URL of the product image.
- `availability`: Availability status (e.g., InStock).
- `gtin`: Global Trade Item Number (GTIN) such as EAN, UPC, ISBN.
- `rawSchema`: The full extracted JSON-LD object for debugging or extra fields.

#### Sample Input

```json
{
    "startUrls": [
        { "url": "https://www.example-store.com/" }
    ],
    "crawlSubpages": true,
    "maxPagesPerCrawl": 200,
    "minRequestDelay": 500,
    "proxyConfiguration": {
        "useApifyProxy": true
    }
}
```

### How it works

1. The crawler visits the `startUrls`.
2. It downloads the raw HTML content of the page.
3. It parses the page content using various strategies:
   - Schema.org (JSON-LD, Microdata)
   - Specific HTML selectors (e.g., for Amazon, Temu if data is in initial HTML)
   - Global JavaScript objects embedded directly in `<script>` tags
4. If a product is found, it extracts relevant fields and saves the item to the dataset.
5. Based on `crawlSubpages`, `roam`, and `allowSubdomains` settings, it finds and adds new links to the crawl queue.

### Common issue when there is no result

- Many websites require the use of a proxy; without it, requests may be redirected to other pages or blocked entirely.
- **JavaScript rendering**: This crawler does not execute client-side JavaScript. If product data is loaded dynamically after the initial page load (e.g., through AJAX calls or complex React/Vue applications that only send a skeleton HTML), this crawler might not find the data.
- Some sites have strict anti-scraping measures that prevent generic crawlers from working. In these cases, a custom scraper may be necessary. This tool is designed as a general-purpose scraper that works on a wide range of sites with products like ecommerce sites.

### Tip

Try setting just one url of your site that you want to scrape in the list of `startUrls` and set `crawlSubpages` to `false`. See if you get any result before going all in.

# Actor input Schema

## `startUrls` (type: `array`):

List of URLs to start crawling from.

## `crawlSubpages` (type: `boolean`):

If checked, the crawler will follow links found on the pages. If unchecked, only the Start URLs will be scraped.

## `maxPagesPerCrawl` (type: `integer`):

Maximum number of pages that the crawler will open. The crawl will stop when this limit is reached. It's good to set this limit to prevent infinite crawling of huge websites.

## `minRequestDelay` (type: `integer`):

Wait at least this many milliseconds between requests.

## `enableSkipping` (type: `boolean`):

If checked, pages matching the 'Skip Patterns' will be ignored.

## `skipPatterns` (type: `array`):

List of strings or regex patterns to exclude from crawling (checks URL and Page Title).

## `roam` (type: `boolean`):

If checked, the crawler will follow links to other domains.

## `allowSubdomains` (type: `boolean`):

If checked, the crawler will follow links to subdomains of the start URLs.

## `proxyConfiguration` (type: `object`):

Use Apify Proxy (recommended for e-commerce sites).

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://www.greats.com/"
    }
  ],
  "crawlSubpages": true,
  "maxPagesPerCrawl": 2,
  "minRequestDelay": 500,
  "enableSkipping": true,
  "skipPatterns": [
    "contact",
    "about",
    "policy",
    "terms",
    "faq",
    "login",
    "signup",
    "cart",
    "checkout",
    "wishlist",
    "blog"
  ],
  "roam": false,
  "allowSubdomains": false
}
```

# Actor output Schema

## `results` (type: `string`):

The scraped products.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://www.greats.com/"
        }
    ],
    "skipPatterns": [
        "contact",
        "about",
        "policy",
        "terms",
        "faq",
        "login",
        "signup",
        "cart",
        "checkout",
        "wishlist",
        "blog"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("datavault/product-finder-crawler-extractor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [{ "url": "https://www.greats.com/" }],
    "skipPatterns": [
        "contact",
        "about",
        "policy",
        "terms",
        "faq",
        "login",
        "signup",
        "cart",
        "checkout",
        "wishlist",
        "blog",
    ],
}

# Run the Actor and wait for it to finish
run = client.actor("datavault/product-finder-crawler-extractor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://www.greats.com/"
    }
  ],
  "skipPatterns": [
    "contact",
    "about",
    "policy",
    "terms",
    "faq",
    "login",
    "signup",
    "cart",
    "checkout",
    "wishlist",
    "blog"
  ]
}' |
apify call datavault/product-finder-crawler-extractor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,datavault/product-finder-crawler-extractor"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/2KNPUttM6ujB8U4XG/builds/R5nvyJ5c4bqoJWgf5/openapi.json