# Google Shopping Scraper (`devcake/google-shopping`) Actor

Google Shopping and Google Lens scraper for price comparison, competitor research and product discovery across 49 countries.

- **URL**: https://apify.com/devcake/google-shopping.md
- **Developed by:** [devcake](https://apify.com/devcake) (community)
- **Categories:** E-commerce
- **Stats:** 2 total users, 1 monthly users, 79.3% runs succeeded, 1 bookmarks
- **User rating**: No ratings yet

## Pricing

from $3.00 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Google Shopping scraper

A small, read-only Google Shopping scraper that replays the public product-search
document with HTTPCloak and parses server-rendered product cards. It does not use
browser cookies, accounts, CAPTCHA solving, private RPCs, or mutation endpoints.

### Search by image

In addition to text `queries`, the Actor accepts product photos and runs one of
two modes (`imageSearchMode`):

| Mode | Behavior |
|------|----------|
| `identifyAndShop` (default) | Upload to Google Lens, derive a query from OCR, `fallbackText`, a descriptive filename, or Lens results, then scrape Google Shopping |
| `lensProducts` | Upload to Google Lens and return Products-tab cards using the Lens parser or the standard Shopping parser, depending on Google's markup |

Image inputs (same contract as the Google Lens Actor):

- `imageUrl` / `imageUrls` — public HTTP(S) image URLs
- `imagesBase64` — raw base64 or `data:image/...;base64,...` URIs
- `imageKvsRecords` — Apify key-value store binary records

Text queries and images can be combined in one run. Dataset rows from image
searches set `query` to the image source (`https://…`, `image:base64:1`, or
`image:kvs:…`), plus `searchMode` and (for identify) `identifiedQuery`.

Example Actor input:

```json
{
  "imageUrl": "https://example.com/eufy-solocam-s340.jpg",
  "imageSearchMode": "identifyAndShop",
  "country": "US",
  "language": "en",
  "maxResults": 40
}
```

When OCR finds no usable text, provide `fallbackText` or use a descriptive
filename such as `koss-headphones.jpg`. Generic names such as `IMG_1234.jpg`
cannot provide a safe query fallback.

Image upload uses Apify `RESIDENTIAL` proxy; Shopping / Lens result fetches use
`GOOGLE_SERP`.

### Install

```bash
python3 -m venv .venv
.venv/bin/pip install -e .
```

### Apify Actor

The repository is deployable as a Python Actor. Actor input accepts one or more
`queries`, stores product rows in the default dataset, and writes sanitized run
metrics to the `SUMMARY` key-value-store record.

The Actor creates proxy URLs through the Apify SDK at runtime. It uses the
`GOOGLE_SERP` group for the SERP route and `RESIDENTIAL` with the requested
country for the residential route. Proxy passwords must not be added to Actor
input or source files.

### Concurrency

The Actor processes independent searches concurrently. `maxConcurrency`
defaults to `8`, can be set from `1` to `16`, and is the global/text-search
limit. Google Lens work is separately capped at `3` because uploads and visual
result parsing are heavier. Text and image jobs are interleaved so a batch of
images cannot starve keyword searches.

Every concurrent job receives its own pool of three sticky residential session
IDs. Dataset writes are serialized after the scrape slot is released, so slow
storage writes neither corrupt a batch nor consume search concurrency. If
Google challenges rise under load, lower `maxConcurrency` without rebuilding
the Actor.

```bash
apify validate-schema
apify push
```

### Proxy configuration

Proxy URLs are loaded only from environment variables. Do not put credentials in
source code, command-line arguments, logs, or committed `.env` files.

```bash
export GOOGLE_SHOPPING_RESIDENTIAL_PROXY='http://user:password@host:port'
export GOOGLE_SHOPPING_SERP_PROXY='http://user:password@host:port'
```

When both variables are set, `--route auto` attempts the SERP route first. The
residential route is used after a connection failure, timeout,
proxy-authentication failure, transient `408`, `425`, `502`, `503`, or `504`
response. A challenge on the first SERP Shopping request opens one fresh SERP
session and retries once; if that retry is also challenged, auto mode tries the
residential route. On Apify, an initial challenge can rotate through up to three
isolated residential sessions; each has a unique sticky-session ID, and a
challenged identity is discarded rather than reused. Use `--route serp` or
`--route residential` in the standalone CLI to prevent cross-route fallback and
make the cost model explicit.

Residential and direct routes perform a same-session warm-up through the
selected route by loading `https://www.google.com/?hl=...`. The SERP route does
not discard a paid Shopping response as a warm-up: its first valid Shopping
response is parsed directly, so the normal SERP path costs exactly one proxy
request. The supplied SERP proxy rejects HTTPS `CONNECT` requests but supports
Google through a plain-HTTP target URL, so only that specialized route uses
`http://www.google.com/search`; residential and direct routes remain HTTPS. The
SERP route uses curl over HTTP/1.1 because this proxy fails before receiving an
HTTP status with HTTPCloak; the other routes use HTTPCloak over HTTP/2. Both
transports keep any same-route cookies in memory. Cookie values are never
printed, saved, or included in output. Use `--warmup-delay` to control the
bounded pause between the residential/direct warm-up and search requests.

A residential `HEAD` request was tested as a low-traffic bootstrap and returned
Google cookies without a response body. Those cookies were all marked `Secure`.
They are deliberately not forwarded to the SERP route because its required
plain-HTTP target cannot carry Secure cookies with browser-correct semantics.

Debug metrics separate `serp_request_count` from
`residential_request_count`. At a SERP price of `$0.003` per request, the normal
single-request path has a `$0.003` proxy cost per query before Actor/platform
fees. `response_bytes` is decoded HTML size and must not be treated as
residential billable wire traffic.

### Pagination

The scraper follows the browser-observed `More results` link returned in each
Shopping document. It does not guess offsets: the next Google `/search` URL,
including its `start` value and page-scoped query values, must be present in the
current response. Pagination is bounded by both `--max-pages` and
`--max-results`, deduplicates Google `data-pid` values across pages, and stops
on a missing/repeated next link, no new products, an access challenge, or schema
drift.

```bash
.venv/bin/google-shopping-scraper \
  "wireless headphones" \
  --route serp \
  --max-pages 2 \
  --max-results 100 \
  --debug
```

Every page in a pagination chain stays on the same transport and proxy session.
SERP page 1 therefore leads to SERP page 2; a residential chain must start and
continue residentially. TLS state cannot be moved between curl and HTTPCloak,
and the client deliberately does not copy cookies or page tokens across proxy
identities after an access challenge.

The bounded browser pilot for `wireless headphones` observed 53
parser-compatible products on page 1 and 40 on page 2, with three overlapping
Google product IDs: 90 unique products across two requests. The saved raw
one-request SERP document now yields 51 unique products with the container-aware
parser. These counts are observations, not guaranteed page sizes.

The client deliberately does **not** change identities after `401`, `403`,
`429`, redirects, unexpected content types, product schema drift, or a challenge
after the first successful page. Those outcomes stop the run. Before any
Shopping page succeeds, auto mode may retry one fresh SERP transport and then
the Actor's bounded pool of isolated residential sessions.

### Run

Write JSONL to stdout:

```bash
.venv/bin/google-shopping-scraper \
  "wireless headphones" \
  --route serp \
  --country US \
  --language en \
  --max-results 20
```

Write a file and print sanitized metrics:

```bash
.venv/bin/google-shopping-scraper \
  "wireless headphones" \
  --max-results 20 \
  --output results.jsonl \
  --debug
```

Each row contains the title, displayed price, seller, optional original price,
discount, delivery/returns text, rating, review count, image URL, and Google
card identifiers observed in the result.

### Tests

```bash
.venv/bin/python -m unittest discover -s tests -v
```

The tests cover browser-observed card parsing, stable-ID deduplication, result
and page limits, next-link pagination, the single-request SERP path,
residential warm-up/search behavior, SERP-to-residential transient fallback,
fresh residential-session rotation before the first successful page, and
fail-closed handling for `403`, post-page challenges, schema drift, global
concurrency, lane limits, stable result order, and non-blocking persistence
handoff.

### Investigation contract

- Goal: retrieve product-search records from public Google Shopping results.
- Surface: `GET https://www.google.com/search` with `q`, `hl`, `gl`, `udm=28`,
  and `shopmd=1`.
- Identity: anonymous public page; no account or session data.
- Methods: one product-search request per followed page on the SERP route;
  residential/direct use one read-only warm-up followed by bounded
  product-search pages in the same session; no detail-page fanout.
- Default bound: at most 20 records and one result page; one request on SERP or
  two on residential/direct at that default.
- Stop conditions: access boundary, challenge, persistent/transient route
  exhaustion, unexpected content type/status, or unrecognized product markup.

Browser observation on July 27, 2026 confirmed product cards exposed as
`role="button"` nodes whose accessible label contains `Current Price:` and
`Go to product viewer for this item.` Stable card IDs were available from the
nearest ancestor's `data-pid`; fields were mapped from descendants and are
covered by the fixture tests.

# Actor input Schema

## `queries` (type: `array`):

Enter one or more product searches. Each query is scraped separately. Optional when image inputs are provided.

## `country` (type: `string`):

Select the Google Shopping country (gl parameter).

## `language` (type: `string`):

Select the Google interface language (hl parameter).

## `maxResults` (type: `integer`):

Stop after this many unique products per query. Keep low for testing.

## `maxConcurrency` (type: `integer`):

Maximum simultaneous searches. Text searches may use the full limit; image searches are capped at 3. Lower this if Google challenges increase.

## `image` (type: `string`):

Upload one PNG, JPG, GIF, or WebP product image. A public URL is also accepted.

## `imageUrl` (type: `string`):

Single public product image URL for search-by-image.

## `imageUrls` (type: `array`):

Public image URLs to process.

## `imagesBase64` (type: `array`):

Raw base64 strings or data:image/...;base64,... URIs.

## `imageKvsRecords` (type: `array`):

KVS references. Omit key to process all binary records in the store.

## `imageSearchMode` (type: `string`):

identifyAndShop: recognize the product via Lens, then scrape Google Shopping. lensProducts: return Lens Products cards directly.

## `fallbackText` (type: `string`):

Optional descriptive query used if Lens visual-search context cannot be preserved.

## Actor input object example

```json
{
  "queries": [
    "wireless headphones"
  ],
  "country": "US",
  "language": "en",
  "maxResults": 40,
  "maxConcurrency": 8,
  "imageSearchMode": "identifyAndShop"
}
```

# Actor output Schema

## `products` (type: `string`):

No description

## `overview` (type: `string`):

No description

## `summary` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "queries": [
        "wireless headphones"
    ],
    "maxResults": 40,
    "maxConcurrency": 8
};

// Run the Actor and wait for it to finish
const run = await client.actor("devcake/google-shopping").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "queries": ["wireless headphones"],
    "maxResults": 40,
    "maxConcurrency": 8,
}

# Run the Actor and wait for it to finish
run = client.actor("devcake/google-shopping").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "queries": [
    "wireless headphones"
  ],
  "maxResults": 40,
  "maxConcurrency": 8
}' |
apify call devcake/google-shopping --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,devcake/google-shopping"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/08OSwkxadzhARSh0B/builds/IWSYrw3kFVtW6LRxr/openapi.json