# Shopify Store Scraper: Collections, Pages & Blogs (`scrapier/shopify-store-scraper`) Actor

Shopify Store Scraper extracts collections, pages, blogs, titles, descriptions, URLs, publication dates, and other public store content. Ideal for ecommerce research, competitor analysis, content discovery, SEO audits, and Shopify store intelligence.

- **URL**: https://apify.com/scrapier/shopify-store-scraper.md
- **Developed by:** [Scrapier](https://apify.com/scrapier) (community)
- **Categories:** E-commerce, Lead generation, Automation
- **Stats:** 2 total users, 0 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $5.99 / 1,000 results

This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

### Shopify Store Scraper — Collections, Pages, Blogs & Verified Counts

Shopify Store Scraper: Collections, Pages & Blogs indexes a storefront's published structure — every collection with its live product count, every content page, every blog article, and every metaobject page — and returns it as one store summary row plus one labelled row per collection. Every response is structured JSON, ready to pass directly to an LLM, load into a spreadsheet, or feed a monitoring pipeline. Each count also states where it came from, so a truncated read is never mistaken for a complete one. Point it at any storefront and get back a structure you can actually audit.

***

### 🧭 What is Shopify Store Scraper: Collections, Pages & Blogs?

It queries one or more Shopify storefront URLs and returns the store's collection tree and content inventory — not the products inside each collection. Shopify's own `/collections.json` endpoint returns collections in batches of 250; a lot of "complete" collection counts out there are really just the first batch. This Actor pages through the whole list and then cross-checks the total it counted against two numbers the store publishes about itself (`publishedCollectionsCount` and `sitemapCollectionUrlCount`), so a wrong count doesn't pass silently. No Shopify account, API key, or store-owner access is required — it reads the same public endpoints and sitemap any visitor's browser can reach. A storefront that is closed to the public is detected and reported as `password_protected`; it is never bypassed.

- **Walks the collection list to exhaustion**, not just the first page, and reports how many batches it took.
- **Cross-checks the collection count three ways** — indexed, store-published, and sitemap-published — and reports all three instead of picking one.
- **Emits one labelled row per collection** with its real size, publish dates, image URL and description length.
- **Builds a content inventory from the store's own sitemap**: standalone pages, blog articles (with the blogs they belong to), and metaobject pages.
- **States the provenance of the product count** it used (`catalogueCountSource`) rather than presenting a single number as ground truth.
- **Runs on a list of storefronts in one job**, and one unreadable store never stops the rest of the run.

***

### 📦 What data can you get with Shopify Store Scraper: Collections, Pages & Blogs?

Each run returns a store profile row, a labelled row per collection, and a content-inventory summary built from the store's own sitemap.

| Result Type | Extracted Fields | Primary Use Case |
| --- | --- | --- |
| Store profile row | `storeDomain`, `storefrontStatus`, `shop_name`, `country`, `currency`, `locale`, `theme_name`, `theme_id`, `theme_version`, `themeSchemaName`, `apps_installed`, `collections_count`, `publishedCollectionsCount`, `sitemapCollectionUrlCount`, `total_products`, `price_min`/`price_avg`/`price_max` | Competitive snapshot of a store's size and setup |
| Collection rows | `title`, `handle`, `collectionUrl`, `productsCount`, `publishedAt`, `updatedAt`, `imageUrl`, `descriptionChars`, `collectionPage` | Collection-tree audit, empty/orphan collection detection |
| Content inventory | `pageCount`, `topLevelPages`, `blogArticleCount`, `blogHandles`, `metaobjectPageCount`, `contentFootprint` | SEO and content-footprint audit |
| Catalogue cross-check | `sitemapProductUrlCount`, `publishedProductsCount`, `catalogueCountSource`, `catalogueIndexState`, `collectionIndexState` | Verifying a count before you report it |

#### Provenance-tracked collection & content counts

Instead of returning one collection number and letting you trust it, the store row carries three independently-sourced counts side by side — `collections_count` (what the Actor actually indexed), `publishedCollectionsCount` (what the store's `meta.json` reports about itself), and `sitemapCollectionUrlCount` (what the store's own sitemap lists). `collectionIndexState` tells you whether the index run was `complete`, `capped` (hit `maxCollectionPages`), `partial`, or `unavailable` — never a silent truncation dressed up as a final number. The same pattern applies to the product catalogue: `catalogueCountSource` names exactly which source (`products_json`, `meta_json`, or `sitemap`) produced `publishedProductsCount` when the direct product listing is unavailable.

```json
{
  "collections_count": 1345,
  "publishedCollectionsCount": 1345,
  "sitemapCollectionUrlCount": 1345,
  "collectionIndexState": "complete",
  "collectionsPaged": 6,
  "catalogueCountSource": "products_json"
}
```

#### Content inventory: pages, blog articles & metaobject pages

Parsed from the storefront's own sitemap index rather than guessed, the content inventory counts standalone content pages, published blog articles (grouped by the blog they belong to), and metaobject pages, then sums them into `contentFootprint`. Locale-specific sitemap mirrors (e.g. a `/en/sitemap.xml` variant) are excluded from these counts so a multi-locale store doesn't get double-counted. Use it to compare how much editorial content sits behind a shop before you decide it's worth a competitive teardown.

***

### 🛠️ Why not build this yourself?

Shopify doesn't publish a single official API for indexing a storefront you don't own. The Storefront API returns rich structured data, but it requires an access token issued by the store itself — it's built for merchants building their own headless frontend, not for reading someone else's store from outside. What's left are the storefront's own public JSON endpoints and its sitemap, and both are harder to read reliably than they look:

- `/collections.json` pages in batches of 250 with no documented total-count field — you only know you're done when a page comes back short or empty.
- Sitemap structure varies by theme: some stores publish a flat sitemap with every URL, others publish a sitemap index with locale-specific child mirrors that need to be excluded from root counts, not summed into them.
- A store's own published counts don't always agree with each other — the self-reported total and the sitemap-listed total can differ by a handful of collections a merchant has marked as not-for-indexing.
- Some storefronts block direct requests outright, which means retry and proxy-tier escalation logic, not just a single fetch.
- Headless storefronts skip the JSON endpoints entirely, so the sitemap becomes the only usable source, and the Actor has to know to fall back to it.

Building and maintaining all of that in-house is a real, ongoing cost. If you need the numbers once and trust them, this Actor already handles the pagination, the fallback logic, and the cross-checks.

***

### 🆚 What's the difference between a Shopify product scraper and a store structure scraper?

A product scraper extracts individual product listings — titles, prices, variants, inventory — one row per SKU. A store structure scraper like this one extracts the shape of the store itself: how many collections it has, how big each one is, and how much standalone content (pages, blog articles) sits around the catalogue. The two answer different questions. A product scraper tells you what's for sale; a structure scraper tells you how the store is organized and how much content backs it. Shopify Store Scraper: Collections, Pages & Blogs returns the structure — collection rows (`type: "collection"`) and a content inventory on the store row — not per-product detail. It still reports catalogue-level totals (`total_products`, `price_min`/`price_max`, vendors, categories) for context, but the row-per-product breakdown a dedicated product scraper provides is out of scope here.

***

### 🚀 How to index a Shopify store with Shopify Store Scraper: Collections, Pages & Blogs

1. Open the Actor on its Apify Store listing and click **Try for free** (or **Run**, if you already have it saved).
2. Enter one or more storefront URLs or bare domains in `shopUrls` — for example `https://www.allbirds.com` or `allbirds.com`.
3. Set the real controls you need: `maxCollections` to cap how many collection rows come back, `collectionMinProducts` and `includeEmptyCollections` to filter by size, `readSitemap`/`countPages`/`countBlogArticles` to turn parts of the content inventory on or off.
4. Click **Start** and watch the run log — it reports each store's status and row counts as it finishes.
5. Download results from the **Dataset** tab as JSON, CSV, Excel, or another Apify-supported export format, or stream them out via the API.

#### How to index multiple stores in one job

Add every storefront URL to `shopUrls` — there is no separate "list" input, just more entries in the same array. Each store is processed independently: `ladder.reset_for_store()` resets connection state per store, so one blocked or unreachable storefront is reported with `storefrontStatus: "unreachable"` and the run continues to the next URL rather than stopping the job.

***

### ⬇️ Input

All fields are optional — leave `shopUrls` empty only if you're relying on the legacy `startUrls` field instead.

| Parameter | Required | Type | Default | Description |
| --- | --- | --- | --- | --- |
| `shopUrls` | No | array (string list) | `["https://www.allbirds.com/"]` (prefill) | Storefront URLs or bare domains to index. Example: `https://www.allbirds.com` or `allbirds.com`. Each entry produces one store row. |
| `startUrls` | No | array (string list) | — | Kept so input files written for the original actor keep working. Entries here are added to the list above and duplicates are removed. Leave empty if you filled in the field above. |
| `indexAllCollections` | No | boolean | `true` | Walk the store's full collection list instead of reading only the first batch of 250. Turn this off to skip collection indexing entirely. |
| `emitCollectionRows` | No | boolean | `true` | Emit a separate labelled row for every collection (title, handle, URL, product count, publish dates, image URL). Turn off to keep only the store-level summary. |
| `includeEmptyCollections` | No | boolean | `true` | Keep collections that currently hold 0 products. They are real published collections, so they are kept by default. Turn off to drop them from the collection rows. The store-level total always counts them. |
| `collectionMinProducts` | No | integer | `0` | Only emit collection rows holding at least this many products. Example: 50 keeps only the store's substantial collections. 0 = no size floor. Minimum: `0`. |
| `maxCollections` | No | integer | `0` | How many collection rows to return for each store address. Enter 25 and you get 25 collection rows per store; enter 500 and you get 500 (or all of them, if the store publishes fewer). Set 0 to return every collection. The store-level totals stay complete whatever you set here. Minimum: `0`. |
| `maxCollectionPages` | No | integer | `20` | Safety cap on how many batches of 250 collections to walk. Example: 20 covers up to 5,000 collections. When the cap is reached the store row says so instead of pretending the list ended. Minimum: `1`. |
| `readSitemap` | No | boolean | `true` | Count the store's published pages, blog articles, metaobject pages and product URLs. This is also the only source that still works on storefronts that do not publish structured store data. |
| `countPages` | No | boolean | `true` | Count the store's standalone content pages and list their handles. Turn off to skip that part of the index. |
| `countBlogArticles` | No | boolean | `true` | Count published blog articles and list the blogs they belong to. Turn off to skip that part of the index. |
| `sitemapChildLimit` | No | integer | `30` | How many sections of the store's published index to read per store. Large stores split their index into many sections. Example: 30 is enough for a store with roughly 150,000 URLs. Minimum: `1`. |
| `proxyConfiguration` | No | object | `{"useApifyProxy": false}` (prefill) | Optional. Leave as-is to run directly. |

#### Example JSON input

```json
{
  "shopUrls": ["https://www.allbirds.com/", "https://blendjet.com/"],
  "indexAllCollections": true,
  "emitCollectionRows": true,
  "includeEmptyCollections": true,
  "collectionMinProducts": 0,
  "maxCollections": 0,
  "maxCollectionPages": 20,
  "readSitemap": true,
  "countPages": true,
  "countBlogArticles": true,
  "sitemapChildLimit": 30,
  "proxyConfiguration": { "useApifyProxy": false }
}
```

**Common pitfall:** `maxCollections` and `maxCollectionPages` control two different things. `maxCollectionPages` caps how far the Actor walks the store's real collection list (the index), while `maxCollections` only trims how many of the already-indexed collections are returned as rows. Lowering `maxCollections` never makes the run faster or the cross-checked totals less accurate — it only shortens the output. If you actually want a faster, capped index, lower `maxCollectionPages` instead, and check `collectionIndexState` for `"capped"` in the result.

***

### ⬆️ Output

Results are pushed to the Actor's default dataset as typed, normalized JSON with a consistent row shape across runs. Export it from the Apify Console as JSON, CSV, Excel, HTML table, or XML, or pull it with the API/`apify_client` in JSON format. Collection rows are also mirrored to a second, per-run dataset named `<runId>-collections`, if you want them isolated from the store rows.

Only the store summary row is a billed event (`row_result`) — one charge per storefront that was actually profiled. Every collection row it emits, and any status row reporting that a store couldn't be reached or that no input was supplied, is pushed for free. To pull only the billed rows from the dataset, filter on `item.type === "store"`; everything else (`type === "collection"` or `type === "status"`) was pushed uncharged.

#### Scraped results

```json
[
  {
    "type": "store",
    "isChild": false,
    "storeDomain": "www.allbirds.com",
    "storefrontStatus": "open",
    "store_domain": "www.allbirds.com",
    "store_url": "https://www.allbirds.com/",
    "shop_name": "weareallbirds.myshopify.com",
    "country": "US",
    "currency": "USD",
    "locale": "en",
    "theme_name": "Allbirds Theme",
    "theme_id": 123456789,
    "theme_version": "1.4",
    "themeSchemaName": "allbirds-theme",
    "apps_installed": ["Elevar Conversion Tracking"],
    "apps_by_category": { "Analytics": ["Elevar Conversion Tracking"], "Other": [] },
    "apps_count": 1,
    "total_products": 291,
    "total_variants": 940,
    "collections": ["Men", "Women", "T-Mobile Tuesdays", "..."],
    "collections_count": 1345,
    "categories": ["Shoes", "Apparel"],
    "vendors": ["Allbirds"],
    "price_min": 20.0,
    "price_avg": 78.45,
    "price_max": 198.0,
    "products_on_sale": 34,
    "discount_pct_avg": 22.5,
    "total_variants_available": 812,
    "inventory_availability_pct": 86.4,
    "estimated_launch_date": "2016-03-01",
    "oldest_product_date": "2016-03-01",
    "newest_product_date": "2026-07-30",
    "products_with_images": 291,
    "avg_images_per_product": 4.2,
    "products_with_description": 291,
    "top_products": [
      { "title": "Tree Runner", "url": "https://www.allbirds.com/products/tree-runner", "price": 98.0 }
    ],
    "api_blocked": false,
    "publishedProductsCount": 291,
    "publishedCollectionsCount": 1345,
    "collectionsPaged": 6,
    "collectionIndexState": "complete",
    "collectionRowsEmitted": 1345,
    "collectionRowsState": "all",
    "catalogueIndexState": "complete",
    "emptyCollectionCount": 423,
    "avgProductsPerCollection": 58.79,
    "largestCollection": { "handle": "t-mobile-tuesdays-discount", "productsCount": 4824 },
    "collectionTop10ByProducts": [
      { "handle": "t-mobile-tuesdays-discount", "title": "T-Mobile Tuesdays", "productsCount": 4824 }
    ],
    "sitemapProductUrlCount": 292,
    "sitemapCollectionUrlCount": 1345,
    "pageCount": 175,
    "blogArticleCount": 80,
    "metaobjectPageCount": 241,
    "sitemapChildCount": 6,
    "hasAgenticDiscoverySitemap": true,
    "contentFootprint": 496,
    "catalogueCountSource": "products_json",
    "blogHandles": ["the-perch", "news"],
    "topLevelPages": ["careers", "fit-guide", "our-commitment"],
    "scraped_at": "2026-08-08T10:03:11+00:00",
    "scrapedAt": "2026-08-08T10:03:11+00:00",
    "scrape_duration_sec": 14.32
  },
  {
    "type": "collection",
    "isChild": true,
    "storeDomain": "www.allbirds.com",
    "collectionId": 166330728514,
    "title": "Men's Shoes",
    "handle": "mens-shoes",
    "collectionUrl": "https://www.allbirds.com/collections/mens-shoes",
    "productsCount": 42,
    "publishedAt": "2020-11-27T17:20:11-08:00",
    "updatedAt": "2026-08-07T04:16:23-07:00",
    "imageUrl": "https://cdn.shopify.com/s/files/1/mens-shoes.jpg",
    "descriptionChars": 210,
    "collectionPage": 1,
    "scrapedAt": "2026-08-08T10:03:11+00:00"
  },
  {
    "type": "collection",
    "isChild": true,
    "storeDomain": "www.allbirds.com",
    "collectionId": 166330728515,
    "title": "Women's Shoes",
    "handle": "womens-shoes",
    "collectionUrl": "https://www.allbirds.com/collections/womens-shoes",
    "productsCount": 51,
    "publishedAt": "2020-11-27T17:20:11-08:00",
    "updatedAt": "2026-08-05T09:02:00-07:00",
    "imageUrl": "https://cdn.shopify.com/s/files/1/womens-shoes.jpg",
    "descriptionChars": 198,
    "collectionPage": 1,
    "scrapedAt": "2026-08-08T10:03:11+00:00"
  },
  {
    "type": "collection",
    "isChild": true,
    "storeDomain": "www.allbirds.com",
    "collectionId": 166330728516,
    "title": "T-Mobile Tuesdays",
    "handle": "t-mobile-tuesdays-discount",
    "collectionUrl": "https://www.allbirds.com/collections/t-mobile-tuesdays-discount",
    "productsCount": 4824,
    "publishedAt": "2019-06-12T11:00:00-07:00",
    "updatedAt": "2026-08-08T02:15:00-07:00",
    "imageUrl": null,
    "descriptionChars": 0,
    "collectionPage": 3,
    "scrapedAt": "2026-08-08T10:03:11+00:00"
  }
]
```

If a store cannot be read at all, a `type: "status"` row is pushed instead, uncharged: `{"type": "status", "isChild": false, "storeDomain": "example.myshopify.com", "store_url": "https://example.com/", "storefrontStatus": "unreachable", "scrapedAt": "..."}`. If no store URL was supplied at all, a single `{"type": "status", "storefrontStatus": "no_input", ...}` row is pushed and the run ends.

***

### 📊 How can I use the data extracted with Shopify Store Scraper: Collections, Pages & Blogs?

- **Ecommerce and SEO analysts:** compare `collections_count`, `emptyCollectionCount`, `pageCount` and `blogArticleCount` across a list of competitors to size up merchandising depth and content investment before a pitch or audit.
- **AI engineers and LLM developers:** a run's JSON rows can be passed directly as grounding context to an LLM — "summarize this store's collection and content structure" — without any parsing step, since the schema is already typed and consistent.
- **Market researchers:** collection titles, handles, and counts across a batch of `shopUrls` build a category-coverage map of how a set of stores organizes the same product space.
- **Agencies and store migration teams:** `collectionTop10ByProducts`, `largestCollection`, `pageCount`, and `blogHandles` give a sized inventory of what has to move before a re-platform, without opening the admin of a store you don't manage.

***

### 📈 How do you monitor a store's structure over time?

Structure monitoring means re-running the same `shopUrls` on a schedule and diffing the store row's counts between runs, rather than trusting a single snapshot. Because `collections_count`, `publishedCollectionsCount`, and `sitemapCollectionUrlCount` are cross-checked every run, a real structural change (a merchant adding or archiving collections) shows up as a moved number on all three, while a scraping glitch shows up as disagreement between them — that distinction is the point of tracking all three, not just one.

The fields worth diffing run-to-run are `collections_count`, `emptyCollectionCount`, `avgProductsPerCollection`, `pageCount`, `blogArticleCount`, `metaobjectPageCount`, and `contentFootprint`. A jump in `blogArticleCount` between two runs is a content-publishing signal; a drop in `collections_count` alongside a stable `publishedCollectionsCount` is worth a second look at `collectionIndexState` before you trust it.

A practical loop: schedule a run across your watch list of `shopUrls` (daily or weekly, via an Apify schedule), export each run's dataset, and compare the current row against the previous run's row for the same `storeDomain`. Alert on the deltas that matter to you — a new collection appearing, `emptyCollectionCount` climbing, or `collectionIndexState` changing from `complete` to `capped`, which would mean the store grew past your `maxCollectionPages` ceiling.

***

### 🔌 Integrate Shopify Store Scraper: Collections, Pages & Blogs and automate your workflow

Shopify Store Scraper: Collections, Pages & Blogs works with any language or tool that can send an HTTP request through the Apify API.

#### REST API with Python

```python
from apify_client import ApifyClient

client = ApifyClient("<YOUR_APIFY_TOKEN>")

run = client.actor("scrapier/shopify-store-scraper-collections-pages-blogs").call(
    run_input={
        "shopUrls": ["https://www.allbirds.com/"],
        "maxCollections": 0,
        "readSitemap": True,
    }
)

for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    if item.get("type") == "collection":
        print(item["title"], item["productsCount"])
```

#### Scheduled monitoring and delivery

The Actor has no built-in scheduler of its own — use an Apify Schedule to run it periodically against the same `shopUrls`, and either poll the dataset via the API after each run or attach a webhook that fires on `ACTOR.RUN.SUCCEEDED` to push results onward to your own endpoint.

***

### ⚖️ Is it legal to scrape a Shopify store's structure?

Yes — Shopify Store Scraper: Collections, Pages & Blogs reads publicly published storefront data (collection listings, sitemap URLs, page and blog counts) that any visitor's browser can already see; it returns nothing that requires a login, and it never bypasses a password-protected storefront. This is business and catalogue data about a store's own structure, not personal data about individuals, so GDPR and CCPA generally do not attach to it — the relevant framework is the target store's terms of service and, in some jurisdictions, database-rights protections over compiled data. Scraping for periodic monitoring of your own competitive set carries a different risk profile than bulk scraping for AI training data at scale. Consult your legal team for commercial use cases involving bulk data storage or redistribution.

***

### ❓ Frequently asked questions

#### Does Shopify Store Scraper: Collections, Pages & Blogs return the products inside each collection?

No. It returns the collection itself — its handle, size, publish dates and image — not the individual products inside it. It's a structure and content index, not a product export.

#### How many collection rows does it return per store?

By default, every published collection (`maxCollections: 0`). Set `maxCollections` to a specific number to cap rows returned per store; `collectionMinProducts` and `includeEmptyCollections` further filter which collections qualify. The store-level totals (`collections_count`, `publishedCollectionsCount`) stay complete regardless of these filters — only the row count changes.

#### What's the real ceiling on how many collections it can index?

`maxCollectionPages` caps the walk at that many batches of 250 collections — the default of 20 covers up to 5,000 collections per store. If a store publishes more than the cap covers, `collectionIndexState` reads `"capped"` instead of silently returning a partial number as final.

#### How does it handle locale-specific sitemap mirrors?

Sitemap child entries under a locale path (e.g. `/en/sitemap.xml`) are detected and excluded from the root content-inventory counts, so a multi-locale store's `pageCount` and `blogArticleCount` reflect the store's structure once, not once per locale mirror.

#### Does it extract the content inventory (pages, blog articles, metaobject pages) as well as collections?

Yes — `pageCount`, `blogArticleCount` (with `blogHandles`), and `metaobjectPageCount` are built from the store's published sitemap and summed into `contentFootprint`. They come back `null`, not `0`, on a store that doesn't publish a sitemap, or if `readSitemap` is turned off.

#### How does it handle a storefront that blocks direct requests?

Each fetch attempt runs through direct connection first, then — only if `proxyConfiguration` supplies Apify Proxy — escalates through a shared proxy tier and then a residential proxy tier, retrying on transient statuses (403, 408, 429, 502, 503, 504) with a short backoff between attempts. A clean 404 or wrong-content-type response is treated as final and not retried.

#### How do I monitor a store's structure over time?

Schedule a run across the same `shopUrls`, then diff the current run's `collections_count`, `pageCount`, `blogArticleCount` and `contentFootprint` against the previous run for the same `storeDomain`. See "How do you monitor a store's structure over time?" above for the full loop.

#### Does it work with Claude, ChatGPT, and AI agent frameworks?

It's callable as an HTTP endpoint through the Apify API by any agent framework that can make a request — an agent can trigger a run, poll or wait for it to finish, and pass the resulting JSON rows to a model as grounded context.

#### Can I use it without managing proxies or a Shopify account?

Yes. No Shopify login, API key, or store-owner access is required — it reads the same public endpoints and sitemap a browser reaches. `proxyConfiguration` is optional and defaults to a direct connection (`useApifyProxy: false`); Apify Proxy is only used, and only escalated through, if you supply it.

#### Is it compatible with the original Shopify Store Scraper's input and output?

Yes. `startUrls` and `proxyConfiguration` are still accepted and merged with `shopUrls`, and the store row still carries the original actor's columns (`total_products`, `price_min`/`price_max`, `vendors`, `categories`, `theme_name`, `locale`, `currency`, `country`) unchanged.

***

### 💬 Your feedback

Found a bug, or a Shopify storefront where a count looks wrong? Let us know at <scrapier.io@gmail.com> — include the store URL and the field that looks off, and we'll take a look.

# Actor input Schema

## `shopUrls` (type: `array`):

Storefront URLs or bare domains to index. Example: https://www.allbirds.com or allbirds.com. Each entry produces one store row.

## `startUrls` (type: `array`):

Kept so input files written for the original actor keep working. Entries here are added to the list above and duplicates are removed. Leave empty if you filled in the field above.

## `indexAllCollections` (type: `boolean`):

Walk the store's full collection list instead of reading only the first batch of 250. Turn this off to skip collection indexing entirely. Default is on.

## `emitCollectionRows` (type: `boolean`):

Emit a separate labelled row for every collection (title, handle, URL, product count, publish dates, image URL). Turn off to keep only the store-level summary. Default is on.

## `includeEmptyCollections` (type: `boolean`):

Keep collections that currently hold 0 products. They are real published collections, so they are kept by default. Turn off to drop them from the collection rows. The store-level total always counts them.

## `collectionMinProducts` (type: `integer`):

Only emit collection rows holding at least this many products. Example: 50 keeps only the store's substantial collections. 0 = no size floor. Default is 0.

## `maxCollections` (type: `integer`):

How many collection rows to return for each store address. Enter 25 and you get 25 collection rows per store; enter 500 and you get 500 (or all of them, if the store publishes fewer). Set 0 to return every collection. The store-level totals stay complete whatever you set here. Default is 0.

## `maxCollectionPages` (type: `integer`):

Safety cap on how many batches of 250 collections to walk. Example: 20 covers up to 5,000 collections. When the cap is reached the store row says so instead of pretending the list ended. Default is 20.

## `readSitemap` (type: `boolean`):

Count the store's published pages, blog articles, metaobject pages and product URLs. This is also the only source that still works on storefronts that do not publish structured store data. Default is on.

## `countPages` (type: `boolean`):

Count the store's standalone content pages and list their handles. Turn off to skip that part of the index. Default is on.

## `countBlogArticles` (type: `boolean`):

Count published blog articles and list the blogs they belong to. Turn off to skip that part of the index. Default is on.

## `sitemapChildLimit` (type: `integer`):

How many sections of the store's published index to read per store. Large stores split their index into many sections. Example: 30 is enough for a store with roughly 150,000 URLs. Default is 30.

## `proxyConfiguration` (type: `object`):

Optional. Leave as-is to run directly.

## Actor input object example

```json
{
  "shopUrls": [
    "https://www.allbirds.com/"
  ],
  "indexAllCollections": true,
  "emitCollectionRows": true,
  "includeEmptyCollections": true,
  "collectionMinProducts": 0,
  "maxCollections": 0,
  "maxCollectionPages": 20,
  "readSitemap": true,
  "countPages": true,
  "countBlogArticles": true,
  "sitemapChildLimit": 30,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}
```

# Actor output Schema

## `results` (type: `string`):

All scraped items in the Actor's default dataset.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "shopUrls": [
        "https://www.allbirds.com/"
    ],
    "proxyConfiguration": {
        "useApifyProxy": false
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("scrapier/shopify-store-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "shopUrls": ["https://www.allbirds.com/"],
    "proxyConfiguration": { "useApifyProxy": False },
}

# Run the Actor and wait for it to finish
run = client.actor("scrapier/shopify-store-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "shopUrls": [
    "https://www.allbirds.com/"
  ],
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}' |
apify call scrapier/shopify-store-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,scrapier/shopify-store-scraper"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/c1zsLjbHyRY2N7Z4G/builds/E2d17Abg4dcEeOjqQ/openapi.json