Shopify Store Scraper: Collections, Pages & Blogs
Pricing
from $5.99 / 1,000 results
Shopify Store Scraper: Collections, Pages & Blogs
Shopify Store Scraper extracts collections, pages, blogs, titles, descriptions, URLs, publication dates, and other public store content. Ideal for ecommerce research, competitor analysis, content discovery, SEO audits, and Shopify store intelligence.
Pricing
from $5.99 / 1,000 results
Rating
0.0
(0)
Developer
Scrapier
Maintained by CommunityActor stats
0
Bookmarked
2
Total users
0
Monthly active users
4 days ago
Last modified
Categories
Share
Shopify Store Scraper — Collections, Pages, Blogs & Verified Counts
Shopify Store Scraper: Collections, Pages & Blogs indexes a storefront's published structure — every collection with its live product count, every content page, every blog article, and every metaobject page — and returns it as one store summary row plus one labelled row per collection. Every response is structured JSON, ready to pass directly to an LLM, load into a spreadsheet, or feed a monitoring pipeline. Each count also states where it came from, so a truncated read is never mistaken for a complete one. Point it at any storefront and get back a structure you can actually audit.
🧭 What is Shopify Store Scraper: Collections, Pages & Blogs?
It queries one or more Shopify storefront URLs and returns the store's collection tree and content inventory — not the products inside each collection. Shopify's own /collections.json endpoint returns collections in batches of 250; a lot of "complete" collection counts out there are really just the first batch. This Actor pages through the whole list and then cross-checks the total it counted against two numbers the store publishes about itself (publishedCollectionsCount and sitemapCollectionUrlCount), so a wrong count doesn't pass silently. No Shopify account, API key, or store-owner access is required — it reads the same public endpoints and sitemap any visitor's browser can reach. A storefront that is closed to the public is detected and reported as password_protected; it is never bypassed.
- Walks the collection list to exhaustion, not just the first page, and reports how many batches it took.
- Cross-checks the collection count three ways — indexed, store-published, and sitemap-published — and reports all three instead of picking one.
- Emits one labelled row per collection with its real size, publish dates, image URL and description length.
- Builds a content inventory from the store's own sitemap: standalone pages, blog articles (with the blogs they belong to), and metaobject pages.
- States the provenance of the product count it used (
catalogueCountSource) rather than presenting a single number as ground truth. - Runs on a list of storefronts in one job, and one unreadable store never stops the rest of the run.
📦 What data can you get with Shopify Store Scraper: Collections, Pages & Blogs?
Each run returns a store profile row, a labelled row per collection, and a content-inventory summary built from the store's own sitemap.
| Result Type | Extracted Fields | Primary Use Case |
|---|---|---|
| Store profile row | storeDomain, storefrontStatus, shop_name, country, currency, locale, theme_name, theme_id, theme_version, themeSchemaName, apps_installed, collections_count, publishedCollectionsCount, sitemapCollectionUrlCount, total_products, price_min/price_avg/price_max | Competitive snapshot of a store's size and setup |
| Collection rows | title, handle, collectionUrl, productsCount, publishedAt, updatedAt, imageUrl, descriptionChars, collectionPage | Collection-tree audit, empty/orphan collection detection |
| Content inventory | pageCount, topLevelPages, blogArticleCount, blogHandles, metaobjectPageCount, contentFootprint | SEO and content-footprint audit |
| Catalogue cross-check | sitemapProductUrlCount, publishedProductsCount, catalogueCountSource, catalogueIndexState, collectionIndexState | Verifying a count before you report it |
Provenance-tracked collection & content counts
Instead of returning one collection number and letting you trust it, the store row carries three independently-sourced counts side by side — collections_count (what the Actor actually indexed), publishedCollectionsCount (what the store's meta.json reports about itself), and sitemapCollectionUrlCount (what the store's own sitemap lists). collectionIndexState tells you whether the index run was complete, capped (hit maxCollectionPages), partial, or unavailable — never a silent truncation dressed up as a final number. The same pattern applies to the product catalogue: catalogueCountSource names exactly which source (products_json, meta_json, or sitemap) produced publishedProductsCount when the direct product listing is unavailable.
Content inventory: pages, blog articles & metaobject pages
Parsed from the storefront's own sitemap index rather than guessed, the content inventory counts standalone content pages, published blog articles (grouped by the blog they belong to), and metaobject pages, then sums them into contentFootprint. Locale-specific sitemap mirrors (e.g. a /en/sitemap.xml variant) are excluded from these counts so a multi-locale store doesn't get double-counted. Use it to compare how much editorial content sits behind a shop before you decide it's worth a competitive teardown.
🛠️ Why not build this yourself?
Shopify doesn't publish a single official API for indexing a storefront you don't own. The Storefront API returns rich structured data, but it requires an access token issued by the store itself — it's built for merchants building their own headless frontend, not for reading someone else's store from outside. What's left are the storefront's own public JSON endpoints and its sitemap, and both are harder to read reliably than they look:
/collections.jsonpages in batches of 250 with no documented total-count field — you only know you're done when a page comes back short or empty.- Sitemap structure varies by theme: some stores publish a flat sitemap with every URL, others publish a sitemap index with locale-specific child mirrors that need to be excluded from root counts, not summed into them.
- A store's own published counts don't always agree with each other — the self-reported total and the sitemap-listed total can differ by a handful of collections a merchant has marked as not-for-indexing.
- Some storefronts block direct requests outright, which means retry and proxy-tier escalation logic, not just a single fetch.
- Headless storefronts skip the JSON endpoints entirely, so the sitemap becomes the only usable source, and the Actor has to know to fall back to it.
Building and maintaining all of that in-house is a real, ongoing cost. If you need the numbers once and trust them, this Actor already handles the pagination, the fallback logic, and the cross-checks.
🆚 What's the difference between a Shopify product scraper and a store structure scraper?
A product scraper extracts individual product listings — titles, prices, variants, inventory — one row per SKU. A store structure scraper like this one extracts the shape of the store itself: how many collections it has, how big each one is, and how much standalone content (pages, blog articles) sits around the catalogue. The two answer different questions. A product scraper tells you what's for sale; a structure scraper tells you how the store is organized and how much content backs it. Shopify Store Scraper: Collections, Pages & Blogs returns the structure — collection rows (type: "collection") and a content inventory on the store row — not per-product detail. It still reports catalogue-level totals (total_products, price_min/price_max, vendors, categories) for context, but the row-per-product breakdown a dedicated product scraper provides is out of scope here.
🚀 How to index a Shopify store with Shopify Store Scraper: Collections, Pages & Blogs
- Open the Actor on its Apify Store listing and click Try for free (or Run, if you already have it saved).
- Enter one or more storefront URLs or bare domains in
shopUrls— for examplehttps://www.allbirds.comorallbirds.com. - Set the real controls you need:
maxCollectionsto cap how many collection rows come back,collectionMinProductsandincludeEmptyCollectionsto filter by size,readSitemap/countPages/countBlogArticlesto turn parts of the content inventory on or off. - Click Start and watch the run log — it reports each store's status and row counts as it finishes.
- Download results from the Dataset tab as JSON, CSV, Excel, or another Apify-supported export format, or stream them out via the API.
How to index multiple stores in one job
Add every storefront URL to shopUrls — there is no separate "list" input, just more entries in the same array. Each store is processed independently: ladder.reset_for_store() resets connection state per store, so one blocked or unreachable storefront is reported with storefrontStatus: "unreachable" and the run continues to the next URL rather than stopping the job.
⬇️ Input
All fields are optional — leave shopUrls empty only if you're relying on the legacy startUrls field instead.
| Parameter | Required | Type | Default | Description |
|---|---|---|---|---|
shopUrls | No | array (string list) | ["https://www.allbirds.com"] (prefill) | Storefront URLs or bare domains to index. Example: https://www.allbirds.com or allbirds.com. Each entry produces one store row. |
startUrls | No | array (string list) | — | Kept so input files written for the original actor keep working. Entries here are added to the list above and duplicates are removed. Leave empty if you filled in the field above. |
indexAllCollections | No | boolean | true | Walk the store's full collection list instead of reading only the first batch of 250. Turn this off to skip collection indexing entirely. |
emitCollectionRows | No | boolean | true | Emit a separate labelled row for every collection (title, handle, URL, product count, publish dates, image URL). Turn off to keep only the store-level summary. |
includeEmptyCollections | No | boolean | true | Keep collections that currently hold 0 products. They are real published collections, so they are kept by default. Turn off to drop them from the collection rows. The store-level total always counts them. |
collectionMinProducts | No | integer | 0 | Only emit collection rows holding at least this many products. Example: 50 keeps only the store's substantial collections. 0 = no size floor. Minimum: 0. |
maxCollections | No | integer | 0 | How many collection rows to return for each store address. Enter 25 and you get 25 collection rows per store; enter 500 and you get 500 (or all of them, if the store publishes fewer). Set 0 to return every collection. The store-level totals stay complete whatever you set here. Minimum: 0. |
maxCollectionPages | No | integer | 20 | Safety cap on how many batches of 250 collections to walk. Example: 20 covers up to 5,000 collections. When the cap is reached the store row says so instead of pretending the list ended. Minimum: 1. |
readSitemap | No | boolean | true | Count the store's published pages, blog articles, metaobject pages and product URLs. This is also the only source that still works on storefronts that do not publish structured store data. |
countPages | No | boolean | true | Count the store's standalone content pages and list their handles. Turn off to skip that part of the index. |
countBlogArticles | No | boolean | true | Count published blog articles and list the blogs they belong to. Turn off to skip that part of the index. |
sitemapChildLimit | No | integer | 30 | How many sections of the store's published index to read per store. Large stores split their index into many sections. Example: 30 is enough for a store with roughly 150,000 URLs. Minimum: 1. |
proxyConfiguration | No | object | {"useApifyProxy": false} (prefill) | Optional. Leave as-is to run directly. |
Example JSON input
Common pitfall: maxCollections and maxCollectionPages control two different things. maxCollectionPages caps how far the Actor walks the store's real collection list (the index), while maxCollections only trims how many of the already-indexed collections are returned as rows. Lowering maxCollections never makes the run faster or the cross-checked totals less accurate — it only shortens the output. If you actually want a faster, capped index, lower maxCollectionPages instead, and check collectionIndexState for "capped" in the result.
⬆️ Output
Results are pushed to the Actor's default dataset as typed, normalized JSON with a consistent row shape across runs. Export it from the Apify Console as JSON, CSV, Excel, HTML table, or XML, or pull it with the API/apify_client in JSON format. Collection rows are also mirrored to a second, per-run dataset named <runId>-collections, if you want them isolated from the store rows.
Only the store summary row is a billed event (row_result) — one charge per storefront that was actually profiled. Every collection row it emits, and any status row reporting that a store couldn't be reached or that no input was supplied, is pushed for free. To pull only the billed rows from the dataset, filter on item.type === "store"; everything else (type === "collection" or type === "status") was pushed uncharged.
Scraped results
If a store cannot be read at all, a type: "status" row is pushed instead, uncharged: {"type": "status", "isChild": false, "storeDomain": "example.myshopify.com", "store_url": "https://example.com", "storefrontStatus": "unreachable", "scrapedAt": "..."}. If no store URL was supplied at all, a single {"type": "status", "storefrontStatus": "no_input", ...} row is pushed and the run ends.
📊 How can I use the data extracted with Shopify Store Scraper: Collections, Pages & Blogs?
- Ecommerce and SEO analysts: compare
collections_count,emptyCollectionCount,pageCountandblogArticleCountacross a list of competitors to size up merchandising depth and content investment before a pitch or audit. - AI engineers and LLM developers: a run's JSON rows can be passed directly as grounding context to an LLM — "summarize this store's collection and content structure" — without any parsing step, since the schema is already typed and consistent.
- Market researchers: collection titles, handles, and counts across a batch of
shopUrlsbuild a category-coverage map of how a set of stores organizes the same product space. - Agencies and store migration teams:
collectionTop10ByProducts,largestCollection,pageCount, andblogHandlesgive a sized inventory of what has to move before a re-platform, without opening the admin of a store you don't manage.
📈 How do you monitor a store's structure over time?
Structure monitoring means re-running the same shopUrls on a schedule and diffing the store row's counts between runs, rather than trusting a single snapshot. Because collections_count, publishedCollectionsCount, and sitemapCollectionUrlCount are cross-checked every run, a real structural change (a merchant adding or archiving collections) shows up as a moved number on all three, while a scraping glitch shows up as disagreement between them — that distinction is the point of tracking all three, not just one.
The fields worth diffing run-to-run are collections_count, emptyCollectionCount, avgProductsPerCollection, pageCount, blogArticleCount, metaobjectPageCount, and contentFootprint. A jump in blogArticleCount between two runs is a content-publishing signal; a drop in collections_count alongside a stable publishedCollectionsCount is worth a second look at collectionIndexState before you trust it.
A practical loop: schedule a run across your watch list of shopUrls (daily or weekly, via an Apify schedule), export each run's dataset, and compare the current row against the previous run's row for the same storeDomain. Alert on the deltas that matter to you — a new collection appearing, emptyCollectionCount climbing, or collectionIndexState changing from complete to capped, which would mean the store grew past your maxCollectionPages ceiling.
🔌 Integrate Shopify Store Scraper: Collections, Pages & Blogs and automate your workflow
Shopify Store Scraper: Collections, Pages & Blogs works with any language or tool that can send an HTTP request through the Apify API.
REST API with Python
Scheduled monitoring and delivery
The Actor has no built-in scheduler of its own — use an Apify Schedule to run it periodically against the same shopUrls, and either poll the dataset via the API after each run or attach a webhook that fires on ACTOR.RUN.SUCCEEDED to push results onward to your own endpoint.
⚖️ Is it legal to scrape a Shopify store's structure?
Yes — Shopify Store Scraper: Collections, Pages & Blogs reads publicly published storefront data (collection listings, sitemap URLs, page and blog counts) that any visitor's browser can already see; it returns nothing that requires a login, and it never bypasses a password-protected storefront. This is business and catalogue data about a store's own structure, not personal data about individuals, so GDPR and CCPA generally do not attach to it — the relevant framework is the target store's terms of service and, in some jurisdictions, database-rights protections over compiled data. Scraping for periodic monitoring of your own competitive set carries a different risk profile than bulk scraping for AI training data at scale. Consult your legal team for commercial use cases involving bulk data storage or redistribution.
❓ Frequently asked questions
Does Shopify Store Scraper: Collections, Pages & Blogs return the products inside each collection?
No. It returns the collection itself — its handle, size, publish dates and image — not the individual products inside it. It's a structure and content index, not a product export.
How many collection rows does it return per store?
By default, every published collection (maxCollections: 0). Set maxCollections to a specific number to cap rows returned per store; collectionMinProducts and includeEmptyCollections further filter which collections qualify. The store-level totals (collections_count, publishedCollectionsCount) stay complete regardless of these filters — only the row count changes.
What's the real ceiling on how many collections it can index?
maxCollectionPages caps the walk at that many batches of 250 collections — the default of 20 covers up to 5,000 collections per store. If a store publishes more than the cap covers, collectionIndexState reads "capped" instead of silently returning a partial number as final.
How does it handle locale-specific sitemap mirrors?
Sitemap child entries under a locale path (e.g. /en/sitemap.xml) are detected and excluded from the root content-inventory counts, so a multi-locale store's pageCount and blogArticleCount reflect the store's structure once, not once per locale mirror.
Does it extract the content inventory (pages, blog articles, metaobject pages) as well as collections?
Yes — pageCount, blogArticleCount (with blogHandles), and metaobjectPageCount are built from the store's published sitemap and summed into contentFootprint. They come back null, not 0, on a store that doesn't publish a sitemap, or if readSitemap is turned off.
How does it handle a storefront that blocks direct requests?
Each fetch attempt runs through direct connection first, then — only if proxyConfiguration supplies Apify Proxy — escalates through a shared proxy tier and then a residential proxy tier, retrying on transient statuses (403, 408, 429, 502, 503, 504) with a short backoff between attempts. A clean 404 or wrong-content-type response is treated as final and not retried.
How do I monitor a store's structure over time?
Schedule a run across the same shopUrls, then diff the current run's collections_count, pageCount, blogArticleCount and contentFootprint against the previous run for the same storeDomain. See "How do you monitor a store's structure over time?" above for the full loop.
Does it work with Claude, ChatGPT, and AI agent frameworks?
It's callable as an HTTP endpoint through the Apify API by any agent framework that can make a request — an agent can trigger a run, poll or wait for it to finish, and pass the resulting JSON rows to a model as grounded context.
Can I use it without managing proxies or a Shopify account?
Yes. No Shopify login, API key, or store-owner access is required — it reads the same public endpoints and sitemap a browser reaches. proxyConfiguration is optional and defaults to a direct connection (useApifyProxy: false); Apify Proxy is only used, and only escalated through, if you supply it.
Is it compatible with the original Shopify Store Scraper's input and output?
Yes. startUrls and proxyConfiguration are still accepted and merged with shopUrls, and the store row still carries the original actor's columns (total_products, price_min/price_max, vendors, categories, theme_name, locale, currency, country) unchanged.
💬 Your feedback
Found a bug, or a Shopify storefront where a count looks wrong? Let us know at scrapier.io@gmail.com — include the store URL and the field that looks off, and we'll take a look.
