Wayfair Products Scraper

Product listings from Wayfair category and keyword pages — name, brand, price and was-price, rating, review count, stock message, promo flags and shipping badge.

HTTP only, no browser, no login.

Input

Category URLs are the precise option:

{
  "mode": "urls",
  "listingUrls": ["https://www.wayfair.com/furniture/sb0/desks-c46129.html"],
  "maxPages": 3,
  "maxItems": 150
}

Keywords also work, with a caveat worth reading below:

{ "mode": "search", "keywords": ["bookcase", "office chair"] }

What you get

recordType	One per	Carries
SEARCH_SUMMARY	keyword / URL	requested vs resolved URL, whether it redirected, pages fetched, rows fetched vs returned, duplicates dropped
PRODUCT	product	SKU, name, brand, price, was-price, rating, review count, stock, badges — plus the raw card metadata
ERROR	failed input	why

{
  "recordType": "PRODUCT",
  "cardSchema": "full",
  "sku": "W114696675",
  "name": "Modern Farmhouse Computer Desk with LED Lights and Power Outlets",
  "brand": "Millwood Pines",
  "currency": "USD", "price": 184.99, "strikethroughPrice": 465.00,
  "ratingAverage": 4.43, "reviewCount": 154,
  "stockMessage": "554 Left in Stock",
  "promoFlagText": "Labor Day Deal",
  "shippingBadgeText": "FREE Delivery"
}

Known limits — read these before you rely on the output

Keyword search is not a keyword search. keyword.php?keyword=desk redirects to whichever single category Wayfair picks for that term — in testing, "desk" landed on unfinished desks. The products are real, but the category may be narrower than your term implies. Every summary reports requestedUrl, resolvedUrl and wasRedirected so you can see where you ended up. For precise control, pass category URLs in listingUrls.

Wayfair also rewrites category slugs from the id: the URL …/sb0/desks-c46129.html resolved to …/sb0/kitchen-dining-tables-c46129.html. The id is authoritative, not the slug you typed.

Wayfair rate-limits bursts with HTTP 429. It is transient, not a block — in testing a rapid burst earned one, and backing off to a request every 3–4 seconds cleared it on the very next request and then ran clean for four consecutive pages. The actor retries 429 with backoff and defaults to one request at a time. If you see repeated failures, raise minRequestInterval rather than lowering it.

Only Chrome TLS fingerprints are served. On the identical URL, chrome124 and chrome131 return the full 3.5 MB page while safari17_0, firefox133 and edge101 all return 429. The actor uses Chrome profiles only. This is worth knowing if you fork it.

Cards repeat, so products are de-duplicated by SKU. A 60-card page holds roughly 48 distinct products; duplicatesDropped tells you how many were removed, both within a page and across pages.

Some products have no name. Wayfair emits two card metadata schemas on the same page. The common one (cardSchema: "full", 48 of 60 in testing) carries name and brand; a second one ("priceOnly", 3 of 60) carries SKU, prices, rating and review count but no name or brand at all. Those are distinct products, not duplicates — measured zero SKU overlap with the named ones — so they are kept rather than dropped, with name left null instead of invented. Filter on cardSchema if you only want named rows.

Prices come from Wayfair's own card tracking data, which is what the site uses for its analytics. It matches the rendered price at fetch time; it is not a quote and does not include shipping or tax.

Technical

Wayfair is a Next.js App Router app that streams RSC, but the product grid is not in the flight stream (that carries only recommendation rails). The grid is server-rendered HTML, and each card carries a data-tracking-metadata JSON attribute holding the whole record — which is what this actor parses, rather than depending on generated class names like class="dq5rf45 dq5rf40" that change on every deploy.

robots.txt was checked at path level, not just the apex: /keyword.php?keyword=… and /furniture/sb0/… are both ALLOWED, with no ClaudeBot/anthropic-ai group and no blanket disallow.

Full recon trail is in CRAWLING_METHOD.md.