Baidu Search Scraper

Playwright-based extraction of public Baidu web results with rich-result classification, stable IDs, real pagination, regional search context, fair query batches, and authorized proxy support.

Highlights

• Single query, deduplicated multiple queries, fair per-query quotas, and exact low-volume mode
• Constructed Baidu pn pagination rather than reliance on a drifting Next selector
• Organic, Baike, video, image, official, academic, finance, live, and other rich-card classification
• Source, snippet, highlighted terms, dates, thumbnails, site links, ratings, and prices when Baidu supplies them
• Stable SHA-256 IDs, canonical URL deduplication, and a typed 44-field output contract
• Consistent Chrome profile, Chinese locale headers, persistent sessions, bounded concurrency/retries/timeouts, and optional Residential proxy
• Explicit no-results/challenge distinction and fail-closed CAPTCHA handling

Input

{
  "queries": ["人工智能", "机器学习"],
  "maxItems": 12,
  "maxItemsPerQuery": 6,
  "maxPages": 2,
  "market": "zh-CN",
  "includeAds": false,
  "includeRelated": false,
  "maxConcurrency": 2,
  "proxyConfiguration": { "useApifyProxy": false }
}

Use query, queries, or both. Duplicate query strings are removed and batch output is interleaved fairly. startUrls remains supported for compatibility, but every URL must be on baidu.com and include the same wd value a user would enter in Baidu's search field.

maxItems is global; maxItemsPerQuery defaults to a fair share. Each Baidu page currently supplies roughly 8–10 usable non-ad records.

Output

Records include stable identity, global/query/page positions, title and optional emphasized HTML, snippet, destination URL, source/domain, thumbnails and site links, rich-result classification, source-backed ad/official flags, dates, ratings/prices when present, query/market context, and run metadata. Null, undefined, empty strings, and empty arrays/objects are removed.

The Actor no longer fabricates a Google favicon URL. Optional fields absent from the Baidu card are omitted.

Proxy and access behavior

Baidu may close connections or return an explicit verification page after repeated requests. Direct mode is supported, but Apify Residential proxy is recommended for cloud production runs:

{
  "proxyConfiguration": {
    "useApifyProxy": true,
    "apifyProxyGroups": ["RESIDENTIAL"],
    "apifyProxyCountry": "CN"
  }
}

GOOGLE_SERP cannot proxy Baidu and is rejected. The Actor does not solve or bypass CAPTCHA/verification pages. Challenges are marked non-retriable and never stored. Logs omit raw HTML, response bodies, destination URLs, cookies, proxy credentials, and signed verification parameters.

Performance

Browser pages run with bounded concurrency and skip media/font resources. Representative Residential cloud runs completed 15 records across two pages in about 53 crawler seconds and a fair 6+6 two-query batch in about 72 seconds, below the requested three-minute target.

Failure semantics

• Invalid input fails before navigation.
• Temporary navigation failures use bounded retries.
• Explicit verification/CAPTCHA pages fail nonzero without challenge data.
• A confirmed no-results page succeeds empty; if Baidu challenges the no-results query, the run fails rather than misclassifying it.
• Partial records buffered before a later challenge are not published as a successful complete run.

Local development

npm ci
npm test
apify validate-schema
apify run --purge --input-file INPUT.json