🔍 Similarweb Scraper

📊 Website intelligence for any domain in seconds. Start with one website, choose traffic/rankings, similar sites, or WHOIS + homepage keywords, then export the results as JSON, CSV, Excel or any other format Apify supports.

💎 What is Similarweb Scraper?

Similarweb Scraper is a fast, captcha-resilient web scraper that pulls public website intelligence without requiring a Similarweb account, login, or API key. Behind the scenes it uses Chrome TLS / JA3 fingerprints via curl_cffi and routes every request through a fresh Apify Residential proxy session. When Similarweb refuses a protected source, the Actor stops quickly and does not save guessed or fallback data as base_data.

You can start with a single domain, then scale to a whole list when you are ready. Pick one dataset mode per run and the Actor returns clean records ready to drop into a spreadsheet, BI tool, warehouse, or AI agent.

🚀 What can Similarweb Scraper do?

• 🗂️ Choose one of three dataset modes for each run:
  • 📊 Base data — global / country / category ranks, monthly visits, bounce rate, pages per visit, time on site, traffic-source split (direct, search, referral, social, paid, mail), top organic keywords with volume and CPC, AI traffic share per LLM (ChatGPT / Claude / Gemini / Perplexity / Copilot).
  • 🪞 Similar sites — competitors and alternatives with their traffic, category, top-country rank and a similarity score that says how close each match actually is.
  • 🆔 AITDK — WHOIS via RDAP (registrar, registration / expiration dates, name servers, EPP status, DNSSEC) plus on-page keyword density analysis of the domain's homepage.
• ⚡ Start small or run in bulk — one domain is enough for a test run; a batch runs as many domains at once as the memory you gave it will hold, so a larger list finishes in about the same wall clock as a small one rather than in proportion to its length.
• 🛡️ Captcha-resilient — uses available public sources directly, without captcha solving. If a protected Similarweb source returns a challenge, the Actor skips that source instead of saving unreliable data.
• 🔄 Evidence-driven IP rotation — requests are served from a pool of Apify Residential proxy exits. An exit is dropped and replaced the moment it returns a block signal (challenge, 403, 429, proxy or timeout error), so a blocked address still costs at most one attempt, while healthy exits are reused instead of paying a new TLS handshake every call.
• 🌐 Three input formats — accepts example.com, www.example.com, or https://example.com. The domain is extracted automatically.

☁️ Remember the Apify platform

Running this Actor on Apify gives you everything that comes with the platform out of the box: managed Residential proxies with global exit IPs, scheduling (run hourly / daily / weekly), free storage in Apify Datasets with export to JSON / CSV / Excel / JSONL / XML / RSS, webhooks and integrations (Make, Zapier, n8n, Google Sheets, Slack, Airtable, Pipedream), and a REST API + Python / JavaScript SDKs to plug results into your own pipelines.

🗝️ What data can this Actor extract?

Field group	Examples
Rankings	Global rank · country rank · category rank
Engagement	Total visits · monthly visits (3 months) · bounce rate · pages / visit · time on site
Traffic sources	Direct · search (organic / paid) · referral · social (organic / paid) · affiliate · display ads · GenAI · mail
AI traffic share	ChatGPT · Claude · Gemini · Perplexity · Copilot — current + 3-month history
AI traffic detail	Ranked list of every named AI referrer (incl. Grok, DeepSeek) · absolute AI-referred visits
Top keywords	Keyword · estimated value · search volume · CPC
Country breakdown	Top countries with share + monthly visit estimates per country
Similar sites	Up to 20 related sites with traffic, their own category, top-country rank, thumbnails and a similarity score
Related apps	Mobile apps associated with the site — platform, store ranking and link
WHOIS (via RDAP)	Registrar · IANA ID · abuse contact · registration / expiration / transfer dates · name servers with glue records · DNSSEC DS records · EPP status
Keyword density	Top-20 non-stopword terms from the homepage with count and density, in any writing system, plus the source each item was read from
Assets	Desktop / mobile screenshots · favicon
Coverage	dataCoverage · isSmall — how much Similarweb actually knows about the domain

🎯 How to use Similarweb Scraper

1. Click Try for free on the Actor's Apify Store page.
2. In the Domains field, enter one website to test, or paste a larger list later — one per line, any format (example.com, www.example.com, or https://example.com).
3. Pick exactly one Dataset to fetch. Start with base_data for the main traffic, rank, engagement, keyword and AI-share overview, or choose another mode when needed.
4. Click Start. A one-domain test run is fine; there is no 10-domain minimum. aitdk takes longer than the other modes because it also fetches RDAP and the homepage.
5. When the run finishes, open Storage → Dataset and export to JSON, CSV, Excel, JSONL, XML or RSS. Or pull the results through the API: https://api.apify.com/v2/datasets/{dataset_id}/items.

📥 Input

The form has two visible fields only — everything else has sensible defaults:

Field	Type	Default
domains	array	20 common domains are prefilled for a quick bulk test
datasetMode	enum	base_data — one of base_data, similar_sites, aitdk

Example input

{
  "domains": ["openai.com", "google.com", "booking.com"],
  "datasetMode": "base_data"
}

API callers may also pass domains through domainsText (newline, comma, or semicolon separated), legacy urls / websites, or Apify-style startUrls request objects. At least one domain source must be provided.

🧠 How much memory to give a run

Measured on 290 domains, each mode at both sizes:

Mode	512 MB (the default)	4096 MB
base_data	8 s · 290/290 complete	8 s · 290/290
similar_sites	12 s · 290/290	10 s · 290/290
aitdk	4 min · 230 complete	82 s · 239 complete

base_data and similar_sites gain nothing from more memory - they wait on the network, not the processor, and the default is both faster to start and far cheaper. aitdk is the one that does: it makes a registry lookup and up to eleven homepage attempts per domain, and Apify hands out CPU in proportion to memory, so 512 MB gives it about a tenth of a core. It finishes either way; it takes about three times as long.

Settings for API callers

These are accepted but deliberately kept out of the form, because the defaults are right for almost every run. Each is clamped to the ceiling shown, so a mistyped value slows a run down rather than breaking it.

Field	Default	Ceiling	What it does
proxy_country_code	US	—	Which country the residential exits come from.
max_concurrency	as many as memory holds	—	Caps domains in flight. Unset, the run widens until the container's memory says stop, which is what makes a large batch finish in about the same wall clock as a small one.
request_timeout_seconds	10	60	How long one request may take.
domain_timeout_seconds	derived from the mode	900	How long one domain may take across all its sources and retries. Derived as four request timeouts for base_data and similar_sites, eight for aitdk, which has more sources to try.
max_attempts_base / _similar / _aitdk	4	8	Retries per source, per domain.
rdap_concurrency	3	10	Requests in flight to any one registry. Registries rate-limit by protocol, and this Actor reaches them from one address.
rdap_min_interval_seconds	none	5	A floor under the pause between two requests to the same registry. There is none by default: the concurrency above is the standing politeness, and beyond it the pause is whatever a registry actually asks for - it grows the moment one answers that it is being asked too often, and follows its Retry-After where it sends one.
proxy_slot_max_uses	50	500	Requests one proxy exit carries before it is retired.

📤 Output

Each domain produces one dataset item. Each item conforms to the dataset schema and is rendered in the Apify Console views that match the selected dataset mode: 📊 Overview · 🪞 Similar sites · 🚦 Traffic sources · 💫 Engagement · 🤖 AI traffic share · 🆔 AITDK (WHOIS and keywords).

Example item (abridged)

{
  "domain": "openai.com",

  "rankGlobal": 207,
  "country": "US",
  "countryRank": 306,
  "category": "ai_chatbots_and_tools",
  "categoryRank": 6,
  "title": "OpenAI",
  "totalVisits": 195737812,
  "bounceRate": 0.5937,
  "pagesPerVisit": 2.59,
  "timeOnSite": 138.72,

  "socialTraffic":   0.0287,
  "searchTraffic":   0.2154,
  "directTraffic":   0.3840,
  "referralTraffic": 0.1038,
  "displayAdsTraffic": 0.0013,
  "genAiTraffic": 0.2358,

  "aiTrafficShareChatgpt": 0.8825,
  "aiTrafficShareClaude":  0.0029,
  "aiTrafficShareGemini":  0.0106,

  "topKeywords": [
    {"keyword": "chatgpt",  "estimatedValue": 20907500.0, "searchVolume": 173339160.0, "cpc": 0.14},
    {"keyword": "chat gpt", "estimatedValue":  5688810.0, "searchVolume":  95011780.0, "cpc": 0.14}
  ]
}

📉 Domains Similarweb barely covers

Not every domain is ranked. Small sites are still returned, but with no global rank and no traffic-source split, and a handful of domains have no Similarweb record at all. Every base_data row carries a dataCoverage value so you can tell those apart from a failed scrape:

Value	Meaning
full	Ranked domain, the whole record is populated.
small_site	Below Similarweb's ranking threshold. Title, description, category, monthly visit estimates and screenshots are still present; rank and traffic split are not.
no_data	Similarweb returned an empty record. The row exists only to confirm the domain was checked.

The run summary reports these under sourceCoverage (ranked, smallSites, noData) — separate from dataQuality, because they describe how much Similarweb knows about a domain, not whether the scrape succeeded. A small_site row is a complete result: it already contains every value the source provides.

🔁 Finishing a bulk run

On large lists a few domains can come back empty because an upstream source was temporarily unavailable. You do not have to guess which ones or repeat the whole batch: every run writes the complete list of domains that produced no dataset item to the default key-value store under FAILED_DOMAINS.

$curl "https://api.apify.com/v2/key-value-stores/<STORE_ID>/records/FAILED_DOMAINS"

{
  "selectedDataset": "base_data",
  "count": 3,
  "domains": ["example.com", "example.org", "example.net"],
  "domainsText": "example.com\nexample.org\nexample.net",
  "failures": [
    {"domain": "example.com", "reason": "aitdk: both WHOIS and keyword density failed for example.com - aitdk(rdap): exhausted RDAP lookup; aitdk(keywords): homepage candidate blocked: HTTP 403 forbidden"}
  ],
  "invalidEntries": []
}

Paste domainsText into the Domains field, or send domains straight back as the Actor input, and only the missing part of the list is scraped. The record is written after every run, so count: 0 is a positive confirmation that nothing is missing. Each reason says what every source actually answered — the status a site refused with, a name that does not resolve, a registry with no record — so a domain that will fail again is easy to tell from one worth re-running. invalidEntries lists input lines that were rejected before any request was made; re-running those would fail again.

🔬 Seeing what a run did

Alongside the summary, every run writes a TIMELINE record holding each request, domain, retry and resource reading with the second it happened on.

$curl "https://api.apify.com/v2/key-value-stores/<STORE_ID>/records/TIMELINE"

It answers what a total cannot: when a batch slowed down or began being refused, which host was refusing, how long each domain took, and how wide the Actor ran. On a very large batch the requests that answered are thinned evenly so the record stays storable — every request that failed is kept, and counts says how many were left out.

🔗 Integrate Similarweb Scraper anywhere

Apify Actors run on a REST API — every run, dataset and webhook is addressable from your code:

# Trigger a run from anywhere
curl -X POST "https://api.apify.com/v2/acts/<USER>~similarweb-scraper/runs?token=<API_TOKEN>" \
     -H "Content-Type: application/json" \
     -d '{"domains": ["openai.com"], "datasetMode": "base_data"}'

# Read results from the run's default dataset
curl "https://api.apify.com/v2/datasets/<DATASET_ID>/items?format=json"

Or use the official Python and JavaScript clients.

❓ FAQ

Can I test it with only one domain? Yes. The Actor needs at least one domain, not ten. The default UI example uses openai.com with base_data, so a new user can click Start immediately.

🔑 Do I need a Similarweb account or API key? No. This Actor talks to Similarweb's public SPA endpoint directly. No login, no API key, and no scraping the captcha-gated in-depth pages.

🆕 Is the data fresh? Yes — it's the same JSON Similarweb's UI loads. The snapshotDate field on every record tells you exactly which month it represents. Similarweb refreshes its traffic data monthly.

⏰ Can I run this on a schedule? Yes — open the Actor in Apify Console, go to Schedules and pick hourly, daily, weekly or a custom cron. Combine with webhooks to push fresh data into Google Sheets, Slack, Make, Zapier or your own backend automatically.

⚖️ Is web scraping legal? Public web pages are generally legal to scrape, but you must respect copyright, terms of service, and personal-data protection laws (GDPR in the EU and similar regulations elsewhere). This Actor only extracts publicly visible data — no personal data is collected. See Apify's legal blog for details.

💬 Support

• Found a bug or have a feature request? Open an issue in the Actor's Issues tab on Apify Console.
• Questions about Apify itself? Visit docs.apify.com or the Apify Discord community.

📝 Changelog

See CHANGELOG.md for the full release history. The Actor follows Semantic Versioning.