Website Crawler Scraper

Turn public website pages into clean text, Markdown, metadata, and link records with a bounded website crawler.

Website Crawler Scraper starts from one or more URLs, follows same-site links within your page and depth limits, and writes one structured dataset record per fetched page. The output is ready for search indexing, LLM knowledge bases, RAG ingestion, content inventories, and recurring documentation refreshes.

What does Website Crawler Scraper do?

The Actor:

• crawls up to 20 public websites in one run;
• follows internal links to a configurable depth;
• extracts title, description, canonical URL, language, and headings;
• exports unique internal and external links;
• removes common navigation and layout boilerplate;
• returns both normalized visible text and structured Markdown;
• calculates a stable SHA-256 content hash for change detection;
• records HTTP status, response time, crawl depth, and failures;
• accepts an optional CSS selector for the main content region;
• supports optional user-configured Apify Proxy without an automatic paid fallback.

It uses lightweight HTTP requests rather than a browser. That makes it a practical website crawler tool for public HTML and Markdown pages where JavaScript rendering is not required.

Who is it for?

AI and RAG teams can prepare source-attributed Markdown for embeddings, vector databases, assistants, and knowledge-base refreshes.

Search engineers can create page-level indexing documents with URLs, metadata, normalized text, and content hashes.

Documentation teams can export a bounded section of public docs before migration, analysis, or scheduled comparison.

Developers and data teams can feed a stable JSON dataset into Apify integrations, webhooks, cloud storage, or an ETL pipeline.

Content operators can inventory public pages and detect changed text by comparing contentHash between scheduled runs.

Why use this Actor?

A general web request gives you HTML. This Actor gives you an integration-ready page record.

• Two useful content formats: cleanedText for indexing and markdown for structure-aware downstream tools.
• Explicit crawl boundaries: maxPages, maxDepth, and maxLinksPerPage prevent accidental unbounded crawls.
• Multi-site runs: refresh several unrelated public sites through one input and one dataset.
• Stable change signal: compare contentHash instead of diffing layout-heavy HTML.
• Transparent failures: blocked, unavailable, non-HTML, or selector-mismatched pages return typed error records instead of disappearing silently.
• SSRF-safe requests: private, local, credential-bearing, and reserved-network targets are rejected, including redirects.
• No hidden paid fallback: direct HTTP is the default; a proxy is used only when you configure one.

What data does it extract?

Field	Meaning
url	Normalized URL requested by the crawler
finalUrl	Final URL after validated redirects
startUrl	Supplied website root that led to the page
depth	Internal-link depth from the start URL
statusCode	Final HTTP status when a response was received
contentType	Response content type
title	HTML title or first Markdown heading
metaDescription	HTML meta description when available
canonicalUrl	Resolved HTML canonical URL when available
language	HTML language attribute when available
headings	Ordered H1-H6 level and text pairs
internalLinks	Unique links inside the website scope
externalLinks	Unique links to other websites
cleanedText	Whitespace-normalized visible content
markdown	Selected content converted to Markdown, or source Markdown
wordCount	Approximate cleaned-text word count
contentHash	SHA-256 hash of cleaned text
responseTimeMs	Request and redirect time in milliseconds
fetchedAt	ISO timestamp for the page record
error	Concise per-page failure reason, otherwise null

Fields that a page does not expose are null or empty arrays. For example, a server-provided Markdown response does not have HTML canonical or language elements.

How to crawl a website

1. Open the Actor input page.
2. Add one or more public HTTP(S) pages under Start URLs.
3. Choose a small Maximum pages value for your first run.
4. Set Maximum link depth to 0 for only supplied pages, 1 for their direct internal links, or a higher bounded depth.
5. Keep query parameters disabled unless they identify genuinely different content.
6. Optionally set a content CSS selector such as main or article.
7. Run the Actor and open the Website content dataset view.
8. Export JSON, CSV, Excel, XML, or connect the dataset to your next system.

Start with low concurrency when crawling a small site. Increase it only when the website can handle the request rate.

Input parameters

startUrls

Required. One to 20 public HTTP(S) start pages. Entries can be request-list objects or URL strings through the API.

Each start URL defines its own crawl scope. Links remain on that hostname unless includeSubdomains is enabled.

maxPages

Maximum number of page records across the entire run. Default: 20. Range: 1 to 10000.

The limit includes success and failure records, so upstream problems cannot make the crawl exceed its requested scope.

maxDepth

Maximum internal-link depth. Default: 2. Range: 0 to 10.

Depth 0 fetches only the supplied URLs. Depth 1 also fetches links found directly on those pages.

maxConcurrency

Maximum requests in flight. Default: 5. Range: 1 to 20.

maxLinksPerPage

Maximum internal links accepted from one page for future crawling. Default: 20.

includeSubdomains

When enabled, links such as docs.example.org can be followed from example.org. The default is false.

followQueryParameters

When enabled, distinct query strings may become distinct crawl targets. Common tracking parameters are always removed. The default is false to reduce duplicate and faceted pages.

contentSelector

CSS selector for the content converted to text and Markdown. The first match is used. Default: body.

Use main, article, or a source-specific selector to reduce layout content. A selector that matches nothing creates an explicit page error.

removeSelectors

Optional CSS selectors removed before conversion. When omitted, the Actor removes scripts, styles, templates, SVG, canvas, iframes, navigation, headers, footers, asides, and aria-hidden elements.

requestTimeoutSecs

Per-page timeout from 5 to 120 seconds. Default: 30.

proxyConfiguration

Optional Apify Proxy or custom proxy configuration. Direct requests are used by default. The Actor never turns on residential proxy transfer automatically.

Example input: website to Markdown

{
  "startUrls": [
    { "url": "https://docs.apify.com/academy/web-scraping-for-beginners" }
  ],
  "maxPages": 5,
  "maxDepth": 1,
  "maxConcurrency": 2
}

For an LLM-focused documentation crawl, add "contentSelector": "main" when the source serves conventional HTML.

Example output

{
  "url": "https://developer.mozilla.org/en-US/docs/Web/HTTP/Overview",
  "finalUrl": "https://developer.mozilla.org/en-US/docs/Web/HTTP/Overview",
  "startUrl": "https://developer.mozilla.org/en-US/docs/Web/HTTP/Overview",
  "depth": 0,
  "statusCode": 200,
  "contentType": "text/html; charset=utf-8",
  "title": "Overview of HTTP - HTTP | MDN",
  "metaDescription": "HTTP is a client-server protocol...",
  "canonicalUrl": "https://developer.mozilla.org/en-US/docs/Web/HTTP/Guides/Overview",
  "language": "en-US",
  "headings": [
    { "level": 1, "text": "Overview of HTTP" }
  ],
  "internalLinks": [
    "https://developer.mozilla.org/en-US/docs/Web/HTTP"
  ],
  "externalLinks": [],
  "cleanedText": "Overview of HTTP HTTP is a client-server protocol...",
  "markdown": "# Overview of HTTP\n\nHTTP is a client-server protocol...",
  "wordCount": 2241,
  "contentHash": "a stable 64-character SHA-256 value",
  "responseTimeMs": 328,
  "fetchedAt": "2026-01-15T12:00:00.000Z",
  "error": null
}

The snippet is shortened for readability. Real Markdown and link arrays contain the page's extracted values.

How much does it cost to crawl website pages?

The Actor uses pay per event pricing:

• one start event per run;
• one item event for each successfully extracted page containing useful text;
• failure and unsupported-content records have no separate item charge.

The final price depends on your Apify plan tier. At the current BRONZE rates, a run has a $0.005 start fee plus $0.007104 per useful page.

Useful pages	BRONZE example cost
1	about 1.21 cents
10	about 7.60 cents
100	about 71.54 cents
1,000	about 710.90 cents

Optional proxy traffic and Apify platform usage may be billed according to your account and proxy configuration. The Actor does not enable paid proxy traffic on its own.

Build an LLM or RAG ingestion workflow

A repeatable knowledge-base refresh can:

1. run this Actor on a schedule;
2. fetch the default dataset through the API;
3. discard rows with error or empty cleanedText;
4. compare contentHash with the prior successful run;
5. chunk only new or changed markdown;
6. attach url, title, and fetchedAt as source metadata;
7. upsert vectors into your search or RAG store.

The Actor returns page-level documents. It does not create embeddings, choose chunk sizes, or write to a particular vector database, so you retain control over your retrieval pipeline.

Refresh a search index

Use cleanedText as the searchable body and retain:

• url as the document key;
• title and metaDescription for result display;
• language for analyzers;
• canonicalUrl for duplicate handling;
• contentHash for incremental updates;
• internalLinks for graph or discovery analysis.

Scheduled runs are independent snapshots. Store the prior dataset or hash map in your own downstream system when you need historical comparisons.

API usage

Replace YOUR_TOKEN with an Apify API token stored securely. Do not commit tokens to source code.

cURL

curl -X POST \
  "https://api.apify.com/v2/acts/automation-lab~website-content-crawler/runs?token=YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"startUrls":[{"url":"https://developer.mozilla.org/en-US/docs/Web/HTTP/Overview"}],"maxPages":5,"maxDepth":1}'

JavaScript

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('automation-lab/website-content-crawler').call({
  startUrls: [{ url: 'https://developer.mozilla.org/en-US/docs/Web/HTTP/Overview' }],
  maxPages: 5,
  maxDepth: 1,
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items.map(({ url, title, contentHash }) => ({ url, title, contentHash })));

Python

import os
from apify_client import ApifyClient

client = ApifyClient(os.environ['APIFY_TOKEN'])
run = client.actor('automation-lab/website-content-crawler').call(run_input={
    'startUrls': [{'url': 'https://developer.mozilla.org/en-US/docs/Web/HTTP/Overview'}],
    'maxPages': 5,
    'maxDepth': 1,
})
items = client.dataset(run['defaultDatasetId']).list_items().items
print([(item['url'], item['title'], item['contentHash']) for item in items])

Use with Apify MCP

Add the Actor to Claude Code:

claude mcp add --transport http apify \
  "https://mcp.apify.com?tools=automation-lab/website-content-crawler"

Claude Desktop, Cursor, and VS Code setup: add this equivalent HTTP MCP server configuration in the client's MCP settings:

{
  "mcpServers": {
    "apify": {
      "type": "http",
      "url": "https://mcp.apify.com?tools=automation-lab/website-content-crawler"
    }
  }
}

Example prompts:

• "Crawl these five documentation pages and return clean Markdown with source URLs."
• "Create a bounded 20-page content inventory for this public website."
• "Refresh these two public documentation sites and show pages whose content hashes changed from my saved list."

MCP runs the same Actor input contract and returns links to its run and dataset.

Limits and responsible crawling

• Only public HTTP(S) targets are supported.
• Private, loopback, link-local, reserved, local-name, credential-bearing, and unsafe redirect targets are rejected.
• The Actor does not sign in, solve CAPTCHAs, bypass access controls, or render JavaScript applications.
• Only HTML, XHTML, plain text, and Markdown responses become useful content records.
• Binary documents, media, and archive links are not crawled.
• The crawler follows links found in fetched content; it does not promise complete sitemap coverage.
• Query-heavy calendars, faceted navigation, and duplicate URL forms may require stricter limits.
• Common layout elements are removed heuristically. Use contentSelector and removeSelectors when a site's structure requires source-specific cleanup.
• The Actor uses bounded transient retries for network failures, HTTP 429, and server errors.

Use conservative limits and concurrency. Review the target site's terms, robots guidance, and acceptable-use rules before crawling.

Troubleshooting

Why did I get only one page?

Check maxDepth. A value of 0 intentionally fetches only start URLs. Also inspect internalLinks: JavaScript-generated navigation may not exist in the HTTP response, or the page may link to a different hostname.

Why is error populated?

The record contains the exact bounded failure, such as a timeout, unsupported content type, invalid selector, HTTP error, or network exhaustion. Review statusCode, contentType, and run logs before retrying.

Why is the Markdown empty or noisy?

Use a narrower contentSelector, commonly main or article. Add site-specific cookie banners, menus, or sidebars to removeSelectors.

Why was my URL rejected?

The Actor intentionally blocks private networks, localhost, reserved IP ranges, credential-bearing URLs, and redirects to those targets. Supply a public HTTP(S) URL.

Should I enable a proxy?

Try direct access first. Configure a proxy only when the public site requires it and you are authorized to use it. There is no automatic proxy fallback.

Legality and privacy

You are responsible for deciding whether you may crawl a target and process its content.

• Crawl public pages you are authorized to access.
• Follow applicable website terms, robots instructions, rate limits, copyright rules, and database rights.
• Do not use the Actor to bypass authentication or technical access controls.
• Avoid collecting personal or sensitive data unless you have a lawful purpose and appropriate safeguards.
• Set page, depth, and concurrency limits that are proportionate to the website.
• Secure exported datasets and delete them when they are no longer needed.

This Actor provides technical extraction tooling, not legal advice.

Related Automation Lab Actors

• Multi-page On-page SEO Audit Crawler for deterministic titles, indexability, content signals, and SEO issue flags.
• Sitewide Broken Link Checker for validating internal and external link targets.
• Multi-Site Article Content Extractor when you already have article URLs and need article-specific fields.
• Website HTML & Text Change Monitor for persisted versioned snapshots and machine-readable additions or removals.

FAQ

Is this a Google website crawler?

No. It crawls public pages from the URLs you supply. It does not imitate Googlebot, submit pages to Google, or access Google Search Console.

Can it crawl several websites in one run?

Yes. Add up to 20 start URLs. maxPages is shared across the run, and each start URL keeps its own website scope.

Does it render JavaScript?

No. The Actor is optimized for static HTML, server-rendered HTML, plain text, and Markdown. Use a browser-focused crawler when essential content exists only after client-side rendering.

Does it compare runs automatically?

No. contentHash makes comparison efficient, but snapshots are independent. Compare hashes in your database, automation, or scheduled workflow.

Are failed pages charged as items?

No. The item event is charged only for successfully extracted pages with useful cleaned text. A typed error row can still appear in the dataset so failures remain visible.

Can it ignore URL query parameters?

Yes. Query parameters are ignored by default. Enable followQueryParameters only when different query strings represent distinct pages.