# Website Intelligence API (`ladra/website-intelligence-api`) Actor

Crawl any public website and turn it into AI-ready intelligence. Extract screenshots, Markdown, HTML, metadata, links, PDFs, compliance evidence, RAG chunks, and structured JSON for sales research, audits, website snapshots, and automation.

- **URL**: https://apify.com/ladra/website-intelligence-api.md
- **Developed by:** [Ladra Software](https://apify.com/ladra) (community)
- **Categories:** AI, Developer tools, SEO tools
- **Stats:** 3 total users, 2 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $2.50 / 1,000 results

This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Website Intelligence API

Turn any public website into structured, AI-ready intelligence.

Website Intelligence API crawls and renders public pages with Playwright, then returns clean website data for research, RAG pipelines, sales workflows, compliance checks, audits, screenshots, and automation.

### Quick start

Use a high-level `intent` instead of configuring dozens of crawler settings:

```json
{
  "url": "https://example.com/",
  "intent": "sales_research"
}
```

For advanced control, override the preset:

```json
{
  "url": "https://example.com/",
  "intent": "rag",
  "advanced": {
    "maxPages": 25,
    "includeScreenshots": false
  }
}
```

### Supported intents

| Intent | Use it for |
|---|---|
| `website_snapshot` | Screenshots, HTML, Markdown, metadata, PDFs, and page assets |
| `rag` | Markdown, chunks, headings, metadata, and internal links |
| `sales_research` | Company info, products, pricing pages, contact details, and socials |
| `documentation` | Docs pages, navigation, tutorials, code-oriented content, and search data |
| `ecommerce` | Product, category, pricing, review, and image discovery |
| `compliance` | Privacy, Terms, Cookie Policy, DPA, Subprocessors, Security, and Trust Center evidence |
| `website_clone` | Visual structure, colors, fonts, screenshots, layouts, navigation, and assets |

### What you get

- Structured `extract-result.json`
- Per-page dataset rows
- Markdown and optional rendered HTML
- Metadata, headings, links, CTAs, and page classifications
- Desktop, mobile, light mode, dark mode, and thumbnail screenshots
- Optional PDFs
- Optional RAG chunks
- Compliance evidence reports
- Snapshot report HTML
- Optional change detection and visual diff reports

### Example output

```json
{
  "company": {},
  "products": [],
  "pricing": [],
  "navigation": [],
  "pages": [],
  "faq": [],
  "contact": {},
  "socials": {},
  "metadata": {},
  "markdown": "",
  "screenshots": {},
  "screenshot": "",
  "pdf": "",
  "html": ""
}
```

### Common use cases

- Build a website knowledge base for RAG
- Research a company before a sales call
- Capture compliance evidence from public pages
- Export screenshots, PDFs, Markdown, and structured JSON
- Monitor public website changes
- Analyze site structure for redesign, migration, or clone workflows

### Advanced options

Most users only need `url` and `intent`.

Advanced users can control crawl size, screenshots, PDFs, Markdown, HTML, wait strategy, concurrency, resource profile, asset loading, visual diff, and change detection through the `advanced` object.

### Notes

This Actor is designed for public websites. It respects robots.txt by default and is intended for website research, public web intelligence, compliance preparation, and AI-ready data extraction workflows.

# Actor input Schema

## `startUrl` (type: `string`):

The website URL where crawling starts. Legacy alias for url.

## `url` (type: `string`):

Website URL for the v2 extract API shape. If startUrl is omitted, this value is used.

## `intent` (type: `string`):

High-level extraction intent. Existing low-level options can still override this preset.

## `mode` (type: `string`):

Snapshot mode controls defaults and summary outputs.

## `qualityMode` (type: `string`):

Preset for speed and output depth. Fast favors text outputs, balanced captures desktop screenshots, complete enables richer report defaults.

## `crawlStrategy` (type: `string`):

Controls how URLs are selected. Compliance-first prioritizes legal, security, trust, status, contact, and support pages.

## `maxPages` (type: `integer`):

Maximum number of pages to capture.

## `crawlDepth` (type: `integer`):

Maximum link depth from the start URL.

## `sameDomainOnly` (type: `boolean`):

Only crawl URLs on the start URL hostname.

## `useSitemap` (type: `boolean`):

Seed the crawler with URLs from sitemap.xml when available.

## `urls` (type: `array`):

Optional explicit URLs to capture in addition to the start URL. These bypass include URL patterns but still respect same-domain, asset, and robots filters.

## `respectRobotsTxt` (type: `boolean`):

Read robots.txt, use its sitemap declarations, and skip disallowed URLs.

## `maxSitemapUrls` (type: `integer`):

Maximum URLs to collect from sitemap files before applying max page limits.

## `includeUrlPatterns` (type: `array`):

Optional path or URL fragments that URLs must match.

## `excludeUrlPatterns` (type: `array`):

Optional path or URL fragments to skip.

## `includeScreenshots` (type: `boolean`):

Capture desktop full-page screenshots.

## `includeMobileScreenshots` (type: `boolean`):

Capture an additional mobile full-page screenshot.

## `includeLightModeScreenshots` (type: `boolean`):

Capture additional screenshots after forcing the page to light color scheme.

## `includeDarkModeScreenshots` (type: `boolean`):

Capture additional screenshots after forcing the page to dark color scheme.

## `screenshotFullPage` (type: `boolean`):

Capture full-page screenshots. Disable to capture only the visible viewport.

## `includeScreenshotThumbnails` (type: `boolean`):

Create viewport thumbnail images for each captured screenshot variant.

## `includeHtml` (type: `boolean`):

Save rendered page HTML files.

## `includeMarkdown` (type: `boolean`):

Extract readable page content and save it as Markdown.

## `includePdf` (type: `boolean`):

Capture each page as a PDF when supported.

## `includeMetadata` (type: `boolean`):

Extract titles, descriptions, headings, links, CTAs, hashes, and page type.

## `includeRagChunks` (type: `boolean`):

Create simple AI-ready text chunks from Markdown.

## `enableAiAnalysis` (type: `boolean`):

Run optional AI-style page analysis tasks such as webpage summaries and CTA highlighting.

## `aiProvider` (type: `string`):

Choose heuristic local analysis or OpenAI Responses API analysis.

## `aiTasks` (type: `array`):

AI tasks to run. Supported values: summarize\_page, highlight\_cta.

## `aiPrompt` (type: `string`):

Optional natural-language prompt, for example: Highlight the CTA, or Summarize this webpage.

## `aiModel` (type: `string`):

OpenAI model used when AI provider is OpenAI.

## `openaiApiKey` (type: `string`):

Optional OpenAI API key. If omitted, the Actor uses the OPENAI\_API\_KEY environment variable.

## `aiMaxInputChars` (type: `integer`):

Maximum page text characters sent to the AI provider per page.

## `removeCookieBanners` (type: `boolean`):

Attempt to accept, reject, close, or remove common cookie consent overlays before capture.

## `removePopups` (type: `boolean`):

Attempt to close or remove common modal, popup, newsletter, and overlay elements before capture.

## `detectBrokenLinks` (type: `boolean`):

Check links found on captured pages and report failed or HTTP 4xx/5xx links.

## `checkExternalLinks` (type: `boolean`):

When broken-link detection is enabled, also check outbound external links.

## `maxLinksToCheckPerPage` (type: `integer`):

Caps broken-link checks per captured page.

## `maxConcurrency` (type: `integer`):

Requested maximum pages to crawl at the same time. The resource profile controls how strictly this is capped.

## `resourceProfile` (type: `string`):

Controls how aggressively the Actor manages memory and concurrency. Safe caps heavy runs conservatively, speed allows more concurrency, and manual honors maxConcurrency with warnings.

## `assetLoading` (type: `string`):

Controls image, media, and font requests. Auto blocks heavy assets only for text-only runs, full loads all assets, and block heavy blocks them even if screenshots are enabled.

## `includeZipArchive` (type: `boolean`):

Create a ZIP archive containing generated screenshots, Markdown, JSON, CSV, and summary files.

## `enableChangeDetection` (type: `boolean`):

Compare this run against a previous snapshot provided as JSON input or URL.

## `previousSnapshot` (type: `object`):

Previous dataset export or object containing pages/items/capturedPages for hash-based comparison.

## `previousSnapshotUrl` (type: `string`):

Optional URL returning previous snapshot JSON for hash-based comparison.

## `webhookUrl` (type: `string`):

Optional endpoint that receives a POST when the snapshot completes.

## `enableVisualDiff` (type: `boolean`):

Compare current desktop screenshots against previous snapshot screenshot URLs.

## `visualDiffThreshold` (type: `number`):

Pixelmatch threshold from 0 to 1. Lower values are more sensitive.

## `includeAuditReport` (type: `boolean`):

Generate audit-report.html and audit-report.pdf summary files.

## `detectCompliancePages` (type: `boolean`):

Classify privacy, legal, security, trust, and related pages.

## `detectMissingPages` (type: `boolean`):

Create a missing-page checklist in compliance evidence mode.

## `waitUntil` (type: `string`):

Page load state to wait for before capture.

## `advanced` (type: `object`):

Optional lower-level overrides for intent presets, such as includePdf, includeScreenshots, waitUntil, resourceProfile, assetLoading, maxConcurrency, enableVisualDiff, and enableChangeDetection.

## Actor input object example

```json
{
  "startUrl": "https://example.com/",
  "url": "https://example.com/",
  "intent": "website_snapshot",
  "mode": "snapshot",
  "qualityMode": "balanced",
  "crawlStrategy": "auto",
  "maxPages": 1,
  "crawlDepth": 2,
  "sameDomainOnly": true,
  "useSitemap": true,
  "respectRobotsTxt": true,
  "maxSitemapUrls": 5000,
  "excludeUrlPatterns": [
    "/cart",
    "/checkout",
    "/account",
    "/wp-admin"
  ],
  "includeScreenshots": true,
  "includeMobileScreenshots": false,
  "includeLightModeScreenshots": false,
  "includeDarkModeScreenshots": false,
  "screenshotFullPage": true,
  "includeScreenshotThumbnails": false,
  "includeHtml": false,
  "includeMarkdown": true,
  "includePdf": false,
  "includeMetadata": true,
  "includeRagChunks": false,
  "enableAiAnalysis": false,
  "aiProvider": "heuristic",
  "aiTasks": [],
  "aiModel": "gpt-5.5",
  "aiMaxInputChars": 12000,
  "removeCookieBanners": true,
  "removePopups": true,
  "detectBrokenLinks": false,
  "checkExternalLinks": false,
  "maxLinksToCheckPerPage": 50,
  "maxConcurrency": 1,
  "resourceProfile": "safe",
  "assetLoading": "auto",
  "includeZipArchive": false,
  "enableChangeDetection": false,
  "enableVisualDiff": false,
  "visualDiffThreshold": 0.1,
  "includeAuditReport": false,
  "detectCompliancePages": true,
  "detectMissingPages": true,
  "waitUntil": "load"
}
```

# Actor output Schema

## `results` (type: `string`):

Default dataset rows, one per captured/analyzed webpage.

## `extractResult` (type: `string`):

Stable v2 structured JSON wrapper with company, pages, navigation, contact, socials, metadata, markdown, screenshots, and output links.

## `crawlSummary` (type: `string`):

Run summary with selected intent, resource plan, warning list, counts, evidence status, and report links.

## `snapshotReport` (type: `string`):

Browser-friendly HTML report with summary metrics, warnings, evidence status, screenshots, and page links.

## `files` (type: `string`):

All records generated in the default key-value store, including screenshots, Markdown, HTML, PDFs, CSV files, and reports.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrl": "https://example.com/",
    "url": "https://example.com/",
    "intent": "website_snapshot",
    "maxPages": 1,
    "crawlDepth": 0,
    "useSitemap": false,
    "includeScreenshots": false,
    "includePdf": false,
    "includeZipArchive": false,
    "waitUntil": "load"
};

// Run the Actor and wait for it to finish
const run = await client.actor("ladra/website-intelligence-api").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrl": "https://example.com/",
    "url": "https://example.com/",
    "intent": "website_snapshot",
    "maxPages": 1,
    "crawlDepth": 0,
    "useSitemap": False,
    "includeScreenshots": False,
    "includePdf": False,
    "includeZipArchive": False,
    "waitUntil": "load",
}

# Run the Actor and wait for it to finish
run = client.actor("ladra/website-intelligence-api").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrl": "https://example.com/",
  "url": "https://example.com/",
  "intent": "website_snapshot",
  "maxPages": 1,
  "crawlDepth": 0,
  "useSitemap": false,
  "includeScreenshots": false,
  "includePdf": false,
  "includeZipArchive": false,
  "waitUntil": "load"
}' |
apify call ladra/website-intelligence-api --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,ladra/website-intelligence-api"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/IGhQFufrs0zQu23LM/builds/cJ2iHjsBVrcxZgaAn/openapi.json