# Website To Markdown (`smart_api/website-to-markdown`) Actor

Convert any webpage into clean, LLM-ready Markdown in seconds — perfect for AI training data, RAG pipelines, and content archiving.

- **URL**: https://apify.com/smart\_api/website-to-markdown.md
- **Developed by:** [SmartApi](https://apify.com/smart_api) (community)
- **Categories:** AI, Integrations, Other
- **Stats:** 27 total users, 2 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: 5.00 out of 5 stars

## Pricing

from $9.00 / 1,000 webpages

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Website to Markdown - LLM-Ready Content Extractor

Transform any webpage into clean, structured Markdown perfect for AI/LLM applications, content archiving, and documentation.

***

### ⚡ Quick Start (30 seconds)

1. Click **Start** in the Apify Console
2. Paste your URLs
3. Click **Run**

**That's it!** No configuration needed.

```json
{
  "urls": ["https://example.com/article"]
}
```

***

### 💰 Pricing

**Pay-Per-Result** - You only pay for what you use.

| Usage | Estimated Cost |
|-------|----------------|
| 100 pages | $1.19 |
| 1,000 pages | $11.99 |
| 10,000 pages | $119.90 |

***

### ✨ What This Actor Does

- ✅ Extract main content from any webpage (articles, blogs, documentation)
- ✅ Convert HTML to clean, well-formatted Markdown
- ✅ Preserve headings, links, images, lists, and tables
- ✅ Remove ads, navigation, footers, and clutter automatically
- ✅ Process multiple URLs concurrently (up to 100 at once)
- ✅ Handle JavaScript-rendered pages with Playwright
- ✅ Bypass bot detection with stealth mode
- ✅ Retry failed requests automatically with exponential backoff

***

### 📊 Sample Output

This is exactly what you'll get:

```json
{
  "recordType": "success",
  "url": "https://example.com/article",
  "finalUrl": "https://example.com/article",
  "title": "Scientists Print Working Electrodes Directly on Skin With Light",
  "markdown": "# Scientists Print Working Electrodes Directly on Skin With Light\n\n...",
  "markdownLength": 4521,
  "processingTimeMs": 2340,
  "timestamp": "2025-01-15T10:30:00.000Z",
  "metadata": {
    "description": "A new study shows visible light can form electrodes from conductive plastics",
    "lang": "en",
    "wordCount": 892,
    "imageCount": 4,
    "linkCount": 12,
    "headingCount": 5
  }
}
```

The `markdown` field contains clean, structured content ready for LLM processing, RAG pipelines, or documentation.

***

### 🎯 Use Cases

#### 🤖 AI & LLM Training Data

Collect clean text from web sources for fine-tuning models, building knowledge bases, or RAG applications.

#### 📚 Documentation Archiving

Convert online documentation, wikis, and help centers into portable Markdown files.

#### 📰 Content Aggregation

Gather articles from multiple news sources for analysis, summarization, or newsletters.

#### 🔬 Research & Analysis

Extract research papers, blog posts, and technical content for systematic analysis.

#### 📝 Blog Migration

Move content from one platform to another by extracting articles as Markdown.

#### 🗂️ Knowledge Management

Build searchable knowledge bases from scattered web resources.

***

### 📝 How to Use

#### Step 1: Prepare Your URLs

Gather the URLs you want to convert. Each URL should:

- Start with `http://` or `https://`
- Point to a page with readable content (articles, blogs, docs)

**Examples:**

```
https://docs.python.org/3/tutorial/index.html
https://blog.example.com/my-article
https://news.ycombinator.com/item?id=12345
```

#### Step 2: Configure the Actor

In the Apify Console, enter your URLs in the **URLs** field. You can:

- Paste URLs one per line
- Use the JSON editor for bulk input

**Minimal configuration:**

```json
{
  "urls": [
    "https://example.com/page1",
    "https://example.com/page2"
  ]
}
```

#### Step 3: Adjust Settings (Optional)

For most websites, **default settings work great**. Adjust only if needed:

- **Max Concurrency**: Increase for faster processing (default: 5)
- **Stealth Mode**: Keep ON for protected sites (default: ON)
- **Proxy**: Enable for rate-limited or geo-restricted sites

#### Step 4: Run the Actor

1. Click the **Start** button
2. Watch the **Log** tab for progress
3. Wait for completion (usually seconds per page)

#### Step 5: Download Results

When finished, go to the **Storage** tab:

- **Dataset**: Click to view all results
- **Export**: Download as JSON, CSV, or Excel
- **API**: Use the dataset ID in your applications

***

### 🔧 Input Parameters (Full Reference)

| Parameter | Required | Type | Default | What It Does |
|-----------|----------|------|---------|--------------|
| `urls` | YES | Array | — | List of webpage URLs to convert to Markdown |
| `maxConcurrency` | No | Integer | 5 | How many pages to process simultaneously (1-100) |
| `pageLoadTimeout` | No | Integer | 30000 | Max time to wait for page load in milliseconds (5000-120000) |
| `maxRetries` | No | Integer | 3 | Retry attempts for failed pages (0-10) |
| `stealthMode` | No | Boolean | true | Enable anti-bot-detection techniques |
| `proxyConfiguration` | No | Object | — | Proxy settings for accessing restricted content |

#### About Stealth Mode

Stealth mode helps access websites that block automated requests:

- Randomizes request timing
- Uses browser fingerprint rotation
- Mimics human browsing patterns

**Keep it ON** for most websites. Only disable for trusted internal sites where speed is critical.

#### About Proxy Configuration

Use proxies when:

- Websites rate-limit your requests
- Content is geo-restricted
- You need to distribute requests across IPs

Configure in the Apify Console using the built-in proxy selector.

***

### 📤 Output Data (What You Get)

Every result includes these fields:

#### ✅ Success Records

| Field | Type | What It Contains |
|-------|------|------------------|
| `recordType` | Text | Always `"success"` for successful extractions |
| `url` | Text | The original URL you provided |
| `finalUrl` | Text | The actual URL after any redirects |
| `title` | Text | Page title from the `<title>` tag |
| `markdown` | Text | The extracted content as Markdown |
| `markdownLength` | Number | Character count of the Markdown content |
| `processingTimeMs` | Number | How long extraction took in milliseconds |
| `timestamp` | Text | When the extraction happened (ISO 8601) |
| `metadata` | Object | Additional page info (description, language, word count, etc.) |

#### ❌ Error Records

| Field | Type | What It Contains |
|-------|------|------------------|
| `recordType` | Text | Always `"error"` for failed extractions |
| `url` | Text | The URL that failed |
| `errorType` | Text | Category: `http_error`, `timeout`, `bot_detection`, etc. |
| `severity` | Text | `"warning"` (retried) or `"error"` (final failure) |
| `message` | Text | Human-readable explanation of what went wrong |
| `httpStatus` | Number | HTTP status code if applicable (404, 500, etc.) |
| `retryCount` | Number | How many retry attempts were made |
| `processingTimeMs` | Number | Time spent before failure |
| `timestamp` | Text | When the error occurred |

***

### 💡 Common Questions

#### ❓ How long does it take to process pages?

Most pages complete in **2-5 seconds**. Complex JavaScript-heavy pages may take up to 30 seconds. Processing 100 pages with default concurrency (5) typically takes **1-2 minutes**.

#### ❓ How much does it cost to run?

Costs depend on page complexity and compute time. Typical costs are around **$0.01 per page**. Enable the free tier or set spending limits in your Apify account settings.

#### ❓ Do I need any API keys or logins?

**No!** This actor works out of the box. No external API keys, no website logins, no configuration files needed.

#### ❓ What if a page fails to load?

The actor automatically retries failed pages up to 3 times (configurable) with exponential backoff. If all retries fail, you'll get an error record explaining what went wrong.

#### ❓ Does this work with JavaScript-heavy websites?

**Yes!** The actor uses Playwright to fully render JavaScript before extraction. Single-page apps (SPAs), React sites, and dynamic content are all supported.

#### ❓ Can I use this commercially?

**Yes!** The extracted content is yours to use. However, always respect the source website's terms of service and copyright.

#### ❓ What happens with paywalled content?

The actor extracts only publicly visible content. It cannot bypass paywalls, login walls, or access restricted content without proper authentication.

#### ❓ Can I process thousands of URLs at once?

**Yes!** Increase `maxConcurrency` (up to 100) for faster processing. For very large jobs, consider using Apify's scheduling and webhook features.

***

### ⚠️ Troubleshooting

#### Problem: "Bot detection" error

**What it means:**
The website identified the request as automated and blocked it.

**Solution:**

1. Make sure **Stealth Mode** is enabled (it's ON by default)
2. Enable **Proxy Configuration** using Apify's residential proxies
3. Reduce **Max Concurrency** to 2-3 to appear more human-like
4. Add delays between requests by lowering concurrency

***

#### Problem: "Timeout" error

**What it means:**
The page took too long to load or render.

**Solution:**

1. Increase **Page Load Timeout** to 60000ms (60 seconds)
2. Check if the URL is valid and the site is online
3. Some pages may require proxy to avoid rate limiting

***

#### Problem: "Extraction failed" error

**What it means:**
The page loaded but the content extractor couldn't find readable content.

**Solution:**

1. Verify the page has actual text content (not just images/video)
2. Some pages (login walls, empty pages) genuinely have no extractable content
3. Check if the URL is correct and the page exists

***

#### Problem: Empty or low-quality Markdown output

**What it means:**
Content was extracted but may not be complete or well-structured.

**Solution:**

1. Check the `markdownLength` and `metadata.wordCount` fields to assess content size
2. Pages with unusual HTML structure may need manual review
3. The content extractor works best with article-style pages

***

#### Problem: "HTTP 403/404/500" errors

**What it means:**

- **403**: Access forbidden (blocked or requires authentication)
- **404**: Page not found (URL is wrong or page deleted)
- **500**: Server error (website is having problems)

**Solution:**

1. For 403: Try enabling proxy; the site may be blocking your region
2. For 404: Verify the URL is correct
3. For 500: Wait and retry; the website may be temporarily down

***

**Still stuck?** Check the error message details in your results. Most issues can be resolved by enabling proxies or adjusting timeout settings.

***

### 🌍 Proxy Configuration

For most users: **Default settings work fine.**

**If you need proxies:**

1. Go to the **Input** tab
2. Scroll to **Proxy Configuration**
3. Select from Apify's proxy options:
   - **Datacenter proxies**: Fast and cheap, good for most sites
   - **Residential proxies**: Better success rate for protected sites

**Example configuration:**

```json
{
  "urls": ["https://example.com/"],
  "proxyConfiguration": {
    "useApifyProxy": true,
    "apifyProxyGroups": ["RESIDENTIAL"]
  }
}
```

**Cost implications:**
Residential proxies cost more but have higher success rates. Start with datacenter proxies and upgrade only if you see bot detection errors.

***

### ❌ Limitations

- 🚫 Cannot bypass paywalls or login-protected content
- 🚫 Cannot extract content from PDFs or non-HTML documents
- 🚫 Cannot process more than 10,000 URLs per run (split into multiple runs)
- 🚫 Cannot guarantee extraction quality for pages with unusual HTML structure
- 🚫 Cannot access content blocked by geographic restrictions without appropriate proxies

***

### 🚀 Getting Started Now

1. **Copy this example:**

```json
{
  "urls": [
    "https://en.wikipedia.org/wiki/Web_scraping",
    "https://docs.apify.com/academy/web-scraping-for-beginners"
  ]
}
```

2. **Paste** into the Input JSON editor

3. **Click Start**

4. **Wait ~10 seconds**

5. **Download results** from the Storage tab as JSON or CSV

That's it! You're done. 🎉

***

### 📞 Support & Maintenance

**Response time:** We respond within 48 hours

**Contact:** Open an issue on the Actor's page in Apify Console

**How we help:**

- Debug failed extractions
- Optimize settings for your use case
- Answer questions about output format

**Maintenance:**

- We actively monitor for website changes affecting extraction
- Updates are rolled out automatically
- You'll see release notes for major changes

***

### 🔄 Version History

**v0.0** — Initial Release

- Core content extraction with Mozilla Readability
- HTML-to-Markdown conversion with Turndown
- Concurrent URL processing with Crawlee
- Stealth mode for bot detection avoidance
- Automatic retry with exponential backoff
- Rich metadata output including word count, image count, and more

***

*Last updated: December 2025*
*We maintain this actor actively and update as websites and web standards evolve.*

# Actor input Schema

## `urls` (type: `array`):

Array of URLs to scrape and convert to Markdown

## `maxConcurrency` (type: `integer`):

Number of URLs to process simultaneously (1-100)

## `pageLoadTimeout` (type: `integer`):

Maximum time to wait for page load (5000-120000 ms)

## `maxRetries` (type: `integer`):

Retry attempts for transient failures with exponential backoff (0-10)

## `stealthMode` (type: `boolean`):

Use stealth techniques to avoid bot detection (recommended for protected sites)

## `proxyConfiguration` (type: `object`):

Configure proxy settings for requests

## Actor input object example

```json
{
  "urls": [
    "https://example.com/"
  ],
  "maxConcurrency": 5,
  "pageLoadTimeout": 30000,
  "maxRetries": 3,
  "stealthMode": true
}
```

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "urls": [
        "https://example.com/"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("smart_api/website-to-markdown").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "urls": ["https://example.com/"] }

# Run the Actor and wait for it to finish
run = client.actor("smart_api/website-to-markdown").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "urls": [
    "https://example.com/"
  ]
}' |
apify call smart_api/website-to-markdown --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,smart_api/website-to-markdown"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/vZCa45kbxdovtpl8J/builds/LFXa3lspSbET2xVfp/openapi.json