# Website Links Graph Generator (`crawlerbros/web-link-graph-visualizer`) Actor

Creates an oriented graph visualizing links between webpages. Outputs: graph.png (visual network diagram) and graph.json (structured data) saved to Key-Value Store, plus detailed dataset of all crawled pages. Configure depth, boundaries, and layout.

- **URL**: https://apify.com/crawlerbros/web-link-graph-visualizer.md
- **Developed by:** [Crawler Bros](https://apify.com/crawlerbros) (community)
- **Categories:** Developer tools, SEO tools, Automation
- **Stats:** 62 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: 5.00 out of 5 stars

## Pricing

from $1.00 / 1,000 results

This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Web Link Graph Visualizer

**Creates oriented graphs visualizing links between webpages**

Crawl a website starting from a URL, extract all links, build a directed graph of the link structure, and export it as a PNG image or JSON file.

***

### 📥 What You'll Get

After the actor completes, you'll receive:

#### 🖼️ **graph.png** - Visual Network Diagram

- **Location:** Key-Value Store → `graph.png`
- **Format:** High-resolution PNG (2000x1600px)
- **Content:** Visual graph with color-coded nodes and directed edges
- **Download:** Click "Actions" → "Download" in Key-Value Store tab

#### 📊 **graph.json** - Structured Data

- **Location:** Key-Value Store → `graph.json`
- **Format:** JSON file with complete graph structure
- **Content:** All nodes, edges, and statistics
- **Use:** Import into analysis tools or custom visualizations

#### 📑 **Dataset** - All Crawled Pages

- **Location:** Dataset tab (Storage section)
- **Format:** JSON records (one per page)
- **Content:** URL, title, depth, all links per page
- **Export:** CSV, JSON, or Excel from Dataset tab

#### 🔍 **Where to Find in Apify Console:**

1. After actor finishes, go to **"Storage"** section
2. **Key-Value Store tab:**
   - Download `graph.png` (your visual graph image)
   - Download `graph.json` (data for analysis)
3. **Dataset tab:**
   - View/export all crawled pages
   - See links extracted from each page

***

### Features

✅ **Smart Crawling:**

- Start from any URL
- Follow links matching a boundary regex
- Configurable depth and page limits
- Respects robots.txt (via Playwright)
- Adjustable request delays

✅ **Graph Building:**

- Directed graph (oriented edges)
- Track internal vs external links
- URL normalization (remove fragments, trailing slashes)
- Depth tracking for each node
- Duplicate link detection

✅ **Visualization:**

- Multiple layout algorithms (hierarchical, spring, circular, random)
- Customizable node labels (URL, path, title, or index)
- Color-coded nodes (internal=blue, external=red)
- High-resolution PNG export
- JSON export for programmatic use

✅ **Statistics:**

- Total nodes and edges
- Average outgoing links per page
- Max depth reached
- Internal vs external link counts

***

### Input Parameters

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `startUrl` | String | **Required** | The URL to start crawling from |
| `boundaryRegex` | String | `.*` | Regex to limit which URLs to crawl |
| `maxDepth` | Integer | `3` | Maximum crawl depth (1-10) |
| `maxPages` | Integer | `50` | Maximum pages to crawl (1-1000) |
| `exportFormat` | Select | `both` | Output format: `both`, `image`, or `json` |
| `graphLayout` | Select | `hierarchical` | Layout: `hierarchical`, `spring`, `circular`, `random` |
| `nodeLabels` | Select | `path` | Label type: `url`, `path`, `title`, `index` |
| `includeExternal` | Boolean | `true` | Show external links in graph |
| `waitForSelector` | String | - | CSS selector to wait for (optional) |
| `requestDelay` | Integer | `1000` | Delay between requests (ms) |

***

### Example Inputs

#### Example 1: Small Website

```json
{
  "startUrl": "https://example.com/",
  "boundaryRegex": "^https://example\\.com/.*",
  "maxDepth": 2,
  "maxPages": 20,
  "exportFormat": "both",
  "graphLayout": "hierarchical",
  "nodeLabels": "path"
}
```

#### Example 2: Documentation Site

```json
{
  "startUrl": "https://docs.python.org/3/",
  "boundaryRegex": "^https://docs\\.python\\.org/3/tutorial/.*",
  "maxDepth": 3,
  "maxPages": 50,
  "exportFormat": "image",
  "graphLayout": "spring",
  "nodeLabels": "title",
  "includeExternal": false,
  "requestDelay": 500
}
```

#### Example 3: Blog with Subdomains

```json
{
  "startUrl": "https://blog.example.com/",
  "boundaryRegex": "^https://.*\\.example\\.com/.*",
  "maxDepth": 2,
  "maxPages": 30,
  "exportFormat": "both",
  "graphLayout": "circular",
  "nodeLabels": "path"
}
```

***

### Output

#### Dataset

Each crawled page is saved to the dataset with:

- `url` - Page URL
- `title` - Page title
- `depth` - Depth from start URL
- `links` - All extracted links
- `internal_links` - Links matching boundary
- `external_links` - Links outside boundary
- `crawled_at` - Timestamp

#### Key-Value Store

**graph.json** (if JSON export enabled):

```json
{
  "graph": {
    "nodes": [
      {
        "id": "https://example.com/",
        "url": "https://example.com/",
        "title": "Example Domain",
        "depth": 0,
        "is_internal": true,
        "outgoing_links": 3
      }
    ],
    "edges": [
      {
        "source": "https://example.com/",
        "target": "https://example.com/page1"
      }
    ],
    "directed": true
  },
  "statistics": {
    "nodes": 15,
    "edges": 42,
    "crawled_pages": 15,
    "external_links": 3,
    "avg_outgoing_links": 2.8,
    "max_depth_reached": 2
  }
}
```

**graph.png** (if image export enabled):

- High-resolution PNG image (2000x1600px)
- Color-coded nodes (blue=internal, red=external)
- Directed edges with arrows
- Legend and statistics

**OUTPUT**:

```json
{
  "start_url": "https://example.com/",
  "statistics": {
    "nodes": 15,
    "edges": 42,
    "crawled_pages": 15
  },
  "exports": {
    "json": true,
    "image": true
  }
}
```

***

### Boundary Regex Examples

| Pattern | Matches |
|---------|---------|
| `^https://example\\.com/.*` | All pages on example.com |
| `^https://example\\.com/blog/.*` | Only blog section |
| `^https://.*\\.example\\.com/.*` | All subdomains |
| `^https://example\\.com/(?!admin).*` | Exclude admin section |
| `.*` | Everything (no boundary) |

***

### Use Cases

🔍 **SEO Analysis:**

- Visualize site structure
- Find orphan pages
- Identify link depth issues

📊 **Content Strategy:**

- Map content relationships
- Find hub pages
- Identify external dependencies

🔗 **Link Building:**

- Discover internal linking opportunities
- Find broken link paths
- Analyze link distribution

🛠️ **Site Migration:**

- Document current structure
- Plan URL redirects
- Validate link integrity

***

### Graph Layouts

#### Hierarchical (Default)

Best for: Sites with clear hierarchy (docs, blogs)

- Top-down structure
- Shows depth clearly

#### Spring (Force-Directed)

Best for: Discovering clusters

- Nodes repel/attract based on connections
- Reveals natural groupings

#### Circular

Best for: Small sites

- Nodes arranged in a circle
- Shows connections clearly

#### Random

Best for: Quick visualization

- Fast to generate
- Good for dense graphs

***

### Node Label Types

| Type | Example | Best For |
|------|---------|----------|
| `url` | `https://example.com/page` | Small graphs |
| `path` | `/blog/post-title` | Medium graphs (default) |
| `title` | `My Blog Post` | Readable labels |
| `index` | `1`, `2`, `3` | Large graphs |

***

### Performance Tips

1. **Start Small:**
   - Use `maxPages: 20` for initial runs
   - Increase gradually

2. **Tight Boundaries:**
   - Use specific regex patterns
   - Avoid crawling entire domains

3. **Adjust Depth:**
   - Depth 2-3 is usually sufficient
   - Depth 4+ can explode exponentially

4. **Request Delays:**
   - Use 1000ms+ for courtesy
   - Reduce for fast sites

5. **External Links:**
   - Set `includeExternal: false` for cleaner graphs
   - Enable to see dependencies

***

### Limitations

- **Max Pages:** 1000 (configurable limit)
- **Max Depth:** 10 (configurable limit)
- **JavaScript:** Rendered via Playwright (may be slow)
- **Image Size:** Large graphs (100+ nodes) may have small labels

***

### Technical Details

**Built With:**

- Python 3.11
- Apify SDK
- Playwright (browser automation)
- BeautifulSoup4 (HTML parsing)
- NetworkX (graph algorithms)
- Matplotlib (visualization)

**Graph Type:**

- Directed graph (DiGraph)
- Nodes = URLs
- Edges = Links (from → to)

**URL Normalization:**

- Removes fragments (#section)
- Removes trailing slashes
- Preserves query strings
- Converts relative to absolute

***

### Example Output

#### Small Site (10 pages)

```
Nodes: 10
Edges: 28
Crawled pages: 10
External links: 3
Avg links per page: 2.8
Max depth reached: 2
```

#### Documentation Site (50 pages)

```
Nodes: 53 (50 internal + 3 external)
Edges: 142
Crawled pages: 50
External links: 3
Avg links per page: 2.7
Max depth reached: 3
```

***

### Troubleshooting

**Issue: No links found**

- Check `waitForSelector` for dynamic sites
- Verify boundary regex matches start URL

**Issue: Too many nodes**

- Reduce `maxPages` or `maxDepth`
- Tighten boundary regex

**Issue: Image labels too small**

- Use `nodeLabels: "index"` for large graphs
- Reduce number of nodes

**Issue: Slow crawling**

- Reduce `requestDelay`
- Decrease `maxPages`
- Check site performance

***

### Support

For issues or questions:

1. Check input parameters
2. Verify boundary regex
3. Test with small `maxPages` first
4. Review dataset for crawl results

***

### License

MIT License - Free for commercial and personal use

***

*Built with ❤️ using Apify SDK*

# Actor input Schema

## `startUrl` (type: `string`):

The URL to start crawling from

## `boundaryRegex` (type: `string`):

Regular expression to limit which URLs to crawl. Only URLs matching this pattern will be followed.

## `maxDepth` (type: `integer`):

Maximum crawl depth from the start URL

## `maxPages` (type: `integer`):

Maximum number of pages to crawl

## `exportFormat` (type: `string`):

Output format for the graph

## `graphLayout` (type: `string`):

Layout algorithm for graph visualization

## `nodeLabels` (type: `string`):

What to show as node labels

## `includeExternal` (type: `boolean`):

Show external links in the graph (won't be crawled, just displayed)

## `waitForSelector` (type: `string`):

Optional CSS selector to wait for before extracting links (for JavaScript-heavy sites)

## `requestDelay` (type: `integer`):

Delay between requests in milliseconds (be nice to servers)

## Actor input object example

```json
{
  "startUrl": "https://example.com/",
  "boundaryRegex": "^https://example\\.com/.*",
  "maxDepth": 3,
  "maxPages": 50,
  "exportFormat": "json",
  "graphLayout": "hierarchical",
  "nodeLabels": "path",
  "includeExternal": true,
  "waitForSelector": "body",
  "requestDelay": 1000
}
```

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrl": "https://example.com/",
    "maxDepth": 1,
    "maxPages": 3,
    "exportFormat": "json",
    "requestDelay": 500
};

// Run the Actor and wait for it to finish
const run = await client.actor("crawlerbros/web-link-graph-visualizer").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrl": "https://example.com/",
    "maxDepth": 1,
    "maxPages": 3,
    "exportFormat": "json",
    "requestDelay": 500,
}

# Run the Actor and wait for it to finish
run = client.actor("crawlerbros/web-link-graph-visualizer").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrl": "https://example.com/",
  "maxDepth": 1,
  "maxPages": 3,
  "exportFormat": "json",
  "requestDelay": 500
}' |
apify call crawlerbros/web-link-graph-visualizer --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,crawlerbros/web-link-graph-visualizer"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/TtddA9bvLVJo8vE7K/builds/Qczxv2iqlF6NE2MGi/openapi.json