# Website URL Finder by Pattern (Recent & Historical) (`igolaizola/website-url-finder`) Actor

Discover recent and historical website URLs by domain, subdomain, or path pattern, with older pages, orphan URLs, and content no longer linked. Search the 10 latest snapshots or choose a year for SEO audits, migrations, content inventories, competitor research, lead generation, and data pipelines

- **URL**: https://apify.com/igolaizola/website-url-finder.md
- **Developed by:** [Iñigo Garcia Olaizola](https://apify.com/igolaizola) (community)
- **Categories:** SEO tools, Developer tools, Integrations
- **Stats:** 4 total users, 0 monthly users, 96.9% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.70 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Website URL Finder (Recent & Historical)

Discover website URLs from recent and historical web records using a domain, subdomain, or path pattern. Build clean URL lists for active sites while also uncovering older pages, orphan URLs, and content that is no longer linked.

### 🤖 What does Website URL Finder do?

Enter patterns such as `example.com/*`, `example.com/blog/*`, or `*.example.com/*` and receive a deduplicated list of matching URLs. Each result was successfully captured at the reported time, making the dataset useful for both recent website discovery and historical research.

- **Find website URLs** across an entire domain or a specific directory
- **Discover subdomains** with a single wildcard pattern
- **Search recent records** using the 12 newest web snapshots automatically
- **Explore URL history** by searching every available snapshot from a chosen year
- **Reveal orphan and legacy pages** that may no longer appear in navigation or sitemaps
- **Export unique results** as JSON, CSV, Excel, XML, or other Apify dataset formats

**Great for:** SEO audits, site migrations, content inventories, competitor research, lead generation, redirect maps, and data pipelines.

> SEO keywords: **website URL finder**, **find all website URLs**, **URL discovery tool**, **historical URL finder**, **orphan page finder**, **domain URL extractor**, **website URL list**, **Apify Actor**.

### 💡 Why discover website URLs?

- 🔎 **Inventory an existing website** — collect known recent URLs without depending on its current navigation
- 🧭 **Find orphan pages** — reveal URLs that are hard to reach from the live site
- 🚚 **Plan site migrations** — build redirect maps and reduce lost traffic during redesigns
- 📈 **Improve SEO audits** — compare recent and historical paths to identify cleanup opportunities
- 🗂️ **Create datasets** — export focused URL lists for enrichment, monitoring, or research
- 🎯 **Research competitors** — discover product, category, campaign, and content structures

### 🧠 How history works

By default, the Actor searches the **12 newest web snapshots**. This is the best starting point for finding URLs associated with an existing website while retaining some recent history.

Set `historyYear` when you need a deeper historical view. The Actor then searches **all available snapshots from that year**, newest first. Results are deduplicated by exact URL across every pattern and snapshot.

The capture time describes when a URL was successfully recorded. It does not guarantee that the URL is still live or returns the same content today.

### 🚀 How to use

1. **Add the Actor** – Open **Website URL Finder (Recent & Historical)** in Apify Store.
2. **Enter URL patterns** – Add one or more values to `urlPatterns`.
3. **Choose the time range** – Leave `historyYear` empty for recent discovery, or select a year for a historical search.
4. **Set a result limit** – Use `maxResults` for a controlled run, or keep `0` to return every unique match.
5. **Run and download** – Export the dataset as JSON, CSV, Excel, XML, or another supported format.

### 💳 Pricing

Apify's **Free plan** includes monthly credits suitable for small searches. Broad whole-domain patterns or full-year history searches can return many URLs, so start with `maxResults` before running an unlimited job.

### 📝 Input parameters

| Parameter | Type | Required | Description |
| --------- | ---- | -------- | ----------- |
| `urlPatterns` | Array of strings | Yes | Domain, subdomain, or path patterns such as `example.com/*`, `example.com/blog/*`, or `*.example.com/*`. |
| `maxResults` | Integer | No | Maximum number of unique URLs returned. Default `0` returns every match. |
| `historyYear` | Integer | No | Search every available web snapshot from this year. When omitted, the 12 newest snapshots are searched. Minimum `2008`. |

#### Example inputs

**Find up to 1,000 URLs for an existing website**

```json
{
  "urlPatterns": ["example.com/*"],
  "maxResults": 1000
}
```

**Build a historical content inventory for selected paths**

```json
{
  "urlPatterns": [
    "example.com/blog/*",
    "example.com/products/*"
  ],
  "historyYear": 2024,
  "maxResults": 5000
}
```

**Discover URLs across subdomains**

```json
{
  "urlPatterns": ["*.example.com/*"],
  "maxResults": 2500
}
```

### 📊 Output results

Each unique URL produces one dataset item:

- **`matchedPattern`** *(string)* — the first input pattern that matched the URL
- **`url`** *(string)* — the discovered absolute URL
- **`capturedAt`** *(string, optional)* — when the URL was successfully recorded, in ISO 8601 format

```json
{
  "matchedPattern": "example.com/blog/*",
  "url": "https://example.com/blog/website-migration-checklist",
  "capturedAt": "2025-07-18T14:23:09Z"
}
```

### 🧭 Patterns and common recipes

- **Whole website:** use `example.com/*` to discover paths associated with the main domain.
- **One directory:** use `example.com/blog/*` or `example.com/products/*` for a focused inventory.
- **Subdomain homepages:** use `*.example.com` to return only the root URL of each subdomain.
- **All subdomains:** use `*.example.com/*` to find URLs on hosts below the domain.
- **One path across subdomains:** use `*.example.com/blog/*` to keep only `/blog/` URLs on subdomains.
- **Multiple areas:** add several patterns; `maxResults` applies to the combined, deduplicated output.
- **Recent discovery:** omit `historyYear` to use the 12 newest snapshots.
- **Historical research:** set `historyYear: 2022` to search all snapshots available from 2022.

Patterns support subdomain wildcards and path prefixes, not arbitrary regular expressions.

For bare subdomain patterns, `capturedAt` is the capture time of the page that proved the subdomain existed; the derived root URL itself may not have been captured.

### ⚙️ Best practices

- **Start with a limit** such as `maxResults: 1000`, then expand after checking relevance.
- **Use narrow path patterns** when a full domain could contain hundreds of thousands of URLs.
- **Treat results as discoveries, not live checks** because older URLs may now redirect, fail, or contain different content.
- **Run separate years for comparisons** when researching how a site's URL structure changed over time.
- **Expect exact-URL deduplication** across all input patterns and searched snapshots.

### 🌍 Proxy usage

Direct access works for most runs. Proxy settings are available for environments that require them, but they are usually unnecessary for normal URL discovery.

### ⚖️ Legal and ethical considerations

- **Respect website terms, robots rules, and applicable data-use restrictions.**
- **Handle personal information responsibly** and comply with privacy laws such as GDPR and CCPA.
- **Keep search scopes reasonable** and avoid unnecessarily broad automated collection.
- **Preserve attribution and ownership** when publishing or redistributing derived URL datasets.
- **Review usage rights** before using discovered data in commercial products.

You are responsible for ensuring that your use of the results complies with applicable laws and third-party rights.

### ❓ FAQ

**Can it find URLs that currently exist?**

Yes. The newest snapshots are useful for finding URLs associated with existing websites. Because this is snapshot-based discovery rather than a live crawl, verify critical URLs before using them operationally.

**Can it find deleted or unlinked pages?**

Yes. A historical year can reveal older and orphan URLs, including pages that are no longer linked or live today.

**Does it download page content?**

No. It returns a focused list of discovered URLs and their capture times.

**Are duplicate URLs returned?**

No. Each exact URL is emitted once, even if it appears in multiple patterns or snapshots.

**Does `*.example.com/*` include the root domain?**

No. It returns only subdomain URLs such as `www.example.com` and `docs.example.com`. Add `example.com/*` as another pattern when you also want root-domain URLs.

**What does `maxResults: 0` mean?**

It removes the result-count limit. The Actor searches every selected pattern across the chosen recent or historical scope.

**Why might a discovered URL not work today?**

The URL was successfully recorded at `capturedAt`, but websites change. It may now redirect, return an error, or have been removed.

### 🛟 Support

Need another output field, pattern option, or specialized URL-discovery workflow? Open an issue or contact me at https://igolaizola.com/#contact

# Actor input Schema

## `urlPatterns` (type: `array`):

Domain, subdomain, or path patterns to search. Use \*.example.com to discover subdomain roots from captured pages or *.example.com/* to return all captured paths across subdomains.

## `maxResults` (type: `integer`):

Maximum number of unique URLs to return across all patterns. Use 0 to return every match.

## `historyYear` (type: `integer`):

Optional year to search across all available web snapshots from that year. Leave empty to search the 12 newest snapshots.

## Actor input object example

```json
{
  "urlPatterns": [
    "*.example.com/*"
  ],
  "maxResults": 0
}
```

# Actor output Schema

## `results` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "urlPatterns": [
        "*.example.com/*"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("igolaizola/website-url-finder").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "urlPatterns": ["*.example.com/*"] }

# Run the Actor and wait for it to finish
run = client.actor("igolaizola/website-url-finder").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "urlPatterns": [
    "*.example.com/*"
  ]
}' |
apify call igolaizola/website-url-finder --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,igolaizola/website-url-finder"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/5DbYWcungwPRuoF3D/builds/Ltc7BxnYh4pgQc7Bx/openapi.json