# Domain Scraper (`ib4ngz/domain-scraper`) Actor

This actor scrapes unique domains from a list of provided URLs. It crawls each page, extracts domains, and stores them in a dataset. The actor respects a defined maximum depth and filters domains based on whether they are ICANN-approved and whether private domains are allowed.

- **URL**: https://apify.com/ib4ngz/domain-scraper.md
- **Developed by:** [Iqbal R](https://apify.com/ib4ngz) (community)
- **Categories:** Automation, Developer tools, SEO tools
- **Stats:** 82 total users, 1 monthly users, 96.9% runs succeeded, 1 bookmarks
- **User rating**: 5.00 out of 5 stars

## Pricing

$10.00/month + usage

To use this Actor, you pay a monthly rental fee to the developer. The rent is subtracted from your prepaid usage every month after the free trial period. You also pay for the Apify platform usage, which gets cheaper the higher Apify subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#rental-actors

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Domain Scraper

This actor scrapes domains from a list of provided URLs. It recursively crawls the pages, extracts unique domains, and stores them in a dataset. The actor respects a defined maximum depth and filters domains based on whether they are ICANN-approved and whether private domains are allowed. Only unique domains are saved, preventing duplicates during the crawling process.

### Features

- **Domain Extraction** : Extracts domains from a list of provided URLs and recursively explores linked pages to gather additional domains.
- **Recursive Crawling**: Crawls web pages to a user-defined maximum depth, enabling detailed exploration while managing resource usage.
- **Domain Filtering**: Processes domains based on ICANN approval and user-defined preferences for private domains.
- **Unique Dataset**: Ensures only unique domains are saved by preventing duplicates during the crawling process.

### Input Schema

- **Start URLs** (required): A list of URLs to start crawling from.
- **Maximum Depth**: The maximum depth for crawling, defining how deep the crawler should explore.
- **Allow Private Domains**: Option to enable or disable crawling of private domains.
- **ICANN Domains Only**: Option to restrict processing to ICANN-approved domains only.
- **Proxy Configuration**: Configuration settings for selecting and using proxies during crawling.
- **Minimum Concurrency**: The minimum number of concurrent requests or pages to process.
- **Maximum Concurrency**: The maximum number of concurrent requests or pages to process.

### Dataset Schema

- **domain**: The full domain name.
- **domainWithoutSuffix**: The domain without the public suffix (e.g., example from example.com).
- **hostname**: The hostname of the domain.
- **isIcann**: Indicates whether the domain is ICANN-approved (boolean).
- **publicSuffix**: The public suffix of the domain (e.g., .com, .org).
- **isPrivate**: Indicates whether the domain is a private domain (boolean).
- **subdomain**: The subdomain part of the domain (e.g., sub.example.com).

### How to Use

1. **Set up the Actor**\
   Start by providing a list of URLs to begin the crawling process. You can either manually input the URLs or provide a list in the actor configuration.

2. **Configure the Input Parameters**

   - **Start URLs**: Provide the initial URLs from which the crawler will start.
   - **Maximum Depth**: Define how deep the crawler should explore.
   - **Allow Private Domains**: Choose whether to allow crawling of private domains.
   - **ICANN Domains Only**: Set whether to crawl only ICANN-approved domains.
   - **Proxy Configuration**: If necessary, configure the proxy settings for your crawler.
   - **Concurrency**: Adjust the minimum and maximum concurrency based on your needs.

3. **Run the Actor**\
   Once the input parameters are configured, run the actor to start the crawling process. The actor will crawl the pages, extract unique domains, and store the results in the dataset.

4. **View Results**\
   After the actor finishes running, you can view the extracted domains in the dataset. The data will be displayed in a table format with the following fields:

   - Domain
   - Domain Without Suffix
   - Hostname
   - ICANN Domain
   - Public Suffix
   - Private Domain
   - Subdomain

5. **Export Data**\
   You can export the dataset for further processing or analysis. The results are saved in a structured format for easy integration with other tools.

6. **Modify Parameters**\
   Adjust the configuration and rerun the actor as needed to gather additional data or refine the crawling process.

### Conclusion

This actor provides an efficient solution for scraping and extracting unique domains from a list of URLs. It recursively crawls the provided pages, extracts domains, and stores them in a dataset. By respecting a defined maximum depth and filtering domains based on ICANN approval and private domain allowance, it ensures only relevant domains are captured.

The actor is optimized to prevent duplicates by saving only unique domains during the crawling process. This makes it a valuable tool for anyone looking to gather domain data in a structured and efficient manner, while maintaining control over the types of domains collected.

# Actor input Schema

## `startUrls` (type: `array`):

URLs to start with.

## `maxDepth` (type: `integer`):

The maximum depth for crawling.

## `allowPrivateDomains` (type: `boolean`):

Whether to allow private domains (default is false).

## `isIcann` (type: `boolean`):

Whether to only process ICANN domains (default is true).

## `crawlerType` (type: `string`):

Select the type of crawler to use.

## `proxyConfiguration` (type: `object`):

Select proxies to be used by your crawler.

## `minConcurrency` (type: `integer`):

The minimum number of concurrent requests or pages being processed.

## `maxConcurrency` (type: `integer`):

The maximum number of concurrent requests or pages being processed.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://apify.com/"
    }
  ],
  "maxDepth": 1,
  "allowPrivateDomains": false,
  "isIcann": true,
  "crawlerType": "Crawlee + Cheerio",
  "minConcurrency": 1,
  "maxConcurrency": 10
}
```

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://apify.com/"
        }
    ],
    "crawlerType": "Crawlee + Cheerio"
};

// Run the Actor and wait for it to finish
const run = await client.actor("ib4ngz/domain-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [{ "url": "https://apify.com/" }],
    "crawlerType": "Crawlee + Cheerio",
}

# Run the Actor and wait for it to finish
run = client.actor("ib4ngz/domain-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://apify.com/"
    }
  ],
  "crawlerType": "Crawlee + Cheerio"
}' |
apify call ib4ngz/domain-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,ib4ngz/domain-scraper"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/Xw9DVMfmjwja3ZHja/builds/dPvyJXGo1fMCDGeK6/openapi.json