How to Build a Web Crawler That Finds Broken Links

A few years ago, I built a small crawler to find broken links on websites.The idea was simple:

Give it a website → crawl the pages → find broken links → tell me where they came from.

It worked.

Until I gave it a larger website.

Then I started running into problems I didn’t expect: out-of-memory errors, slow UI rendering, JavaScript-heavy websites, Puppeteer memory usage, and crawling performance.

Recently, I came back to the project and decided to approach it differently.

Instead of building another dashboard around the crawler, I wanted the simplest possible interface:

npx opencrawler <domain>

That’s how the new OpenCrawler CLI came together.

The original architecture

The first version wasn’t a CLI.

It was built using:

• Next.js
• NestJS
• BullMQ
• Redis
• Puppeteer

The basic architecture looked like this:

Next.js
                   │
                   ▼
                NestJS
                   │
                   ▼
             BullMQ + Redis
                   │
                   ▼
              Crawler Job
                   │
                   ▼
               Puppeteer
                   │
                   ▼
            Broken Link Report

The frontend had two different approaches.

One could crawl from the browser, while the server mode could push a crawling job into BullMQ and process it asynchronously.

At the time, this made sense.

But as I started testing it against larger websites, I discovered that the crawler itself wasn’t necessarily the hardest part.

The surrounding architecture was becoming part of the problem.

Problem #1: Out-of-memory errors

The first major issue was memory.

In the frontend, I was maintaining the discovered pages in React state.

The approach was essentially:

setAllPages(prev => [...prev, newPage])

It looks harmless.

But imagine you’ve already discovered thousands of URLs.

Every time a new URL arrives, you’re creating a new array containing the previous items plus the new one.

Conceptually:

[1]

[1, 2]
[1, 2, 3]
[1, 2, 3, 4]
...
[1, 2, 3, ..., 10000]

As the crawler continued running, this became increasingly expensive.

Eventually:

💥 JavaScript heap out of memory

The crawler could work perfectly against a small website and then fall over when the number of pages increased.

That was my first big lesson:

A crawler is fundamentally a data-management problem as much as it is a networking problem.

I started reducing unnecessary state updates, using Sets for lookups, and batching updates instead of pushing every discovered URL directly into React state.

Problem #2: The UI became slow

The next issue wasn’t the crawler itself.

It was the UI.

The crawler could discover thousands of URLs, and I was rendering them all.

For example:

10 URLs
100 URLs
1,000 URLs
10,000 URLs

The browser doesn’t care that the user can only see maybe 10 rows on the screen.

If React creates thousands of DOM nodes, it still has to manage them.

Scrolling became noticeably slower.

The solution was list virtualization.

Instead of rendering:

10,000 DOM nodes

the UI renders roughly:

10 visible rows

and reuses them as you scroll.

That was another useful lesson:

The amount of data you have is not necessarily the amount of data you should render.

Problem #3: A normal HTTP request doesn’t see everything

Then I hit a different kind of problem.

Consider a React application.

You request:

https://example.com

and receive HTML.

You parse the HTML looking for:

<a href="/pricing">

That works for traditional websites.

But modern applications can generate parts of the page after JavaScript executes.

For example:

Initial HTML
     ↓
React executes
     ↓
Navigation component renders
     ↓
Dropdown appears
     ↓
Links appear

A simple HTTP request doesn’t execute React.

So the crawler may see:

HTML
└── a few links

while the actual browser sees:

Rendered DOM
├── navigation
├── dropdown
├── dynamic links
├── content
└── more links

This is where Puppeteer became important.

Why Puppeteer?

With Puppeteer, the crawler can launch a real browser and allow JavaScript to execute.

Then instead of asking:

“What links exist in the raw HTML?”

we can ask:

“What <a> elements exist in the rendered page?"

The CLI uses:

page.$$eval('a', anchors =>
  anchors.map(a => a.href)
)

So the flow becomes:

URL
 ↓
Headless Chrome
 ↓
JavaScript executes
 ↓
DOM rendered
 ↓
Extract <a> elements
 ↓
Discover URLs

This makes it much more useful for JavaScript-heavy websites.

I also started thinking about multi-threading

At one point I started looking into worker threads and multi-threading.

Node.js runs JavaScript on a main thread, so it was tempting to think:

“Maybe the solution is simply to use multiple threads.”

But this turned out to be a useful distinction.

Not every performance problem is a CPU problem.

In my case, I was dealing with several different bottlenecks:

Memory
  ↓
React state management

Rendering
  ↓
Too many DOM nodes
Network
  ↓
Too many requests
Browser
  ↓
Puppeteer / Chrome memory
JavaScript
  ↓
Rendered SPA content

Adding more threads wouldn’t automatically solve those problems.

It reinforced an important engineering principle for me:

Find the actual bottleneck before adding concurrenc

Then I stepped back

The more I worked on the project, the more complicated it became.

The original setup had:

Next.js
NestJS
BullMQ
Redis
Puppeteer
Frontend state
Backend jobs
Email reports

But what was the actual thing I wanted to do?

It was simply:

Scan a website
       ↓
Find broken links
       ↓
Show the result

So I started thinking about the interface again.

What if I didn’t need a dashboard for the basic use case?

What if I could just run:

npx opencrawler example.com

and get the result?

That became the direction for the new version.

The new OpenCrawler CLI

The new version is a standalone Node.js CLI.

The architecture is now much simpler:

npx opencrawler
                     │
                     ▼
                Node.js CLI
                     │
            ┌────────┴────────┐
            │                 │
            ▼                 ▼
        HTTP mode        Puppeteer mode
            │                 │
            ▼                 ▼
        HTML source      Rendered DOM
            │                 │
            └────────┬────────┘
                     ▼
                Link extraction
                     │
                     ▼
                  BFS queue
                     │
                     ▼
              Status checking
                     │
                     ▼
             Broken link report

The CLI starts as a normal Node executable:

#!/usr/bin/env node

and reads arguments directly from the command line.

How the crawler works

The crawling process is essentially a Breadth-First Search (BFS).

Get Naeemudheen p’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

I start with the target domain:

example.com

The queue contains:

example.com

The crawler requests that page.

Suppose it discovers:

/pricing
/blog
/docs

The queue becomes:

/pricing
/blog
/docs

Then it processes those pages and continues discovering new URLs.

Conceptually:

example.com
│
├── /pricing
│   ├── /pricing/enterprise
│   └── /pricing/legacy
│
├── /blog
│   ├── /blog/2025/post-1
│   └── /blog/2025/post-2
│
└── /docs
    ├── /docs/getting-started
    └── /docs/api

The crawler keeps going until the queue is empty or the maximum page limit is reached.

By default, that limit is:

500 pages

and can be changed with:

npx opencrawler example.com --max-pages 1000

The CLI supports this limit directly to prevent an uncontrolled crawl.

Tracking visited URLs

The crawler uses a Set:

const visited = new Set();

Before crawling a URL:

Have I already visited this URL?

If yes, skip it.

If not:

Add to visited
      ↓
Fetch
      ↓
Extract links
      ↓
Add new links to queue

This is important because websites contain many repeated links.

For example:

Home
 ├── Header → /pricing
 ├── Footer → /pricing
 └── CTA    → /pricing

We don’t want to crawl /pricing three times.

URL normalization

Another small but important part is URL normalization.

These URLs may effectively represent the same page:

https://example.com/pricing
https://example.com/pricing/
https://example.com/pricing#plans
https://example.com/pricing?utm_source=test

The crawler removes fragments and query parameters and normalizes trailing slashes before adding URLs to the crawl queue.

This reduces duplicate crawling.

Staying inside the target domain

The crawler is designed to crawl the target domain rather than wandering across the entire internet.

If the target is:

example.com

and a page contains:

example.com/pricing
example.com/docs
github.com/project
google.com

only the URLs belonging to the target origin are added to the crawl.

This is an important boundary for a crawler.

Finding the parent of a broken link

One feature I particularly wanted in the CLI was parent-source tracking.

Finding this:

404 /pricing/legacy

is useful.

But finding this:

404 /pricing/legacy

Parent:
https://example.com/pricing

is much more actionable.

So instead of putting only a URL into the queue, the crawler stores:

{
  url: link,
  parentUrl: currentUrl
}

Now every discovered page knows:

“Which page contained the link that led me here?”

When a page is broken, that relationship is preserved in the report.

The result can look like:

[404] https://example.com/pricing/legacy

↳ Parent Source:
  https://example.com/pricing

This makes fixing the problem much easier.

HTTP mode vs Puppeteer mode

The CLI has two ways to crawl.

1. HTTP mode

Default:

npx opencrawler example.com

It uses Node’s HTTP/HTTPS functionality to request the page and inspect the HTML.

This is lightweight and fast.

The response body is collected and links are extracted from the HTML.

2. Puppeteer mode

For JavaScript-heavy websites:

npx opencrawler example.com --puppeteer

The CLI launches headless Chrome and extracts links from the rendered DOM.

This gives you a useful trade-off:

Simple website
      ↓
HTTP mode
      ↓
Lower overhead

JavaScript-heavy website
      ↓
Puppeteer
      ↓
Real browser rendering

Reducing Puppeteer overhead

A browser is significantly heavier than an HTTP request.

So when Puppeteer is used, the crawler blocks unnecessary resources such as:

Images
Stylesheets
Fonts
Media

The goal is to render enough of the page to discover links without loading resources that aren’t important to the crawler.

The browser instance is reused while the crawl runs, and individual pages are closed after processing.

Request throttling

Another feature I added after thinking about crawling real websites is request throttling.

You can run:

npx opencrawler example.com --delay 200

The crawler waits 200 milliseconds between requests.

The delay is implemented before processing subsequent URLs.

This matters because:

A crawler shouldn’t behave like a denial-of-service tool.

If you’re scanning a website you don’t control, you should also respect its terms, robots policies where applicable, and rate limits.

The CLI therefore includes a reminder that users should only scan domains they own or have permission to test.

The final output

The whole point of simplifying the architecture was to make the result immediately useful.

A completed scan produces a summary containing:

Target Domain
Total Scanned Pages
Valid Pages
Broken Pages
Duration

followed by the broken URLs and their parent sources.

What’s next?

There are still plenty of things I’d like to improve.

For example:

• Better concurrency control
• More efficient queue management
• Sitemap support
• Better handling of redirects
• Robots.txt awareness
• More robust URL normalization
• CI/CD integration
• Better reporting
• More efficient crawling of very large websites
• Improved handling of JavaScript-heavy applications

And there are still performance optimizations to explore.

For example, the current queue checks whether a URL is already queued using a linear search:

queue.some(q => q.url === link)

That works, but on very large crawls it can become another bottleneck.

A dedicated Set for queued URLs would make that lookup constant-time.

So even after getting the CLI working, there are still interesting engineering problems left to solve.

The result

What started as a small broken-link checker eventually became a lesson in:

memory management, crawling algorithms, browser automation, React rendering, concurrency, and system design.

And after going through all of that, the interface I ended up with is surprisingly simple:

Give it a domain. Find the broken links.

That’s the whole idea.

Try it

npx opencrawler example.com

GitHub: https://github.com/naeemudheenp/opencrawler-cli