I Reverse Engineered Screaming Frog SEO Spider: Meet Crawlwise Spider

Screaming Frog SEO Spider is the tool most technical SEOs open first. You give it a URL, it crawls the site, and it shows you every broken link, missing title, bad canonical and redirect chain. It is a great tool. It is also a closed, paid Java app.

I wanted to know how a tool like that works on the inside. So I studied it with a reverse engineering toolkit, wrote down how it behaves, and then built my own crawler from zero. The result is Crawlwise Spider: a free, open source (MIT) desktop crawler for technical SEO audits, built with Electron, React, TypeScript and SQLite.

This post explains what Crawlwise Spider is, and how I built it.

What is Crawlwise Spider?

Crawlwise Spider is a Windows desktop app. You enter a URL, click Start, and it crawls the site like a search engine bot. Then it shows the SEO issues it found, ranked by priority, and you can export everything to CSV or Excel.

All crawl data stays on your computer. The app stores each crawl in a local SQLite file and sends no telemetry.

Download from here direct exe file: https://github.com/Tajammal-Maqbool/crawlwise-spider/releases/latest

What it can do?

Look at here all list of the features:

The screenshots in this post come from a real crawl of my own site, tajammalmaqbool.com: 453 URLs, with the issues sorted by priority.

How I built it?

Step 1: Study the original, but do not copy it

I used REA (reverse-engineer-anything), a reverse engineering toolkit that runs inside Claude Code as a skill plus an MCP server. I pointed it at Screaming Frog SEO Spider 23.3.

Before I started, I set one hard rule: study the structure and the behaviour, never the code. Copying decompiled code into my own project is a licence risk, and it is also not how you learn. So I drew a clear line:

Allowed:

• Listing the JAR files and the packages inside them.
• Reading class and method signatures and constants (javap -p -constants).
• Mapping the dependencies between modules (jdeps).
• Reading the bundled resources: labels, filter names, config defaults.
• Reading the database schema it uses to store crawls.
• Running a real crawl against a small test site on 127.0.0.1 and watching what it does.

Not allowed:

• Decompilers.
• Bytecode dumps (javap -c or -v).
• Reading any method body.

This is a clean room approach. The output of the research was a set of behaviour specs: plain descriptions such as “a page title over this length gets this warning” or “redirect chains are reported from this tab”. No code moved from their app to mine. Every line of Crawlwise Spider is new.

The research answered the questions that matter for a crawler like this:

• Which tabs and filters exist, and what each filter really checks.
• How a crawl is stored, so you can open a 100,000 URL crawl and still scroll it fast.
• Which settings a crawl needs (scope, depth, rate limits, user agent, rendering).
• How issues are grouped and ranked.

Step 2: Design before code

With the specs ready, I wrote the design in order: the UI layout, then the features by phase, then error handling and testing. I picked a stack where every part runs on the user’s machine:

• Electron for a real desktop app with a Chromium renderer built in.
• React 19 and TypeScript (strict) for the UI.
• SQLite through Node’s built-in node:sqlite, so there is no native database module to compile.
• cheerio for fast HTML parsing, robots-parser for robots.txt, axe-core for accessibility, ExcelJS for exports.

Step 3: The architecture

The app has three processes, plus a worker thread for the heavy work:

src/
  main/       Electron main process: crawler, database, exports, Google, scheduling
  preload/    Safe bridge between the main process and the UI
  renderer/   React user interface
  shared/     Types, constants and helpers used by all processes
  cli/        Command line entry point

A few design choices made the biggest difference:

Get Tajammal Maqbool’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

The crawler runs in a worker thread. The crawl engine (fetch, parse, follow links, store) runs in a worker_threads worker. The UI stays smooth even when the crawler is busy with thousands of pages.

SQLite is the single source of truth. Every crawl is one SQLite file in WAL mode, with tables for URLs, links, headers, hreflang, structured data, sitemap entries, custom values, accessibility results, near duplicates, screenshots and page sources. The grid does not hold the crawl in memory. It asks the database for the rows it needs.

One filter registry for everything. Each tab and each filter (“Page Titles: Missing”, “Response Codes: 4xx”) is defined one time in a shared registry. The grid, the issues report, the exports and the CLI all read the same definitions. A filter can never show one number in the grid and a different number in the export.

Polite crawling. A per host throttle respects the configured rate and the robots.txt crawl-delay. Failed requests retry with backoff. The crawler stays inside the scope you set.

Rendering in a locked down browser. JavaScript rendering uses a pool of hidden Electron windows in a separate session. That session blocks downloads and denies every permission request, so a page you crawl cannot do anything on your machine.

Near duplicates with MinHash. The app breaks each page’s text into 3 word shingles and builds a 128 value MinHash signature. Then it compares signatures, not full pages, so duplicate detection stays fast on big sites.

Your own Google client. The app does not ship a shared OAuth client. You create a free Desktop client one time, and the app signs in with PKCE on a local loopback address. Tokens and API keys are encrypted with the operating system’s secure storage (Electron safeStorage).

Crawlwise Report of a site

Step 4: Build with AI agents, in parallel

I built Crawlwise Spider with Claude Code as my pair programmer. The workflow was strict:

1. Brainstorm the feature and agree on the design.
2. Write a plan with small, testable tasks.
3. Run the tasks with subagents, each in its own git worktree, so independent work runs in parallel without file conflicts.
4. Verify with the real mechanism (the real build, the real crawl, the real database query) before I call anything done.

The codebase follows a strict file layout: types/, interfaces/, constants/, utils/, services/, hooks/ and one component per file. Small files make it easy for both people and agents to find and change the right thing.

The project has about 11,000 lines of TypeScript in src/ and about 160 tests in unit, integration and end-to-end suites (Vitest, plus Playwright for the Electron app).

Step 5: Review, fix, repeat

After the first release, I ran a full code review and split the findings into four branches, one concern each:

• fix/security: move PageSpeed and AI API keys into the main process with safeStorage, harden the render session, sitemap gzip handling and scheduler input, lock main window navigation, and add a single instance lock.
• fix/crawler: correct robots.txt failure handling, crawl-delay, the average crawl rate, stopping during analysis, and stale crawl state after a crash.
• fix/exports: stream CSV and XLSX exports so big crawls do not fill memory, and neutralize spreadsheet formulas so a crawled page cannot inject a formula into your Excel file.
• fix/renderer: loading and error states, focus traps in dialogs, better contrast in the light theme, a context menu that stays on screen, and a debounced grid search.

These branches merged into version 0.1.1.

What I learned

• Reverse engineering is about behaviour, not code. Signatures, resources, schemas and a real test crawl told me almost everything I needed. I never needed a method body.
• The database design is the product. A crawler is only as fast as its storage. Putting every crawl in SQLite and querying only the visible rows is what makes large crawls usable.
• One definition, many consumers. The shared filter registry removed a whole class of “the numbers do not match” bugs.
• AI agents are fast, but verification is not optional. Parallel subagents wrote a lot of code quickly. The review round still found real security and correctness issues. Running the real thing is the only proof.

Try it

• Download for Windows: latest release. There is an installer and a portable exe. The builds are not code signed yet, so Windows SmartScreen can show a warning: click More info, then Run anyway.
• Source code: github.com/Tajammal-Maqbool/crawlwise-spider
• Run from source:

git clone https://github.com/Tajammal-Maqbool/crawlwise-spider.git
cd crawlwise-spider
npm install
npm run dev

If you find a bug or want a feature, open an issue on GitHub.

Crawlwise Spider is an independent project. It is not affiliated with, endorsed by, or connected to Screaming Frog Ltd. “Screaming Frog” and “SEO Spider” are trademarks of their owner. No Screaming Frog code is part of this project.

Written by Tajammal Maqbool, website and game developer.