Building a Highly Concurrent, Dependency-Free Email DNS Auditor in Go

Building a Highly Concurrent, Dependency-Free Email DNS Auditor in Go

When you send or receive an email, a silent and complex dance of protocols happens entirely behind the scenes. Mechanisms like MX, SPF, DMARC, DKIM, BIMI, and MTA-STS exist to verify identity, enforce security policies, and prevent spoofing. However, misconfigurations are incredibly common.

To truly understand how these protocols operate under the hood, I decided to step away from abstract RFC documents and build an operational tool: email-checker.

The goal was clear: build a command-line interface (CLI) tool capable of auditing domain email DNS postures at scale, utilizing a stdlib-first approach with zero external dependencies, driven by Test-Driven Development (TDD), and highly optimized for concurrency.

In this article, I will pull back the curtain on the core architecture, the concurrency hurdles I faced (such as the thundering herd problem), and how sticking tightly to standard library interfaces resulted in a production-grade developer tool.

The Architectural Blueprint: Interface-Driven Design

One of the foundational design choices for email-checker was ensuring that the network layer remained completely decoupled from the validation logic. Instead of hardcoding direct calls to Go's native net package, everything revolves around a composable DNSResolver interface:

type DNSResolver interface {
    LookupTXT(ctx context.Context, name string) ([]string, error)
    LookupMX(ctx context.Context, name string) ([]*net.MX, error)
    // Extended lookups...
}

By enforcing an interface-based boundary, I was able to build a layered resolver stack. Each component has a singular responsibility and wraps the underlying resolver seamlessly:

1. netDNSResolvernetDNSResolver: The base layer executing raw network queries using the Go standard library.
2. RateLimitedResolverRateLimitedResolver: A wrapper that acts as a guard. Because DNS auditing is heavily I/O-bound, spawning thousands of raw requests simultaneously can crash public DNS servers or trigger strict rate limits. This layer uses a buffered channel as a semaphore to cap maximum in-flight requests.
3. CachedDNSResolverCachedDNSResolver: A TTL-backed caching layer that intercepts queries before they hit the network, tracking results locally to optimize throughput.

Solving the Concurrency Bottleneck: The Thundering Herd

When writing highly concurrent Go programs, allocating work across a massive worker pool introduces unique race conditions. In a large batch-processing pipeline, multiple parallel workers might simultaneously attempt to audit sub-dimensions of the exact same root domain.

If that domain’s records aren’t in our CachedDNSResolver yet, every single worker will experience a cache miss at the exact same moment. They will all hit the network simultaneously to resolve the identical record. This is known as the thundering herd problem.

To resolve this efficiently without introducing massive external libraries, I implemented a custom singleflight pattern using a generic flightGroup[V] struct.

Get pouyasadri’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

When a cache miss occurs, the worker registers its intent to fetch the key. If another worker requests the same key while the first query is still in flight, the subsequent workers don’t execute a new network request. Instead, they block, wait for the original request to complete, and safely share the single returned result.

Combined with a shared http.Client pool that reuses a single http.Transport across all external MTA-STS HTTPS policy fetches, this completely eliminated connection cold-starts and redundant network overhead.

Moving with Confidence via Test-Driven Development (TDD)

Developing a custom validation or linting engine means dealing with massive string parsing and strict RFC compliance rules. For instance, parsing SPF records requires recursively validating nested include mechanisms while strictly enforcing the RFC 7208 limit of 10 or fewer recursive DNS lookups.

Trying to code this type of logic blindly and checking it via manual CLI execution is a recipe for endless regressions.

By employing Test-Driven Development (TDD), I defined the boundaries first. Using the interface-driven design mentioned earlier, I wrote a fakeDNSResolver directly within the test packages. This mock resolver allowed me to simulate intricate DNS landscapes—such as circular SPF loops, unresolvable records, or valid/invalid DMARC external destination authorizations—entirely in memory.

func TestSPFLintLimit(t *testing.T) {
    mockResolver := &fakeDNSResolver{
        TXTRecords: map[string][]string{
            "example.com": {"v=spf1 include:nested1.com ~all"},
            "nested1.com": {"v=spf1 include:nested2.com ~all"},
            // Simulate deep nesting up to the RFC threshold
        },
    }
    
    // Assert that exceeding 10 lookups reliably flags a specific Lint Finding.
}

Writing table-driven unit tests for every individual lint rule ensured that as the scoring engines and output stream writers evolved, the underlying protocol validations remained rock solid.

Production Grade Traits in a Standard-Library App

What transforms a basic utility script into a reliable tool designed for production data pipelines? For email-checker, it boiled down to three principles:

• Deterministic Output Ordering: While workers execute concurrently and finish tasks out of order based on network latencies, the final merge layer matches the output order exactly to the original input index. This makes the tool perfectly suited for traditional Unix streaming pipelines utilizing stdin and stdout.
• Context Propagation: Every single low-level function that triggers network I/O accepts a context.Context as its primary argument. If a per-domain timeout occurs, the resources are instantly reclaimed, preventing goroutine leaks.
• Stream-First Writing: Outputting data into formats like Newline-Delimited JSON (NDJSON) or CSV happens via stream writers. This means memory usage remains completely flat, whether auditing 10 domains or 100,000 domains.

Lessons Learned and Next Frontiers

Building this project from scratch reinforced a powerful truth about Go: the standard library is incredibly capable when combined with solid architectural patterns. You don’t need a heavy dependency graph to build high-performance, resilient development tools.

The current version handles core audits, linting configurations, and custom risk-scoring profiles dynamically. As I continue exploring low-level networking, the next enhancements for this engine will target deeper active checks, including active SMTP STARTTLS negotiations directly with target MX hosts, certificate chain validations, and parsing real-time DNS TTL responses natively.

The full implementation, golden test fixtures, and architectural layout are open source. Feel free to explore the code, open an issue, or contribute to the project over on GitHub:

🔗 GitHub Repository: github.com/pouyasadri/email-checker-tool