<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by steven b on Medium]]></title>
        <description><![CDATA[Stories by steven b on Medium]]></description>
        <link>https://medium.com/@steveinatorx_49018?source=rss-b93ca585f7ac------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*rvOjVDjY3ercvgPD.</url>
            <title>Stories by steven b on Medium</title>
            <link>https://medium.com/@steveinatorx_49018?source=rss-b93ca585f7ac------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 10:58:10 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@steveinatorx_49018/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Building a Financial RAG System Pt 6/6: From Prototype to Production Ready]]></title>
            <link>https://medium.com/@steveinatorx_49018/building-a-financial-rag-system-pt-6-6-from-prototype-to-production-ready-6bf9708b7ce3?source=rss-b93ca585f7ac------2</link>
            <guid isPermaLink="false">https://medium.com/p/6bf9708b7ce3</guid>
            <category><![CDATA[rags]]></category>
            <category><![CDATA[mlops]]></category>
            <dc:creator><![CDATA[steven b]]></dc:creator>
            <pubDate>Thu, 05 Feb 2026 20:28:52 GMT</pubDate>
            <atom:updated>2026-02-05T20:29:40.329Z</atom:updated>
            <content:encoded><![CDATA[<p>Building a working RAG system is one thing. Making it production-ready is another.</p><p><strong>Dagster Orchestration</strong></p><p>I used <a href="https://dagster.io/">Dagster</a> to orchestrate the pipeline. It solves reproducibility (same inputs equal same outputs), adds observability (see what ran, when, why), incremental updates (only rerun what changed), backfills (reprocess historical data), and error handling (track failures, retries).</p><p>Each step is an asset with clear dependencies: docs_raw → docs_clean → chunks → embeddings → faiss_index → eval_results. Benefits: visual dependency graph, can materialize incrementally, track lineage (know what produced what), backfill capabilities.</p><p><strong>Available jobs:</strong> rag_offline_job (complete offline RAG pipeline, runs without API keys) and rag_sweep_job (hyperparameter sweep, evaluates all configurations, generates plots).</p><p><strong>UI benefits:</strong> visual asset graph, materialization history, error tracking, performance metrics. Start with `pipenv run dagster dev` or `make dagster`, then open <a href="http://localhost:3000/">http://localhost:3000</a>.</p><p><strong>Latency Optimization</strong></p><p>Current performance: P50 latency less than 0.001s (sub millisecond), P95 latency less than 0.003s (still very fast).</p><p>Optimization strategies: embedding caching (cache embeddings for documents, only recompute when documents change), index optimization (FAISS IndexFlatL2 is fast enough for our scale, can upgrade to IVF for millions of vectors), generation optimization (extractive method is fast and free, LLM only when needed), and parallel processing (generate embeddings in parallel, batch queries together, use GPU when available).</p><p><strong>Cost Management</strong></p><p><strong>Offline path (default):</strong> embeddings $0 (local sentence transformers), generation $0 (extractive method), total $0 per query.</p><p><strong>Enhanced path (with API keys):</strong> embeddings ~$0.0001 per 1K tokens, generation ~$0.002 per query (GPT-3.5), total ~$0.002 per query.</p><p><strong>At scale (estimated):</strong> 1,000 queries/day equals $2/day ($60/month), 10,000 queries/day equals $20/day ($600/month), 100,000 queries/day equals $200/day ($6,000/month).</p><p><strong>Cost optimization strategies:</strong> use offline first (default to free local models), cache embeddings (don’t recompute for same documents), batch requests (process multiple queries together), gate LLM usage (only use when quality matters), and monitor usage (track costs and optimize).</p><p><strong>Monitoring</strong></p><p><strong>Metrics to track</strong>: quality metrics (context recall should be greater than 0.8, faithfulness should be greater than 0.9), performance metrics (P50/P95 latency should be less than 1s, error rate should be less than 1%), and cost metrics (cost per query, total daily/monthly cost, cost by component).</p><p><strong>Alerting:</strong> set up alerts for degradation (recall/faithfulness drops), high error rates (greater than 5% failures), latency spikes (P95 greater than 2s), and cost overruns (daily cost exceeds threshold).</p><p><strong>Dashboards:</strong> Dagster UI for pipeline health and asset materialization, custom dashboards for quality metrics and costs, logs for error tracking and debugging.</p><p><strong>Deployment Considerations</strong></p><p><strong>Model serving:</strong> embeddings (local: load model once, reuse; API: use OpenAI/Cohere for better quality; hybrid: local default, API fallback), index (FAISS on disk: load on startup; in-memory: faster but uses more RAM), generation (extractive: always available; LLM API: when quality matters; hybrid: extractive fallback if LLM fails).</p><p><strong>Scaling:</strong> horizontal scaling (multiple instances behind load balancer, stateless design, share index across instances), vertical scaling (larger instances for more RAM, GPU for faster embeddings, more CPU for parallel processing), caching (embedding cache, index cache, query result cache).</p><p><strong>Reliability:</strong> error handling (graceful degradation, retries with exponential backoff, clear error messages), health checks (index loaded correctly, embedding model available, API keys valid if using), fallbacks (local models if API fails, extractive if LLM fails, cached results if generation fails).</p><p><strong>Best Practices</strong></p><p><strong>Start simple:</strong> offline first (no API dependencies), extractive generation (fast, free, reliable), local embeddings (no external calls). Then enhance: add LLM generation when needed, use OpenAI embeddings for better quality, fine tune for domain adaptation.</p><p><strong>Evaluate early:</strong> build evaluation from day one (don’t add it later), measure everything (recall, faithfulness, latency), iterate based on metrics (data-driven improvements).</p><p><strong>Optimize incrementally:</strong> profile first (find bottlenecks), optimize bottlenecks (don’t optimize everything), measure improvements (verify optimizations help).</p><p><strong>Production readiness:</strong> Dagster for orchestration (observability and reproducibility), monitoring and alerting (know when things break), cost tracking (don’t get surprised by bills), error handling (graceful degradation).</p><p><strong>Lessons Learned</strong></p><p><strong><em>Evaluation is essential </em></strong>— can’t improve what you don’t measure. Systematic evaluation revealed structure aware chunking wins (counterintuitive), more chunks doesn’t mean better recall, and chunk strategy matters most.</p><p><strong><em>Offline first enables reliability</em></strong> — no API dependencies equals no API failures. Complete pipeline works without external services.</p><p><strong><em>Dagster provides observability </em></strong>— know what’s running, when, why. Can debug issues, track performance, backfill data.</p><p><strong><em>Advanced chunking matters </em></strong>— custom structure aware chunking achieved 0.877 recall (87.7%) on 10 real SEC filings vs 0.759–0.826 for others. Worth the investment.</p><p><strong><em>Cost management is critical</em></strong> — at scale, API costs add up. Offline-first keeps costs at $0, with optional enhancements when needed.</p><p><strong>Production Checklist</strong></p><p>Evaluation framework (metrics, sweeps, visualization), orchestration (Dagster assets and jobs), error handling (graceful degradation, retries), monitoring (metrics, alerts, dashboards), cost tracking (monitor API usage), documentation (README, guides, examples), testing (unit tests, integration tests), deployment (model serving, scaling, caching).</p><p><strong>Limitations &amp; Threats to Validity</strong></p><p>This evaluation has several limitations worth acknowledging:</p><p><strong><em>Gold set scope: </em></strong>The 42-question gold set, while diverse across 10 companies, may not capture all failure modes. Some companies (notably Apple) have more questions than others, which could introduce bias if their filing structure is atypical. The questions focus on factual, verifiable answers — interpretive or analytical questions are not covered.</p><p><strong><em>Metric proxies:</em></strong> Context recall and faithfulness are proxies, not ground truth. They measure whether relevant information is retrieved and whether answers are grounded, but don’t capture semantic correctness or completeness. A system could retrieve the right chunks but still generate incorrect answers.</p><p><strong><em>Document domain: </em></strong>Evaluation is limited to SEC 10-K filings (2023). Results may not generalize to other financial documents (earnings calls, analyst reports) or non-financial domains. The structure-aware chunking strategy was specifically tuned for hierarchical financial documents.</p><p><strong><em>Temporal questions:</em></strong> Most temporal questions rely on multi-year data within a single filing. True cross-filing temporal questions (e.g., comparing FY 2022 to FY 2023 across different documents) are underrepresented and may have different failure modes.</p><p><strong><em>Multi-document reasoning:</em></strong> While multi-document questions test cross-company comparisons, they don’t test complex reasoning requiring synthesis across many documents. The current set focuses on factual comparisons rather than analytical synthesis.</p><p><strong><em>Evaluation scale: </em></strong>42 questions across 10 documents is sufficient for initial validation but may miss edge cases that appear at larger scale. Production systems should expand gold sets iteratively as new failure modes emerge.</p><p>These limitations don’t invalidate the results, but they bound the claims. The evaluation demonstrates that structure-aware chunking improves retrieval for financial documents, not that it solves all RAG problems universally.</p><p><strong>Takeaways</strong></p><p>High quality RAG requires more than just a pipeline. Orchestration can provide observability. Evaluation enables optimization. Cost management is critical. Monitoring prevents degradation. Offline-first enables reliability.</p><p>This blog series covered architecture and design decisions, RAG pipeline components, evaluation framework, advanced chunking strategies, and production considerations.</p><p><strong>Key message: </strong>evaluation first, offline first, production ready. Build with evaluation from day one. Use offline first for reliability. Make it production ready with orchestration, monitoring, and cost management.</p><p>Thank you so much for reading my blog series ! Open to any suggestions for improvement</p><p><a href="https://medium.com/@steveinatorx_49018/building-a-finance-rag-system-why-start-with-evaluation-188f521f6779"><strong>Part 1 of this series</strong></a></p><p><a href="https://github.com/steveinatorx/finance-rag-eval">Github repo</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6bf9708b7ce3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a Financial RAG pipeline Part 4/6: The Chicken-and-Egg Problem of RAG Evaluation]]></title>
            <link>https://medium.com/@steveinatorx_49018/building-a-financial-rag-pipeline-part-4-6-the-chicken-and-egg-problem-of-rag-evaluation-971fc3168fb0?source=rss-b93ca585f7ac------2</link>
            <guid isPermaLink="false">https://medium.com/p/971fc3168fb0</guid>
            <category><![CDATA[ml-evaluation]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[rags]]></category>
            <dc:creator><![CDATA[steven b]]></dc:creator>
            <pubDate>Thu, 05 Feb 2026 20:26:53 GMT</pubDate>
            <atom:updated>2026-02-05T20:26:53.768Z</atom:updated>
            <content:encoded><![CDATA[<p>You want to evaluate your RAG chunking strategy. But to evaluate chunking, you need a gold set (question and answer pairs). And to create a good gold set, you need good chunking — otherwise, you’re evaluating on poorly chunked documents.</p><p><strong>The Real Question</strong></p><p>How do you evaluate chunking quality before you have a gold set? And once you do have a gold set, how do you know if your evaluation methodology is sound?</p><p>One approach is to use intrinsic metrics (evaluate chunking directly) plus extrinsic metrics (evaluate via retrieval performance), using synthetic data to bootstrap and real data to validate.</p><p><strong>Two Types of Evaluation</strong></p><p><strong>Extrinsic evaluation</strong> measures retrieval performance (recall, faithfulness) with different chunking strategies. It directly measures what matters (retrieval performance) and is real-world relevant. But it requires a gold set (chicken and egg problem), doesn’t explain why chunking is good or bad, and is slow to iterate (full retrieval pipeline).</p><p><strong>Intrinsic evaluation</strong> measures chunking quality directly (coherence, boundaries, structure). It requires no gold set (can evaluate immediately), explains why chunking is good or bad, and is fast to iterate (just analyze chunks). But it doesn’t directly measure retrieval performance, and you need to validate correlation with recall.</p><p><strong>How They Work Together</strong></p><p>Use intrinsic metrics to bootstrap evaluation (evaluate chunking before creating gold set), understand failures (why did recall drop? check intrinsic metrics), iterate quickly (test chunking changes fast), and debug (identify specific chunking issues).</p><p>Use extrinsic metrics to validate (confirm intrinsic metrics correlate with recall), final evaluation (measure real-world performance), and compare strategies (which chunking leads to best retrieval?).</p><p><strong>The Bootstrap Strategy: Synthetic → Real Data</strong></p><p><em>Phase 1: Synthetic data bootstrap.</em> Generate synthetic financial documents with known structure. Evaluate chunking strategies with intrinsic metrics. Create synthetic gold set (since I know what’s in docs). Validate intrinsic metrics correlate with recall. Establish baselines: what does good chunking look like?</p><p><em>Phase 2: Real data validation.</em> Start with small real gold set (10–15 questions). Compare intrinsic metrics: synthetic vs real. Compare extrinsic metrics: synthetic vs real. Identify gaps: what works in synthetic but not real? Iterate: fix issues, expand gold set.</p><p><em>Phase 3: Production evaluation.</em> Expand gold set (25+ questions). Cover edge cases (multi-document, temporal queries). Ongoing evaluation (track metrics over time). Monitor for regressions.</p><p><strong>Intrinsic Metrics</strong></p><p>Chunk size distribution: are chunks reasonably sized? Not too small (fragmented) or too large (unwieldy)?</p><p>Semantic coherence: are sentences within a chunk semantically related? Split chunks into sentences, generate embeddings, calculate pairwise similarity within chunks. Higher similarity means more coherent chunks.</p><p>Boundary quality: are chunk boundaries at natural breaks (sentence/paragraph boundaries)? Check if chunks end with sentence punctuation and if next chunk starts with capital letter. Natural boundaries preserve meaning.</p><p>Structure preservation: are document structures (headers, tables) preserved with context? Detect headers in chunks, detect tables, calculate preservation ratio. Headers and tables need context — isolated table cells are meaningless.</p><p>Document coverage: is all document content chunked? Any gaps? Calculate coverage ratio. Missing content can’t be retrieved.</p><p><strong>Intrinsic vs Extrinsic Correlation</strong></p><p>I found strong correlation between intrinsic and extrinsic metrics across all tested chunking strategies. Structure aware achieved the highest boundary quality (0.721) and highest recall (0.877). Fixed had boundary_quality=0.674 and recall=0.866. Recursive had boundary_quality=0.680 and recall=0.822. Semantic had boundary_quality=0.650 and recall=0.759.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*n8GAhabt64iZmcFvY8Pd1A.png" /></figure><p>The correlation plots show how intrinsic metrics predict extrinsic performance. The left plot shows boundary quality vs context recall — higher boundary quality correlates with higher recall. The right plot shows structure preservation vs recall. Structure aware (red) achieves the highest boundary quality and recall, validating that intrinsic metrics can predict retrieval performance before you need a gold set.</p><p>While this analysis does not claim causal proof, boundary quality and structure preservation showed a consistent monotonic relationship with recall across tested configurations. The correlation is strong enough to use intrinsic metrics for rapid iteration before building a gold set, but extrinsic validation remains essential for final evaluation.</p><p><strong>Real-World Workflow</strong></p><p><em>Step 1: Bootstrap with synthetic data</em>. Generate synthetic documents, evaluate chunking strategies with intrinsic metrics (no gold set needed), create synthetic gold set (easy: I know what’s in docs), validate with recall.</p><p><em>Step 2: Validate on real data.</em> Start with small real gold set (10–15 questions, manageable), test best strategy from synthetic, compare intrinsic metrics (similar to synthetic?), compare extrinsic metrics (similar recall?).</p><p><em>Step 3: Expand and iterate. </em>Expand gold set as system improves, ongoing evaluation, monitor for regressions, check intrinsic metrics if recall drops.</p><p><strong>Metrics Summary</strong></p><p>Intrinsic metrics are predictive — they correlate strongly with recall. Synthetic data is valuable — lets you develop evaluation framework quickly, test edge cases systematically, establish baselines, validate methodology. But you must validate on real data — synthetic results don’t always translate.</p><p>Small gold sets are enough to start — you don’t need 100+ questions. 10–15 well-chosen questions can validate your approach. Expand iteratively as you learn.</p><p>Both metrics matter — intrinsic for fast feedback and explaining why, extrinsic for validating performance and measuring what matters. Use both together for comprehensive evaluation.</p><p>The chicken and egg problem is solvable — you can evaluate chunking quality without a gold set using intrinsic metrics. Then validate with a small gold set. Then expand. The key is starting with intrinsic metrics.</p><p><a href="https://medium.com/@steveinatorx_49018/building-a-financial-rag-system-pt-5-how-i-fixed-chunking-to-reach-90-recall-7f1158e934a9"><strong>Next: chunking strategies themselves and how I evaluated them to reach 0.90+ recall on diverse real-world data.</strong></a></p><p><a href="https://medium.com/@steveinatorx_49018/building-a-financial-rag-system-pt-3-how-i-evaluated-my-rag-system-c5bd562b33dc"><strong>Link to Previous Part 3 of this series</strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=971fc3168fb0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a Financial RAG System Pt 5/6: How I Fixed Chunking to Reach 90% Recall]]></title>
            <link>https://medium.com/@steveinatorx_49018/building-a-financial-rag-system-pt-5-how-i-fixed-chunking-to-reach-90-recall-7f1158e934a9?source=rss-b93ca585f7ac------2</link>
            <guid isPermaLink="false">https://medium.com/p/7f1158e934a9</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[rags]]></category>
            <category><![CDATA[chunking]]></category>
            <dc:creator><![CDATA[steven b]]></dc:creator>
            <pubDate>Wed, 04 Feb 2026 19:25:20 GMT</pubDate>
            <atom:updated>2026-02-05T20:34:10.926Z</atom:updated>
            <content:encoded><![CDATA[<p>Chunking is often overlooked, but it is a critical component of RAG quality. Bad chunking equals bad retrieval equals bad answers.</p><p>Most tutorials show you fixed-size chunking and call it a day. But for production systems — especially financial documents — you need better.</p><p><strong>Why Basic Chunking Can Fail</strong></p><p>Fixed size chunking splits related content. Revenue breakdown gets split across chunks. Headers are separated from their content. You get mid word splits like “0 million” instead of “$125 million”. And there’s no semantic understanding — it doesn’t know what belongs together.</p><p>Example: a document with “Revenue Breakdown — Technology Division: $85 million (68% of total revenue) — Services Division: $30 million (24% of total revenue)” might get split into “Revenue Breakdown — Technology Division: $85” and “million (68% of total revenue) — Services Division: $30”. Query “What is the revenue breakdown?” retrieves incomplete chunks.</p><p><strong>What I Tried</strong></p><p>I implemented and evaluated five different chunking strategies: fixed (baseline, 0.826 recall), recursive (0.822 recall), structure-aware (highest recall: 0.877), semantic (0.759 recall), and hybrid (0.759 recall).</p><p>Structure aware chunking detects document structure — headers, sections, paragraphs — then chunks respecting that structure. It keeps headers with their content, preserves section boundaries, and doesn’t split mid section.</p><p>Results: 0.877 context recall (87.7%) on 10 real SEC filings, highest of any strategy.</p><p><strong>Why it wins:</strong> financial documents have clear structure. Questions often require understanding full sections. Preserving document structure and context matters more than granularity.</p><p>Semantic chunking groups sentences by semantic similarity using embeddings. It uses a cosine similarity threshold of 0.5 to group related concepts together. Results: 0.759 recall — lower than structure aware, but more granular. It creates many small chunks (29 vs 3), which may fragment related information.</p><p>Hybrid chunking combines structure aware and semantic. It first applies structure aware (larger chunks), then refines large chunks with semantic chunking. For our documents, structure aware already creates optimal chunks, so semantic refinement didn’t help. Results: 0.759 recall (same as semantic).</p><p>Recursive chunking recursively splits by separators: try paragraphs, then sentences, then sentence endings, then words, fallback to fixed-size. Results: 0.822 recall — good baseline.</p><p>Fixed size chunking is simple character based splitting with word or sentence boundary detection. Results: 0.826 recall — surprisingly good.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*reF3xJEv9Ca2O88GTGtAuA.png" /></figure><p>The bar chart shows structure aware chunking achieving the highest recall (0.877) compared to all other strategies. Fixed and recursive perform similarly (0.826, 0.822), while semantic and hybrid lag behind (0.759).</p><p><strong>A Counterintuitive Result ?</strong></p><p>Semantic created many small chunks but had lower recall (0.759) than structure aware (0.877). More chunks doesn’t mean better recall — context recall requires finding all relevant information. Fragmenting information across many chunks makes it harder to retrieve everything needed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Q4aaFeOWz7kFKZSCjKYyPg.png" /></figure><p>The scatter plot visualizes the counterintuitive relationship: structure aware creates only 3 chunks but achieves 0.877 recall, while semantic creates 29 chunks but only achieves 0.759 recall. The negative trend line shows that more chunks correlate with lower recall — the opposite of what you might expect. Preserving document structure and context matters more than creating many small chunks.</p><p><strong>The Bug I Found</strong></p><p>When evaluating on real SEC filings, recall was stuck at 0.526 (52.6%). Investigation revealed chunks containing answer numbers were pure HTML table markup with zero semantic content.</p><p><strong>Example:</strong> query “What was Apple’s total net sales for fiscal year 2023?” The chunk with number (383,285) was “bottom”&gt;&lt;span style=”color:#000000…”&gt;383,285&amp;#160;&lt;/span&gt;&lt;/td&gt;&lt;td…”. Cosine similarity: 0.0705 (very low). BM25 score: 0.0000 (no keyword overlap). Not retrieved even with k=50.</p><p><strong>Root cause:</strong> HTML tables were being chunked as raw markup. Tables split into tiny chunks of pure HTML. No semantic content, just HTML tags and style attributes. Structure aware chunking didn’t recognize tables — only detected headers and paragraphs. Tables treated as regular paragraphs, split into meaningless pieces.</p><p><strong>The Fix</strong></p><p>I added HTML table extraction. Detect HTML table elements, extract table headers (rows with years, dates, or header keywords), format tables as readable text: “Table: Header1 | Header2 | Header3”, preserve data rows with pipe separators.</p><p>Before: raw HTML markup. After: “Table: 2025 | 2024 | 2023\nTotal net sales | 416,161 | 391,035 | 383,285”.</p><p>I enhanced structure detection to recognize table rows by pipe separators, detect patterns (dollar amounts plus years), treat tables as special structure elements, and group consecutive table rows together.</p><p>I modified structure aware chunking to chunk tables with their surrounding headers, include semantic context, and prevent isolation in HTML only chunks.</p><p><strong>Results</strong></p><p>Before fix: recall 0.526 (52.6%), chunks with numbers 0/4 retrieved (even with k=50), pure HTML markup with no semantic content.</p><p>After fix: recall 0.738 (73.8%) — 40% improvement. Chunks with numbers 3/3 retrieved at ranks #1–3. Tables include headers and semantic context.</p><p>After metric refinement and expanding to 10 documents: recall 0.877 (87.7%) on 42 questions. The metric was refined with better normalization (remove HTML entities, remove HTML tags, normalize whitespace), smarter number matching (extract numbers, normalize, check if normalized number appears), and weighted scoring (70% phrases, 30% numbers).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*E5WOj3YFPN1HNyyAe_yQaQ.png" /></figure><p>The improvement chart shows the dramatic impact of fixing HTML table extraction. Recall jumped from 0.526 to 0.738 (40% improvement) after extracting tables as readable text, then to 0.877 after refining the evaluation metric. This demonstrates how fixing chunking quality directly translates to better retrieval performance.</p><p><strong>What I Learned</strong></p><p>Chunking quality directly impacts retrieval quality. Structure aware wins for financial documents (0.877 recall on 10 real SEC filings). More chunks doesn’t mean better recall — context preservation matters. Evaluation reveals counterintuitive results — always measure. And custom implementation shows deep understanding — not just using tools.</p><p><a href="https://medium.com/@steveinatorx_49018/building-a-financial-rag-system-pt-6-6-from-prototype-to-production-ready-6bf9708b7ce3"><strong>Next- Part 6/6: production considerations and lessons learned.</strong></a></p><p><a href="https://github.com/steveinatorx/finance-rag-eval">Github repo link</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7f1158e934a9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a Financial RAG System Pt 3/6: How I Evaluated my RAG system]]></title>
            <link>https://medium.com/@steveinatorx_49018/building-a-financial-rag-system-pt-3-how-i-evaluated-my-rag-system-c5bd562b33dc?source=rss-b93ca585f7ac------2</link>
            <guid isPermaLink="false">https://medium.com/p/c5bd562b33dc</guid>
            <category><![CDATA[chunking]]></category>
            <category><![CDATA[pareto]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[steven b]]></dc:creator>
            <pubDate>Wed, 04 Feb 2026 19:24:27 GMT</pubDate>
            <atom:updated>2026-02-05T20:33:23.161Z</atom:updated>
            <content:encoded><![CDATA[<p>How do you know if your RAG system is actually good? Most projects answer: “it works on a few examples.” That’s not enough for production.</p><p>I needed systematic evaluation with quantitative metrics to measure quality objectively, compare different configurations, identify improvements, and make data-driven decisions.</p><p><strong>The Metrics</strong></p><p>Context recall (proxy) measures whether retrieved context contains the answer. I extract key phrases from the gold answer and check if they appear in retrieved chunks, calculating the percentage of phrases found. This ensures relevant information is retrieved — if the answer isn’t in the retrieved chunks, the system can’t be correct.</p><p>Faithfulness (proxy) measures whether generated answers are supported by retrieved context. I split the answer into sentences, check word overlap with retrieved chunks, and calculate the percentage of sentences with sufficient overlap. This prevents hallucination — ensures answers are grounded in retrieved context, not made up.</p><p>Latency measures how fast the system is. I track P50 (median latency) and P95 (95th percentile latency) to catch outliers. Production systems need to be fast — sub-second latency is critical for interactive use.</p><p><strong>The Sweep</strong></p><p>I systematically evaluated chunk_size (256, 512, 1024), chunk_strategy (fixed, recursive, structure_aware, semantic), retriever (cosine, mmr), top_k (3, 5, 10), and rerank (False, True). That’s 3 × 4 × 2 × 3 × 2 = 144 configurations.</p><p><strong>What I Found</strong></p><p>Structure aware chunking achieved the highest context recall: 0.877 (87.7%) on 10 real SEC filings with 42 questions, outperforming fixed-size (0.826), recursive (0.822), and semantic (0.759) strategies. **This despite creating fewer chunks** than semantic chunking — more chunks doesn’t mean better recall. Preserving document structure and context matters more.</p><p><strong>Chunk size:</strong> 512 works best for our documents — not too small, not too large.</p><p><strong>Retrieval strategy: </strong>Cosine is sufficient for most cases. Use MMR when you need diverse results.</p><p><strong>Top-k:</strong> 5 is optimal balance for this use case.</p><p><strong>Reranking:</strong> Helps but isn’t always worth the latency cost.</p><p><strong>Visualization</strong></p><p>I generated three key visualizations to understand the tradeoffs across configurations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6p5pY3JL_Sn1PTgdpYjJ2Q.png" /></figure><p>The faithfulness vs latency scatter plot shows the quality-speed tradeoff. Each point represents a configuration, colored by chunking strategy. Structure aware (red) achieves high faithfulness with low latency, while semantic (orange) has lower faithfulness.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*F_DaRX2gNCe0eTMxAOLgSg.png" /></figure><p>The recall vs chunk size line plot shows how chunk size affects context recall across different strategies. Structure aware consistently outperforms others, with chunk size 512 being optimal. More chunks doesn’t mean better recall — preserving context matters more than granularity.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*i8vufqj_ZUc3n99N_psxvQ.png" /></figure><p>The Pareto frontier identifies optimal configurations balancing multiple objectives. Points on the red frontier represent configurations where you can’t improve faithfulness without increasing latency, or vice versa. The Pareto frontier showed no single “best” configuration. Tradeoffs depend on priorities: maximum recall (structure aware, chunk_size=512, top_k=10), maximum speed (fixed, chunk_size=256, top_k=3, no rerank), or balanced (structure aware, chunk_size=512, top_k=5).</p><p><strong>The Gold Set</strong></p><p>I created 42 question and answer pairs based on 10 real SEC filings from diverse companies (Apple, Microsoft, Google, Amazon, Tesla, JPMorgan, Visa, Johnson &amp; Johnson, Walmart, ExxonMobil). Questions cover revenue, expenses, segments, ratios, fiscal year information, and multi-document comparisons. Answers are factual and verifiable.</p><p>The gold questions were intentionally designed to be answerable via explicit spans or tables, making context recall a meaningful proxy for retrieval quality. This design choice ensures that if the answer isn’t in the retrieved chunks, the system cannot be correct — eliminating ambiguity about whether failures are retrieval or generation problems.</p><p>Building a gold set: start with sample documents, create questions that can be answered, verify answers are in documents, expand over time, and cover different question types (single-document, multi-document, temporal).</p><p><strong>Key Insights</strong></p><p>Evaluation reveals counterintuitive results — **structure aware wins despite fewer chunks**. More chunks doesn’t mean better recall — context preservation matters more. Chunk strategy matters most — bigger impact than other parameters. There’s no free lunch — tradeoffs everywhere (speed vs quality). And systematic evaluation is essential — you can’t optimize what you don’t measure.</p><p><strong>Failure Analysis by Question Type</strong></p><p>Not all questions are equally difficult. Here’s how different question types performed:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pREeKNdEMVgZa3wXwvA9jg.png" /></figure><p>Table-based numeric questions were the hardest — initially failing because HTML tables were chunked as raw markup with no semantic content. After fixing table extraction, recall improved significantly. Multi-document questions require retrieving and synthesizing information across filings, which is inherently harder. Temporal questions struggle with year disambiguation and require tracking changes across periods.</p><p><strong>Takeaways</strong></p><p>Evaluation is not optional for production RAG. Proxy metrics are useful even if imperfect. Systematic sweeps reveal insights you wouldn’t find manually. Visualization helps understand tradeoffs. And gold set quality matters — invest in good question and answer pairs.</p><p><a href="https://medium.com/@steveinatorx_49018/building-a-financial-rag-pipeline-part-4-6-the-chicken-and-egg-problem-of-rag-evaluation-971fc3168fb0"><strong>Next — Part 4/6: advanced chunking strategies deep dive.</strong></a></p><p><a href="https://medium.com/@steveinatorx_49018/building-a-finance-rag-system-part-2-rag-pipeline-deep-dive-what-actually-works-c7c33f665bca">Link to Part 2 of this series</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c5bd562b33dc" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a Finance RAG System Part 2: RAG Pipeline Deep Dive -What Actually Works]]></title>
            <link>https://medium.com/@steveinatorx_49018/building-a-finance-rag-system-part-2-rag-pipeline-deep-dive-what-actually-works-c7c33f665bca?source=rss-b93ca585f7ac------2</link>
            <guid isPermaLink="false">https://medium.com/p/c7c33f665bca</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[rags]]></category>
            <dc:creator><![CDATA[steven b]]></dc:creator>
            <pubDate>Mon, 02 Feb 2026 23:51:30 GMT</pubDate>
            <atom:updated>2026-02-05T20:19:01.617Z</atom:updated>
            <content:encoded><![CDATA[<h3>Building a Finance RAG System Part 2/6: RAG Pipeline Deep Dive -What Actually Works</h3><p>In <a href="https://medium.com/@steveinatorx_49018/building-a-finance-rag-system-why-start-with-evaluation-188f521f6779"><strong>Part 1</strong></a> — I covered the high-level RAG pipeline and architecture. Now let’s look at what actually works in practice.</p><p><strong>Document Ingestion</strong></p><p>SEC filings often come as HTML. The challenge is parsing that HTML and extracting clean text while preserving structure. I used BeautifulSoup for parsing and focused on text/HTML (avoiding PDFs and OCR complexity).</p><p>The key decision: simple cleaning that normalizes whitespace and preserves structure, while keeping source metadata for citations and provenance.</p><p><strong>Chunking: Where I Spent Most of My Time</strong></p><p>Chunking quality directly impacts retrieval quality. I tried five different strategies:</p><p>Fixed-size chunking is simple and predictable — character-based with word/sentence boundary detection. It’s a good baseline, but may split related content.</p><p>Recursive chunking tries to preserve structure by splitting on paragraph breaks, then sentences, then words. It’s better than fixed, but still size-based.</p><p>Structure aware chunking detects headers, sections, and paragraphs, then chunks respecting document structure. It keeps sections together — for example, “Revenue Breakdown” stays with revenue data. This achieved 0.877 context recall (87.7%) on 10 real SEC filings, the highest of any strategy.</p><p>Semantic chunking groups sentences by semantic similarity using embeddings. It uses a cosine similarity threshold of 0.5 to group related concepts together. This achieved 0.759 recall — lower than structure-aware, but more granular.</p><p>Hybrid chunking combines structure aware and semantic. It first applies structure aware (larger chunks), then refines large chunks with semantic chunking. For our documents, structure aware already creates optimal chunks, so semantic refinement didn’t help.</p><p><strong>The Counterintuitive Result</strong></p><p>Structure aware achieved the highest recall (0.877) <strong>despite creating fewer chunks !</strong> Semantic created many small chunks but had lower recall (0.759) than structure aware (0.877), which preserves document structure better.</p><p>Why? Context recall requires finding all relevant information. Fragmenting information across many chunks makes it harder to retrieve everything needed. Financial documents have clear hierarchical structure — headers to sections to paragraphs. Questions often require understanding full sections. Structure-aware preserves these relationships.</p><p><strong>Embeddings</strong></p><p>I used sentence transformers (all-MiniLM-L6-v2) as the default. It’s 384 dimensions, about 80MB, fast, offline, and good quality. It runs locally on CPU or GPU.</p><p>OpenAI embeddings (text-embedding-ada-002) are optional. They’re 1536 dimensions with better semantic understanding, but cost money and require an API. I recommend starting with sentence transformers, fine tuning if needed, and using commercial LLM/Bedrock for production if budget allows.</p><p><strong>Indexing with FAISS</strong></p><p>FAISS (Facebook AI Similarity Search) is a library for efficient similarity search and clustering of dense vectors. It’s designed to quickly find the most similar vectors in large collections — exactly what you need for RAG retrieval. Instead of comparing your query embedding against every chunk embedding one by one (which would be slow), FAISS uses optimized data structures and algorithms to search millions of vectors in milliseconds.</p><p>I used IndexFlatL2 for simplicity (can upgrade to IVF for scale), normalize embeddings for cosine similarity, and save/load for persistence. FAISS handles the heavy lifting of vector search, making retrieval fast even as your document collection grows.</p><p><strong>Retrieval Strategies</strong></p><p>Cosine similarity computes cosine similarity between query embedding and all chunk embeddings, returning top-k most similar chunks. It’s fast, simple, and effective for clear intent and fast queries.</p><p>MMR (Maximal Marginal Relevance) balances relevance and diversity, reducing redundant results. It iteratively selects chunks that are relevant but different. Use it for exploratory queries when you need diverse results.</p><p><strong>Reranking</strong></p><p>Cross encoder reranking uses ms-marco-MiniLM-L-6-v2. It’s more accurate than bi encoder because it sees query and chunk together, but slower since it runs on all retrieved chunks. It’s optional — use it for critical queries where quality matters more than speed.</p><p><strong>Answer Generation</strong></p><p>Extractive method (default) finds relevant sentences from retrieved chunks using keyword matching and scoring, then combines top sentences. It’s fast (no API calls), free, deterministic, and works offline. The downside is it may be choppy and less natural than LLM.</p><p>LLM generation (optional) sends query plus retrieved chunks to LLM, which generates a natural answer. It’s more coherent and natural, handles complex queries better, but requires an API key, costs money, is slower, and can potentially be non-deterministic.</p><p><strong>Chunking Summary</strong></p><p>Chunking matters most — quality of chunking directly impacts retrieval quality. Structure aware wins for financial docs (0.877 recall vs 0.759–0.826 for others on 10 real SEC filings). More chunks doesn’t mean better recall — preserving context beats granularity. Embeddings are critical — better embeddings mean better retrieval. Retrieval strategy depends on use case — cosine vs MMR. Reranking helps but adds latency. And extractive generation works — for many cases, no LLM needed.</p><p><a href="https://medium.com/@steveinatorx_49018/building-a-financial-rag-system-pt-3-how-i-evaluated-my-rag-system-c5bd562b33dc"><strong>Next in the blog series — Part 3/6 of Building a Financial RAG system: evaluation framework and results</strong></a></p><p><a href="https://github.com/steveinatorx/finance-rag-eval">Github Repo</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c7c33f665bca" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a Finance RAG System: Why Start With Evaluation ?]]></title>
            <link>https://medium.com/@steveinatorx_49018/building-a-finance-rag-system-why-start-with-evaluation-188f521f6779?source=rss-b93ca585f7ac------2</link>
            <guid isPermaLink="false">https://medium.com/p/188f521f6779</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[rags]]></category>
            <category><![CDATA[ml-evaluation]]></category>
            <dc:creator><![CDATA[steven b]]></dc:creator>
            <pubDate>Mon, 02 Feb 2026 23:39:19 GMT</pubDate>
            <atom:updated>2026-02-05T20:18:35.400Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>Building a Finance RAG System Part 1/6: Why Start With Evaluation ?</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3bf8idSG8Ib3XzoE4DUgug.png" /></figure><p>I built a RAG system to process SEC filings, and quickly realized that “it works on a few examples” wasn’t good enough. Equity analysts can spend hours manually scanning 100–300 page filings to find specific facts. A RAG system can solve this, but how do you know if yours is actually good?</p><p>The only way to determine if a RAG system is any good is systematic evaluation done before deployment. <strong>Deploying directly to production is not evaluation, that is production debugging</strong> !</p><p><strong>The Problem</strong></p><p>RAG evaluation is genuinely hard. Teams often skip it because: Ground truth is expensive to create. Manually creating high-quality question and answer pairs requires domain expertise and time. “Correctness” is fuzzy — especially with long-form answers, there may be multiple valid ways to express the same information. LLM-based judges feel circular or untrustworthy. And retrieval plus generation failures are intertwined — when an answer is wrong, is it because retrieval failed to find the right context, or because generation failed to synthesize it correctly?</p><p>This leads to a common anti-pattern: “If users complain, we’ll fix it.” That’s not evaluation — that’s production debugging. By the time users complain, you’ve already deployed a system that may be fundamentally broken.</p><p><strong>Why Finance?</strong></p><p>Financial documents present unique challenges. They’re dense and technical, full of financial terminology, numbers, and relationships. Regulatory compliance means answers must be accurate and verifiable. Cost sensitivity matters as API costs add up quickly at scale. Having offline data can help in development ease and cost control.</p><p>This project tackles these challenges with an evaluation-first approach and offline-first design.</p><p><strong>What I Built</strong></p><p>The system works completely offline. Local models for embeddings (sentence transformers), extractive answer generation (no LLM required), and optional API enhancements when needed. The tradeoff is quality vs. cost, but you can enhance with APIs when it matters.</p><p>I used Dagster for orchestration. Asset-based pipeline with clear dependencies, visual UI for monitoring and debugging, and the ability to backfill, rerun, and track lineage.</p><p>The evaluation framework measures context recall (does retrieved context contain the answer?), faithfulness (is the answer supported by retrieved context?), and latency (P50/P95 for production readiness). I ran systematic hyperparameter sweeps across 36+ configurations and used Pareto frontier analysis to find optimal tradeoffs.</p><p><strong>The Results</strong></p><p>After systematic evaluation across five chunking strategies (fixed-size, recursive, structure-aware, semantic, and hybrid), structure-aware chunking achieved the highest context recall: <strong>0.877 (87.7%)</strong> on 10 real SEC filings with 42 questions. This outperformed fixed-size <strong>(0.826),</strong> recursive <strong>(0.822)</strong>, and semantic <strong>(0.759)</strong> chunking strategies.</p><p>The counterintuitive finding: **structure-aware chunking won despite creating fewer chunks** than semantic chunking. More chunks doesn’t mean better recall — preserving document structure and context matters more than granularity for financial documents.</p><p><strong>What’s Next ?</strong></p><p>This is Part 1 of a 6-part series. Next, I’ll dive into the RAG pipeline components, evaluation framework, chunking strategies, and production considerations. <a href="https://medium.com/@steveinatorx_49018/building-a-finance-rag-system-part-2-rag-pipeline-deep-dive-what-actually-works-c7c33f665bca">Here is part 2.</a></p><p>The complete pipeline runs offline — no API keys required. See the README for full documentation here: <a href="https://github.com/steveinatorx/finance-rag-eval">https://github.com/steveinatorx/finance-rag-eval</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=188f521f6779" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Rust vs Go: A Fair-ish Performance Fight using a Link Shortener App]]></title>
            <link>https://medium.com/@steveinatorx_49018/rust-vs-go-a-fair-ish-performance-fight-using-a-link-shortener-app-e405a3fc3b0d?source=rss-b93ca585f7ac------2</link>
            <guid isPermaLink="false">https://medium.com/p/e405a3fc3b0d</guid>
            <category><![CDATA[hashmap]]></category>
            <category><![CDATA[rust]]></category>
            <category><![CDATA[performance]]></category>
            <category><![CDATA[golang]]></category>
            <dc:creator><![CDATA[steven b]]></dc:creator>
            <pubDate>Sat, 03 Jan 2026 22:20:18 GMT</pubDate>
            <atom:updated>2026-01-03T22:34:11.438Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Uo1xlxl_I7K9o0FDUXx3Ag.png" /></figure><p>I’ve always been curious about the real-world performance differences between Rust and Go. Not the marketing claims or microbenchmarks that test one specific thing, but an actual apples-to-apples comparison doing real work. So I built a benchmark that tries to be as fair as possible.</p><p>The idea was simple: build a link shortener’s core engine — the part that maps short codes to URLs. No HTTP, no network stack, no serialization. Just the data structure operations that matter: lookups and inserts, under load, with multiple threads competing for the same data.</p><p>Why skip HTTP? Because I wanted to measure the <strong><em>language</em></strong> differences, not the HTTP library differences. HTTP parsing, response formatting, and network I/O would add noise that obscures what we’re actually trying to compare. By focusing purely on the in-memory data structure operations, we get a cleaner signal about how Rust and Go perform at the core algorithmic level.</p><p><strong>Why Hash Maps?</strong></p><p>For a link shortener, you need fast lookups: given a short code like “aB3xY9mK”, find the corresponding URL. Hash maps are the obvious choice here — O(1) average-case lookups, simple to implement, and what you’d actually use in production. I could have used a B-tree or some other fancy data structure, but that would be optimizing for the wrong thing. Hash maps are the right tool for this job, and they let us focus on what we’re actually comparing: how well each language handles the same fundamental operations.</p><p><strong>The Setup</strong></p><p>Both implementations use the exact same approach:</p><p>- <strong>Sharded hash maps</strong> (128 shards by default) to reduce lock contention<br>- <strong>FNV-1a 64-bit hash </strong>for shard selection (implemented identically in both languages)<br>- <strong>Same workload</strong> — both benchmarks read from the same pre-generated operation files<br>- <strong>Same concurrency model</strong> — worker threads/goroutines that each process their slice of operations</p><p><strong>Why FNV-1a?</strong> I needed a hash function I could implement identically in both languages to ensure fairness. FNV-1a is simple, fast, and deterministic — perfect for shard selection. Both implementations use the exact same algorithm, so any differences in performance come from the language runtime, not from different hash implementations. I could have used each language’s built-in hash function, but that would introduce another variable. By implementing FNV-1a myself in both languages, I guarantee we’re comparing apples to apples.</p><p>The workload is 95% reads, 5% writes, which matches a typical link shortener split you might see in production. I tested two key distributions:</p><p>- <strong>Uniform</strong>: Keys chosen randomly (baseline case)<br>- <strong>Hot</strong>: 80% of reads hit 20% of keys (realistic cache behavior)</p><p>I ran tests across different thread counts (1, 2, 4, 8, 12) to see how each language scales. Each test ran for 10 seconds after a 2-second warmup, and I measured both throughput and latency percentiles.</p><h3><strong>The Results</strong></h3><p>Let’s start with the big picture: <strong>throughput:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iw0m57Gxr1XW-aiejFxTLA.png" /></figure><p>Rust consistently outperforms Go, but the gap narrows as we add more threads. At single-threaded, Rust is about 1.5x faster. By 12 threads, that advantage shrinks to just 5%. This makes sense — at higher concurrency, you’re more likely to be bottlenecked by lock contention or memory bandwidth than by the language runtime.</p><p>You might notice throughput still increases past 8 threads on my 8-core (virtual) machine. This is likely due to a few factors: Apple Silicon’s efficiency cores might be helping, but more importantly, when threads are waiting on locks (especially write locks), having extra threads ready to run can help keep the CPU busy. With 128 shards, there’s enough parallelism that oversubscribing threads can actually help — while one thread waits for a lock on one shard, another thread can work on a different shard. Still, the gains diminish significantly past 8 threads, which is what you’d expect.</p><p>The interesting part is how reads and writes behave differently:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mUeoT7JX31bkyYkGuwUY5w.png" /><figcaption>Read vs Write OPS</figcaption></figure><p>Reads scale beautifully in both languages. Writes… not so much. This is expected — reads can use shared locks (RwLock in Rust, RWMutex in Go), so multiple readers can proceed in parallel. Writes need exclusive locks, so they serialize.</p><p>What surprised me was how close the write performance is between Rust and Go. The difference is much smaller than for reads. My theory: writes are already bottlenecked by the lock, so the language overhead becomes less significant.</p><p><strong>Latency: The Real Story</strong></p><p>Throughput is one thing, but latency is what users feel. Here’s where things get interesting:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*G5Es7ZqQgdr7Rx72VYPCnw.png" /></figure><p>The latency story is more nuanced. At low thread counts, both languages are essentially identical — operations complete in under a microsecond. As we add threads, Go’s P99 latency starts to creep up more than Rust’s, but we’re still talking about sub-millisecond differences.</p><p>The read vs write latency breakdown tells a clearer story:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*czC-WujQSiRLdtGXrdx8Ow.png" /><figcaption>Read vs Write Latency</figcaption></figure><p>Writes are consistently slower than reads (no surprise — exclusive locks), but the gap between Rust and Go is smaller for writes. Again, I think this is because writes are already limited by lock contention, so the language runtime overhead matters less.</p><p><strong>What I Learned</strong></p><p>1. <strong>Rust is faster, but not dramatically so</strong> for this workload. The 1.05x to 1.5x speedup is real, but it’s not the 10x difference you sometimes see in microbenchmarks.</p><p>2. <strong>Scalability is similar</strong>. Both languages scale well with thread count, and both hit diminishing returns around 8–12 threads on my 8-core machine.</p><p>3. <strong>Reads benefit more from Rust’s performance</strong> than writes do. If your workload is write-heavy, the gap between languages narrows.</p><p>4. <strong>The “fairness” matters</strong>. By using identical algorithms, hash functions, and workloads, I’m confident these results reflect real differences, not implementation quirks.</p><p><strong>Real-World Throughput Estimates</strong></p><p>So what does 25 million ops/sec mean in practice? How many actual HTTP requests per second could this handle?</p><p>The honest answer: it depends on your HTTP stack. But we can make some rough estimates.</p><p>Our benchmark shows Rust hitting ~25M ops/sec and Go hitting ~18M ops/sec for the core data structure operations. In a real link shortener, you’d add:</p><p>- HTTP request parsing (headers, path, query params)<br>- Response serialization (JSON, redirect headers)<br>- Network I/O overhead<br>- Connection handling</p><p>Typical HTTP overhead is <strong><em>much</em></strong> larger than the hash map lookup. A well-optimized HTTP server might add 5–20 microseconds per request. Our core operations are sub-microsecond, so the HTTP layer dominates.</p><p>If we assume ~10μs of HTTP overhead per request (reasonable for a fast HTTP server), that’s roughly 100,000 requests/sec per core before you even hit the data structure. With our 8-core machine and 4 threads hitting ~25M ops/sec, you’re looking at roughly <strong>6–10 million requests/sec</strong> for the core engine alone.</p><p>But realistically? Your HTTP server will bottleneck long before the hash map does. Most production link shorteners handle thousands to tens of thousands of requests per second per server, not millions. The data structure performance we’re measuring here is more than enough — you’ll be limited by network bandwidth, HTTP parsing, or database connections long before you hit these numbers.</p><p>The takeaway: both languages give you headroom. The performance difference matters more if you’re building something that needs to squeeze every microsecond out of the system, or if you’re running at massive scale where small differences compound.</p><p><strong>The Methodology</strong></p><p>Here is what I did to make this comparison as fair as possible:</p><p>- <strong>Same hash function</strong>: FNV-1a implemented identically in both languages<br>- <strong>Same sharding strategy</strong>: 128 shards, same hash-to-shard mapping<br>- <strong>Same workload</strong>: Pre-generated operation sequences that both benchmarks read<br>- <strong>Same measurement</strong>: 1μs-resolution histograms, proper warmup, identical timing</p><p>The code is open source <a href="https://github.com/steveinatorx/link-shortener-bakeoff">https://github.com/steveinatorx/link-shortener-bakeoff</a> if you want to dig into the details or run it yourself.</p><p><strong>Should You Choose Rust or Go?</strong></p><p>Honestly? For this workload, either language works fine. The performance difference exists, but it’s not huge. The bigger question is: what matters more to your team?</p><p>- <strong>Choose Rust</strong> if you need every bit of performance, want memory safety without GC pauses, or are building systems where latency spikes are unacceptable.</p><p>- <strong>Choose Go</strong> if you value faster development, easier concurrency primitives, or have a team that’s more comfortable with Go’s simpler model.</p><p>For a link shortener specifically? Go’s performance is more than adequate, and you’ll probably ship faster. But if you’re building something where microseconds matter — high-frequency trading, real-time systems, or anything that needs predictable latency — Rust’s advantages become more compelling.</p><p><strong>The Code</strong></p><p>All the benchmark code, results, and analysis scripts are available on GitHub. You can run the full test matrix yourself, generate the plots, and see if you get similar results on your hardware.</p><p>The benchmark suite includes:</p><p>- Workload generators (Python)<br>- Rust and Go implementations<br>- Plotting scripts (matplotlib)<br>- Analysis tools for generating insights</p><p>Check it out: <a href="https://github.com/steveinatorx/link-shortener-bakeoff">https://github.com/steveinatorx/link-shortener-bakeoff</a></p><p>—</p><p><strong><em>Benchmark run on: Apple Silicon M2 (8 cores), 16g ram, macOS. Rust 1.75.0, Go 1.22.0. Results averaged across multiple runs with uniform and hot key distributions.</em></strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e405a3fc3b0d" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>