<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Islam Taha on Medium]]></title>
        <description><![CDATA[Stories by Islam Taha on Medium]]></description>
        <link>https://medium.com/@islamtaha-29281?source=rss-d741b64b39b0------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*Q-xxfriX5OBy_RJEqNQshQ.jpeg</url>
            <title>Stories by Islam Taha on Medium</title>
            <link>https://medium.com/@islamtaha-29281?source=rss-d741b64b39b0------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 17:30:39 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@islamtaha-29281/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Two Times Claude Steered Me Wrong — and What They Had in Common]]></title>
            <link>https://islamtaha-29281.medium.com/two-times-claude-steered-me-wrong-and-what-they-had-in-common-ce06cabfc4ee?source=rss-d741b64b39b0------2</link>
            <guid isPermaLink="false">https://medium.com/p/ce06cabfc4ee</guid>
            <category><![CDATA[engineering-leadership]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[ai-coding-assistant]]></category>
            <category><![CDATA[software-engineering]]></category>
            <dc:creator><![CDATA[Islam Taha]]></dc:creator>
            <pubDate>Mon, 25 May 2026 11:01:00 GMT</pubDate>
            <atom:updated>2026-05-25T11:01:00.828Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EqVKPJDwu7xRiBSm2b4AEg.png" /></figure><p><em>Recently I was debugging a Microsoft Fabric Copilot issue with Claude, when it casually wrote a phrase like “it’s not 2026 yet” — except we are roughly a third of the way through 2026. A small slip, fixed in seconds once I called it out. But the moment reminded me of a much larger version of the same problem that cost me three weeks of my life last fall. The pattern across both isn’t really about LLM limitations — it’s about which engineering skills these tools are quietly making more valuable, not less.</em></p><h3>The F1/F2 saga</h3><p>The larger episode was a <a href="https://medium.com/@islamtaha-29281/why-fpgas-cant-accelerate-sumo-traffic-simulation-and-what-i-learned-trying-c245a6f3b0d2">side project</a>: a hardware-accelerated traffic simulator targeting AWS FPGA instances. My background is in digital design — I built FPGA systems for my undergrad capstone (a portable object tracker on a Xilinx Virtex-4) and later worked on SDR with custom Verilog blocks during my research lab years. Returning to FPGA work in the cloud felt natural, and Claude Code was supposed to make it easier.</p><p>I started work in late November 2025. My default AWS region is Canada Central — closer to home, latency advantage, the natural choice. For the first six weeks of the project, Claude Code helped me architect everything around F1 — Amazon’s first-generation FPGA EC2 platform, launched in 2017 and powered by Xilinx Virtex UltraScale+ VU9P chips. Early on, Claude Code recommended I deploy in us-east-1: “best F1 availability,” with us-west-2 (Oregon) and eu-west-1 (Ireland) as fallbacks. Canada Central was not on the list. I followed the recommendation, migrated to us-east-1, sketched the deployment plan for f1.2xlarge instances on c5.4xlarge dev machines, wrote a detailed Vitis flow integration document, went through the AWS quota approval process for F1, and got it.</p><p>Then I picked the work back up in mid-January 2026 after a holiday pause, and Claude Code surfaced — for the first time — a fact it had presumably known the whole time: <strong>AWS had announced </strong><a href="https://repost.aws/questions/QU1hNwhenmSse2Np7UrQzN3A/instance-type-f1-2xlarge-is-not-supported-in-your-requested-availability-zone"><strong>F1 end-of-life for December 31, 2025</strong></a>. The announcement was nine months old by then. My six weeks of F1 architecture work was, in effect, planning a deployment for a platform that no longer accepted new customers. Claude Code’s January 18 session note framed the pivot calmly: “AWS has deprecated F1 instances in favor of F2. F1 is end-of-life (December 2025). F2 offers 60% better price/performance.” Calmly delivered, but the implication was that the previous six weeks had been wasted setup.</p><p>The pivot to F2 — AMD Virtex UltraScale+ HBM VU47P, 16 GiB of high-bandwidth memory per FPGA, a new shell, a new developer kit, a different binary format — surfaced its own set of problems. The first F2 xclbin load failed with an interface UUID mismatch: I had a three-way version conflict between the xclbin built with one platform string, the F2 hardware shell at a different version, and the AMI platform at a third version. That took the better part of a week to untangle and ultimately required pinning the AMI to 1.16.1 with a matching platform shell.</p><p>Then the data path itself didn’t work. Memory-mapped I/O over BAR2/PCIS — the path that worked fine on F1 — returned only address echoes on F2, never BRAM contents. Claude Code’s first suggestion for the fix was XDMA, the canonical DMA driver for FPGA work on AWS, with years of documentation and examples behind it. We built it, loaded it, and watched it fail to enumerate F2’s PCIe endpoint. After more digging, Claude Code surfaced the actual constraint: XDMA isn’t supported on F2 — the standard driver expects a PCIe configuration that F2 doesn’t present. Its fallback was MMIO via BAR0/OCL using peek/poke primitives, which worked but at the cost of one full PCIe round-trip per 32-bit word. Functional. Glacial. About 1 ms total for a 20 KB transfer.</p><p>The fix came from me, not the model. I asked Claude Code about SDE — Streaming Data Engine, an HDK soft IP that does CL-initiated DMA via the PCIM interface rather than the shell-initiated DMA via PCIS that XDMA uses, and which the F2 HDK explicitly supports. Once I named it, Claude Code immediately agreed it was the right path, drafted an implementation plan, and we eventually built a custom PCIM DMA engine that delivered a 6x speedup over the MMIO fallback — around 158 µs end-to-end for the same 20 KB transfer, against the 1 ms baseline. SDE was in the corpus. The F2 HDK ships a cl_sde example. It just wasn&#39;t where Claude Code reached by default — XDMA was the dominant DMA pattern in the training data, and SDE was the less-traveled correct answer for this specific platform. First successful CPU/FPGA round-trip on F2 hardware came on February 17, a month after the pivot; the SDE-based speedup landed a few weeks later after its own debugging arc through timing violations and a Management PF configuration fix.</p><p>Separately — and this is the piece I most want to call out for the corpus-gravity argument — Claude Code’s deployment recommendations after the F2 pivot still listed only us-east-1, us-west-2, and eu-west-1. Canada Central never came up. Not as the recommended region, not as a fallback, not as worth checking. When I went to launch F2 instances and looked at regional availability for myself, I discovered F2 had been <a href="https://aws.amazon.com/about-aws/whats-new/2025/11/amazon-ec2-f2-instances-four-additional-aws-regions">available in Canada Central since November 12, 2025</a> — two months before that January deployment plan was written. The information was sitting in AWS’s own “What’s New” feed. The model wasn’t operating on stale data. It was operating on the dominant pattern in its corpus, where us-east-1 has been the canonical FPGA-on-AWS region since 2017. ca-central-1 <em>exists</em> in the corpus. It just doesn’t dominate. Corpus gravity won — silently, by omission.</p><p>When I switched the project back to ca-central-1 on my own initiative, Claude Code was perfectly happy to update everything and standardize on it. It just hadn’t suggested it. By the time the project was running, ca-central-1 was hard-coded as the project default.</p><h3>The Fabric Copilot debug</h3><p>The Fabric Copilot situation was a much smaller version of the same thing. I’m leading a Microsoft Fabric implementation, and I wanted to check four tenant-level Copilot settings — whether they were on or off — without having tenant admin privileges. Claude and I worked through behavioral tests: try the chat-magic in a notebook, check the Copilot button in the ribbon, look for the Data Agent option. Standard stuff.</p><p>At one point Claude told me Fabric Copilot is “not very strong yet” and that Microsoft will close the gap on its in-product Copilots — but the framing was “it’s not 2026 yet.” We are <em>in</em> 2026. Roughly a third through it. The current date sits in Claude’s context, but it doesn’t sit in Claude’s intuitions. The model’s prose patterns leak out of a pre-cutoff worldview unless it consciously corrects for the actual date — which it doesn’t always do.</p><p>I called it out. The model owned it, explained the underlying reason (training cutoff lag, plus uncalibrated confidence about its own coverage gaps), and we kept moving. Total cost: maybe ten minutes of friction. But it was a smaller version of the same shape as the F1/F2 saga.</p><h3>What the failures had in common</h3><p>Both involved confident answers from a model operating on stale priors. But they’re actually different failure modes, and it’s worth naming them separately because the fix is different.</p><p><strong>Cutoff lag.</strong> The model literally doesn’t know X happened, because X happened after the training data was collected. This is the version everyone writes about, and it’s the easier one. The Fabric Copilot “it’s not 2026 yet” slip is a flavor of this — the model knew the current date in context but its prose intuitions were stuck pre-cutoff. The Fabric Copilot preview-vs-GA matrix shifts month by month, so any unsupplemented answer about which workload is GA today is liable to be wrong.</p><p><strong>Training-corpus gravity.</strong> The model knows X but keeps defaulting to Y because Y dominates the training corpus. This is the F1/F2 failure mode and it’s much harder to catch. F1 launched in 2017; F2 launched in December 2024. F1’s EOL was publicly announced by AWS at least nine months before my pivot. None of this was beyond Claude Code’s training cutoff. The information was <em>available</em>. But by the time Claude Code was trained, there were seven years of “use F1” content — forums, GitHub issues, Stack Overflow answers, AWS blog posts, vendor tutorials, conference talks — competing against one year of F2 content and a handful of EOL announcement threads. The model wasn’t lying when it recommended F1. It was <em>technically correct that F1 existed and worked</em>, but the implicit recommendation — the path of least resistance, the dominant pattern in the corpus — was F1. Up to and past the EOL cliff.</p><p>The region miss has the same shape. ca-central-1 was real and current; us-east-1 was the canonical FPGA-on-AWS region for seven years. The model didn’t say Canada was unavailable. It just never put Canada on the list, then or after the F2 pivot. Default attractors don’t have to argue. They just have to fill the slot.</p><p>This is the failure mode I’d push other practitioners to learn to recognize. The model doesn’t suggest the wrong thing explicitly. It suggests <em>both</em> options and quietly makes the old one easier — sometimes by omitting the new one entirely. Default attractors win when the user doesn’t know enough to swim against the current.</p><p><em>A small admission, because it’s the right kind of evidence:</em> while I was drafting this article in conversation with Claude, Claude confidently asserted that F1 wasn’t formally deprecated, that customers were still running it, that the F1 ecosystem was just in “maintenance.” That assertion was wrong on the substance — AWS publicly EOL’d F1 on December 31, 2025 — and the correct answer was sitting in my own project notes from four months earlier. The model wasn’t operating on stale data. It was operating on the <em>dominant pattern</em> in its corpus, which is “F1 is a thing that exists and people use.” It only corrected after I pushed back with citations. Same failure mode as the original F1/F2 saga, demonstrated live, while writing about it.</p><p>To be honest about my share of this: the verification was within reach in every one of these cases — a two-minute look at AWS’s region table or the deprecation feed would have surfaced the truth on its own. I didn’t run those checks because the model’s confidence on platform questions quietly suppressed my own instinct to verify. The lesson cuts both ways. The model needs to know its limits, and so does the user.</p><h3>Why this happens</h3><p>There’s a useful decomposition that’s helped me reason about when to trust LLM output. The user-visible lag in any given conversation is the sum of several components:</p><ol><li><strong>Data collection cutoff</strong> — the model’s training data was frozen at some date.</li><li><strong>Training and post-training duration</strong> — weeks to months of compute, plus RLHF, safety evals, red-teaming.</li><li><strong>Release lag</strong> — staged rollout, documentation, productization.</li><li><strong>Time since release</strong> — the model sits in service for months or longer.</li></ol><p>For frontier models, components 1–3 add up to roughly six months from cutoff to public release. Component 4 grows monotonically. Six months after release, the user-visible lag is closer to a year. A year after, eighteen months.</p><p>The other thing worth knowing is that the <em>effective</em> cutoff for any given topic can be much earlier than the <em>reported</em> cutoff. Knowledge cutoffs are not uniform across topics. Mainstream English-language content with heavy duplication is well-represented up to the cutoff. Niche, recent, non-English, or paywalled content is underrepresented. F2 documentation existed before Claude Code’s cutoff, but a year of F2 content does not compete equally with seven years of F1 content. Topic-corpus skew matters as much as the headline date.</p><h3>How I work with LLMs now</h3><p>A few heuristics that have sharpened up since the F1/F2 episode:</p><p>For <strong>structural and mechanical reasoning</strong> — how does X interact with Y, what trade-offs apply, how should this be architected — LLMs are reliable, because the underlying mechanics change slowly. This is where their value is highest.</p><p>For <strong>time-sensitive facts</strong> — current product status, who holds a role, what shipped last month, current pricing, whether a setting is GA or Preview — assume the answer is a hypothesis worth verifying. Demand sources. Use search tools where available. If the model is confidently asserting from memory in a fast-moving domain, that confidence is undercalibrated.</p><p>For <strong>niche technical domains</strong> where the model could be subject to training-corpus gravity — newer cloud SKUs, fresh API versions, recent library releases, anything that displaced a long-established predecessor — pin the model explicitly to the new platform and demand it justify every suggestion against current documentation. Don’t trust that “I know F2 exists” translates to “I will give you F2-correct answers.” The default attractors are stronger than they look.</p><p>And: <strong>call the model out when it drifts</strong>. The Fabric session took ten minutes to recover because I noticed and pushed back. Last fall’s F1/F2 saga took three weeks because I didn’t. That difference is most of the value.</p><h3>The skill that compounds</h3><p>This is the part of the conversation about LLMs that gets undersold. The dominant narrative is that LLMs commoditize engineering work — that they make junior engineers cheaper, eventually mid-level engineers redundant, and over time pressure senior roles too. There’s some truth in the execution layer. But in practice, watching how teams actually use these tools, I think the opposite is happening at the architectural layer: <strong>LLMs sharply widen the gap between engineers who can architect and engineers who can only execute.</strong></p><p>The F1/F2 episode is a clean example. An engineer without FPGA domain knowledge — handed Claude Code, the AWS docs, and a deadline — would have shipped something on F1. The code would have compiled. The synthesis would have run. The benchmarks would have looked plausible. And the project would have hit a performance ceiling six months later that nobody would have correctly diagnosed, because the diagnostic question (“are we even on the right platform?”) doesn’t get asked unless someone in the room knows enough about the domain to ask it. The LLM was never going to ask it. The LLM’s defaults had F1 baked in.</p><p>What the model can’t reliably do is two related things. The first is <strong>reach for less-traveled correct paths in its own corpus when the dominant path is wrong</strong>. Claude Code didn’t reach for SDE on its own. SDE was in the training data — the AWS HDK ships an example of it — but XDMA was the dominant DMA pattern by orders of magnitude, so XDMA is what got recommended. It took my asking, with FPGA-architecture vocabulary, to surface a solution that had been sitting in the model’s training data the whole time. The second is the part that isn’t in the corpus at all: trade-offs that depend on <em>your</em> system’s constraints, <em>your</em> team’s strengths and gaps, <em>your</em> existing technical debt, <em>your</em> business timeline, <em>your</em> compliance environment. Those facts live in your head and in conversations with stakeholders. The LLM doesn’t have them and can’t ask the right questions about them — partly because it doesn’t know to, and partly because the moment it tries, it falls back on generic “best practice” framings that pattern-match to whatever was loudest in its training data. That’s where confident-but-wrong architectural advice comes from.</p><p>So the practitioner skill that compounds in an LLM-saturated environment isn’t writing code faster. It’s <strong>knowing what to ask the model to build, knowing the better paths in its corpus that it won’t surface on its own, knowing what to refuse to delegate, and knowing when to throw the model’s suggestion out and start over.</strong> That’s system design and architecting — the work of structuring a problem so that execution is the easy part. LLMs collapse the cost of execution. They do not collapse the cost of judgment. If anything, they raise the premium on it, because cheap execution lets weak architecture scale faster than it used to.</p><p>This is opinion, not fact. But it’s an opinion I’ve now seen evidence for in three different settings: my own FPGA work, a Microsoft Fabric program I lead, and conversations with peers in data engineering across financial services and infrastructure consulting. The teams getting durable value from LLM tools are not the ones with the cleverest prompts. They’re the ones whose senior engineers spend more time on system architecture, contracts between components, and explicit failure-mode reasoning — and use the LLM to compress everything downstream of those decisions.</p><h3>Closing</h3><p>The 5% of the time when the model is confidently wrong, the cost of catching it is the cost of having someone in the room who knows the domain well enough to smell a wrong suggestion — and stubborn enough to push back when the model insists otherwise.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ce06cabfc4ee" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why FPGAs Can’t Accelerate SUMO Traffic Simulation (And What I Learned Trying)]]></title>
            <link>https://islamtaha-29281.medium.com/why-fpgas-cant-accelerate-sumo-traffic-simulation-and-what-i-learned-trying-c245a6f3b0d2?source=rss-d741b64b39b0------2</link>
            <guid isPermaLink="false">https://medium.com/p/c245a6f3b0d2</guid>
            <category><![CDATA[performance-engineering]]></category>
            <category><![CDATA[hardware-acceleration]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[fpga]]></category>
            <dc:creator><![CDATA[Islam Taha]]></dc:creator>
            <pubDate>Tue, 19 May 2026 04:13:24 GMT</pubDate>
            <atom:updated>2026-05-19T04:13:24.057Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/672/1*boz5VluXdhy1vZKRglLwdQ.png" /></figure><p><em>A case study companion to </em><a href="https://medium.com/towardsdev/why-most-data-pipelines-shouldnt-use-fpgas-and-when-they-should-b1740f792df1"><em>“Why Most Data Pipelines Shouldn’t Use FPGAs — and When They Should”</em></a></p><p>A few weeks ago I wrote about the boundary between when hardware acceleration helps a pipeline and when it makes things worse. The argument rested on Amdahl’s Law, the iceberg of integration cost, and the six gating criteria that separate a real acceleration candidate from a tempting one.</p><p>This article is the case study behind it. For more than six months, I built a complete FPGA acceleration pipeline for SUMO — a popular open-source traffic micro-simulator — on AWS F2 (Xilinx Virtex UltraScale+ VU47P). Everything worked: HLS kernels, custom PCIe DMA, parallel batch processing, the lot. I measured rigorously on real hardware.</p><p>Then I did the math and concluded the project was not viable. Here’s the story.</p><h3><strong>Why SUMO looked like a great FPGA candidate</strong></h3><p>The function I targeted, MSVehicle::planMoveInternal, plans one vehicle’s next move per timestep. It’s compute-heavy: lookahead through 32 links, evaluate traffic lights and stops, run car-following physics with double-precision floating point. SUMO profiling on a large network showed it taking up to 15% of total CPU time.</p><p>It also had textbook FPGA-friendly properties:</p><p>- <strong>Per-vehicle independent:</strong> Vehicle <em>i</em>’s plan doesn’t depend on vehicle <em>i+1</em>’s plan. Embarrassingly parallel.</p><p>- <strong>Bounded loops:</strong> No dynamic allocation, no recursion, fixed-size data structures.</p><p>- <strong>Stable algorithm:</strong> The Krauss car-following model hasn’t changed in years.</p><p>- <strong>Frequent invocation</strong> Called once per vehicle per timestep — hundreds of thousands of calls in a real simulation.</p><p>On paper this was a clean acceleration candidate. The kernel synthesized to HLS in a few hundred lines. The arithmetic intensity was high. The data shape was regular.</p><p>The first gating criterion in my earlier article asked: <em>”Does the workload dominate end-to-end latency?”</em> The answer was <em>kind of</em> — 15% is below the 30% threshold I recommended, but it was the single largest function in the profile. I thought function fusion (offloading several hot functions together) could push it higher. So I proceeded.</p><p>That was decision number one I’d later have to revisit.</p><h3><strong>The early wins that seemed to validate it</strong></h3><p>The engineering progressed in clean phases:</p><p><strong>Phase 1: MMIO transport (Feb 2026):</strong> Got an HLS kernel running end-to-end on F2. Single vehicle: 968us per call. Correct results matching CPU bit-for-bit on simple and multi-lane test scenarios. Felt great.</p><p><strong>Phase 2: PCIM DMA transport (Mar 2026):</strong> Replaced 32-bit MMIO writes with a custom PCIM DMA engine doing 8-beat bursts. Single vehicle dropped to 88us per call — an 11x improvement. The bottleneck became the kernel itself (~68us), not the data transfer.</p><p><strong>Phase 3: Parallel batch (late Mar 2026):</strong> Built cl_batch_dispatcher (scatter/gather DMA to N BRAMs) and cl_batch_controller (hardware-driven parallel kernel start). Two parallel instances completed in 44us total — 22us per vehicle, a 4x improvement over single-vehicle DMA. The kernel time dropped from 68us to 12us because both kernels ran in parallel and the controller stopped polling individually.</p><p>Numbers looked great in isolation:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/316/1*HGn1TA8xZzjIphV5KbKZeA.png" /></figure><p>I had a working pipeline that could process 45,000 vehicles per second through the FPGA. Time to plan scaling to 8 instances and start the function fusion work.</p><p>Then I paused and asked the right question.</p><h3><strong>The Amdahl’s Law moment</strong></h3><p>The question was: <em>what is the CPU time per call?</em></p><p>I had been comparing FPGA configurations against each other (great progress!) without comparing the FPGA against the CPU I was supposedly accelerating.</p><p>planMove_HLS on a modern x86 CPU runs in <strong>about 150 nanoseconds per call</strong> (measured on the same source compiled with -O2). My best FPGA configuration was 22 microseconds per vehicle. The FPGA was <strong>roughly 150 times slower per call</strong> than the thing I set out to make faster.</p><p>This wasn’t a measurement error. The FPGA runs at 250 MHz with double-precision FP. The CPU runs at 3+ GHz with superscalar execution, out-of-order scheduling, branch prediction, and SIMD. For a single sequential floating-point algorithm, the CPU wins by a wide margin. The FPGA’s edge is parallelism and custom datapaths — but to exploit it, I needed the <em>parallelism advantage</em> to outweigh the <em>per-call disadvantage</em>.</p><p>It doesn’t. Here’s the math.</p><p>If I offload planMove (P = 15% of CPU time) and the FPGA was infinitely fast (S = infinity):</p><p>Speedup = 1 / ((1 — P) + P/S) = 1 / 0.85 = <strong>1.18x</strong></p><p>That’s the theoretical ceiling. With my actual hardware where S is <em>less than 1</em> (slower per call), I get <strong>net slowdown</strong>.</p><p>If I offload all four parallelizable hot functions — planMove, updateOccupancy, getLeaderInfo, blockedByFoe — together (P = 33.5%, ~6 more weeks of engineering):</p><p>Max speedup = 1 / (1–0.335) = <strong>1.50x</strong></p><p>Half a decade of CPU progress, in exchange for months of FPGA engineering, AWS bills, and a much harder system to debug. Not a deal anyone should take.</p><p>This is exactly the situation my earlier article warned about: a single visible-in-profile stage that <em>feels</em> large but is actually a minority of end-to-end time. Hardware accelerators look much better in isolation than in system context.</p><h3><strong>Why I couldn’t just offload more</strong></h3><p>The natural next thought is: <em>fine, then offload everything.</em> Move the full vehicle update loop to the FPGA so P approaches 1.0 and Amdahl’s Law stops mattering.</p><p>This is where the second gating criterion — <em>”can the boundary be cleanly isolated?”</em> — kills the project.</p><p>The execution phase (executeMove, lane changes, occupancy updates) does things that are fundamentally incompatible with HLS synthesis:</p><p>- <strong>Dynamic memory:</strong> Vehicle lists, lane occupancy queues, route paths — all std::vector, std::deque, std::map. HLS needs fixed-size arrays.</p><p>- <strong>Pointer chasing:</strong> vehicle-&gt;myLane-&gt;getEdge().getToJunction()-&gt;getInternalLanes(). HLS can’t traverse arbitrary heap graphs.</p><p>- <strong>Virtual dispatch: </strong>Car-following models use polymorphism. HLS can’t resolve virtual calls at synthesis time.</p><p>- <strong>Global shared state:</strong> Lane occupancy arrays, traffic signal states, detector counters — read-modify-write from many places, hard to model in hardware.</p><p>- <strong>Spatial algorithms:</strong> Collision detection requires R-trees or similar dynamic structures. Dijkstra-based rerouting needs graph algorithms.</p><p>To move executeMove to FPGA you’d need to rewrite SUMO’s core simulation engine in hardware-friendly C++ — flat data structures, explicit type switches instead of virtual dispatch, no exceptions, no dynamic memory. That’s not an “FPGA acceleration project,” that’s “build a new traffic simulator from scratch.” Six to twelve months for a team. At which point the FPGA isn’t accelerating SUMO; you’ve replaced it.</p><h3><strong>The integration tax was real (and quantifiable)</strong></h3><p>If the Amdahl’s Law argument is the ceiling, the integration tax is the floor. My earlier article called the kernel “the tip of the iceberg” — about 15% of the total work. This project gave me a precise number.</p><p>Across roughly six months of engineering, I built:</p><p>- 1 HLS kernel (~600 lines of HLS-friendly C++)</p><p>- 4 RTL modules: BRAM bridge, OCL bridge, PCIM DMA engine, stream-to-BRAM adapter</p><p>- 2 RTL modules added in batch phase: dispatcher, controller</p><p>- 1 host runtime (~1000 lines of C++) with MMIO, DMA, and batch APIs</p><p>- SUMO integration via wrapper function</p><p>I also produced <strong>29 AFI builds</strong>, each taking about an hour of Vivado time and an hour of AWS AFI creation. Three of those AFIs worked. The other 26 hit bugs that had nothing to do with the algorithm:</p><p>- HLS interface pragmas silently changed the AXI-Lite port structure</p><p>- HLS struct deserialization through 512-bit AXI exploded LUT usage from 5% to 88%</p><p>- AW+W race condition in OCL caused DMA register writes to be silently dropped</p><p>- PCIM read path stuck because Bus Master Enable was not set on the Management PF</p><p>- Batch dispatcher’s per-instance byte count register defaulted to 4096 but DMA engine expected 2368</p><p>- Batch controller’s AXI-Lite pending bits were reassigned every cycle, never clearing</p><p>- Port A mux for instance 0 bypassed pipeline registers, dispatcher saw raw signals</p><p>Each of these took anywhere from a day to a week to diagnose. None of them are “FPGA algorithm” problems. All of them are <em>boundary</em> problems — the cost of moving data across the shell, the kernel, the host, and the OS, while keeping AXI protocols intact and the hardware-software contract in sync.</p><p>I cataloged 22 distinct lessons in the project’s docs/LESSONS_LEARNED.md. The vast majority describe boundary failures, not computation failures. This is what the “iceberg” looks like up close.</p><h3><strong>What I built anyway (and why it matters)</strong></h3><p>Despite the project not delivering a speedup, the engineering produced real artifacts. The complete pipeline works correctly and is fully reproducible:</p><p>- <strong>Custom PCIM DMA engine</strong> with 8-beat bursts, validated at 158us per round-trip with full software-managed bus mastering</p><p>- <strong>Parallel batch architecture </strong>with hardware-driven kernel start (no software polling per instance)</p><p>- <strong>End-to-end SUMO integration</strong> showing identical CPU-vs-FPGA results on multi-lane congestion scenarios</p><p>- <strong>Comprehensive lessons-learned document</strong> that should save future teams weeks of debugging</p><p>The infrastructure is reusable. If I ever target a workload that genuinely fits the FPGA model — high arithmetic intensity, deterministic latency requirements, dominant in end-to-end time, clean boundary — the DMA engine, the batch dispatcher, the host runtime, and the build pipeline all transfer.</p><p>The feasibility study itself is also a deliverable. It saves the next team — possibly someone reading this — from repeating the same path. That alone justifies the investment, in my view.</p><h3><strong>Three honest paths forward for SUMO</strong></h3><p>If the goal is <em>actually accelerating SUMO</em> rather than <em>using an FPGA</em>, three approaches make more sense than what I built:</p><p>1. <strong>GPU acceleration:</strong> GPUs have much lower per-call overhead than FPGAs (no PCIe round-trip per call once data is resident in GPU memory) and similar parallelism. The same per-vehicle independence that I exploited on FPGA would work well on a GPU with much less integration cost.</p><p>2. <strong>Multi-core CPU parallelism: </strong>SUMO has limited threading support. Parallelizing the planning phase across CPU cores with OpenMP could achieve ~8x on the planning phase, which gives ~1.13x overall by Amdahl — comparable to function fusion on FPGA with a fraction of the engineering effort.</p><p>3. <strong>A custom simulator built for FPGA from day one:</strong> Not retrofitting SUMO, but designing a traffic micro-simulator around fixed-point arithmetic, flat memory layout, and a fully pipelined update loop with all vehicle state in on-chip BRAM. This is the only path that breaks through Amdahl’s ceiling, because it makes the FPGA the <em>whole</em> system rather than an accelerator for one stage. But it’s a 6–12 month project requiring traffic simulation domain expertise plus FPGA design.</p><p>The recurring lesson is the one in my earlier article: the FPGA decision is an <em>architectural</em> decision, not a <em>function-level</em> one. Bolting an accelerator onto a CPU-shaped simulator inherits all the CPU-shaped constraints.</p><h3><strong>What I’d do differently</strong></h3><p>In hindsight, the project went sideways at the very first gating criterion. I saw planMove at the top of the SUMO profiler and reached for hardware before asking whether even a <em>perfect</em> FPGA could help.</p><p>Three checks would have surfaced the issue in a day instead of six months:</p><p>1. <strong>Measure CPU time per call before measuring anything else: </strong>Not the function’s share of total time, but the absolute per-call latency in microseconds. I knew planMove was 15% of CPU time; I did not know it ran in ~150 nanoseconds per call until much later. Per-call CPU time is what acceleration must beat, and PCIe round-trip alone is in tens of microseconds. The math could have been done on day one.</p><p>2. <strong>Run Amdahl’s calculation before writing a single line of HLS: </strong>With P = 0.15 and S = infinity, max speedup is 1.18x. That number should have been pinned to the wall before any kernel work started. If the team isn’t excited about a 1.18x ceiling, the project is over before it began.</p><p>3. <strong>Audit the offload boundary, not just the kernel: </strong>The execution phase of SUMO is a maze of C++ objects, virtual dispatch, and global state. Even before debugging a single AXI handshake, a structural audit would have shown that the boundary couldn’t be cleanly drawn around planMove in a way that would ever let P grow large enough to matter.</p><p>None of this is hindsight bias. It’s exactly the framework I wrote down in the <a href="https://medium.com/towardsdev/why-most-data-pipelines-shouldnt-use-fpgas-and-when-they-should-b1740f792df1">general article</a> — the same six gating criteria, applied to the project I was about to start. If I had run my own checklist against my own project, I would have killed it on day one.</p><p>The good news is the framework worked: when I eventually ran the analysis, it produced a clear, unambiguous “no.” The bad news is I ran the analysis six months too late.</p><h3><strong>The reusable takeaways</strong></h3><p>If you’re considering FPGA acceleration for a similar workload — a function that profiles as “compute-heavy” but lives inside a larger CPU-shaped system — here are the things I wish someone had told me before I started:</p><p>- <strong>A “hot” function in your CPU profile is necessary but not sufficient.</strong> It needs to be both hot <em>and</em> dominant in end-to-end time. 15% is not enough. 30% is the lower bound; 50%+ is where things start to be interesting.</p><p>- <strong>Per-call CPU time is the real benchmark.</strong> Not “this function takes 5% of CPU time” but “this function takes 800 nanoseconds per call.” If your accelerator’s round-trip latency is bigger than your CPU’s per-call cost, you’ve already lost. PCIe DMA round-trips are tens of microseconds on the best day. CPU calls to optimized C++ are often under a microsecond.</p><p>- <strong>The boundary is the thing.</strong> A clean boundary — pure compute in, pure result out, no shared state, no virtual dispatch, no pointer chasing — is the difference between an FPGA project that ships and one that doesn’t. SUMO’s planMove had a clean boundary in isolation; the <em>system</em> around it did not.</p><p>- <strong>The iceberg is real and roughly the same size every time.</strong> Across the projects I’ve seen, the HLS kernel ends up being 15–25% of the total work. The host integration, debugging, observability, and operational concerns make up the rest. Budget accordingly.</p><p>- <strong>A failed acceleration project that produces good documentation is still a success.</strong> The feasibility study and lessons-learned file from this project will save the next team from repeating the same path. That’s a real deliverable, even if it’s not the one anyone planned for.</p><h3><strong>Closing</strong></h3><p>Not every accelerated pipeline that works is one that should ship. The hardest engineering judgment is recognizing when “we built it and it works” doesn’t mean “we should put it in production.” Amdahl’s Law is rarely the dramatic constraint people imagine — it’s usually a quiet, boring number that closes a project before it starts.</p><p>The complete project — RTL, HLS source, host runtime, feasibility study, all 22 lessons learned, and the final 2-instance batch DCP — is preserved as a reference implementation. If you want to see what a “shouldn’t ship” FPGA acceleration project actually looks like under the hood, the <a href="https://github.com/tahaislam/sumo-fpga-case-study">repository</a> is open.</p><p>And if you find yourself reaching for an accelerator, I’d suggest reading the <a href="https://medium.com/towardsdev/why-most-data-pipelines-shouldnt-use-fpgas-and-when-they-should-b1740f792df1">general article</a> first, walking through the six gating criteria honestly, and only then deciding whether to start the project. It’s much cheaper than running 29 AFI builds to reach the same conclusion.</p><p><em>If you found this useful, the companion repository for the general article includes a reusable Amdahl’s Law calculator and integration cost model — both of which would have flagged this project as not-viable on day one. Use them.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c245a6f3b0d2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why Most Data Pipelines Shouldn’t Use FPGAs — and When They Should]]></title>
            <link>https://towardsdev.com/why-most-data-pipelines-shouldnt-use-fpgas-and-when-they-should-b1740f792df1?source=rss-d741b64b39b0------2</link>
            <guid isPermaLink="false">https://medium.com/p/b1740f792df1</guid>
            <category><![CDATA[fpga]]></category>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <dc:creator><![CDATA[Islam Taha]]></dc:creator>
            <pubDate>Mon, 30 Mar 2026 00:22:13 GMT</pubDate>
            <atom:updated>2026-03-30T10:29:07.177Z</atom:updated>
            <content:encoded><![CDATA[<p>For engineers working on data and ML infrastructure, performance pressure is constant. Pipelines grow, datasets expand, SLAs tighten, and eventually someone asks the inevitable question: <em>Should we accelerate this with specialized hardware?</em></p><p>GPUs and FPGAs promise massive throughput gains on paper. But in practice, most data pipelines do not benefit meaningfully from hardware acceleration — and some become slower, more fragile, or more expensive as a result.</p><p>This article is about finding that boundary: the point where CPU-based pipelines genuinely hit a wall, and what happens when you cross it.</p><h3>The Acceleration Temptation</h3><p>Modern hardware is impressive. GPUs deliver enormous parallelism. FPGAs offer fine-grained control, deterministic latency, and excellent performance per watt. Cloud providers like AWS now offer FPGA instances (F2, powered by Xilinx Virtex UltraScale+ VU47P with HBM2) on demand — no hardware lab required.</p><p>It’s tempting to assume that moving “heavy computation” onto an accelerator will naturally improve pipeline performance. You profile a stage, see it’s compute-heavy, and think: <em>This would fly on an FPGA.</em></p><p>That assumption is often wrong — and I’ve watched it mislead teams (including my own past self) more than once.</p><h3>What Actually Slows Down Data Pipelines</h3><p>Here’s a pattern I’ve seen repeatedly across production data systems: a team identifies a “slow” pipeline, profiles a single stage, and immediately starts discussing acceleration. But when you instrument the full end-to-end path, the picture changes dramatically.</p><p>Consider a typical analytics pipeline: ingest from object storage, deserialize Parquet, run transformations, write results back. The transformation kernel might take 200ms. But the full pipeline takes 14 seconds. Where does the time go?</p><p><strong>I/O and data movement</strong> dominate. Reading from S3, databases, or message queues isn’t just slow — it’s variable. Network jitter, throttling, and cold-start penalties on serverless storage add unpredictable latency that no amount of compute acceleration can fix.</p><p><strong>Serialization overhead</strong> is the silent killer. I’ve seen pipelines where converting between formats — JSON to Parquet, Avro to internal representations, Power Query M to SQL-compatible structures — consumed more time than the actual analytical logic. During a recent migration from Power BI dataflows to a warehouse-based SQL engine, the biggest performance win came not from faster compute, but from eliminating an unnecessary serialization hop between dataflow stages.</p><p><strong>Orchestration tax</strong> is real. Task scheduling, dependency resolution, retry logic, and backpressure management in tools like Airflow or Fabric pipelines add latency that’s invisible in microbenchmarks but dominant in production. When I built a pipeline orchestration system with parameterized refresh types (handling both daily incremental and monthly historical loads), the orchestration logic itself was the most latency-sensitive component — not the transformations.</p><p><strong>Stage isolation failures</strong> amplify bottlenecks. A fast kernel embedded in a poorly isolated pipeline doesn’t move the needle. If your accelerated stage has to wait for a slow upstream dependency or contend for shared resources, the speedup evaporates at the system level.</p><p>In many production pipelines, truly compute-bound work represents 5–15% of end-to-end latency. Accelerating that slice — even by 100x — barely registers in the overall picture.</p><h3>The Boundary That Matters: Amdahl’s Law in Practice</h3><p>This is where engineers often get tripped up, and it’s worth being precise about it.</p><p>Amdahl’s Law tells us that if 10% of your pipeline is compute-bound and you accelerate that portion to near-zero, your maximum possible speedup is ~1.11x. That’s an 11% improvement in exchange for introducing an entirely new hardware paradigm into your stack.</p><p>Think of it like upgrading the engine in a car that spends 90% of its time stuck in traffic. The engine isn’t the bottleneck — the road is.</p><p>But Amdahl’s Law also works in reverse: when a compute-bound stage genuinely dominates (say, 70%+ of end-to-end time), acceleration becomes transformative. A 10x speedup on that stage yields ~3x system-level improvement. Now the investment starts to make sense.</p><p>The critical skill isn’t knowing <em>how</em> to accelerate — it’s knowing <em>whether</em> you should, based on rigorous measurement of where time actually goes. (The companion repository includes a <a href="https://github.com/tahaislam/hardware-acceleration-in-data-pipelines/blob/main/tools/crossover_analysis.py">heatmap visualization</a> that maps this relationship across both dimensions — compute fraction and kernel speedup — making the narrow “sweet spot” for acceleration visually obvious.)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NiRGrRrYoA6KRP4gezaDIA.png" /><figcaption>Pipeline speedup as a function of compute fraction and kernel speedup (Amdahl’s Law).</figcaption></figure><h3>Identifying a Real Acceleration Candidate</h3><p>Before reaching for hardware, stress-test your assumptions with these questions:</p><p><strong>Is the bottleneck persistent and measurable?</strong> Transient spikes, GC pauses, or poorly instrumented systems often mislead optimization efforts. I’ve seen teams chase “slow stages” that were actually just noisy measurements from inadequate observability. Profile under sustained load, not just peak.</p><p><strong>Does the workload dominate end-to-end latency?</strong> If your candidate stage is less than 30% of total pipeline time, you’re unlikely to see meaningful system-level gains from acceleration — no matter how impressive the kernel speedup looks in isolation.</p><p><strong>Is the computation stable and well-defined?</strong> Acceleration — especially FPGA acceleration — favors deterministic workloads with limited branching, high arithmetic intensity, and consistent data shapes. If your transformation logic changes every sprint, you’ll spend more time re-synthesizing bitstreams than you save in execution.</p><p><strong>Can the boundary be cleanly isolated?</strong> This is the question that kills most acceleration projects. Moving data across CPU–FPGA boundaries (via PCIe, or through shell–kernel interfaces on AWS F2) introduces latency that can exceed the compute savings. On the F2 instance architecture, the FPGA communicates through 512-bit HBM AXI channels — fast, but only if your data access patterns are aligned and your transfer sizes are large enough to amortize the setup cost.</p><p>If you can’t answer these confidently, you’re not ready for acceleration. You’re ready for better profiling.</p><h3>The Hidden Costs No One Warns You About</h3><p>Let’s say you’ve done the analysis and acceleration looks promising. The kernel works in simulation. The HLS synthesis completes. The numbers look great.</p><p>Now the real work begins.</p><p><strong>Integration complexity is non-linear.</strong> Getting a working HLS kernel is maybe 30% of the project. The remaining 70% is host-side integration: memory management, synchronization, error handling, and making it work reliably inside a pipeline that was designed for CPU execution. In my experience with FPGA-accelerated simulation work, the kernel logic was straightforward — it was the data marshaling and result collection across the shell-kernel boundary that consumed the most engineering time.</p><p><strong>Debugging across hardware-software boundaries is painful.</strong> When something goes wrong in a CPU pipeline, you attach a debugger, add logging, trace the execution. When something goes wrong across a CPU-FPGA boundary, you’re often looking at waveform traces and trying to correlate them with software-side state. The mental context-switch alone slows down incident response.</p><p><strong>Observability degrades.</strong> Standard pipeline monitoring tools (Prometheus, Datadog, CloudWatch) don’t natively understand FPGA execution. You’ll need custom instrumentation, which means custom dashboards, custom alerts, and a team that understands both worlds.</p><p><strong>Operational rigidity increases.</strong> FPGA bitstreams are not containerized microservices. You can’t blue-green deploy them. Rolling back means re-flashing, which takes minutes, not milliseconds. In production systems where uptime matters, this rigidity is a real operational cost.</p><p>For production systems, these costs persist long after the initial prototype succeeds. Operational simplicity is often more valuable than raw performance — and the total cost of ownership for an accelerated pipeline is almost always higher than it appears during the proof-of-concept phase. (I’ve cataloged these costs in a structured <a href="https://github.com/tahaislam/hardware-acceleration-in-data-pipelines/blob/main/docs/integration-cost-model.md">integration cost model</a> with an estimation template — useful for pressure-testing acceleration proposals before committing.)</p><h3>When Acceleration Genuinely Transforms Performance</h3><p>With all those caveats, there are workloads where acceleration isn’t just helpful — it’s the only viable path to meeting requirements.</p><p><strong>Simulation-heavy workloads</strong> are the classic case. Traffic simulation, Monte Carlo analysis, computational fluid dynamics — these are dominated by repetitive, compute-intensive kernels with predictable data flows. An FPGA can execute a custom simulation pipeline at wire speed while a CPU is still fetching cache lines.</p><p><strong>Inference paths with strict latency budgets</strong> benefit when the model is stable and the latency requirement is hard (not “fast” — hard, as in “must respond within 2ms or the downstream system fails”). FPGAs excel here because they offer deterministic, worst-case latency guarantees that GPUs typically cannot.</p><p><strong>Repeated analytics kernels in large pipelines</strong> — the case where a specific transformation runs millions of times daily across terabytes of data — can justify acceleration when the compute fraction is dominant and the kernel is stable across releases.</p><p>The common thread: the compute is the bottleneck, it’s measurable, it’s stable, and the integration boundary is clean.</p><p>This pattern shows up consistently in the companies that have made FPGA acceleration work at scale. Microsoft’s Project Catapult placed FPGAs inline on the network path in every Azure server — not beside the CPU, but in the data flow itself — to accelerate networking and Bing search ranking. Illumina’s DRAGEN platform uses FPGAs to process a full human genome in under five minutes, with the acceleration embedded directly in the sequencing instrument. In both cases, the workload was compute-dominant, the kernel was stable across releases, and the integration boundary was architectural, not bolted on.</p><h3>A Systems View: CPU, GPU, and FPGA as Complementary Tools</h3><p>Effective systems don’t treat accelerators as CPU replacements. They treat them as specialized stages in a heterogeneous architecture.</p><p><strong>CPUs</strong> remain the right tool for orchestration, I/O management, complex control flow, and any logic that changes frequently. They’re the generalists — flexible, well-tooled, and easy to debug.</p><p><strong>GPUs</strong> shine when you have massive data parallelism with regular access patterns: batch inference, matrix operations, large-scale aggregations. Their programming model (CUDA, OpenCL) is more accessible than FPGA development, which matters for team velocity.</p><p><strong>FPGAs</strong> excel at custom data paths with deterministic timing requirements. They’re the tool of last resort in the best sense: when nothing else can meet your requirements, an FPGA can be shaped to do exactly what you need, nothing more and nothing less.</p><p>The key skill for modern data engineers isn’t hardware specialization — it’s <em>hardware literacy</em>. Understanding enough about each accelerator’s strengths and integration costs to make informed architectural decisions. You don’t need to write HLS kernels to know when an FPGA might help. You do need to understand data movement costs, Amdahl’s Law, and the operational overhead of heterogeneous systems.</p><h3>What I Think Matters Going Forward (Opinion)</h3><p>I believe the industry is approaching an inflection point. As data volumes grow and latency requirements tighten, more teams will hit the CPU ceiling — not because CPUs are slow, but because the workloads are genuinely compute-bound at scale.</p><p>When that happens, the engineers who can think across the full stack — from pipeline orchestration down to data path design — will have a significant advantage. Not because everyone needs to become a hardware engineer, but because the <em>decision</em> of when to cross the hardware boundary will become increasingly consequential.</p><p>The most impactful performance gains in data systems still come from better architecture: cleaner stage isolation, smarter data movement, eliminating unnecessary work. But selective, targeted acceleration — applied with engineering judgment and rigorous measurement — is an increasingly important tool in the kit.</p><h3>Further Reading</h3><p>I’ve built a companion repository that goes deeper on the ideas in this article:</p><p><a href="https://github.com/tahaislam/hardware-acceleration-in-data-pipelines"><strong>Hardware Acceleration in Data Pipelines — GitHub</strong></a></p><p>What’s in it:</p><ul><li>A structured <strong>decision framework</strong> with six gating criteria for evaluating acceleration candidates — designed to be walked through during architecture reviews.</li><li>An <strong>integration cost model</strong> that catalogs the costs acceleration proposals typically undercount (data marshalling, fallback paths, observability gaps, team ramp-up) with an estimation template.</li><li>A <strong>worked example</strong> comparing CPU vs. FPGA for bitfield packing — including a compilable C baseline, an annotated HLS sketch, and a system-level trade-off analysis showing how a kernel-level speedup degrades after transfer overhead.</li><li>A <strong>crossover analysis tool</strong> (Python) that generates parameterized visualizations of Amdahl’s Law and batch-size crossover points — plug in your own workload numbers to explore the trade-offs.</li><li>Three <strong>architectural diagrams</strong>: pipeline acceleration zones, the decision tree, and the integration “iceberg” (the kernel is ~15% of the work).</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dt5OQVuukaTHKjsxcXCy5A.png" /><figcaption>System-level speedup vs. batch size for the bitfield packing example.</figcaption></figure><h3>Closing Thought</h3><p>Most data pipelines don’t need hardware acceleration. Some absolutely do. And a few are stuck in a painful middle ground where they <em>might</em> benefit but the integration cost isn’t justified.</p><p>Knowing which category your system falls into — and having the measurement discipline to prove it — is where engineering judgment lives.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b1740f792df1" width="1" height="1" alt=""><hr><p><a href="https://towardsdev.com/why-most-data-pipelines-shouldnt-use-fpgas-and-when-they-should-b1740f792df1">Why Most Data Pipelines Shouldn’t Use FPGAs — and When They Should</a> was originally published in <a href="https://towardsdev.com/">Towards Dev</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Learning Prolog by Stress-Testing It on a Real Transit Routing Problem]]></title>
            <link>https://towardsdev.com/learning-prolog-by-stress-testing-it-on-a-real-transit-routing-problem-314d002351fb?source=rss-d741b64b39b0------2</link>
            <guid isPermaLink="false">https://medium.com/p/314d002351fb</guid>
            <category><![CDATA[computer-science]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Islam Taha]]></dc:creator>
            <pubDate>Sat, 31 Jan 2026 06:29:53 GMT</pubDate>
            <atom:updated>2026-02-09T07:35:46.094Z</atom:updated>
            <content:encoded><![CDATA[<h4><strong>What happens when you deliberately use a declarative language for a problem most people would solve imperatively?</strong><br>This is a story about logic programming, algorithmic fit, and learning where a paradigm stops scaling.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DzSyQ9sPA0Ads2s_E4iBmA.png" /></figure><h3>Why Prolog, and Why Transit Routing?</h3><p>I wanted to understand declarative programming beyond toy examples. Prolog, despite its age, still appears in AI research, rule engines, and symbolic reasoning systems. Its promise is appealing: describe <em>what</em> is true, not <em>how</em> to compute it.</p><p>Rather than building a contrived demo, I decided to stress-test Prolog on a real, data-heavy problem: <strong>public transit routing</strong>.</p><p>The goal was to build a prototype transit route optimizer over Toronto’s TTC network using GTFS (General Transit Feed Specification) data, with longer-term ambitions around accessibility and equity analysis. Concretely, the project aimed to:</p><ul><li>Parse and model GTFS data declaratively</li><li>Implement time-dependent routing using A* search</li><li>Support multimodal routing (subway, bus, streetcar)</li><li>Compute reachable areas within time budgets (isochrones)</li></ul><p>This was not an attempt to compete with production planners. It was an experiment: <strong>how far can a logic-first approach go before the cracks show?</strong></p><blockquote>This project also reinforced a broader pattern I’ve seen across data systems and AI pipelines: abstractions that prioritize correctness and expressiveness often encounter hard limits when scale and performance dominate. Understanding where those limits lie is just as important as mastering the abstraction itself.</blockquote><h3>Where Prolog Shines: Modeling and Querying</h3><p>GTFS data maps surprisingly well to Prolog. Stops, trips, routes, and schedules naturally become facts, and relationships emerge through unification rather than explicit joins.</p><p>Once loaded, querying feels almost conversational:</p><ul><li>“Which stops look like subway stations?”</li><li>“Which trips depart from this stop after 9 AM?”</li><li>“What are the valid next stops from here, given time and transfer constraints?”</li></ul><p>Instead of loops and filters, you express constraints. The engine handles the rest.</p><p>This declarative style made the early phase of the project genuinely productive. Within hours, I had a working loader for TTC GTFS data:</p><ul><li>~9,300 stops</li><li>223 routes</li><li>~130,000 trips</li><li>~4.3 million stop_times</li></ul><p>At this stage, Prolog felt like an excellent fit. The code was compact, readable, and closely aligned with the domain.</p><h3>The Real Test: Routing at Scale</h3><p>Things became interesting when I moved from data modeling to <strong>route finding</strong>.</p><p>I implemented a time-dependent A* search: each state represented a stop, a timestamp, transfer count, and accumulated cost. In theory, this is a natural fit for Prolog — tree search is one of its strengths.</p><p>And initially, the results were encouraging:</p><ul><li><strong>Subway-to-subway routing:</strong> under 1 second</li><li><strong>Simple multimodal trips:</strong> workable</li></ul><p>But performance degraded rapidly as the search space expanded.</p><h3>When the Search Space Explodes</h3><p>The problem wasn’t A* itself. It was the <strong>shape of the transit graph</strong>.</p><p>Subway networks are small and structured:</p><ul><li>~75 stations</li><li>Limited branching</li><li>Predictable schedules</li></ul><p>Surface transit is the opposite:</p><ul><li>9,000+ stops</li><li>Multiple routes per stop</li><li>Vehicles every 5–15 minutes</li><li>Dense temporal overlap</li></ul><p>In a one-hour window:</p><ul><li>A subway stop might expose ~100–200 candidate departures</li><li>A busy surface stop might expose <strong>400–800</strong></li></ul><p>Each A* expansion had to consider all valid next departures. Multiply that by transfers, time windows, and schedule constraints, and the frontier grows extremely fast.</p><p>The result:</p><ul><li>Subway-to-surface routing: ~30 seconds</li><li>Surface-to-surface routing: frequently exceeded time limits</li></ul><p>At that point, the behavior was consistent and reproducible. The approach had stopped scaling.</p><h3>Optimization Didn’t Change the Outcome</h3><p>I explored several optimization strategies:</p><ul><li>Dynamically limiting search depth based on distance</li><li>Detecting high-frequency stops and tightening time windows</li><li>Collapsing multiple platform stops into logical stations for caching</li></ul><p>Each helped locally. None changed the fundamental behavior.</p><p>This was the key realization: <strong>the bottleneck wasn’t a missing micro-optimization — it was a mismatch between algorithm, data scale, and execution model.</strong></p><h3>The Honest Conclusion</h3><p>Prolog wasn’t “too slow.” It was simply the wrong tool for <em>this class of problem at this scale</em>.</p><p>Three factors mattered most:</p><ol><li><strong>Data cardinality</strong><br>Millions of schedule records with time-dependent joins are outside Prolog’s performance envelope.</li><li><strong>Algorithmic fit</strong><br>Transit routing has specialized algorithms (RAPTOR, Connection Scan) designed to avoid exactly this kind of combinatorial explosion.</li><li><strong>Paradigm tension</strong><br>I was forcing imperative performance tricks into a declarative system — a sign that the abstraction was breaking down.</li></ol><p>At that point, continuing would have meant fighting the language rather than learning from it.</p><h3>What Prolog Is Excellent For</h3><p>This project clarified where Prolog excels:</p><ul><li>Knowledge representation and rule-based systems</li><li>Constraint solving with moderate state spaces</li><li>Symbolic reasoning and validation</li><li>Rapid prototyping of complex logical relationships</li></ul><p>And where it struggles:</p><ul><li>Large-scale, data-intensive, time-dependent search problems</li><li>Workloads that require tight control over memory layout and execution order</li></ul><p>Understanding that boundary is valuable in itself.</p><h3>What I Learned (Beyond Prolog)</h3><p>This project was far from wasted. It delivered several durable lessons:</p><h4>1. Declarative Thinking Changes How You Design Systems</h4><p>Instead of asking <em>“How do I iterate through this?”</em>, you ask <em>“What must be true for this state to be valid?”</em><br> That shift carries over to SQL, rule engines, and even system design discussions.</p><h4>2. Unification Is More Powerful Than It Looks</h4><p>The same predicate can validate, query, or generate data depending on how variables are bound. That flexibility is rare in imperative languages.</p><h4>3. Tool–Problem Fit Matters More Than Elegance</h4><p>A beautiful abstraction that doesn’t scale is still the wrong abstraction. Knowing when to pivot is an engineering skill, not a failure.</p><h4>4. Boundary-Finding Is a Legitimate Outcome</h4><p>Stress-testing a paradigm until it breaks teaches you more than staying safely within its comfort zone.</p><h3>The Code Lives On</h3><p>The project is open source:<br> <a href="https://github.com/tahaislam/prolog-transit-optimizer"><strong>https://github.com/tahaislam/prolog-transit-optimizer</strong></a></p><p>It includes:</p><ul><li>A complete GTFS loader</li><li>A working A* router for small to medium networks</li><li>Station bundling and caching infrastructure</li><li>Documentation of what scaled — and what didn’t</li></ul><p>If you’re learning Prolog, it’s a good place to experiment, extend, or refactor.</p><h3>Final Thoughts</h3><p>Would I use Prolog again? Absolutely — for the right problems.</p><p>Expert systems. Rule engines. Configuration validators. Symbolic reasoning. Anywhere relationships and constraints matter more than raw throughput.</p><p>But for routing across millions of time-indexed records? I’d reach for a tool designed for that domain from the start.</p><p>The most valuable learning projects don’t just teach you <em>how</em> to use a tool.<br>They teach you <strong>when not to</strong>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=314d002351fb" width="1" height="1" alt=""><hr><p><a href="https://towardsdev.com/learning-prolog-by-stress-testing-it-on-a-real-transit-routing-problem-314d002351fb">Learning Prolog by Stress-Testing It on a Real Transit Routing Problem</a> was originally published in <a href="https://towardsdev.com/">Towards Dev</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Beyond the Dashboard: Architecting a Modular, Metadata-Driven Data Observability Framework]]></title>
            <link>https://towardsdev.com/beyond-the-dashboard-architecting-a-modular-metadata-driven-data-observability-framework-8f08247c9cec?source=rss-d741b64b39b0------2</link>
            <guid isPermaLink="false">https://medium.com/p/8f08247c9cec</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data-quality]]></category>
            <category><![CDATA[data-analytics]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[microsoft-fabric]]></category>
            <dc:creator><![CDATA[Islam Taha]]></dc:creator>
            <pubDate>Sun, 14 Dec 2025 02:59:20 GMT</pubDate>
            <atom:updated>2025-12-17T06:17:55.747Z</atom:updated>
            <content:encoded><![CDATA[<h4>Part 2: Upgrading from Power BI hacks to a Metadata-Driven Engine in Microsoft Fabric.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kaelr061fNjSIj6X3P4QRw.jpeg" /></figure><p>Earlier this year, I shared how I built a “Data Health Hub” entirely within Power BI. It was a resourceful solution born out of necessity — using M code for statistical analysis and recursive dataflows to hack together historical logging. It worked, but as our data landscape grew, the limitations became clear. Maintaining complex logic inside fifty separate Power BI queries is not scalable.</p><p>We needed to evolve. We needed to move from <strong>“hard-coded checks”</strong> to a <strong>“metadata-driven framework.”</strong></p><p>This article details that evolution. I’ll walk through how we redesigned our observability architecture to be modular, generic, and tech-stack agnostic. While I implemented this version using <strong>Microsoft Fabric</strong> (Python &amp; SQL), the design patterns here are universal and can be easily ported to Snowflake, Databricks, or any modern data stack.</p><h3>The Core Philosophy: “Validator” as an Abstraction</h3><p>The biggest shift was architectural. Instead of writing specific code like <em>“Check if FactSales is updated”</em>, we abstracted the problem into a generic concept: the <strong>Validator</strong>.</p><p>A Validator is a black box engine. It doesn’t know about “Sales” or “Finance.” It only knows:</p><ol><li><strong>Input:</strong> A dataset and a rule configuration (e.g., “Max Age = 24 hours”).</li><li><strong>Process:</strong> Execute the math (Python/SQL).</li><li><strong>Output:</strong> A standardized log entry.</li></ol><p>We organized these checks into <strong>5 Universal Pillars of Data Health</strong>:</p><ol><li><strong>Timeliness:</strong> Is the data fresh?</li><li><strong>Completeness:</strong> Are there gaps in the timeline?</li><li><strong>Integrity:</strong> Are keys unique?</li><li><strong>Quality:</strong> Are there nulls or statistical outliers?</li><li><strong>Consistency:</strong> Is the volume stable compared to historical trends?</li></ol><h3>The Architecture: Separation of Concerns</h3><p>To make this scalable, we decoupled the system into three distinct layers. This prevents the “spaghetti code” that plagues many data quality projects.</p><ol><li><strong>The Control Layer (The Brain):</strong> A SQL table (RuleRegistry) that defines <em>what</em> to check. This is our &quot;Low Code&quot; interface.</li><li><strong>The Compute Layer (The Muscle):</strong> A Python Class (DataHealthValidator) that knows <em>how</em> to execute the checks.</li><li><strong>The Storage Layer (The Memory):</strong> A generic Log table (ValidationLog) that stores history for trending.</li></ol><h3>1. The Control Layer: Defining Rules as Data</h3><p>We stopped writing Python code for every new table. Instead, we treat monitoring as a configuration problem. We created a RuleRegistry table in our warehouse. Onboarding a new table is now as simple as running a SQL INSERT statement.</p><pre>CREATE TABLE dq.RuleRegistry (<br>    Layer VARCHAR(50),           -- &#39;Bronze&#39;, &#39;Gold&#39;<br>    SchemaName VARCHAR(50),      -- &#39;dbo&#39;<br>    TableName VARCHAR(100),      -- &#39;FactSales&#39;<br>    CheckType VARCHAR(50),       -- &#39;Timeliness&#39;, &#39;Consistency&#39;, &#39;Quarantine_Ratio&#39;<br>    Threshold FLOAT,             -- e.g., 24 (hours), 0.05 (5%)<br>    Parameter VARCHAR(50),       -- Extra config (e.g., &#39;D&#39; for Daily frequency)<br>    IsActive BIT DEFAULT 1<br>);</pre><p><strong>Example Configuration:</strong></p><pre>-- Rule 1: Alert if FactSales data is older than 24 hours<br>INSERT INTO dq.RuleRegistry (Layer, SchemaName, TableName, CheckType, Threshold, Parameter)<br>VALUES (&#39;Gold&#39;, &#39;dbo&#39;, &#39;FactSales&#39;, &#39;Timeliness&#39;, 24, &#39;hours&#39;);<br><br>-- Rule 2: Alert if Raw Sales volume drops by &gt; 50% vs 30-day average<br>INSERT INTO dq.RuleRegistry (Layer, SchemaName, TableName, CheckType, Threshold)<br>VALUES (&#39;Bronze&#39;, &#39;dbo&#39;, &#39;Sales_Raw&#39;, &#39;Consistency&#39;, 0.5);</pre><h3>2. The Compute Layer: The Python “Validator” Class</h3><p>In Microsoft Fabric, we implemented the logic using a Python class. However, we used a <strong>“Push-down” strategy</strong>. Instead of pulling millions of rows into a DataFrame to calculate an average (which is slow and expensive), we generate a lightweight SQL query and ask the Data Warehouse engine to do the heavy lifting.</p><p>Here is the clean implementation of the Validator class. Notice how it is completely generic.</p><pre>class DataHealthValidator:<br>    def __init__(self, sql_endpoint, database_name):<br>        self.server = sql_endpoint<br>        self.database = database_name<br>        self.results = []<br><br>    def run_query(self, query):<br>        &quot;&quot;&quot;Helper to execute SQL on the Warehouse and return a DataFrame&quot;&quot;&quot;<br>        # Connection logic (pyodbc/JDBC) goes here...<br>        return pd.read_sql(query, self.conn)<br><br>    # --- PILLAR 1: TIMELINESS ---<br>    def check_timeliness(self, table_name, date_col, threshold, unit=&#39;hours&#39;):<br>        # We push the aggregation to SQL for maximum performance<br>        query = f&quot;SELECT MAX({date_col}) as MaxDate FROM {table_name}&quot;<br>        df = self.run_query(query)<br>        <br>        latest = pd.to_datetime(df[&#39;MaxDate&#39;].iloc[0])<br>        age_hours = (datetime.now(timezone.utc) - latest).total_seconds() / 3600<br>        limit = threshold * 24 if unit == &#39;days&#39; else threshold<br>        <br>        status = &quot;Pass&quot; if age_hours &lt;= limit else &quot;Fail&quot;<br>        self._log_result(table_name, &quot;Timeliness&quot;, &quot;DataAgeHours&quot;, age_hours, limit, status)<br><br>    # --- PILLAR 2: CONSISTENCY (Volume Stability) ---<br>    def check_consistency(self, table_name, deviation_threshold=0.5):<br>        # 1. Get Current Volume<br>        current = self.run_query(f&quot;SELECT COUNT(*) as C FROM {table_name}&quot;)[&#39;C&#39;].iloc[0]<br>        <br>        # 2. Get History from our own Log (Self-Learning!)<br>        query_hist = f&quot;&quot;&quot;<br>            SELECT AVG(MetricValue) as AvgVol FROM dq.ValidationLog <br>            WHERE TableName = &#39;{table_name}&#39; AND MetricName = &#39;RowCount&#39;<br>            AND RunDate &gt;= DATEADD(day, -30, GETDATE())<br>        &quot;&quot;&quot;<br>        avg = self.run_query(query_hist)[&#39;AvgVol&#39;].iloc[0]<br>        <br>        # 3. Compare<br>        dev = abs(current - avg) / avg<br>        status = &quot;Pass&quot; if dev &lt;= deviation_threshold else &quot;Warning&quot;<br>        self._log_result(table_name, &quot;Consistency&quot;, &quot;VolumeDeviation&quot;, dev, deviation_threshold, status)<br><br>    def save_to_warehouse(self):<br>        &quot;&quot;&quot;Flushes results to dq.ValidationLog table&quot;&quot;&quot;<br>        # Batch insert logic...</pre><h3>3. The Orchestration: Connecting the Dots</h3><p>The final piece is a simple loop. It reads the registry, decides which function to call, and executes it. This script never needs to change, even if you add 100 new tables to the registry.</p><pre># Initialize<br>validator = DataHealthValidator(ENDPOINT, DB)<br><br># Fetch Active Rules<br>rules = validator.run_query(&quot;SELECT * FROM dq.RuleRegistry WHERE IsActive = 1&quot;)<br><br># Execute<br>for i, row in rules.iterrows():<br>    table = f&quot;{row[&#39;SchemaName&#39;]}.{row[&#39;TableName&#39;]}&quot;<br>    check = row[&#39;CheckType&#39;]<br>    <br>    if check == &#39;Timeliness&#39;:<br>        validator.check_timeliness(table, row[&#39;ColumnName&#39;], row[&#39;Threshold&#39;], row[&#39;Parameter&#39;])<br>    elif check == &#39;Consistency&#39;:<br>        validator.check_consistency(table, deviation_threshold=row[&#39;Threshold&#39;])<br>    # ... handle other check types ...<br><br># Save History<br>validator.save_to_warehouse()</pre><h3>Why build this instead of using “Great Expectations”?</h3><p>A common question is: <em>“Why not use an off-the-shelf library like Great Expectations (GX)?”</em></p><p>GX is a fantastic tool for data testing, and we evaluated it. However, we chose a custom framework for three specific reasons related to <strong>Observability</strong> vs. <strong>Testing</strong>:</p><ol><li><strong>The Output Schema:</strong> GX outputs complex JSON objects designed for its own HTML reports. We needed a flat, simple SQL table (ValidationLog) to feed our existing Power BI dashboard. Building a parser for GX JSON was more effort than writing the generic SQL checks ourselves.</li><li><strong>Push-Down Performance:</strong> We wanted to leverage our Data Warehouse’s compute. GX often pulls data into memory (Pandas/Spark) to validate it. Our check_timeliness runs a simple SELECT MAX(Date) on the server, which is instant and costs almost nothing in data transfer.</li><li><strong>Dependencies:</strong> In a restricted enterprise environment, installing heavy Python dependencies can be a hurdle. Our framework uses standard libraries (pandas, pyodbc) that are native to almost every environment.</li></ol><h3>Beyond Fabric: A Universal Pattern</h3><p>While I built this on Microsoft Fabric, the architecture is <strong>technology agnostic</strong>. The logic resides in the Python class, and the rules reside in a SQL table.</p><p>To port this to <strong>Snowflake</strong> or <strong>Databricks</strong>, you simply change the _get_db_connection method in the Python class to use the respective connector. The Rule Registry, the Logic Pillars, and the Power BI dashboard remain exactly the same.</p><h3>Conclusion</h3><p>By modularizing our data quality logic, we transformed a set of rigid Power BI workarounds into a flexible, enterprise-grade engine. We now have a system where:</p><ul><li><strong>Data Engineers</strong> can add checks via SQL.</li><li><strong>Analysts</strong> can visualize health trends in Power BI.</li><li><strong>The Business</strong> can trust the data because every table has a transparent, auditable health record.</li></ul><p>This is the difference between “checking” data and “observing” it.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8f08247c9cec" width="1" height="1" alt=""><hr><p><a href="https://towardsdev.com/beyond-the-dashboard-architecting-a-modular-metadata-driven-data-observability-framework-8f08247c9cec">Beyond the Dashboard: Architecting a Modular, Metadata-Driven Data Observability Framework</a> was originally published in <a href="https://towardsdev.com/">Towards Dev</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why My RAG System Failed Randomly — And How I Fixed It]]></title>
            <link>https://towardsdev.com/why-my-rag-system-failed-randomly-and-how-i-fixed-it-ac35971d9cc4?source=rss-d741b64b39b0------2</link>
            <guid isPermaLink="false">https://medium.com/p/ac35971d9cc4</guid>
            <category><![CDATA[rags]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Islam Taha]]></dc:creator>
            <pubDate>Sat, 29 Nov 2025 08:36:30 GMT</pubDate>
            <atom:updated>2025-12-01T10:39:06.937Z</atom:updated>
            <content:encoded><![CDATA[<h3>Why My RAG System Failed Randomly — And How I Fixed It</h3><h4><em>A deep dive into LLM non-determinism, table-heavy documents, hybrid retrieval, and what it actually takes to make RAG production-ready</em></h4><p>Most Retrieval-Augmented Generation (RAG) tutorials look clean and simple:<br>chunk text → embed → vector search → answer.</p><p>I followed the same formula when I started building my own RAG system.</p><p>It passed tests.<br>It worked on demo PDFs.<br>It looked production-ready.</p><p>And then… something impossible happened.</p><p>My tests started failing <strong>randomly</strong>.</p><p>Same code.<br>Same tests.<br>Same inputs.<br>Different outputs.</p><p>Not just occasionally — <em>consistently inconsistently</em>.</p><p>This article is the story of how I discovered the real reasons RAG systems break in the real world, what I found under the hood, and how I redesigned the architecture from the ground up.</p><h3>🚨 The Symptom: Random Test Failures</h3><p>I had just finished building an end-to-end ingestion and query pipeline for table-heavy PDFs — documents like financial reports and research papers where the <strong>most important information sits inside structured tables</strong> that typical RAG systems mishandle. <em>(See architecture diagram below.)</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VFVeQobVtG79H0C3FxcqWQ.png" /><figcaption>Complete System Architecture</figcaption></figure><p>I wrote 20 solid test cases.</p><p>Then the madness began:</p><pre>Run 1 → 3 failures  <br>Run 2 → 2 failures  <br>Run 3 → 0 failures  <br>Run 4 → 1 failure  <br>Run 5 → 0 failures</pre><p>I changed <em>nothing</em>.</p><p>How do you debug a system that doesn’t behave the same way twice?</p><h3>🕵️ The Investigation</h3><p>To stay sane, I created scripts to inspect what the system was doing at each step.</p><h3>MongoDB</h3><p>All tables extracted correctly.<br>Data looked clean.<br>Indexes worked.</p><h3>Qdrant</h3><p>Vector embeddings were consistent.<br>Chunk storage looked good.<br>Semantic search returned relevant text.</p><h3>LLM Context Window</h3><p>The model <strong>received the correct context every time</strong>.</p><p>So why did the model sometimes answer correctly and sometimes hallucinate or omit data?</p><p>This is when I realized:</p><p>The pipeline wasn’t broken.<br><strong>The LLM was.</strong></p><h3>🔍 The Root Causes (3 Hidden Problems)</h3><p>After days of debugging, I found <strong>three issues combining into one nightmare</strong>.</p><h3>1. LLM Non-Determinism</h3><p>I assumed Ollama would default to a low temperature.</p><p>It didn’t.</p><p>Default temperature ≈ 0.8.</p><p>That means:</p><p>Same prompt → different answer<br>Same data → different reasoning<br>Same table → different interpretation</p><p>RAG usually <em>looks</em> deterministic, but it isn’t. A temperature of 0.8 makes borderline cases fail unpredictably. This alone explained half of my “random” failures.</p><h3>2. Duplicate Table Data (The Silent Killer)</h3><p>PDFs are messy.</p><p>The table:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/656/1*-6aj3TAoFxtQGo51ev7ftw.png" /></figure><p>…also appears as:</p><pre>Table data: Phase Requirements 2024-01-15 Review docs […]</pre><p>So the LLM saw both:</p><ul><li>a clean, structured table</li><li>a garbled text-chunk version</li></ul><p>Same data, two formats, contradictory cues.</p><p>The model had no idea which version to trust.<br>Sometimes it extracted from the real table.<br>Sometimes from the noisy text chunk.<br>Sometimes mixed both.</p><p>This caused the most perplexing intermittent failures.</p><h3>3. My Prompt Was Too Polite</h3><p>I originally wrote something like:</p><blockquote><em>“Use the tables provided. Consider all rows.”</em></blockquote><p>LLMs treat this as a suggestion.</p><p>In borderline cases, the model ignored:</p><ul><li>first rows</li><li>parenthetical notes</li><li>document titles</li><li>date columns</li><li>list completeness</li></ul><p>For narrative text, vague prompts work fine.</p><p>For structured data, vague prompts break everything.</p><h3>🛠️ The Fix: Rebuilding for Determinism</h3><p>Once I understood the problems, the solutions became clear.</p><h3>✔️ 1. Force Deterministic Output</h3><p>I introduced fixed temperature presets:</p><pre>class QueryEngine:<br>    TEMPERATURE_DETERMINISTIC = 0.0  # default</pre><p>With temperature = 0.0:</p><ul><li>Same query → same output</li><li>Tests became reproducible</li><li>No more randomness</li></ul><p>This instantly stabilized the system.</p><h3>✔️ 2. Intelligent Table-Chunk Filtering</h3><p>I built heuristics to identify and remove table-like chunks:</p><ul><li>“Table data:” prefix</li><li>3+ YYYY-MM-DD patterns</li><li>3+ currency formats</li><li>alternating text/number patterns</li><li>abnormal spacing/tab density</li></ul><p>This removed <strong>duplicated table noise</strong> before embedding.</p><p>The LLM now sees <strong>only one version</strong> of each table — the correct one.</p><h3>✔️ 3. Ultra-Explicit Instructions for the LLM</h3><p>I rewrote the prompt like a strict set of rules:</p><ol><li><strong>Document titles matter</strong> (temporal context)</li><li><strong>Read every row in each table</strong></li><li><strong>Extract <em>all</em> values when asked</strong></li><li><strong>Never skip items in lists</strong></li><li><strong>Always include parenthetical notes</strong> (e.g., “(extended)”)</li></ol><p>This eliminated 100% of table-reading errors.</p><h3>💡 The Architecture (Hybrid RAG)</h3><p>Once everything was stable, the final ingestion and query flows looked like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*f6JRBc3jt9DCUEiavkNI2Q.png" /><figcaption>Detailed Ingestion Flow</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/927/1*ZNZ6XS3AQCy1qiRWPt61pg.png" /><figcaption>Detailed Query Flow</figcaption></figure><p>Why hybrid storage?</p><ul><li>Tables need SQL-like precision</li><li>Text needs semantic similarity search</li><li>Combining both yields near-perfect recall</li></ul><h3>🧪 Results: 20/20 Deterministic Test Passes</h3><p>After these changes:</p><pre>Run 1: 20/20  <br>Run 2: 20/20  <br>Run 3: 20/20</pre><p>Consistent.<br>Deterministic.<br>Production-ready.</p><p>Average response time: <strong>~2 seconds</strong>.<br>All processing done <strong>100% locally</strong> using Ollama.<br>8 PDFs included, covering 5 distinct structures.</p><h3>🏗️ What This Project Is Actually About (Besides RAG)</h3><p>Yes, I built a table-aware RAG system.</p><p>But the real takeaway is this:</p><h3>Real-world RAG requires engineering discipline, not just embeddings.</h3><p>What I learned:</p><ul><li>LLMs are not deterministic by default</li><li>PDF tables will break your RAG unless extracted separately</li><li>Noisy duplicates poison LLM reasoning</li><li>Prompting needs to be strict when dealing with structured data</li><li>Hybrid retrieval (semantic + structured) is essential at scale</li></ul><p>Most RAG content online doesn’t cover these realities.</p><h3>📦 Want to Try It? (Open Source, MIT License)</h3><p>👉 <strong>Repo:</strong> <a href="https://github.com/tahaislam/hybrid-rag-parser">https://github.com/tahaislam/hybrid-rag-parser</a><br>👉 Dual storage (MongoDB + Qdrant)<br>👉 Full Docker setup<br>👉 Local LLM (Ollama)<br>👉 20 deterministic tests<br>👉 Smart table filtering<br>👉 3 temperature presets</p><p>This is a production-grade baseline for RAG systems that must handle more than plain text.</p><h3>🎯 Final Thoughts</h3><p>If you’re building RAG for real documents — not blog posts — you’ll face the same issues:</p><ul><li>tables mixed with text</li><li>inconsistent formatting</li><li>LLM randomness</li><li>extraction ambiguity</li><li>retrieval mismatches</li></ul><p>The good news?</p><p>You can engineer your way out of all of them.</p><p>By combining deterministic LLM settings, smart preprocessing, and hybrid retrieval, you can build RAG systems that are <strong>stable</strong>, <strong>accurate</strong>, and <strong>production-ready</strong>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ac35971d9cc4" width="1" height="1" alt=""><hr><p><a href="https://towardsdev.com/why-my-rag-system-failed-randomly-and-how-i-fixed-it-ac35971d9cc4">Why My RAG System Failed Randomly — And How I Fixed It</a> was originally published in <a href="https://towardsdev.com/">Towards Dev</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building an AI-Powered Data Catalog with Prolog and Python]]></title>
            <link>https://towardsdev.com/building-an-ai-powered-data-catalog-with-prolog-and-python-24f56990e88f?source=rss-d741b64b39b0------2</link>
            <guid isPermaLink="false">https://medium.com/p/24f56990e88f</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[metadata]]></category>
            <category><![CDATA[prolog]]></category>
            <category><![CDATA[data-governance]]></category>
            <dc:creator><![CDATA[Islam Taha]]></dc:creator>
            <pubDate>Sun, 09 Nov 2025 10:31:25 GMT</pubDate>
            <atom:updated>2025-11-11T07:59:36.895Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Why Your Data Team Needs a Natural Language Interface</strong></p><blockquote><em>A practical guide to building an intelligent metadata management system using logic programming</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vk3vO2OsrO8bIzPz0muFgg.png" /></figure><p>Data catalogs like Microsoft Purview and Collibra cost $100K+ per year. They’ll auto-scan your databases and show pretty lineage graphs. But ask them <em>“find Gold datasets with PII from SAP lacking stewards”</em> and watch them struggle.</p><p>Let me show you a different approach: <strong>Prolog + AI, zero cost, infinite flexibility</strong>.</p><h3>The Problem</h3><p>Modern data platforms are complex. You have hundreds of tables across Bronze, Silver, and Gold layers. Data analysts ask questions like:</p><ul><li>“Where does this column come from?”</li><li>“What breaks if I change this table?”</li><li>“Where is our customer PII stored?”</li><li>“Which datasets lack proper governance?”</li></ul><p>These questions require traversing complex relationships, checking business rules, and analyzing data lineage. Traditional SQL queries or BI dashboards fall short because they can’t handle:</p><ol><li><strong>Recursive relationships</strong> (full lineage chains)</li><li><strong>Complex business rules</strong> (governance policies)</li><li><strong>Graph traversal</strong> (impact analysis)</li><li><strong>Natural language</strong> (talking to your data catalog)</li></ol><h3>The Solution: Logic Programming Meets Python</h3><p>I built a metadata management system that combines:</p><ul><li><strong>Prolog</strong>: For logic-based queries and rules</li><li><strong>Python</strong>: For data processing and user interface</li><li><strong>Natural Language</strong>: Ask questions in plain English</li><li><strong>Local AI</strong>: Intelligence without sending data to the cloud</li></ul><p>The result? A system that lets you ask:</p><pre>💬 &quot;Show me all Gold datasets&quot;<br>💬 &quot;Trace lineage for customer_orders.email&quot;<br>💬 &quot;Find governance violations&quot;<br>💬 &quot;Where is confidential data?&quot;</pre><p>And get instant, accurate answers — <strong>with your data never leaving your infrastructure</strong>.</p><h3>Architecture Overview</h3><h3>The Three Layers</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/693/1*dSByeBZqDdsHywWSHcxqFg.png" /></figure><h3>1. Metadata Storage (Prolog Facts)</h3><p>Instead of traditional databases, I use Prolog facts:</p><pre>% Dataset fact<br>dataset(&#39;V0001&#39;, &#39;gold_customer_360&#39;, &#39;Gold&#39;, <br>        &#39;Customer 360 View&#39;, &#39;Customer&#39;, &#39;Customer Data&#39;,<br>        &#39;CRM Platform&#39;, &#39;CRM001&#39;, &#39;Alice Johnson&#39;, <br>        &#39;Bob Smith&#39;, &#39;Data Engineering&#39;, &#39;Emily Brown&#39;).<br><br>% Data point fact with lineage<br>datapoint(&#39;DP00001&#39;, &#39;Approved&#39;, &#39;2024-03-15&#39;, &#39;Active&#39;,<br>          &#39;Customer&#39;, &#39;V0001&#39;, &#39;gold_customer_360&#39;, <br>          &#39;email&#39;, &#39;Email Address&#39;, &#39;...&#39;, &#39;Production&#39;,<br>          &#39;Complete&#39;, &#39;string&#39;, &#39;Customer email address&#39;,<br>          &#39;DP00050&#39;, &#39;silver_customer_crm&#39;, &#39;email&#39;,<br>          ..., &#39;Confidential&#39;, &#39;Contains PII&#39;, &#39;Yes&#39;).</pre><p>Why Prolog? Because it excels at:</p><ul><li>Pattern matching</li><li>Recursive queries</li><li>Rule-based reasoning</li><li>Graph traversal</li></ul><h3>2. Business Logic (Prolog Rules)</h3><p>Define complex rules once, use them everywhere:</p><pre>% Find datasets without proper governance<br>governance_violation(ViewName, &#39;Missing Reviewer&#39;) :-<br>    dataset(_, ViewName, _, _, _, _, _, _, Reviewer, _, _, _),<br>    var(Reviewer).<br><br>% Trace full lineage recursively with cycle detection<br>full_lineage(ViewName, ColumnName, Lineage) :-<br>    full_lineage_helper(ViewName, ColumnName, <br>                       [ViewName-ColumnName], Lineage).<br><br>full_lineage_helper(ViewName, ColumnName, Visited, []) :-<br>    \+ immediate_source(ViewName, ColumnName, _, _).<br><br>full_lineage_helper(ViewName, ColumnName, Visited, <br>                   [SourceView-SourceCol|Rest]) :-<br>    immediate_source(ViewName, ColumnName, SourceView, SourceCol),<br>    \+ member(SourceView-SourceCol, Visited),<br>    full_lineage_helper(SourceView, SourceCol, <br>                       [SourceView-SourceCol|Visited], Rest).<br><br>% Find all downstream dependencies<br>downstream_impact(SourceView, SourceCol, TargetView, TargetCol) :-<br>    immediate_source(TargetView, TargetCol, SourceView, SourceCol).</pre><h3>3. Natural Language Interface (Python)</h3><p>Translate English to Prolog queries with <strong>optional AI enhancement</strong>:</p><pre>class NaturalLanguageInterface:<br>    def __init__(self, use_ai=False):<br>        &quot;&quot;&quot;Initialize with optional Ollama AI enhancement&quot;&quot;&quot;<br>        self.prolog = Prolog()<br>        self.use_ai = use_ai<br>        if use_ai:<br>            self._init_ollama()  # Connect to local Ollama<br>    <br>    def translate_to_prolog(self, question):<br>        &quot;&quot;&quot;Translate natural language to Prolog&quot;&quot;&quot;<br>        <br>        # Try AI translation first if enabled<br>        if self.use_ai and self.ollama_available:<br>            query, message = self.translate_with_ai(question)<br>            if query:<br>                return query, message<br>        <br>        # Fall back to pattern matching<br>        question_lower = question.lower()<br>        <br>        if &#39;gold datasets&#39; in question_lower:<br>            return &quot;datasets_in_layer(&#39;Gold&#39;, ViewName)&quot;<br>        <br>        if &#39;lineage for&#39; in question_lower:<br>            match = re.search(r&#39;lineage for (\w+)\.(\w+)&#39;, <br>                            question_lower)<br>            if match:<br>                view, col = match.groups()<br>                return f&quot;full_lineage(&#39;{view}&#39;, &#39;{col}&#39;, Lineage)&quot;<br>        <br>        # ... more patterns</pre><p><strong>Two modes available:</strong></p><p>1.<strong>Pattern Matching</strong> (default) — Fast, works offline</p><pre>python nl_interface.py &quot;Show Gold datasets&quot;</pre><p>2. <strong>AI Enhanced</strong> (with Ollama) — Superior understanding</p><pre>python nl_interface.py --ai &quot;what is the impact if I changed gold_customer_data.email&quot;</pre><p>The AI mode uses Ollama (local LLM) to understand complex questions, handle typos, and interpret context. It’s optional but provides a much better user experience!</p><h3>Real-World Use Cases</h3><h3>Use Case 1: Data Lineage</h3><p><strong>Question</strong>: “Where does gold_revenue.amount come from?&quot;</p><p><strong>System traces through</strong>:</p><pre>gold_revenue.amount<br>  ↑ silver_orders.total_amount<br>    ↑ bronze_sales.order_total<br>      ↑ raw_transactions.amount</pre><p><strong>Prolog makes this easy</strong> because it handles recursion naturally:</p><pre>full_lineage(&#39;gold_revenue&#39;, &#39;amount&#39;, Lineage)</pre><p>Try doing that in SQL!</p><h3>Use Case 2: Impact Analysis</h3><p><strong>Question</strong>: “What breaks if I change raw_customers.customer_id?&quot;</p><p><strong>System finds</strong>:</p><ul><li>47 downstream columns affected</li><li>Across 12 different tables</li><li>Spanning all 3 layers</li></ul><p><strong>Why this matters</strong>: Before making schema changes, you know exactly what’s impacted.</p><h3>Use Case 3: Governance Compliance</h3><p><strong>Question</strong>: “Find all PII without proper data stewards”</p><p><strong>System checks</strong>:</p><pre>confidential_data(ViewName, ColumnName),<br>datapoint_without_steward(ViewName, ColumnName)</pre><p><strong>Results</strong>: Immediate list of compliance violations to fix.</p><h3>Implementation Guide</h3><h3>Step 1: Model Your Metadata</h3><p>Create an Excel template with two sheets:</p><p><strong>Dataset Sheet</strong>:</p><ul><li>ViewID, ViewName, DataLayer, SubjectArea, DataSource, Reviewer, Validator, etc.</li></ul><p><strong>DataPoint Sheet</strong>:</p><ul><li>DatapointID, ViewName, ColumnName, SourceViewName, SourceColumnName, SensitivityLabel, DataSteward, DataOwner, etc.</li></ul><h3>Step 2: Convert to Prolog Facts</h3><pre>def convert_excel_to_prolog(excel_file):<br>    datasets_df = pd.read_excel(excel_file, sheet_name=&#39;Dataset&#39;)<br>    <br>    with open(&#39;metadata_facts.pl&#39;, &#39;w&#39;) as f:<br>        for _, row in datasets_df.iterrows():<br>            fact = f&quot;dataset(&#39;{row[&#39;ViewID&#39;]}&#39;, &quot; \<br>                   f&quot;&#39;{row[&#39;ViewName&#39;]}&#39;, ...).\n&quot;<br>            f.write(fact)</pre><h3>Step 3: Define Business Rules</h3><pre>% Your domain-specific rules<br>high_risk_dataset(ViewName) :-<br>    dataset(_, ViewName, &#39;Gold&#39;, _, _, _, _, _, _, _, _, _),<br>    confidential_data(ViewName, _),<br>    \+ has_reviewer(ViewName).</pre><h3>Step 4: Build the Interface</h3><pre>interface = NaturalLanguageInterface()<br>interface.ask(&quot;Find high-risk datasets&quot;)</pre><h3>Why This Approach Works</h3><h3>1. Separation of Concerns</h3><ul><li><strong>Data</strong>: Prolog facts (auto-generated)</li><li><strong>Logic</strong>: Prolog rules (business logic)</li><li><strong>Interface</strong>: Python (user interaction)</li></ul><h3>2. Declarative Programming</h3><p>Write <strong>what</strong> you want, not <strong>how</strong> to get it:</p><pre>% What you want<br>complete_pipeline(SubjectArea) :-<br>    has_bronze(SubjectArea),<br>    has_silver(SubjectArea),<br>    has_gold(SubjectArea).<br><br>% vs. Imperative (Python)<br>def complete_pipeline(subject_area):<br>    bronze = check_layer(subject_area, &#39;Bronze&#39;)<br>    if not bronze:<br>        return False<br>    silver = check_layer(subject_area, &#39;Silver&#39;)<br>    if not silver:<br>        return False<br>    gold = check_layer(subject_area, &#39;Gold&#39;)<br>    return gold</pre><p>The Prolog version is clearer and maintainable.</p><h3>3. Natural Recursion</h3><p>Lineage and impact analysis require recursion. Prolog handles it elegantly:</p><pre>% One predicate for any depth<br>full_lineage(View, Col, Lineage) :- ...</pre><p>Try writing that in SQL with CTEs for arbitrary depth!</p><h3>4. Extensibility</h3><p>Adding new rules is trivial:</p><pre>% New rule: Find orphaned data<br>orphaned_data(ViewName) :-<br>    dataset(_, ViewName, &#39;Gold&#39;, _, _, _, _, _, _, _, _, _),<br>    \+ has_downstream_usage(ViewName).</pre><p>No schema changes, no migrations needed.</p><h3>5. AI Enhancement (Optional)</h3><p>With Ollama integration:</p><ul><li>Pattern matching: Good for standard queries, instant</li><li>AI enhanced: Handles any question, ~1–2 seconds</li></ul><p><strong>Pattern matching struggles:</strong></p><pre>❌ &quot;Show me tables with PII and no stewards&quot;  <br>❌ &quot;What are the problematic customer datasets?&quot;</pre><p>AI handles easily:</p><pre>✅ Understands compound conditions<br>✅ Interprets &quot;problematic&quot; as governance violations<br>✅ Maps natural language to precise Prolog</pre><p>Just add --ai flag:</p><pre>python nl_interface.py --ai &quot;Find compliance issues&quot;</pre><h3>6. Privacy-First AI</h3><p>This is critical: <strong>Your data never leaves your infrastructure.</strong></p><p><strong>Traditional AI tools:</strong></p><ul><li>Alation: Proprietary AI, data sent to vendor servers</li><li>Collibra: GenAI descriptions, cloud-based processing</li><li>Commercial tools: Your metadata goes to their cloud</li></ul><p><strong>Your tool with Ollama:</strong></p><ul><li>Runs 100% locally on your machine</li><li>No internet connection required</li><li>No data sent to external servers</li><li>Free, unlimited queries</li><li>Full control over model and data</li></ul><p><strong>Why this matters:</strong></p><pre># Your sensitive query<br>&quot;Find datasets with customer PII and credit card data&quot;<br><br># Commercial tool: Sent to vendor cloud for processing<br># Your tool: Processed locally, never leaves your laptop</pre><p>For regulated industries (finance, healthcare, government), this isn’t just nice to have — <strong>it’s a requirement</strong>.</p><h3>Performance Considerations</h3><p><strong>Question</strong>: “Isn’t Prolog slow?”</p><p><strong>Answer</strong>: Not for this use case.</p><ul><li>Metadata is relatively small (thousands of facts, not millions)</li><li>Queries execute in milliseconds</li><li>Perfect for interactive use</li><li>Can handle 10K+ datapoints easily</li></ul><p>For massive scale (100K+ tables), you might need optimizations:</p><ul><li>Indexing (Prolog supports this)</li><li>Caching common queries</li><li>Pre-computing expensive relations</li></ul><p>But for most organizations, vanilla Prolog is plenty fast.</p><h3>How This Compares to Commercial Tools</h3><p>You might be wondering: <em>“How does this stack up against established solutions like Microsoft Purview, Collibra, or Alation?”</em></p><p>Fair question. Here’s an honest comparison:</p><h3>What This Approach Does Differently</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/694/1*-mvR3RlAR0z4X-rqqB6_Ig.png" /></figure><h3>What Commercial Tools Do Better</h3><p>Let’s be clear about trade-offs:</p><p><strong>Automated Scanning</strong> ✅ Them</p><ul><li>Purview/Collibra automatically scan databases and capture metadata</li><li>This tool requires manual Excel maintenance</li><li><strong>Trade-off:</strong> You get simplicity and control, they get automation</li></ul><p><strong>Visual Interfaces</strong> ✅ Them</p><ul><li>Beautiful dashboards and interactive lineage graphs</li><li>This tool is text-based</li><li><strong>Trade-off:</strong> You get speed and scriptability, they get visuals</li></ul><p><strong>Enterprise Features</strong> ✅ Them</p><ul><li>Full RBAC, SSO, audit trails, multi-tenancy</li><li>This tool has basic capabilities</li><li><strong>Trade-off:</strong> You get simplicity, they get enterprise-grade security</li></ul><p><strong>Scale</strong> ✅ Them</p><ul><li>Designed for 1000+ users across organizations</li><li>This tool is optimized for smaller teams (5–50 users)</li><li><strong>Trade-off:</strong> You get agility, they get massive scale</li></ul><h3>When to Choose Each Approach</h3><p><strong>Choose this tool if you:</strong></p><ul><li>Have a small-medium team (5–50 people)</li><li>Need complex, custom governance logic</li><li>Value code-first, scriptable solutions</li><li>Want zero licensing costs</li><li>Need privacy-first AI (local processing)</li><li>Can maintain metadata in Excel</li></ul><p><strong>Choose commercial tools if you:</strong></p><ul><li>Have enterprise scale (100+ users)</li><li>Need automated metadata scanning</li><li>Prefer GUI-driven workflows</li><li>Require comprehensive audit trails</li><li>Have budget for licensing</li></ul><p><strong>The key insight:</strong> These aren’t competing solutions — they serve different markets. Commercial tools are excellent for large enterprises with big budgets. This approach is excellent for everyone else who needs power, flexibility, and privacy at zero cost.</p><h3>Lessons Learned</h3><h3>What Worked Well</h3><ol><li><strong>Prolog for rules</strong>: Expressing complex logic is much easier</li><li><strong>Python interface</strong>: Users don’t need to know Prolog</li><li><strong>Excel as source</strong>: Non-technical users can maintain metadata</li><li><strong>Automatic fact generation</strong>: No manual Prolog coding needed</li></ol><h3>What Was Challenging</h3><ol><li><strong>Prolog learning curve</strong>: Team needs training</li><li><strong>Debugging</strong>: Prolog errors can be cryptic</li><li><strong>Tooling</strong>: IDE support is limited compared to Python</li><li><strong>Integration</strong>: Some friction with existing Python stack</li></ol><h3>Would I Do It Again?</h3><p><strong>Absolutely</strong>. The benefits far outweigh the challenges:</p><ul><li>Complex queries that were impossible are now trivial</li><li>Business rules are clear and maintainable</li><li>Natural language interface delights users</li><li>System is extensible and powerful</li></ul><h3>Try It Yourself</h3><p>I’ve open-sourced the full system:</p><p><strong>GitHub</strong>: <a href="https://github.com/tahaislam/prolog-data-catalog">https://github.com/tahaislam/prolog-data-catalog</a></p><p>Includes:</p><ul><li>✅ Sample data generator</li><li>✅ Excel to Prolog converter</li><li>✅ Complete rule library</li><li>✅ Natural language interface</li><li>✅ Comprehensive tests</li><li>✅ User guide</li></ul><p><strong>Quick Start</strong>:</p><pre>git clone https://github.com/tahaislam/prolog-data-catalog.git<br>pip install -r requirements.txt<br>python generate_sample_data.py<br>python excel_to_prolog.py<br>python nl_interface.py</pre><h3>Conclusion</h3><p><strong>Data governance doesn’t have to be painful</strong>. By combining:</p><ul><li>Logic programming (Prolog)</li><li>Modern Python tooling</li><li>Privacy-first AI (local Ollama)</li></ul><p>You can build a system that’s both powerful and user-friendly.</p><p>The key insight: <strong>Use the right tool for each layer</strong>:</p><ul><li>Prolog for complex logic and rules</li><li>Python for data processing and UX</li><li>Natural language for accessibility</li><li>Local AI for intelligence without compromise</li></ul><p><strong>Your data team will thank you</strong> when they can ask questions like:</p><blockquote><em>“Find all Gold datasets from the ERP system in the Finance subject area that contain PII but don’t have data stewards assigned”</em></blockquote><p>And get instant answers.</p><h3>A Final Word</h3><p>I’m not trying to replace Microsoft Purview, Collibra, or Alation. Those are excellent tools for large enterprises with the budgets to match.</p><p><strong>I’m serving a market they overlook:</strong> teams that need power without complexity, intelligence without subscriptions, and governance without vendor lock-in.</p><p>If you’re a data engineer tired of clicking through UIs, a startup that can’t afford six-figure licenses, or a privacy-conscious organization that wants AI to stay local — this approach is for you.</p><p>The code is open source. The innovation is proven. The cost is zero.</p><p><strong>Data governance, democratized.</strong> 🚀</p><h3>Next Steps</h3><p>Interested in applying this to your organization?</p><ol><li><strong>Start small</strong>: Model one subject area</li><li><strong>Define core rules</strong>: Governance, lineage, compliance</li><li><strong>Build interface</strong>: Start with simple queries</li><li><strong>Iterate</strong>: Add more rules as you learn</li></ol><p><strong>Questions or feedback?</strong> Leave a comment or open an issue on GitHub!</p><p>If you found this useful, please give the repo a ⭐ and share with your data team!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=24f56990e88f" width="1" height="1" alt=""><hr><p><a href="https://towardsdev.com/building-an-ai-powered-data-catalog-with-prolog-and-python-24f56990e88f">Building an AI-Powered Data Catalog with Prolog and Python</a> was originally published in <a href="https://towardsdev.com/">Towards Dev</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a Modern Data Warehouse: A Hands-On Guide with Snowflake, dbt, and Python]]></title>
            <link>https://towardsdev.com/building-a-modern-data-warehouse-a-hands-on-guide-with-snowflake-dbt-and-python-936271bae875?source=rss-d741b64b39b0------2</link>
            <guid isPermaLink="false">https://medium.com/p/936271bae875</guid>
            <category><![CDATA[fivetran]]></category>
            <category><![CDATA[dbt]]></category>
            <category><![CDATA[snowflake]]></category>
            <category><![CDATA[data-pipeline]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Islam Taha]]></dc:creator>
            <pubDate>Mon, 18 Aug 2025 07:10:12 GMT</pubDate>
            <atom:updated>2025-08-21T14:58:04.328Z</atom:updated>
            <content:encoded><![CDATA[<h4>From Python scripts and Fivetran to a fully automated Snowflake and dbt pipeline, here’s a complete guide for data engineers.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/932/1*PLf9yAtkDybXZoUBb8fKUQ.png" /></figure><h3>Introduction</h3><p>The best way to master data engineering is to build a real system from the ground up. That’s why I’m excited to share my latest portfolio project — a complete, end-to-end data pipeline that you can explore, clone, and run yourself.</p><p><strong>You can find all the code and setup instructions on GitHub </strong><a href="https://github.com/tahaislam/snowflake-elt-pipeline"><strong>here</strong></a></p><p>This isn’t just a theoretical exercise; it’s a practical blueprint that covers multiple ingestion patterns and two different orchestration philosophies. We’ll go beyond theory and build a multi-faceted system that ingests data from various sources, transforms it into analytics-ready models, and automates the entire workflow. Let’s dive in.</p><h3>Part 1: The Foundation — A Well-Architected Snowflake Environment</h3><p>Before any data can flow, we need a solid foundation. Instead of creating objects on the fly, this project uses a series of numbered SQL scripts (01_ to 05_) to build the Snowflake environment in a repeatable and organized way.</p><h4>Role-Based Access Control (RBAC)</h4><p>A key principle here is the <strong>principle of least privilege</strong>. We don’t use powerful roles like SYSADMIN for our tools. Instead, we create dedicated service roles:</p><ul><li>FIVETRAN_ROLE: This role has the minimal permissions needed for Fivetran to do its job: USAGE on the database and CREATE TABLE on the specific raw schemas it writes to.</li><li>TRANSFORMER_ROLE: This is the role our dbt project uses. It has read-only access to the raw &quot;bronze&quot; schemas and write access to our &quot;silver&quot; and &quot;gold&quot; transformation schemas.</li></ul><p>This separation is critical for security and manageability in a production environment.</p><h4>The Bronze, Silver, and Gold Schema Design</h4><p>Our database (PROJECT_DB) is organized into logical layers:</p><ul><li><strong>Bronze (The </strong><strong>RAW_* schemas):</strong> This layer holds raw, untouched data exactly as it was ingested by our sources (Fivetran, Python scripts).</li><li><strong>Silver (The </strong><strong>STAGING schema):</strong> This is where our dbt staging models live. They perform the initial cleaning, renaming, and type-casting of the raw data.</li><li><strong>Gold (The </strong><strong>ANALYTICS schema):</strong> This is the final, user-facing layer. Our dbt marts models create aggregated, business-ready tables here, perfect for connecting to a BI tool.</li></ul><h3>Part 2: Ingestion — Custom Python Scripts</h3><p>While tools like Fivetran are powerful, a data engineer must know how to build custom ingestion pipelines. This project includes two distinct Python scripts to demonstrate this.</p><h4>The Simple Approach: ingest_data.py</h4><p>This script uses the pandas and SQLAlchemy libraries to fetch data from the JSONPlaceholder API and load it directly into a Snowflake table using the to_sql() method.</p><ul><li><strong>Key takeaway:</strong> This method is great for simplicity and smaller datasets. However, as we discovered, it requires careful transaction management (connection.commit()) to ensure data isn&#39;t silently rolled back when the connection closes.</li></ul><h4>The Production Pattern: ingest_nyc_taxi.py</h4><p>This script demonstrates a more robust and scalable approach, ideal for larger files like the NYC Taxi dataset.</p><ol><li>The script downloads the data and saves it locally as a <strong>Parquet file</strong>.</li><li>It then uses the native snowflake-connector-python to execute a PUT command, uploading the Parquet file to a Snowflake internal stage.</li><li>Finally, it runs a COPY INTO command, which leverages Snowflake&#39;s massively parallel processing capabilities to bulk-load the data from the stage into the final table.</li></ol><ul><li><strong>Key takeaway:</strong> This PUT/COPY pattern is the industry standard for loading large volumes of data into Snowflake efficiently.</li></ul><h3>Part 3: Transformation — A Structured dbt Project</h3><p>This is the heart of the project. Instead of a messy folder of SQL, we use a structured dbt project to manage our transformations.</p><h4>Layered Modeling (staging &amp; marts)</h4><ul><li><strong>Staging Models:</strong> Located in the models/staging/ folder, these models have a 1:1 relationship with a raw source table. They perform basic cleaning and standardization and are always materialized as views.</li><li><strong>Marts Models:</strong> Located in models/marts/, these are our final &quot;gold&quot; layer tables. They join multiple staging models, perform aggregations, and apply business logic. They are materialized as tables for fast query performance.</li></ul><h4>Configuration as Code (.yml files)</h4><ul><li><strong>dbt_project.yml:</strong> This is the project&#39;s blueprint. We use it to define our model paths and, crucially, to configure our models to build in different schemas (STAGING for staging models, ANALYTICS for marts models).</li><li><strong>sources.yml:</strong> This file acts as a map to our raw data. It declares our Fivetran and Python-loaded tables as sources, allowing us to test them for freshness and quality and to build a clear lineage graph using dbt&#39;s source() macro.</li></ul><h3>Part 4: Automation &amp; Orchestration</h3><p>Finally, we demonstrate two distinct ways to run our pipeline.</p><h4>Scenario 1: The Local Development Workflow</h4><p>This is the workflow an analytics engineer would use daily. By running dbt run and dbt test from the command line, we can quickly build and validate our models. For a production scenario, this local execution would be replaced by an external orchestrator like <strong>Apache Airflow</strong> or <strong>dbt Cloud</strong>.</p><h4>Scenario 2: The Serverless Cloud Workflow</h4><p>This pipeline runs entirely within Snowflake, requiring no external tools.</p><ol><li><strong>Snowflake Stream:</strong> We create a STREAM on the raw Google Sheets table. This object acts as a changelog, recording any new rows that Fivetran loads.</li><li><strong>Snowflake Task:</strong> We create a TASK that runs on a 5-minute schedule. The task is configured with a WHEN SYSTEM$STREAM_HAS_DATA(...) clause, which means it only wakes up and consumes warehouse credits if the stream has new data.</li></ol><ul><li><strong>Key takeaway:</strong> This creates a highly efficient, event-driven pipeline that automatically transforms data the moment it arrives, demonstrating a powerful serverless architecture.</li></ul><h3>Conclusion</h3><p>This project covers a wide range of essential data engineering concepts, from foundational setup to advanced automation. By building it, you’ll gain practical experience with a complete, modern data stack. I hope this serves as a valuable starting point and a clear guide for your own data engineering journey.</p><p>You can find all the code and detailed setup instructions in the <a href="https://github.com/tahaislam/snowflake-elt-pipeline">GitHub repository</a>. Feedback and questions are always welcome!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=936271bae875" width="1" height="1" alt=""><hr><p><a href="https://towardsdev.com/building-a-modern-data-warehouse-a-hands-on-guide-with-snowflake-dbt-and-python-936271bae875">Building a Modern Data Warehouse: A Hands-On Guide with Snowflake, dbt, and Python</a> was originally published in <a href="https://towardsdev.com/">Towards Dev</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Low-Code Data Validation Framework for Power BI]]></title>
            <link>https://towardsdev.com/a-low-code-data-validation-framework-for-power-bi-b9fd72738a17?source=rss-d741b64b39b0------2</link>
            <guid isPermaLink="false">https://medium.com/p/b9fd72738a17</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[automation]]></category>
            <category><![CDATA[power-bi]]></category>
            <category><![CDATA[etl]]></category>
            <category><![CDATA[data-quality]]></category>
            <dc:creator><![CDATA[Islam Taha]]></dc:creator>
            <pubDate>Sat, 02 Aug 2025 08:18:52 GMT</pubDate>
            <atom:updated>2025-08-06T14:44:57.741Z</atom:updated>
            <content:encoded><![CDATA[<h3>A practical guide for data engineers to automate data quality checks and de-risk deployments without leaving the Power BI ecosystem.</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*tnheWD7LddiVrPFT-AUXcQ.png" /></figure><p>Data engineers spend countless hours building robust pipelines in Power BI. Intricate dataflows are created, semantic models are fine-tuned, and powerful datamarts are deployed. Yet, a critical question often remains: “Is the data <em>correct</em>?” A single upstream error can corrupt a dashboard, erode user trust, and lead to hours of manual debugging.</p><p>Power BI is a premier tool for transformation and visualization, but it lacks a native, declarative framework for data validation. The common alternatives are manual spot-checks or complex, one-off Power Query logic that is difficult to manage and scale.</p><p>This article introduces a centralized, scalable, and low-code data validation framework built directly within Power BI, using only a Dataflow and two SharePoint lists.</p><h3>The Challenge: The Data Validation Gap in Power BI</h3><p>A proper validation system needs to automatically answer critical data quality questions:</p><ul><li>Is the total sales amount for today within an expected range?</li><li>Are there any NULL values in a CustomerID column?</li><li>Does the row count in the development version of the FactSales table match the production version?</li></ul><p>Performing these checks manually is not a <strong>scalable solution</strong>. An automated engine is required to run these tests and provide a clear report of any failures.</p><h3>The Framework: A Two-Part Solution</h3><p>This framework consists of two simple but powerful components:</p><ol><li><strong>The Rule Engine (Two SharePoint Lists):</strong> A centralized list where every validation rule is defined. It specifies <em>what</em> to test.</li><li><strong>The Validation Engine (A Power BI Dataflow):</strong> A single dataflow that reads the rules from SharePoint, executes them against Power BI artifacts (Dataflows, Semantic Models, Datamarts), and reports the results. It specifies <em>how</em> to test.</li></ol><h3>Part 1: The Rule Engine via SharePoint</h3><p>The foundation of this framework is a set of two SharePoint lists that act as the “rule engine.” This two-list approach separates the high-level <em>what-to-test</em> configuration from the low-level <em>how-to-test</em> rules, making the system cleaner and easier to manage.</p><h4>List 1: Validation Settings</h4><p>This primary list is designed to be simple, containing only a single item that tells the dataflow which artifacts to validate for the current run. This is where the user specifies the validation job.</p><p><strong>Columns:</strong></p><ul><li><strong>Artifact1Name</strong> (Text): The name of the source artifact to be validated (e.g., “Sales Staging Dataflow”).</li><li><strong>Artifact1Type</strong> (Choice: Dataflow Gen1, Datamart (Preview), SemanticModel): The type of the source artifact.</li><li><strong>Artifact1Workspace</strong> (Text): The name or ID of the Power BI workspace for the source artifact.</li><li><strong>Artifact2Name</strong> (Text): (Optional) The name of the target artifact for comparison (e.g., “Sales Production Dataflow”).</li><li><strong>Artifact2Type</strong> (Choice: Dataflow Gen1, Datamart (Preview), SemanticModel): (Optional) The type of the target artifact.</li><li><strong>Artifact2Workspace</strong> (Text): (Optional) The name or ID of the workspace for the target artifact.</li></ul><p><strong>Logic:</strong></p><ul><li>If the <strong>Artifact2</strong> columns are left empty, the framework runs in <strong>single-artifact validation mode</strong>. It will validate the tables in Artifact 1 against the predefined rules in the Validation Rules list.</li><li>If the <strong>Artifact2</strong> columns are filled, the framework runs in <strong>comparison mode</strong>, validating Artifact 1 against Artifact 2 on a rule-by-rule basis.</li></ul><h4>List 2: Validation Rules</h4><p>This list contains the specific, granular rules that will be applied to the artifacts defined in the Validation Settings list.</p><p><strong>Columns:</strong></p><ul><li><strong>ArtifactName</strong> (Text): The name of the artifact this rule applies to. This must match the name used in Validation Settings.</li><li><strong>TableName</strong> (Text): The specific table or query within the artifact to be validated.</li><li><strong>ColumnName</strong> (Text): The column to which the aggregation will be applied.</li><li><strong>AggregationType</strong> (Choice: Sum, CountBlank, CountNull, Average, Max, Min): The aggregation function to execute on the specified column.</li><li><strong>ExpectedValue</strong> (Text): A flexible text field defining the success criteria. This is only used in single-artifact validation mode.</li></ul><h4>Defining ExpectedValue Rules</h4><p>The ExpectedValue column is designed to be highly flexible by parsing text-based conditions. This allows for a wide range of validation scenarios without changing the underlying M code. The framework can be built to interpret the following formats:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/698/1*q3oK6jMbu3jjXsak1o4I6g.png" /></figure><p>This two-list setup provides a robust and scalable foundation for the validation framework, clearly separating the high-level configuration from the detailed rule definitions.</p><p>Of course. Here is the updated “Part 2” section, rewritten to align with the two-list setup and the specific columns you defined.</p><h3>Part 2: The Validation Engine via Power BI Dataflow</h3><p>A single Power BI Dataflow orchestrates the entire validation process. It acts as the engine that reads the configuration, connects to the data sources, executes the rules, and generates a final report. The logic inside the dataflow follows these steps:</p><ol><li><strong>Ingest Configuration and Determine Mode<br></strong>The process begins by connecting to the ValidationSettings SharePoint list. Since this list contains only one item, the dataflow reads that row to get the key attributes: Artifact1Name, Artifact1Type, Artifact1Workspace, and the optional Artifact2 details. The engine immediately checks if Artifact2Name is empty. This determines whether to run in single-artifact validation or two-artifact comparison mode.</li><li><strong>Load Rules and Access Artifacts<br></strong>Next, the dataflow connects to the ValidationRules list. It filters this list to retrieve only the rules where the ArtifactName column matches the Artifact1Name from the settings list. Using custom M functions, the dataflow then dynamically connects to the specified Power BI artifacts (Dataflow, Datamart, or Semantic Model) to access the tables targeted by the rules.</li><li><strong>Execute Validation Logic<br></strong>This is the core execution step, where the logic diverges based on the mode:<br><strong>• In Single-Artifact Mode</strong>: For each rule, the engine applies the specified AggregationType (e.g., Sum, CountNull) to the ColumnName within the TableName. The resulting value is then validated against the condition in the ExpectedValue column (e.g., checking if the sum is &gt;=100).<br><strong>• In Comparison Mode</strong>: For each rule, the engine performs the same aggregation on the same table and column in <strong>both</strong> Artifact 1 and Artifact 2. The ExpectedValue column from the rules list is ignored. The validation simply checks if the result from Artifact 1 is identical to the result from Artifact 2.</li><li><strong>Generate a Unified Report<br></strong>The final step is to consolidate the results into a single, clean output table. This table serves as the source for a monitoring dashboard and clearly shows the outcome of each test. The output typically includes columns like TableName, ColumnName, AggregationType, the actual result from Artifact 1, the expected/comparison result from Artifact 2, and a final status (e.g., “Passed” or “Failed”).</li></ol><h3>Core Framework Functions (M Code)</h3><p>The heart of the validation engine is a set of reusable Power Query functions. Below are code snippets of the key functions that drive the framework.</p><h4>Data Access Functions</h4><p>These functions connect to a given artifact and list its tables/queries. Below are three functions to cover Power BI Dataflows Gen1, Datamarts, and Semantic Models.</p><pre>// GetDataflowTables: Connects to a dataflow and returns a list of its tables.<br>(WorkspaceName as text, dataflowName as text) =&gt; let<br>  Source = PowerPlatform.Dataflows([]),<br>  Workspaces = Source{[Id = &quot;Workspaces&quot;]}[Data],<br>  CurrentWorkspace = Table.SelectRows(Workspaces, each [workspaceName] = WorkspaceName),<br>  Dataflows = CurrentWorkspace{[]}[Data],<br>  UpstreamDataflow = Table.SelectRows(Dataflows, each [dataflowName] = dataflowName),<br>  Tables = UpstreamDataflow{[]}[Data],<br>  Output = Table.RenameColumns(<br>    Table.SelectColumns(Tables, {&quot;entity&quot;, &quot;Data&quot;}),<br>    {&quot;entity&quot;, &quot;Name&quot;}<br>  )<br>in<br>  Output</pre><pre>// GetDatamartTables: Connects to a datamart and returns a list of its tables.<br>(workspaceName as text, datamartName as text) =&gt; let  <br>  Source = PowerBI.Datamarts(null),<br>  Datamarts = Source{[workspaceName = workspaceName]}[Data],<br>  Datamart = Table.SelectRows(Datamarts, each [displayName] = datamartName),<br>  Tables = Datamart{[]}[Data],<br>  Output = Table.RenameColumns(<br>    Table.SelectColumns(Tables, {&quot;Item&quot;, &quot;Data&quot;}),<br>    {&quot;Item&quot;, &quot;Name&quot;}<br>  )<br>in<br>  Output</pre><pre>// GetSemanticModelTables: Connects to a semantic model and returns its tables.<br>(WorkspaceName as text, ModelName as text) =&gt; let<br>  // Connect to the database<br>  Databases = Table.FromRecords({<br>    [WorkspaceName = &quot;Workspace1&quot;, Database = AnalysisServices.Databases(&quot;powerbi://api.powerbi.com/v1.0/myorg/Workspace1&quot;)],<br>    [WorkspaceName = &quot;Workspace2&quot;, Database = AnalysisServices.Databases(&quot;powerbi://api.powerbi.com/v1.0/myorg/Workspace2&quot;)]<br>  }),<br>  Workspace = Databases{[WorkspaceName = WorkspaceName]}[Database],<br>  Database = Workspace{[Name = ModelName]},<br>  // Connect to the semantic model<br>  Model = Database[Data],<br>  // Connect to the folder<br>  Folder = Model{[Id = &quot;Model&quot;]}[Data],<br>  // Connect to the cube<br>  Cube = Folder{[Id = &quot;Model&quot;]}[Data],  <br>  // Return the list of tables<br>  Tables = Table.SelectColumns(Cube.Dimensions(Cube), {&quot;Name&quot;, &quot;Data&quot;})<br>in<br>  Tables</pre><h4>Summarization Functions</h4><p>These functions iterate through the rules and apply them to the target tables.</p><pre>// SummarizeDataflow: Applies validation rules to a specified dataflow&#39;s tables.<br>(ValidationRules as table, ArtifactName as text, ArtifactWorkspace as text) =&gt;<br>let<br>  // Get all tables within this dataflow<br>  AllTables = GetDataflowTables(ArtifactWorkspace, ArtifactName),<br>  // Join tables&#39; data with validation rules<br>  AllTablesWithData = Table.SelectColumns(<br>    Table.Join(<br>        ValidationRules, {&quot;TableName&quot;},<br>        AllTables, {&quot;Name&quot;}<br>    ),<br>    {&quot;Id&quot;, &quot;TableName&quot;, &quot;ColumnName&quot;, &quot;AggregationType&quot;, &quot;Data&quot;, &quot;ExpectedValue&quot;}<br>  ),<br>  // Apply the aggregation function/operator<br>  AddAggregated = Table.AddColumn(AllTablesWithData, &quot;Result&quot;, each AggregateColumn([Data], [ColumnName], [AggregationType]), type number),<br>  // Remove extra columns<br>  Output = Table.RemoveColumns(AddAggregated, {&quot;Data&quot;})<br>in<br>  Output</pre><pre>// SummarizeDatamart: Applies validation rules to a specified datamart&#39;s tables.<br>(ValidationRules as table, ArtifactName as text, ArtifactWorkspace as text) =&gt;<br>let<br>  // Get all tables within this datamart<br>  AllTables = GetDatamartTables(ArtifactWorkspace, ArtifactName),<br>  AllTablesWithData = Table.SelectColumns(<br>    Table.Join(<br>        ValidationRules, {&quot;TableName&quot;},<br>        AllTables, {&quot;Name&quot;}<br>    ),<br>    {&quot;Id&quot;, &quot;TableName&quot;, &quot;ColumnName&quot;, &quot;AggregationType&quot;, &quot;Data&quot;, &quot;ExpectedValue&quot;}<br>  ),<br>  AddAggregated = Table.AddColumn(AllTablesWithData, &quot;Result&quot;, each AggregateColumn([Data], [ColumnName], [AggregationType]), type number),<br>  Output = Table.RemoveColumns(AddAggregated, {&quot;Data&quot;})<br>in<br>  Output</pre><pre>// SummarizeSemanticModel: Applies validation rules to a specified semantic model&#39;s tables.<br>(ValidationRules as table, ArtifactName as text, ArtifactWorkspace as text) =&gt;<br>let<br>  // Get all tables within this semantic model<br>  AllTables = GetSemanticModelTables(ArtifactWorkspace, ArtifactName),<br>  AllTablesWithData = Table.SelectColumns(<br>    Table.Join(<br>        ValidationRulesWithSummary, {&quot;TableName&quot;},<br>        AllTables, {&quot;Name&quot;}<br>    ),<br>    {&quot;Id&quot;, &quot;TableName&quot;, &quot;ColumnName&quot;, &quot;AggregationType&quot;, &quot;Data&quot;, &quot;ExpectedValue&quot;}<br>  ),<br>  AddAggregated = Table.AddColumn(AllTablesWithData, &quot;Result&quot;, each AggregateColumn([Data], &quot;[&quot; &amp; [TableName] &amp; &quot;].[&quot; &amp; [ColumnName] &amp; &quot;].[&quot; &amp; [ColumnName] &amp; &quot;]&quot;, [AggregationType]), type number),<br>  Output = Table.RemoveColumns(AddAggregated, {&quot;Data&quot;})<br>in<br>  Output</pre><h4>Core Logic Functions</h4><p>These are the fundamental building blocks for executing a single validation test.</p><pre>// AggregateColumn: Applies a specified aggregation (e.g., Sum, Max, Count) to a table column.<br>(tbl as table, colName as text, op as text) as any =&gt;<br>let<br>    // Extract the column<br>    colRaw = Table.Column(tbl, colName),<br>    // Convert to number if needed<br>    colNumeric = List.Transform(colRaw, each try Number.From(_) otherwise null),<br>    // Choose the appropriate column based on operator<br>    col = if List.Contains({&quot;Sum&quot;, &quot;Average&quot;, &quot;Min&quot;, &quot;Max&quot;}, op) then colNumeric else colRaw,<br>    // Apply the aggregation function<br>    result = <br>        if op = &quot;Sum&quot; then List.Sum(col)<br>        else if op = &quot;CountBlank&quot; then List.Count(List.Select(col, each _ = &quot;&quot;))<br>        else if op = &quot;CountNull&quot; then List.Count(List.Select(col, each _ = null))<br>        else if op = &quot;Average&quot; then List.Average(col)<br>        else if op = &quot;Max&quot; then List.Max(col)<br>        else if op = &quot;Min&quot; then List.Min(col)<br>        else null<br>in<br>    result</pre><pre>// ValidateRecord: Compares the actual result to the expected value(s) from a rule.<br>(result as number, expectedValue as text) =&gt; let<br>  // Validate the result based on the rule `expectedValue`. Here&#39;s some examples of the supported rules and their meaning:<br>  // Rule Format    | Meaning<br>  // &gt;=100          | Greater than or equal to 100<br>  // &lt;50            | Less than 50<br>  // =75            | Equal to 75<br>  // 50-100         | Between 50 and 100 (inclusive)<br>  // &gt;=50 and &lt;=100 | Compound condition (inclusive range)<br>  rule = Text.Trim(expectedValue),<br>  <br>  // Check for compound condition<br>  isCompound = Text.Contains(rule, &quot;and&quot;),<br>  isRange = Text.Contains(rule, &quot;-&quot;) and not isCompound,<br><br>  // Handle compound condition<br>  compoundParts = if isCompound then Text.Split(rule, &quot;and&quot;) else {},<br>  compoundCheck = if isCompound then<br>    List.AllTrue(List.Transform(compoundParts, each <br>      let<br>        part = Text.Trim(_),<br>        op = Text.Start(part, Text.PositionOfAny(part, {&quot;0&quot;..&quot;9&quot;})),<br>        val = Number.FromText(Text.Range(part, Text.PositionOfAny(part, {&quot;0&quot;..&quot;9&quot;})))<br>      in<br>        if op = &quot;&gt;=&quot; then result &gt;= val<br>        else if op = &quot;&lt;=&quot; then result &lt;= val<br>        else if op = &quot;&gt;&quot; then result &gt; val<br>        else if op = &quot;&lt;&quot; then result &lt; val<br>        else if op = &quot;=&quot; then result = val<br>        else false<br>    ))<br>    else null,<br><br>  // Handle range like &quot;50-100&quot;<br>  rangeParts = if isRange then Text.Split(rule, &quot;-&quot;) else {},<br>  rangeCheck = if isRange then<br>    let<br>      lower = Number.FromText(rangeParts{0}),<br>      upper = Number.FromText(rangeParts{1})<br>    in<br>      result &gt;= lower and result &lt;= upper<br>  else null,<br><br>  // Handle simple condition<br>  simpleOp = Text.Start(rule, Text.PositionOfAny(rule, {&quot;0&quot;..&quot;9&quot;})),<br>  simpleVal = Number.FromText(Text.Range(rule, Text.PositionOfAny(rule, {&quot;0&quot;..&quot;9&quot;}))),<br>  simpleCheck = if not isCompound and not isRange then<br>      if simpleOp = &quot;&gt;=&quot; then result &gt;= simpleVal<br>      else if simpleOp = &quot;&lt;=&quot; then result &lt;= simpleVal<br>      else if simpleOp = &quot;&gt;&quot; then result &gt; simpleVal<br>      else if simpleOp = &quot;&lt;&quot; then result &lt; simpleVal<br>      else if simpleOp = &quot;=&quot; then result = simpleVal<br>      else false<br>    else null<br>in<br>  if isCompound then compoundCheck<br>  else if isRange then rangeCheck<br>  else simpleCheck</pre><h4>Master Validation Query</h4><p>This master query orchestrates the entire validation process. It connects to the SharePoint lists, determines if it’s a single artifact validation or a two-artifact comparison, and then uses the previously defined functions to generate a final summary output.</p><pre>let<br>    // 1. Connect to the SharePoint site holding the configuration lists<br>    Source = SharePoint.Tables(&quot;https://your-company.sharepoint.com/sites/your-site&quot;, [ApiVersion = 15]),<br><br>    // 2. Load the main validation settings (specifies which artifacts to test)<br>    // Replace &#39;your-validation-settings-list-id&#39; with the actual GUID of your settings list<br>    ValidationSettings = Source{[Id=&quot;your-validation-settings-list-id&quot;]}[Items],<br><br>    // 3. --- Process the First Artifact (Source) ---<br>    Artifact1Name = Table.First(ValidationSettings)[Artifact1Name],<br>    Artifact1Workspace = Table.First(ValidationSettings)[Artifact1Workspace],<br>    Artifact1Type = Table.First(ValidationSettings)[Artifact1Type],<br>    <br>    // Load the validation rules for the first artifact<br>    // Replace &#39;your-validation-rules-list-id&#39; with the actual GUID of your rules list<br>    ValidationRules = Source{[Id=&quot;your-validation-rules-list-id&quot;]}[Items],<br>    ValidationRules1 = Table.SelectRows(ValidationRules, each ([ArtifactName] = Artifact1Name)),<br>    <br>    // Summarize the first artifact using the appropriate function based on its type<br>    // Note: SummarizeArtifact would be a wrapper function that calls SummarizeDataflow, SummarizeDatamart, etc.<br>    Artifact1Summary = SummarizeArtifact(ValidationRules1, Artifact1Name, Artifact1Workspace, Artifact1Type),<br><br>    // 4. --- Process the Second Artifact (Target/Comparison) ---<br>    Artifact2Name = Table.First(ValidationSettings)[Artifact2Name],<br>    Artifact2Workspace = Table.First(ValidationSettings)[Artifact2Workspace],<br>    Artifact2Type = Table.First(ValidationSettings)[Artifact2Type],<br><br>    // If a second artifact is defined, generate its rules by replacing the name from the first set<br>    ValidationRules2 = if Artifact2Name &lt;&gt; null then Table.ReplaceValue(ValidationRules1, Artifact1Name, Artifact2Name, Replacer.ReplaceValue, {&quot;ArtifactName&quot;}) else null,<br>    Artifact2Summary = if Artifact2Name &lt;&gt; null then SummarizeArtifact(ValidationRules2, Artifact2Name, Artifact2Workspace, Artifact2Type) else null,<br><br>    // 5. --- Compare Results and Format Output ---<br>    Result = if Artifact2Name = null then<br>        // --- SINGLE ARTIFACT VALIDATION ---<br>        // The result is compared against the &#39;ExpectedValue&#39; from the rules list<br>        Table.RenameColumns(<br>            Artifact1Summary,<br>            {{&quot;Result&quot;, &quot;ActualValue&quot;}, {&quot;ExpectedValue&quot;, &quot;ExpectedValue&quot;}}<br>        )<br>    else<br>        // --- TWO ARTIFACT COMPARISON (e.g., Dev vs. Prod) ---<br>        // Join the two summaries on the rule keys<br>        let<br>            Comparison = Table.NestedJoin(<br>                Artifact1Summary, {&quot;TableName&quot;, &quot;ColumnName&quot;, &quot;AggregationType&quot;},<br>                Artifact2Summary, {&quot;TableName&quot;, &quot;ColumnName&quot;, &quot;AggregationType&quot;},<br>                &quot;ComparisonData&quot;,<br>                JoinKind.LeftOuter<br>            ),<br>            Expanded = Table.ExpandTableColumn(Comparison, &quot;ComparisonData&quot;, {&quot;Result&quot;}, {&quot;Artifact2Result&quot;}),<br>            Renamed = Table.RenameColumns(Expanded, {{&quot;Result&quot;, &quot;Artifact1Result&quot;}})<br>        in<br>            Table.SelectColumns(Renamed, {&quot;Id&quot;, &quot;TableName&quot;, &quot;ColumnName&quot;, &quot;AggregationType&quot;, &quot;Artifact1Result&quot;, &quot;Artifact2Result&quot;}),<br><br>    // 6. --- Add Final Validation Status Column ---<br>    AddIsMatchColumn = Table.AddColumn(<br>        Result,<br>        &quot;IsMatch&quot;,<br>        each if Artifact2Name = null then<br>                 // For single artifact, validate the actual value against the expected value from the rule<br>                 ValidateRecord([ActualValue], [ExpectedValue], null, [AggregationType])<br>             else<br>                 // For two artifacts, check if their results are identical<br>                 [Artifact1Result] = [Artifact2Result],<br>        type logical<br>    ),<br><br>    // 7. --- Set Final Column Types for the Output ---<br>    FinalOutput = Table.TransformColumnTypes(<br>        AddIsMatchColumn,<br>        {<br>            {&quot;TableName&quot;, type text},<br>            {&quot;ColumnName&quot;, type text},<br>            {&quot;AggregationType&quot;, type text}<br>        }<br>    )<br><br>in<br>    FinalOutput</pre><h3>Key Use Cases Unlocked 🚀</h3><p>This framework is incredibly flexible and supports two primary use cases.</p><h4>1. Continuous Data Quality Monitoring</h4><p>By scheduling the validation dataflow to run after main ETL pipelines, it provides an automated check on production assets. If a row count in a critical table suddenly drops by 50%, the system will flag it immediately, often before business users are impacted.</p><h4>2. Pre-Deployment Sanity Checks (Dev vs. Prod)</h4><p>Before deploying changes from a development workspace to production, the framework can validate the new version against the old one. Rules can be configured to compare artifacts, for example:</p><ul><li>Check if the CountBlank of Table_DEV equals the CountBlank of Table_PROD.</li><li>Validate that the Sum of a key metric column is within a small variance (e.g., +/- 2%) between the dev and prod semantic models.</li></ul><p>This process drastically reduces the risk of deploying a change that introduces data errors.</p><h3>Why This Approach Shines 🌟</h3><ul><li><strong>Centralized &amp; Scalable:</strong> All rules live in one place. As the data estate grows, one simply adds rows to SharePoint, not complexity to dataflows.</li><li><strong>Low-Code:</strong> Business analysts or data stewards can add new validation rules without writing M code.</li><li><strong>Native &amp; Cost-Effective:</strong> It uses tools already available within the Power BI ecosystem.</li><li><strong>Flexible:</strong> It works across the most common Power BI artifacts: Dataflows, Datamarts, and Semantic Models.</li></ul><p>By investing a small amount of time to set up this framework, organizations can shift from a reactive “fire-fighting” approach to data quality to a proactive, automated one that builds trust and saves valuable engineering time.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b9fd72738a17" width="1" height="1" alt=""><hr><p><a href="https://towardsdev.com/a-low-code-data-validation-framework-for-power-bi-b9fd72738a17">A Low-Code Data Validation Framework for Power BI</a> was originally published in <a href="https://towardsdev.com/">Towards Dev</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Data Team’s Horizon: From Building Pipelines to Rewiring the Business]]></title>
            <link>https://islamtaha-29281.medium.com/the-data-teams-horizon-from-building-pipelines-to-rewiring-the-business-f9384c7af83d?source=rss-d741b64b39b0------2</link>
            <guid isPermaLink="false">https://medium.com/p/f9384c7af83d</guid>
            <category><![CDATA[career-advice]]></category>
            <category><![CDATA[data-maturity]]></category>
            <category><![CDATA[career-development]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Islam Taha]]></dc:creator>
            <pubDate>Fri, 13 Jun 2025 20:02:03 GMT</pubDate>
            <atom:updated>2025-06-13T20:02:03.179Z</atom:updated>
            <content:encoded><![CDATA[<p>Your data team built the pipelines. Now what? Exploring the shift from engineering to data science and beyond.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JVIxViddmfb06pGRZsvVLg.jpeg" /></figure><p>If you lead a data team, you know the grind. The initial phase is a feat of engineering grit: wrangling disparate sources, building resilient pipelines, and architecting a data warehouse that can stand the test of time. In an industry like capital projects, this means taming a beast of project performance data, financial reports, and field logs. You build the tracks, and finally, the data starts to flow reliably.</p><p>Then comes the question that signals a new dawn: “What can we <em>do</em> with all this data?”</p><p>This isn’t a sign of scope creep; it’s the sign of success. Your organization is moving up the data maturity curve, and your team’s purpose is about to evolve. This transition from a data engineering-heavy function to one that enables data science is not just common; it’s a critical, value-unlocking progression.</p><p>But it’s a journey fraught with challenges. How do you manage this shift without leaving your foundational engineering talent behind? And what comes <em>after</em> data science? Does the team eventually shrink into a small operational core?</p><p>The answer is no. The future isn’t smaller — it’s more integrated, more strategic, and far more impactful.</p><h4>The Natural Evolution: From Foundation to Insight</h4><p>The journey of a data team can be seen as a clear, phased progression:</p><p><strong>Phase 1: The Foundation (Data Engineering-Heavy)</strong> This is all about creating a single source of truth. The heroes are <strong>Data Engineers</strong>, and their focus is on reliability, scalability, and data quality. For a company managing massive infrastructure projects, this means creating pipelines that deliver accurate, on-time data about costs, timelines, and resource allocation. Success is measured by pipeline uptime, data freshness, and stakeholder trust in the numbers.</p><p><strong>Phase 2: The Insight (Data Science &amp; Analytics Emerge)</strong> With reliable data now accessible, the organization’s ambition grows. The focus shifts from providing data to using it for a competitive advantage. <strong>Data Scientists</strong> and <strong>Analysts</strong> join the fray, partnering with engineers to tackle complex questions.</p><p>This is where the magic happens in infrastructure projects:</p><ul><li><strong>Predictive Maintenance:</strong> Forecasting equipment failure on a construction site to prevent costly delays.</li><li><strong>Project Risk Modeling:</strong> Identifying which projects are at the highest risk of budget overruns based on a decade of historical data.</li><li><strong>Supply Chain Optimization:</strong> Optimizing the delivery of steel and concrete to minimize storage costs and prevent work stoppages.</li></ul><p>The challenge for leads here is twofold: managing the new, more experimental nature of data science work and beginning the process of integrating two different skill sets and mindsets.</p><h4>The Leadership Challenge: Evolving Your Team, Not Replacing It</h4><p>As the demand for data science grows, a common fear among Data Engineers is that their skills are becoming obsolete. A great leader knows the opposite is true: their role is more critical than ever. Data science models are nothing without a rock-solid data foundation.</p><p>Here’s how to manage this crucial transition and keep your team whole:</p><p><strong>1. Reframe the Narrative:</strong> This is an <em>expansion</em> of capabilities, not a replacement. Your DE team built the railroad; now you’re collectively building the high-speed, intelligent trains that will run on it. Their knowledge of the data’s context, flaws, and lineage is the most valuable asset your new data scientists will have.</p><p><strong>2. Invest in Hybrid Roles:</strong> The future isn’t a hard line between engineering and science. It’s a spectrum. Create clear career paths for your engineers to evolve:</p><ul><li><strong>Analytics Engineer:</strong> A perfect next step for a DE who enjoys working closer to business problems. They use software engineering principles to transform raw data into pristine, well-modeled datasets that are optimized for analysis.</li><li><strong>Machine Learning (ML) Engineer:</strong> The ideal evolution for a systems-thinking DE. They take the models built by data scientists and productionize them, building the robust infrastructure for deployment, monitoring, and retraining.</li></ul><p><strong>3. Create a Culture of Cross-Pollination:</strong></p><ul><li><strong>Paired Projects:</strong> Embed your engineers and scientists on the same project teams. Let them learn each other’s languages and appreciate each other’s challenges.</li><li><strong>Internal Upskilling:</strong> Empower your team to learn. Provide budgets for courses, encourage certifications, and have your data scientists run workshops on statistical concepts or ML libraries.</li><li><strong>Shared Metrics:</strong> While DEs care about uptime and latency, and DSs care about model accuracy, create shared team goals around business impact. When a model successfully reduces project delays, the entire team wins.</li></ul><h4>Beyond Data Science: The True Endgame</h4><p>So, you’re running successful data science projects. Is that the end of the line? Not at all. The next phases are what separate a company that <em>uses</em> data from one that is <em>defined</em> by it.</p><p><strong>Phase 3: The Factory (MLOps &amp; AI-Powered Operations)</strong> Here, the focus shifts from one-off projects to building a scalable “model factory.” The <strong>ML Engineer</strong> is the star, building automated systems for deploying and monitoring models. The goal is to make machine learning a reliable, repeatable, and governed part of the company’s operations.</p><p><strong>Phase 4: The Data-Driven Culture (Democratization &amp; Embedded Analytics)</strong> This is the ultimate destination. The centralized data team evolves from being the sole “doers” to being “enablers.”</p><ul><li><strong>Decentralization:</strong> The team’s structure changes. A lean <strong>Center of Excellence (CoE)</strong> may focus on strategy and advanced R&amp;D, while many analysts and data product owners are embedded directly into business units. Frameworks like the <strong>Data Mesh</strong> formalize this, making business domains responsible for their own “data products.”</li><li><strong>Data as a Product:</strong> Key datasets are no longer just tables in a warehouse; they are treated as products, with managers, documentation, and quality guarantees.</li><li><strong>Democratization:</strong> The goal is to empower the entire organization. Project managers don’t just receive a report; they interact with a self-service dashboard or see a “project risk score” embedded directly in their project management tool. Widespread <strong>data literacy</strong> becomes a core company value.</li></ul><p>The data team doesn’t shrink into a maintenance crew. It becomes the strategic nervous system of the organization. The routine work is automated or democratized, freeing up your highly skilled professionals to work on the next generation of challenges.</p><p>As a lead, your role will shift from managing projects to managing capabilities; from overseeing pipelines to cultivating a culture. The horizon for a data team isn’t a finish line — it’s the continuous integration of data into every decision, making the entire organization smarter, faster, and more resilient.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f9384c7af83d" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>