<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Aymenfkir on Medium]]></title>
        <description><![CDATA[Stories by Aymenfkir on Medium]]></description>
        <link>https://medium.com/@aymenfkir23?source=rss-b4e6f95a3a16------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*Fbo6iFAEshABsqky</url>
            <title>Stories by Aymenfkir on Medium</title>
            <link>https://medium.com/@aymenfkir23?source=rss-b4e6f95a3a16------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 05:12:27 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@aymenfkir23/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Cracking the LLM Drift Problem: Building a Dynamic Context-Branching Pipeline in Go]]></title>
            <link>https://medium.com/@aymenfkir23/cracking-the-llm-drift-problem-building-a-dynamic-context-branching-pipeline-in-go-44a829114f54?source=rss-b4e6f95a3a16------2</link>
            <guid isPermaLink="false">https://medium.com/p/44a829114f54</guid>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[context-engineering]]></category>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[Aymenfkir]]></dc:creator>
            <pubDate>Sun, 24 May 2026 22:12:20 GMT</pubDate>
            <atom:updated>2026-05-24T22:12:20.390Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>Context management is arguably the most underrated discipline in AI agent architecture. Most developers follow a linear paradigm: you spin up a session, stream user text, append responses, and pass the entire trailing history back to the LLM on every turn.</p><p>Humans love to talk. We shift topics, inject tangents, and flush lifetime conversations into a single session. As the token trajectory grows, the LLM loses focus. Its attention mechanism gets diluted because the context no longer points toward a single, unified objective.</p><p>To prevent this drift, we are usually presented with two industry-standard strategies:</p><ol><li><strong>LLM Summarization:</strong> Compresses the long historical trajectory into a compact summary turn.</li><li><strong>Observation Masking:</strong> Hides the environment’s raw observations while preserving the agent’s actions and internal chain-of-thought.</li></ol><p>But both strategies suffer from a fatal flaw: <strong>They propagate noise.</strong></p><h3>The Noise Propagation Trap</h3><p>Imagine an autonomous agent with a primary objective. Suddenly, due to an ambiguous tool output or user tangent, the agent drifts off track. The moment it drifts, your historical trajectory becomes corrupted.</p><p>If you summarize that history, you compress the corruption into the summary. If you mask the observations, the flawed reasoning still remains. The noise is locked into the timeline, actively dragging your model further away from its destination. Your agent either wastes tokens spinning in circles, or breaks down entirely.</p><h3>The Breakthrough: Dynamic Context Branching</h3><p>After obsessing over this problem for a few days, I realized we need to move away from rigid, linear history. Instead of compressing noise, <strong>we must isolate it.</strong></p><p>When a drift from the main objective is detected, the agent should surgically look back at its history, sever the problematic trajectory, and spin it off into an entirely separate, isolated session with a freshly defined sub-objective.</p><p>Now, you have two contexts running concurrently. The main pipeline remains untainted, waiting at the fork, while the model dedicates its focus to resolving the sub-objective in the secondary branch. Once the sub-objective is cleared, a evaluation strategy decides whether that branch’s resolution is clean enough to be merged back into the main timeline, or if it should be discarded completely.</p><p>To build a high-throughput framework capable of splitting, evaluating, and merging text streams concurrently, I chose <strong>Go</strong>. It provides native speed, simplicity, and structural control over concurrency primitives.</p><h3>Architecture: Constructing the Channels in Go</h3><p>A pipeline in Go is a series of execution stages connected by channels, where groups of concurrent goroutines process inbound values, apply transformations, and emit values downstream.</p><p>To make context branching work, the engine relies on two foundational concurrency patterns:</p><ul><li><strong>The Tee Pattern (Fan-Out):</strong> Duplicating incoming data streams into isolated, independent channels so multiple candidate strategies can analyze context concurrently without competing for data.</li><li><strong>The Fan-In Pattern (Merge):</strong> Safely multiplexing multiple processing paths back into a unified data sink once sub-objectives are completed.</li></ul><p>Here is how I structured the core concurrent components in my internals package to handle safe execution and data distribution.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/613/1*L-5oeNB53U9T-Gq2Jub4ww.png" /><figcaption>current pipeline</figcaption></figure><h4>1. The Generic Worker Node (stage_generator.go)</h4><p>Each processing stage spins up a dedicated worker pool. We use a standard for range loop to naturally exhaust the input channel, backed by context.Context to instantly stop upstream execution if the agent aborts down-stream.</p><pre>package internals<br>import (<br> &quot;context&quot;<br> &quot;sync&quot;<br>)<br>// Execute runs the core transformation function over the channel data<br>func Execute[T any](ctx context.Context, in &lt;-chan T, out chan T, fn callable[T]) {<br> for val := range in {<br>  select {<br>  case &lt;-ctx.Done():<br>   return<br>  default:<br>   result, err := fn(val)<br>   if err != nil {<br>    continue // Handle or log execution errors safely<br>   }<br>   select {<br>   case &lt;-ctx.Done():<br>    return<br>   case out &lt;- result:<br>   }<br>  }<br> }<br>}<br>// Stage provisions a pool of concurrent workers processing data streams<br>func Stage[T any](ctx context.Context, in &lt;-chan T, fn callable[T], workers int) &lt;-chan T {<br> out := make(chan T)<br> var wg sync.WaitGroup<br> for i := 0; i &lt; workers; i++ {<br>  wg.Add(1)<br>  go func() {<br>   defer wg.Done()<br>   Execute(ctx, in, out, fn)<br>  }()<br> }<br> go func() {<br>  wg.Wait()<br>  close(out)<br> }()<br> return out<br>}</pre><h4>2. The Broadcast System (fanin_fanout.go)</h4><p>To achieve true context isolation without data-starvation, we use an array of channels ([]chan T). A background routine reads an item once from the source channel and copies it to all sub-stages.</p><pre>package internals<br>import &quot;context&quot;<br>func initTeas[T any](numberSplits int) []chan T {<br> stageInputs := make([]chan T, numberSplits)<br> for i := 0; i &lt; numberSplits; i++ {<br>  stageInputs[i] = make(chan T)<br> }<br> return stageInputs<br>}<br>func copyData[T any](ctx context.Context, in &lt;-chan T, stageInputs []chan T) {<br> defer func() {<br>  for _, ch := range stageInputs {<br>   close(ch)<br>  }<br> }()<br> for val := range in {<br>  for _, ch := range stageInputs {<br>   select {<br>   case &lt;-ctx.Done():<br>    return<br>   case ch &lt;- val:<br>   }<br>  }<br> }<br>}<br>func Split[T any](ctx context.Context, in &lt;-chan T, fns []callable[T], workers []int, numberSplits int) []&lt;-chan T {<br> outs := make([]&lt;-chan T, 0, numberSplits)<br> stageInputs := initTeas[T](numberSplits)<br> go copyData[T](ctx, in, stageInputs)<br> for i := 0; i &lt; numberSplits; i++ {<br>  out := Stage(ctx, stageInputs[i], fns[i], workers[i])<br>  outs = append(outs, out)<br> }<br> return outs<br>}</pre><h4>3. Context-Aware Multiplexing (merge_channels.go)</h4><p>When sub-branches complete their execution, they fan-in safely. This version explicitly listens to ctx.Done(), protecting the engine from memory leaks if a downstream evaluator cuts the execution short.</p><pre>package internals<br>import (<br> &quot;context&quot;<br> &quot;sync&quot;<br>)<br>func Merge[T any](ctx context.Context, in []&lt;-chan T) &lt;-chan T {<br> var wg sync.WaitGroup<br> out := make(chan T)<br> output := func(c &lt;-chan T) {<br>  defer wg.Done()<br>  for info := range c {<br>   select {<br>   case &lt;-ctx.Done():<br>    return<br>   case out &lt;- info:<br>   }<br>  }<br> }<br> wg.Add(len(in))<br> for _, c := range in {<br>  go output(c)<br> }<br> go func() {<br>  wg.Wait()<br>  close(out)<br> }()<br> return out<br>}</pre><h3>Next Horizons</h3><p>Right now, this architecture provides a fast, thread-safe concurrent engine. The current blocks are static building blocks, but they provide the exact structural framework needed to implement dynamic runtime logic.</p><p>The next phase of my iteration is designing the classification strategies that sit inside the pipeline to analyze token streams, pinpoint the exact structural indices where a drift occurs, and automate the merge/discard decisions.</p><p>Are you building complex pipelines for AI agents? How are you tackling the problem of long-term history corruption? Let’s discuss in the comments.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=44a829114f54" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Data Management Iceberg: Medallion Architecture, XGBoost, and Real-Time Dashboards [Part 2]]]></title>
            <link>https://medium.com/@aymenfkir23/the-data-management-iceberg-medallion-architecture-xgboost-and-real-time-dashboards-part-2-abd8c65aa790?source=rss-b4e6f95a3a16------2</link>
            <guid isPermaLink="false">https://medium.com/p/abd8c65aa790</guid>
            <category><![CDATA[data-management]]></category>
            <category><![CDATA[hackathons]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[tunisiatech]]></category>
            <dc:creator><![CDATA[Aymenfkir]]></dc:creator>
            <pubDate>Mon, 18 May 2026 12:22:39 GMT</pubDate>
            <atom:updated>2026-05-18T12:22:39.086Z</atom:updated>
            <content:encoded><![CDATA[<p>In <a href="https://medium.com/@aymenfkir23/8-hours-to-stream-building-a-real-time-iot-data-pipeline-with-kappa-architecture-c77e3f81dd03">[Part 1]</a> of our NRTF Energy Hackathon journey, my teammate Mohamed Boughrara and I built a high-velocity real-time ingestion firehose using Kafka and Kappa architecture. But here is the brutal truth about data engineering: <strong>collection is just the visible tip of the iceberg.</strong></p><p>The hidden, painful part? Managing that data once it arrives.</p><p>Dumping raw data into a data lake doesn’t solve your problems. In a real industrial environment, you aren’t just dealing with clean streams; you are hit with a chaotic mix of structured, semi-structured, and legacy files. For Challenge 2 of the hackathon, our mission was to ingest these multi-source data files, design a bulletproof data management strategy, train an AI model for anomaly detection, and spin up an interactive monitoring dashboard.</p><p>Here is how we conquered the rest of the iceberg.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yMvGQmcy71x1Xx3ao-oXEA.png" /><figcaption>Full data management System</figcaption></figure><h3>The Ingestion Nightmare: Wrestling with Excel and PDFs</h3><p>Before you can model data, you have to load it. We had to digest data from completely different sources text-heavy PDFs and formatted Excel sheets.</p><p>Let’s be honest for a second — trying to load a heavily formatted corporate Excel sheet filled with custom colors, merged cells, and weird structures using standard pandas.read_excel() is an absolute nightmare. To preserve our sanity and our 8-hour timeline, we built a lightweight pipeline to cleanly extract tabular data from these multi-format files and immediately isolate the raw data into our data lake&#39;s foundational layer.</p><h3>The Strategy: Implementing Medallion Architecture</h3><p>To prevent our data lake from turning into a data swamp, we implemented a strict <strong>Medallion Architecture</strong>. This framework allowed us to organize and clean the data sequentially through three logical layers:</p><pre>[Raw Files: PDF/Excel] <br>            ↓<br>  📊 BRONZE LAYER (Raw, Unprocessed Data Lake)<br>            ↓  --&gt; Reshape &amp; Remove N/As<br>  🥈 SILVER LAYER (Cleaned &amp; Standardized)<br>            ↓  --&gt; Data Quality (Carbon Integrity &amp; Forecast Continuity)<br>  🏆 GOLD LAYER   (ML-Ready &amp; Dashboard Optimized)</pre><ol><li><strong>Bronze Layer (The Raw Ingest):</strong> This layer stores all incoming data exactly as it arrived — unverified, uncleaned, and raw. Its sole purpose is to act as a historical record of truth.</li><li><strong>Silver Layer (The Cleansing Ground):</strong> To transition from Bronze to Silver, we ran processing scripts to strip out missing values (N/A), fix structural anomalies, and reshape the tables into uniform dataframes.</li><li><strong>Gold Layer (The Analytics Hub):</strong> This is where the highest-quality data lives. Before data was promoted to Gold, we subjected it to rigorous data quality checks, specifically evaluating <strong>carbon integrity</strong> and <strong>forecast continuity</strong>. Only clean, validated records reached this layer, making it perfectly optimized for business intelligence and machine learning.</li></ol><h3>The Intelligence Layer: XGBoost for Anomaly Detection</h3><p>With high-quality data safely resting in our Gold layer, we moved on to predictive modeling. We trained an <strong>XGBoost</strong> model to analyze historical industrial energy footprints and spot environmental non-compliance.</p><p>The model was trained to predict energy profiles and immediately flag factories exhibiting abnormal energy consumption behavior. By coupling this predictive ML model with deterministic threshold rules, the system could instantly identify critical engine states.</p><h3>The Visual Interface: FastAPI meets React</h3><p>Data is only valuable if operators can read it in real time. We built a premium, dark-mode <strong>Industrial Energy Monitoring Dashboard</strong> powered by a <strong>FastAPI</strong> backend and a <strong>React (Vite) + Recharts</strong> frontend.</p><p>The backend simulates high-frequency industrial engines by streaming data every 2 seconds via <strong>WebSockets</strong>. For every single tick of incoming data, the system performs live thermodynamic calculations, tracking emissions and operational efficiency in real-time.</p><p>For instance, thermal efficiency is computed dynamically using the following formula:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/773/1*KLshoQwd4kP6YJI77EDjpw.png" /><figcaption>formula</figcaption></figure><p>If efficiency drops below 35% while the gas is flowing, or if an engine enters an “Idle Power” state (burning gas but producing zero net power), the frontend instantly logs a critical anomaly event on the UI.</p><h3>Final Thoughts: The 2-Part Sprint Conclusion</h3><p>Looking back at the entire weekend hackathon, we started with raw sensor signals at the edge and ended with automated AI insights flashing on a live dashboard. Moving through the layers of the Medallion architecture while fighting formatting bugs taught us that design strategy matters just as much as writing clean code.</p><p>Massive credit to my partner Mohamed Boughrara building an end-to-end event-driven architecture, automated data cleaning pipeline, and a full React application in a weekend is an engineering marathon we won’t soon forget.</p><p><a href="https://github.com/aymen-fkir/Data_Mangment/tree/main">GitHub Repository: Data Management &amp; Dashboard</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=abd8c65aa790" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[8 Hours to Stream: Building a Real-Time IoT Data Pipeline with Kappa Architecture]]></title>
            <link>https://medium.com/@aymenfkir23/8-hours-to-stream-building-a-real-time-iot-data-pipeline-with-kappa-architecture-c77e3f81dd03?source=rss-b4e6f95a3a16------2</link>
            <guid isPermaLink="false">https://medium.com/p/c77e3f81dd03</guid>
            <category><![CDATA[edge-computing]]></category>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[microservices]]></category>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Aymenfkir]]></dc:creator>
            <pubDate>Mon, 04 May 2026 09:59:03 GMT</pubDate>
            <atom:updated>2026-05-04T09:59:03.395Z</atom:updated>
            <content:encoded><![CDATA[<p>This weekend, my teammate Mohamed Boughrara and I jumped into the NRTF Energy Hackathon here in Tunisia. Two challenges. Eight hours. Questionable sleep decisions. This article is about Challenge 1 the one where we basically tried to outpace a firehose of industrial data without breaking everything.</p><h3><strong>The Problem: When Data Comes at You Like a DDoS (But It’s Legit)</strong></h3><p>In Industrial IoT (IIoT), collecting data is the easy part. The real problem? Handling <em>millions</em> of sensor events flying in like they all had a deadline at the same time.<br> Think factories, turbines, power grids everything screaming data 24/7.<br> Our mission was simple (on paper):<br> <strong>Build a real-time ingestion pipeline that doesn’t choke under pressure.</strong><br> Simple… until you realize you have 8 hours and caffeine becomes your primary dependency.</p><h3>The Mindset: Speed First, Perfection Later</h3><p>We made a very intentional decision early on:</p><blockquote><em>If the pipeline is slow, the data is gone. Forever.</em></blockquote><p>So yeah, we didn’t aim for perfection. We aimed for <strong>survival under load</strong>.</p><ul><li>No over-engineering</li><li>No “let’s optimize later” (there is no later in a hackathon)</li><li>Just raw throughput and stability</li></ul><p>Also, we went with <strong>Kappa Architecture</strong>. Why? Because:</p><ul><li>Lambda = more layers, more complexity, more headaches</li><li>Kappa = <em>everything is a stream</em> → simpler, cleaner, faster to implement</li></ul><p>And when the clock is ticking, simplicity wins.</p><h3>The Architecture: Keep It Clean, Keep It Moving</h3><p>Before writing a single line of code, we designed the system like actual adults (rare moment, I know).</p><p>Here’s the flow 👇</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UHn0u97zQIaNGS---DpHOg.png" /><figcaption>data collection diagram</figcaption></figure><h3>1. Edge Collection Layer (Where Data Is Born)</h3><p>We designed the system to ingest data as close to the physical sensors as possible. By placing a <strong>local MQTT broker</strong> right at the edge, we achieved:</p><ul><li><strong>Minimized Latency:</strong> Immediate data capture without the round-trip delay.</li><li><strong>Traffic Optimization:</strong> Dramatically reduced the load on our central infrastructure.</li></ul><p>The philosophy was simple: if you don’t need to send it across the entire network immediately, don’t. Process locally, scale globally.</p><h3>2. Bridge Gateway (The Middle Guy Nobody Talks About)</h3><p>We didn’t just connect the edge to the core; we bridged them. By utilizing a second MQTT broker as an intermediary gateway, we created a critical abstraction layer between our field sensors and the processing servers.</p><p><strong>Why this was a non-negotiable choice:</strong></p><ul><li><strong>Architectural Decoupling:</strong> The edge environment remains agile and autonomous. If the backend undergoes maintenance or scaling, the edge continues to buffer data without failing.</li><li><strong>Protocol Isolation:</strong> It allows us to manage local traffic differently from long-haul data transfers, making future scaling significantly less painful.</li><li><strong>The Security Gauntlet:</strong> This is also the primary point of failure for unauthorized access. We treated <strong>SSL/TLS implementation</strong> as a non-negotiable production requirement. We spent a significant chunk of our 8 hours debugging certificates and handshakes like they had a personal vendetta against us but in IIoT, if it isn’t encrypted, it isn’t production-ready.</li></ul><h3>3. Unified Stream Engine (Kafka Doing Kafka Things)</h3><p>This is where the pipeline moves from simple ingestion to true industrial-grade orchestration. To handle the potential “firehose” of millions of data points, we didn’t just pipe the data; we partitioned it.</p><p><strong>The Engineering Logic:</strong></p><ul><li><strong>Partitioned Parallelism:</strong> By splitting each topic into <strong>4 distinct partitions</strong>, we effectively quadrupled our theoretical throughput. This allows multiple consumer instances to read from the same topic simultaneously without locking issues.</li><li><strong>Topic Separation:</strong> We isolated data streams into two dedicated topics to ensure that a surge in “Valve” telemetry wouldn’t create a lag spike for critical “Gas” sensor monitoring.</li><li><strong>Latency Reduction:</strong> We utilized <strong>Kafka in KRaft mode</strong> to remove the Zookeeper overhead, streamlining the metadata management and making the cluster leaner and faster.</li></ul><p>Let’s be honest: in the heat of a hackathon, aggressively throwing partitions at a throughput problem feels like a power move but when you watch the consumer latency drop to near-zero under load, you know it was the right architectural call.</p><h3>4. Processing + Storage (Where Data Gets a Purpose)</h3><p>The final stage of the pipeline is where raw bytes become actionable intelligence. Our Python consumers act as the gatekeepers between the volatile stream and our long-term storage.</p><p><strong>The Transformation Logic:</strong></p><ul><li><strong>Integrity Verification:</strong> Before any data touches the database, our consumers perform a <strong>SHA-256 payload verification</strong>. In an industrial context, data that has been tampered with or corrupted at the edge is worse than no data at all.</li><li><strong>The Power of Time-Series:</strong> We chose <strong>InfluxDB</strong> as our destination. Standard SQL databases struggle with the high-write frequency of IoT, but InfluxDB’s LSM-tree based engine is purpose-built for this. It allows us to handle thousands of writes per second while maintaining high-speed queryability.</li><li><strong>Dashboard Readiness:</strong> By the time the data is persisted, it’s structured, tagged by site_id and sensor_id, and optimized for immediate visualization in tools like Grafana.</li></ul><h3>The Tech Stack: Simple, Solid, No Drama</h3><p>We didn’t try to be fancy. We tried to be effective.</p><ul><li><strong>Mosquitto (MQTT)</strong> → lightweight, reliable, does the job</li><li><strong>Python</strong> → fast to write, easy to debug (and yes, we debugged a lot)</li><li><strong>Kafka</strong> → the backbone of the whole system</li><li><strong>InfluxDB</strong> → because time-series data deserves respect</li></ul><p>And of course… Docker everywhere, because:</p><blockquote><em>“It works on my machine” is not a valid deployment strategy.</em></blockquote><h3>What Actually Made This Work</h3><p>A few key decisions saved us:</p><h4>1. Decoupling Everything</h4><p>Each component does one job:</p><ul><li>MQTT → transport</li><li>Kafka → streaming</li><li>Python → logic</li><li>InfluxDB → storage</li></ul><p>If one breaks, the whole system doesn’t collapse.</p><h4>2. Parallelism Is Not Optional</h4><p>Kafka partitions = <strong>free performance boost</strong></p><p>Without it, we’d be watching our pipeline crawl like it’s on 2G internet.</p><h4>3. Edge Computing Is Underrated</h4><p>Processing closer to the source:</p><ul><li>reduces latency</li><li>reduces load</li><li>makes everything smoother</li></ul><h4>4. Accept Imperfection</h4><p>We didn’t chase perfect parsing or validation.<br> We chased:</p><ul><li><strong>no data loss</strong></li><li><strong>consistent throughput</strong></li></ul><p>And honestly? That’s what matters in real systems too.</p><h3>The Reality Check (a.k.a. What Almost Broke Us)</h3><ul><li>TLS between brokers? Yeah… not as plug-and-play as you hope</li><li>Debugging distributed systems in 8 hours = character development</li><li>Sleep deprivation = creative problem solving (or chaos, depends)</li></ul><p>At one point, the system worked… then didn’t… then worked again.<br> Classic.</p><h3>Final Thoughts</h3><p>Building this in 8 hours was essentially sprinting a marathon while assembling IKEA furniture without instructions and debugging the manual in real-time.</p><p>The result? A fully operational real-time pipeline, a rock-solid Kappa architecture, and a system capable of handling industrial-scale data flows. Not a bad way to spend a weekend.</p><p><strong><em>Note : </em></strong>This was just <strong>Challenge 1</strong>.In Part 2, things get even more interesting 👀 Stay tuned.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c77e3f81dd03" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From Tables to Graphs: Building a TensorFlow GNN Sampler]]></title>
            <link>https://medium.com/@aymenfkir23/from-tables-to-graphs-building-a-tensorflow-gnn-sampler-5ca44f48440e?source=rss-b4e6f95a3a16------2</link>
            <guid isPermaLink="false">https://medium.com/p/5ca44f48440e</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[tensorflow]]></category>
            <category><![CDATA[sampler]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[gnn]]></category>
            <dc:creator><![CDATA[Aymenfkir]]></dc:creator>
            <pubDate>Mon, 27 Oct 2025 11:47:30 GMT</pubDate>
            <atom:updated>2025-10-27T12:29:15.958Z</atom:updated>
            <content:encoded><![CDATA[<p>Previously, I worked on data preparation and feature engineering. I ended up with <strong>14 new features</strong>, and now the dataset is ready for use. This week, I started exploring <strong>model development</strong>, transforming the data into <strong>graph data</strong>, and setting up the <strong>sampler model</strong>. Let’s dig into it!</p><h3>Graph Data</h3><p>Creating graph data is actually quite simple and TensorFlow makes it even easier. There’s already a predefined function that transforms tabular data into graph data <strong>tfgnn.GraphTensor.from_pieces</strong></p><p>To use this function, I need to define a few key components:</p><ul><li><strong>Node Sets : </strong>A node set is basically a dictionary containing the name of the node along with its size and features</li></ul><pre>{<br>    &quot;customers&quot;: tfgnn.NodeSet.from_fields(<br>        sizes=tf.convert_to_tensor([len(df_customers_unique)]),<br>        features=customers_features,  # type: ignore<br>    ),<br>    &quot;terminals&quot;: tfgnn.NodeSet.from_fields(<br>        sizes=tf.convert_to_tensor([len(df_terminals_unique)]),<br>        features=terminals_features,  # type: ignore<br>    ),<br>}</pre><ul><li><strong>Edge Sets : </strong>Edge sets have a similar structure, but with source and target fields to define the connections between nodes , In my case, i created two edge sets (one from customer → terminal and another from terminal → customer ) . This setup is crucial for implementing my sampling strategy later.</li></ul><pre>edge_sets = {<br>    &quot;transactions&quot;: tfgnn.EdgeSet.from_fields(<br>        sizes=tf.convert_to_tensor([num_transactions]),<br>        adjacency=tfgnn.Adjacency.from_indices(<br>            source=(&quot;customers&quot;, source_indices),  # type: ignore<br>            target=(&quot;terminals&quot;, target_indices),  # type: ignore<br>        ),<br>        features=transactions_features,  # type: ignore<br>    ),<br><br>    &quot;reversed_transactions&quot;: tfgnn.EdgeSet.from_fields(<br>        sizes=tf.convert_to_tensor([num_transactions]),<br>        adjacency=tfgnn.Adjacency.from_indices(<br>            source=(&quot;terminals&quot;, target_indices),  # type: ignore<br>            target=(&quot;customers&quot;, source_indices),  # type: ignore<br>        ),<br>    ),<br>}</pre><ul><li><strong>Context Set : </strong>This usually holds the labels for each transaction</li></ul><pre>context=tfgnn.Context.from_fields(<br>            features={&quot;tx_fraud&quot;: tf.constant(target.values, dtype=tf.float32)} # type: ignore<br>        )</pre><p><strong>A Few Important Notes</strong></p><ul><li>Customer nodes must be unique , for example:</li></ul><pre>df = df.drop_duplicates(subset=[&quot;customer_id&quot;])</pre><ul><li><strong>Node IDs ≠ Customer IDs.</strong></li></ul><p>The node ID refers to the _index_ of the node, not the customer ID. Mixing them up will cause chaos (and mysterious TensorFlow errors). <br> Fun fact: I learned this the hard way after spending _three days_ debugging why my graph wouldn’t connect properly. 😅</p><h3>Splitting Data</h3><p>After defining the graph, I split the data into train and validation sets. <br>The important part:</p><blockquote>A customer should not appear in both sets this prevents data leakage.</blockquote><pre>transactions = sample_df.set_index(&quot;node_id&quot;)<br>x = transactions.drop([&quot;tx_fraud&quot;, &quot;tx_fraud_scenario&quot;], axis=1)<br>y = transactions[&quot;tx_fraud&quot;]<br>groups = transactions.index<br><br>train_ids, validation_ids = next(<br>    GroupShuffleSplit(test_size=0.33, n_splits=2, random_state=42).split(x, y, groups)<br>)</pre><p>Using GroupShuffleSplit from sklearn, we ensure proper separation. <br>Now, it’s time for the fun part .</p><h3>Sampler</h3><p>The sampler is a critical part of the process. <br>You don’t want to train on the entire graph every step that’s wildly inefficient (and your GPU will cry).</p><p>Instead, we use a sampler to extract a fixed number of neighbor nodes for each node during training.</p><p><strong>My strategy is simple:</strong></p><ul><li>For each customer node, sample 10 terminals that had transactions with it</li><li>For each of those terminals, sample 5 other customers who transacted with the same terminal.</li></ul><p>This is called <strong>neighbor sampling</strong> using nearby nodes to gather contextual information.</p><p>TensorFlow GNN provides a handy function <em>sampler.create_sampling_model_from_spec()</em></p><p>It takes four arguments :</p><ul><li><strong>Graph schema</strong> defines node and edge sets (retrieved via <em>tfgnn.create_schema_pb_from_graph_spec</em>)</li><li><strong>Sampling spec</strong> specifies how many neighbors to sample</li><li><strong>Edge sampler factory</strong> defines how to sample edges</li><li><strong>Node features accessor factory</strong> maps nodes to their features</li></ul><p>Here’s how the code looks:</p><pre>def create_sampling_model(full_graph_tensor: tfgnn.GraphTensor, sizes: Mapping[str, int], graph_schema) -&gt; tf.keras.Model:<br>    def edge_sampler(sampling_op: tfgnn.sampler.SamplingOp):<br>        edge_set_name = sampling_op.edge_set_name<br>        sample_size = sampling_op.sample_size<br>        return sampler.InMemUniformEdgesSampler.from_graph_tensor(<br>            full_graph_tensor, edge_set_name, sample_size=sample_size, name=sampling_op.op_name<br>        )<br><br>    def get_features(node_set_name: tfgnn.NodeSetName | tfgnn.EdgeSetName):<br>        return sampler.InMemIndexToFeaturesAccessor.from_graph_tensor(<br>            full_graph_tensor, node_set_name<br>        )<br><br>    # Sampling logic<br>    sampling_spec_builder = tfgnn.sampler.SamplingSpecBuilder(<br>        graph_schema, default_strategy=tfgnn.sampler.SamplingStrategy.RANDOM_UNIFORM<br>    )<br>    seed = sampling_spec_builder.seed(&quot;customers&quot;)<br><br>    terminals_collected = seed.sample(sizes[&quot;transactions&quot;], &quot;transactions&quot;)<br>    customers_2hop = terminals_collected.sample(sizes[&quot;reversed_transactions&quot;], &quot;reversed_transactions&quot;)<br><br>    sampling_spec = sampling_spec_builder.build()<br><br>    model = sampler.create_sampling_model_from_spec(<br>        graph_schema, sampling_spec, edge_sampler, get_features, seed_node_dtype=tf.int64<br>    )<br>    return model</pre><p>Each step of the sampling logic defines how to expand from the initial seed nodes to their neighbors kind of like building a mini <strong>ego network</strong> around each node.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZfC-vR5z8BysvN6J-oTu4A.png" /><figcaption>ego network</figcaption></figure><h3>Dataset Provider</h3><p>Now that we have the sampler, we can build a dataset provider for training and validation. <br>This class will handle subgraph extraction automatically:</p><pre>from tensorflow_gnn import runner<br>import functools<br><br>class SubgraphSampler(runner.DatasetProvider):<br>    def __init__(self, graph: tfgnn.GraphTensor, sampling_sizes: dict[str, int], node_seed: tf.Tensor):<br>        super().__init__()<br>        self.graph = graph<br>        self.sampling_sizes = sampling_sizes<br>        self.schema = tfgnn.create_schema_pb_from_graph_spec(graph.spec)<br>        self.sampling_model = create_sampling_model(<br>            full_graph_tensor=graph, sizes=sampling_sizes, graph_schema=self.schema<br>        )<br>        self.node_seed = tf.data.Dataset.from_tensor_slices(tf.random.shuffle(node_seed))<br><br>    def get_dataset(self, context: tf.distribute.InputContext) -&gt; tf.data.Dataset:<br>        ds = self.node_seed.shard(num_shards=context.num_input_pipelines, index=context.input_pipeline_id)<br>        ds = ds.batch(32)<br>        ds = ds.map(<br>            functools.partial(self.sample),<br>            num_parallel_calls=tf.data.AUTOTUNE,<br>            deterministic=False,<br>        )<br>        return ds.unbatch().prefetch(tf.data.AUTOTUNE)<br><br>    def sample(self, seeds: tf.Tensor) -&gt; tfgnn.GraphTensor:<br>        seeds = tf.cast(seeds, tf.int64)<br>        batch_size = tf.size(seeds)<br>        seeds_ragged = tf.RaggedTensor.from_row_lengths(seeds, tf.ones([batch_size], tf.int64))<br>        return self.sampling_model(seeds_ragged)<br><br>train_ds_provider = SubgraphSampler(graph, train_sampling_sizes, train_nodes)<br>val_ds_provider = SubgraphSampler(graph, validation_sample_sizes, val_nodes)</pre><p>This class does two main things:</p><ul><li><strong>get_dataset()</strong> → creates batched datasets for parallel training</li><li><strong>sample()</strong> → transforms node IDs into subgraphs using the sampling model</li></ul><h3>Wrapping Up</h3><p>And that’s it — the data is ready, the sampler works, and I’m _almost_ ready to start training.I’m super happy with the progress. I’m just one step away from training my first model on graph data.</p><p>Wish me luck, folks I’m diving deep into the graph universe next! 🚀😜</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5ca44f48440e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From Transactions to Graphs: Constructing Fraud Networks with Graph Neural Networks]]></title>
            <link>https://medium.com/@aymenfkir23/from-transactions-to-graphs-constructing-fraud-networks-with-graph-neural-networks-d7f8394f6c4c?source=rss-b4e6f95a3a16------2</link>
            <guid isPermaLink="false">https://medium.com/p/d7f8394f6c4c</guid>
            <category><![CDATA[feature-engineering]]></category>
            <category><![CDATA[gnn]]></category>
            <category><![CDATA[graph]]></category>
            <dc:creator><![CDATA[Aymenfkir]]></dc:creator>
            <pubDate>Mon, 20 Oct 2025 09:27:52 GMT</pubDate>
            <atom:updated>2025-10-20T09:27:52.199Z</atom:updated>
            <content:encoded><![CDATA[<p>Last time, I worked on the simulation setup, including the Kafka server, the consumer, and the producer. I also fixed the previous bug and you won’t believe it 😅 it turned out to be something really simple and silly: I was appending to the array twice! Once I spotted that , I fixed it, and everything is running smoothly now</p><p>This week wasn’t heavy on coding but more about reading and understanding the problem I’m working on. I spent my time studying the fundamentals of Graph Neural Networks (GNNs) what graphs are, how to train a GNN, what algorithms and models can be used, how to construct a graph dataset, and finally, how to perform feature engineering.So, let’s get started</p><h4><strong>Graph Neural Networks (GNNs)</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sJsLoikMmSL9Zkv6CTvBWA.png" /><figcaption>Graph</figcaption></figure><p>A graph is a data structure that represents a set of entities (called nodes or vertices) and the relationships between them (called edges).</p><ul><li><strong>Nodes </strong>represent entities such as people, atoms, locations, users, or terminals</li><li><strong>Edges</strong> represent relationships or interactions between nodes, such as friendships, chemical bonds, transactions, or citations</li></ul><p>Graphs are well-known for preserving relationships between entities, which makes them excellent for understanding and analyzing connections in data</p><p>Graphs can also carry attributes or features on nodes, edges, or even globally across the entire graph. For example, a node might have a feature vector describing its properties, edges might have weights or types, and the graph itself might have features that summarize overall characteristics</p><h4><strong>Types of Graphs</strong></h4><ul><li>Directed graphs (edges have direction)</li><li>Indirect graphs (edges are symmetric)</li><li>Multi-graphs (multiple edges between the same nodes)</li><li>Hyper-graphs (edges connecting more than two</li><li>Hierarchical or nested graphs</li></ul><p>Many real-world systems are naturally represented as graphs social networks, molecules, knowledge graphs, communication systems, or fraud detection networks. In our case, we’ll be building a fraud network graph</p><p><strong>What is a Graph Neural Network (GNN)?</strong></p><p>A Graph Neural Network is a neural architecture designed to work on graph-structured data. Unlike traditional neural networks (which work on grids or sequences), GNNs can handle arbitrary connectivity between nodes and edges.</p><p>In graphs, information is distributed across nodes. Each node knows only about itself, but context often matters so information must be shared. This is where message passing comes in: a process of sharing and aggregating information across connected nodes and edges. Through this, each node learns not just its own properties, but also the structure and features of its neighbors.</p><p>A typical message passing neural network works in three steps:</p><ul><li><strong>Message or Gather :</strong> Each node collects messages from its neighbors. Messages are computed from neighbor embeddings and possibly edge features.</li><li><strong>Aggregate :</strong> The gathered messages are combined (using sum, mean, max, etc.) in a permutation-invariant way (so the order of neighbors doesn’t matter).</li><li><strong>Update :</strong> The aggregated message and the node’s current embedding are used to compute a new node embedding, often through a multilayer perceptron (MLP).</li></ul><p>GNNs can handle three main types of prediction tasks <strong>Graph-level tasks </strong>where the model predict the properties of an entire graph , <strong>Node-level tasks </strong>where it predict the identity or role of individual nodes and finally <strong>Edge-level tasks </strong>where the model predict the existence or type of an edge between nodes</p><p>now we grasped the fundamentals of graph and GNN it’s time to dig into constructing the dataset</p><h4>Constructing a Graph Dataset</h4><p>I’ll be using TensorFlow since I’m more familiar with it.</p><p>To convert a regular dataset into a graph dataset, we represent the data as interconnected entities and relationships. In TensorFlow (especially TensorFlow GNN), this usually involves three main sets of tensors:</p><ul><li><strong>Node sets</strong> Contain information about individual nodes (e.g., price, category, or shop type). Each tensor represents the attributes of an entity.</li><li><strong>Edge sets</strong> Represent the relationships or connections between nodes, often using adjacency information. Edges can also include features such as transaction amount, timestamp, or weight.</li><li><strong>Context set</strong>s Stores global or aggregated information about the graph as a whole. This can summarize statistics or represent shared embeddings.</li></ul><p>Together, these three components nodes, edges, and context form the foundation of a graph dataset, allowing GNNs to learn from both structure and attributes.</p><p>I haven’t built this part yet, though! 😅</p><p>As describe in the previous blog the dataset that i’m working with lack features both the consumer node and the terminal node , that’s why the next step is crustal for the model development</p><h4>Feature Engineering</h4><p>My dataset contains the following columns:</p><pre>transaction_id, tx_datetime, customer_id, terminal_id, tx_amount,<br>tx_time_seconds, tx_time_days, tx_fraud, tx_fraud_scenario</pre><p>In this setup, we have two types of nodes <strong>clients</strong> and <strong>terminal</strong>, and one type of edge <strong>transactions</strong> , The transaction’s datetime and amount will serve as features for the transaction edge.However, our nodes don’t have properties yet, so we need to create some.</p><p>Here are a few examples i created:</p><ul><li><strong>TX_DURING_WEEKEND</strong> 1 if the transaction happened on a weekend, else 0</li></ul><pre>def is_weekend(tx_time: pd.Timestamp) -&gt; int:<br>    weekday = tx_time.weekday()<br>    is_weekend = weekday &gt;= 5<br>    return int(is_weekend)</pre><ul><li><strong>TX_DURING_NIGHT</strong> 1 if the transaction happened at night, else 0</li></ul><pre>def is_night(tx_time: pd.Timestamp) -&gt; int:<br>    hour = tx_time.hour<br>    is_night = (hour &lt;= 6)<br>    return int(is_night)</pre><p>To better understand the customer behavior, I’ll create these features:</p><ul><li><strong>CUSTOMER_ID_NB_TX</strong> Number of transactions in a specific time window</li><li><strong>CUSTOMER_ID_AVG_AMOUNT</strong> Average transaction amount in that time window</li></ul><pre>def get_spending_behaviour(data:pd.DataFrame,<br>                           window_sizes: list[int] = [1,7,30]<br>                           ) -&gt; pd.DataFrame:<br>    data = data.sort_values(by=&quot;tx_datetime&quot;)<br>    data = data.set_index(&quot;tx_datetime&quot;)<br>    for window in window_sizes:<br>        rolling = data[&quot;tx_amount&quot;].rolling(window=f&quot;{window}d&quot;)<br>        rolling_sum = rolling.sum()<br>        rolling_count = rolling.count()<br>        avg_spend = rolling_sum / rolling_count<br>        data[f&quot;number_of_transactions_last_{window}_days&quot;] = list(rolling_count)<br>        data[f&quot;avg_spend_last_{window}_days&quot;] = list(avg_spend)<br>    <br>    data = data.reset_index()<br>    return data</pre><p><strong>Note:</strong> Make sure to sort the data by date and use a rolling function to avoid data leakage.</p><p>For terminal nodes, we can create features like:</p><ul><li><strong>TERMINAL_ID_RISK</strong> Average number of fraudulent transactions on a terminal within a time window</li><li><strong>TERMINAL_ID_NB_TX</strong> Number of transactions per time window</li></ul><pre>def get_count_risk_rolling_window(data:pd.DataFrame,<br>                                  delay=7,<br>                                  window_sizes: list[int] = [1,7,30]<br>                                  ) -&gt; pd.DataFrame:<br><br>    data = data.sort_values(by=&quot;tx_datetime&quot;)<br>    data = data.set_index(&quot;tx_datetime&quot;)<br>    <br>    nb_fraud_delay = data[&quot;tx_fraud&quot;].rolling(window=f&quot;{delay}d&quot;).sum()<br>    nb_tx_delay = data[&quot;tx_fraud&quot;].rolling(window=f&quot;{delay}d&quot;).count()<br>    for window in window_sizes:<br>        rolling = data[&quot;tx_fraud&quot;].rolling(window=f&quot;{delay+window}d&quot;)<br>        rolling_sum = rolling.sum()<br>        rolling_count = rolling.count()<br>        nb_fraud_window = rolling_sum - nb_fraud_delay<br>        nb_tx_window = rolling_count - nb_tx_delay<br>        risk_window = nb_fraud_window / nb_tx_window<br><br>        data[f&quot;terminal_id_risk_last_{window}_days&quot;] = list(risk_window)<br>        data[f&quot;number_of_transactions_last_{window}_days&quot;] = list(rolling_count)<br>    <br>    data = data.reset_index()<br>    data.fillna(0,inplace=True)<br>    return data</pre><p>I’ve defined three time windows daily, weekly, and monthly.</p><p>Now comes the fun part developing the model itself!</p><p>Wish me luck, folks 😁</p><p><strong>Resources I used : </strong><a href="https://distill.pub/2021/gnn-intro/">read more gnn</a> , <a href="https://www.youtube.com/watch?time_continue=3&amp;v=JqWROPYeqjA&amp;embeds_referring_euri=https%3A%2F%2Fwww.google.com%2F&amp;source_ve_path=MjM4NTE">read more about tenseflow gnn</a> , <a href="http://fraud-detection-handbook.github.io/fraud-detection-handbook/Chapter_3_GettingStarted/BaselineFeatureTransformation.html">read more about feature engineering</a> (hope it help you as it helped me)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d7f8394f6c4c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Producer, Consumer, and Database (Kafka , go)]]></title>
            <link>https://medium.com/@aymenfkir23/producer-consumer-and-database-kafka-go-cbb26766c0b7?source=rss-b4e6f95a3a16------2</link>
            <guid isPermaLink="false">https://medium.com/p/cbb26766c0b7</guid>
            <category><![CDATA[neo4j]]></category>
            <category><![CDATA[go]]></category>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[postgres]]></category>
            <dc:creator><![CDATA[Aymenfkir]]></dc:creator>
            <pubDate>Tue, 14 Oct 2025 08:07:20 GMT</pubDate>
            <atom:updated>2025-10-14T08:07:20.828Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>Producer, Consumer, and Database (Kafka , go)</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lPYfI-Ia4g0krFM4Q9fPUw.png" /><figcaption>High Level Architect</figcaption></figure><p>Previously in Part 1, I talked about setting up the simulation I’ll use to test the model. Well, it’s almost there! But first, let me show you what I built this week after generating all that glorious synthetic data.</p><p>The first order of business was to chop up the data into three distinct subsets (because one size definitely doesn’t fit all): train validation and simulation .</p><pre>| Subset         | Purpose                                                                                                              | Size            |<br>| -------------- | -------------------------------------------------------------------------------------------------------------------- | --------------- |<br>| **Train**      | For model development—the bulk of the work.                                                                          | 55% of the data |<br>| **Validation** | To validate the model&#39;s performance _during_ the simulation, like a mid-level manager checking on a project.         | 25% of the data |<br>| **Simulation** | This data doesn&#39;t have the **fraud label** on it and will be used to test the end-to-end simulation. The final boss. | 20% of the data |</pre><p>With the data split, it was time to prepare my unsung heroes: the Producer and the Consumer.</p><h3><strong>The Producer: The Transaction Courier</strong></h3><p>The producer is our client in this scenario. Every transaction it makes is sent off to the Kafka server, which saves it to a partition in a dedicated topic.</p><p>Our producer simply reads the data from a JSON file. The data contains the following columns a simple tale of a transaction:</p><blockquote>[transaction_id, tx_datetime,customer_id, terminal_id, tx_amount, tx_time_seconds, tx_time_days]</blockquote><p>Each row represents a single transaction. So, in short, our producer loads the data from the file and then sends every record one at a time. (For now, I’m doing this <strong>synchronously</strong> but I’m planning to switch to <strong>Go routines</strong> soon so it becomes closer to a real-life transaction flow. This will be a great way to truly test Kafka’s mettle!)</p><p>The producer follows this folder structure:</p><pre>transaction_producer/<br>├── cmd<br>│   └── main.go<br>├── data<br>│   ├── test<br>│   │   └── test.json<br>│   └── validation<br>│       └── val.json<br>├── go.mod<br>├── go.sum<br>├── pkg<br>│   └── producer<br>│       └── work.go<br>└── util<br>    └── utils.go</pre><p>The <strong>util</strong> file contains the TransactionRecord type again, as well as a LoadEnv function that pulls in the environment variables (you know, the secrets).</p><p>The <strong>pkg</strong> directory holds our consumer Worker type, which is a bit more of a behemoth, taking a series of parameters to manage the whole process:</p><pre>type Worker struct {<br><br> messages []*util.TransactionRecord <br> <br> consumer *kafka.Consumer<br> <br> COMMIT_RETRY_DELAY time.Duration<br> <br> wg sync.WaitGroup<br> <br> dbconnection *sql.DB // The Postgres connection<br> <br> neo4jDriver neo4j.DriverWithContext<br> <br> ctx context.Context // The context controller<br> <br> neo4jSemaphore chan struct{} // Concurrency controller for Neo4j<br> <br> postgresSemaphore chan struct{} // Concurrency controller for Postgres<br> <br> BATCH_SIZE int<br> <br> neo4jdatabase string<br><br>}</pre><ul><li><strong>ctx</strong> a context object that coordinates the lifetime and cancellation across Go routines.</li><li><strong>neo4jSemaphore and postgresSemaphore </strong>: Two semaphores that control concurrency. Their job is to politely prevent us from bombarding the databases with requests when things get heated.</li></ul><p>This <strong>Worker</strong> type contains five methods, clustered into two key jobs:</p><ul><li><strong>ConsumeMessages:</strong> The workhorse method that waits indefinitely for incoming messages from the producer. In case of error, it logs the issue and keeps going no quitting on this job.</li></ul><p><strong>The four handler methods for incoming messages :</strong></p><ul><li><strong>BatchProcessMessages </strong>: Inserts a batch of records into the databases, using two helper functions.</li><li><strong>ProcessMessage </strong>: Decodes incoming messages and appends them to an array.</li><li><strong>Neo4jHandler</strong>: The logic for inserting into the Neo4j database.</li><li><strong>PostgresHandler</strong>: The logic for inserting into the Postgres database.</li></ul><p>The <strong>internal</strong> folder separates the two database handlers <strong>neo4j</strong> and <strong>postgres</strong>.</p><h3><strong>The Database Security Setup</strong></h3><p>Before we could start writing to the databases, I needed to set up the proper configuration. For each database, this meant creating a user, a role, and granting access.</p><p>For b, I created a user called… wait for it… <strong>user</strong> (Not my most creative moment, I know 😅. I’ll fix it later, promise!) Then I created a role. The commands went something like this:</p><pre>postgres=# CREATE ROLE app_rw NOLOGIN;<br>CREATE ROLE<br>postgres=# \c simulation <br>You are now connected to database &quot;simulation&quot; as user &quot;postgres&quot;.<br>simulation=# GRANT CONNECT ON DATABASE simulation TO app_rw;<br>GRANT<br>simulation=# GRANT USAGE ON SCHEMA public TO app_rw;<br>GRANT<br>simulation=# GRANT SELECT, INSERT, UPDATE, DELETE ON ALL TABLES IN SCHEMA public TO app_rw;<br>GRANT<br>simulation=# ALTER DEFAULT PRIVILEGES IN SCHEMA public<br>GRANT SELECT, INSERT, UPDATE, DELETE ON TABLES TO app_rw;<br>ALTER DEFAULT PRIVILEGES<br>simulation=# CREATE USER app_user WITH LOGIN PASSWORD &#39;******&#39;;<br>CREATE ROLE<br>simulation=# GRANT app_rw TO app_user;<br>GRANT ROLE</pre><p>The setup for <strong>Neo4j</strong> was similar:</p><pre>CREATE USER goserver SET PASSWORD &#39;******&#39; CHANGE NOT REQUIRED;<br>CREATE ROLE dev;<br>GRANT ACCESS ON DATABASE &quot;db_name_with_no_qoutes&quot; TO dev  <br>GRANT NAME MANAGEMENT ON DATABASE &quot;db_name_with_no_qoutes&quot; TO dev  <br>GRANT ALL GRAPH PRIVILEGES ON GRAPH &quot;db_name_with_no_qoutes&quot; TO dev<br>GRANT ROLE dev TO goserver;</pre><p>To learn more on access and role on neo4j you can refer to the documentation or you can read this blog that i read <a href="https://medium.com/neo4j/role-based-access-control-in-neo4j-4-1-3e65d5b3f45">Role Based Access Control in Neo4j</a></p><h3><strong>Batch Inserts with Transactions</strong></h3><p>Now, with access granted, it was time to code the logic for inserting data into each one. Crucially, we are inserting in batches.Of course, I used transactions. You may be asking, “What exactly is a transaction?”</p><blockquote>A transaction is a sequence of one or more database operations (like `INSERT`, `UPDATE`, `DELETE`) that are treated as a single, atomic unit of work.</blockquote><p>It’s a much faster and more reliable way to insert data. Why faster? Simply because you only have to <strong>commit once</strong> when the entire batch is finished, which is far better than committing record-by-record.</p><p>With that in mind, let’s dive into the code’s structure:</p><ul><li><strong>Postgres Handler contains three functions </strong>: NewDBConnection, insertRecord (a helper), and the all-important <strong>InsertData</strong>, which starts the transaction and inserts a batch of data.</li><li><strong>Neo4j Handler is relatively similar :</strong> CreateConnection, CreateOneRelation (a helper), and CreateRelations, which creates multiple relationships in a batch.</li></ul><p>As you can see, we’re doing different things in each database. For the <strong>Graph Database (Neo4j)</strong>, we create a relationship called <strong>TRANSACTION</strong> between existing nodes (Customer and Terminal). For each incoming record, we link the corresponding nodes. Why are the nodes already there? Because I created them separately using Python. Why that ? Well, because in a real-world scenario, the Customer and Terminal nodes would likely exist at the start of the simulation!</p><p>For the Postgres database, I’m just saving the transaction record exactly as it is.The simulation is almost finished! I just have one small bug to fix related to data duplication (that pesky little gremlin). But still, next week, I’m starting on the GNN models and reading a lot of papers.</p><p>Wish me luck! And byyy! 👋</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cbb26766c0b7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Great Graph Adventure: Using GNNS to Catch Digital Crooks]]></title>
            <link>https://medium.com/@aymenfkir23/the-great-graph-adventure-using-gnns-to-catch-digital-crooks-76a9b82e3662?source=rss-b4e6f95a3a16------2</link>
            <guid isPermaLink="false">https://medium.com/p/76a9b82e3662</guid>
            <category><![CDATA[fraud-detection]]></category>
            <category><![CDATA[gnn]]></category>
            <category><![CDATA[microservices]]></category>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[golang]]></category>
            <dc:creator><![CDATA[Aymenfkir]]></dc:creator>
            <pubDate>Mon, 06 Oct 2025 09:02:14 GMT</pubDate>
            <atom:updated>2025-10-06T09:02:14.091Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/979/1*_2A9czPQ72_qR-ahrh2wew.png" /><figcaption>i don’t know what i’m building 😅</figcaption></figure><p>Read a book, build a thing.” That’s my motto, or at least it was for the past week. I dove headfirst into the world of graphs not the bar chart kind, but the cool, interconnected kind. Specifically, <strong>Graph Neural Networks (GNNs)</strong>.</p><p>Turns out, these things are _fantastic_ at <strong>fraud detection</strong>. Why? Because transactions aren’t solo acts; they’re relationships. A GNN can sniff out the connection between a suspicious sender and a sketchy receiver much better than traditional models.</p><p>So, naturally, I thought: “I must build a ridiculously complicated Rube Goldberg machine to prove this!” My mission: create a fool-scale, high-speed transactional system simulation, plug in a GNN, and watch the magic (or the bugs) happen.</p><p>This whole glorious mess will be built in four acts:</p><ul><li>Data Collection &amp; Prep (The ‘Where Do I Find Fake Money?’ Stage)</li><li>Setting Up the Bank Simulation (The ‘Make It Go Zoom’ Stage)</li><li>Creating the GNN Model (The ‘Actual Brains’ Stage)</li><li>Connecting the Pieces (The ‘Please Don’t Explode’ Stage)</li></ul><h3><strong>Section 1: The Quest for Fake Fraud Data</strong></h3><p>To build a fraud-catching robot, you need <strong>data</strong> specifically, data with sender IDs, receiver IDs, and a big red flag that says “FRAUD!” or a tiny green flag that says “All Good.”Did I find a perfect, clean, production-ready dataset? <strong>Nope</strong>. Did I find something better? Maybe!</p><p>I stumbled upon this <a href="https://fraud-detection-handbook.github.io/fraud-detection-handbook/Chapter_3_GettingStarted/SimulatedDataset.html#customer-profiles-generation">amazing handbook</a> It contains a whole pipeline to generate a synthetic fraud dataset that is so messy and real-world-like, it’s perfect for torture testing a model.The generated data is the good kind of bad</p><blockquote>They will include class imbalance (less than 1% of fraudulent transactions), a mix of numerical and categorical features (with categorical features involving a very large number of values), non-trivial relationships between features, and time-dependent fraud scenarios.</blockquote><p>That’s code for: “This will be a realistic headache.” Perfect!</p><p>After generating my synthetic gold, I split it into two piles: one for training the GNN, and one to feed into my high-speed banking simulation. Data prepped, mission GO!</p><h3><strong>Section 2: Building a Bank That Never Sleeps (Literally)</strong></h3><p>When you decide to simulate a bank with transactions flying around like digital confetti, you can’t just use a for loop. You need speed, reliability, and the ability to handle data coming in fast.</p><p>After a quick Google session, I realized I was building a microservices architecture and needed a tool for streaming data. Enter Kafka!</p><p><strong>Meet the Tech Stack &amp; The Gang of Four</strong></p><pre>| Component      | Role                                      | Tech Used                    |<br>| -------------- | ----------------------------------------- | ---------------------------- |<br>| **The Client** | Sends transactions _concurrently_ (fast!) | **Golang**                   |<br>| **Storage 1**  | Relational Database (The Ledger)          | **PostgreSQL**               |<br>| **Storage 2**  | Graphical Database (The Connections)      | **Neo4j**                    |<br>| **The Brain**  | GNN Inference Server (The Fraud Catcher)  | _(TBD, but probably Python)_ |<br>| **The Glue**   | Connects everything in real-time          | **Kafka**                    |</pre><p>I spent the past week setting up the Client (in lightning-fast <strong>Golang</strong>) and getting Kafka to talk to it. It was a journey of producer.send() and consumer.poll()!</p><p><strong>Kafka: The Reliable Middleman</strong></p><p>Think of <strong>Kafka</strong> as a giant, distributed post office for data. It’s designed to handle massive volumes of real-time events.</p><p>The Main Concepts (Simplified):</p><ul><li><strong>Event:</strong> A record of something that happened (“John sent to Jane”)</li><li><strong>Topic:</strong> A category where events are stored (e.g., a “Bank_Transactions” topic).</li><li><strong>Producer:</strong> The guy who writes the events to the topic (My Golang Client).</li><li><strong>Consumer:</strong> The guys who read and process the events (Postgres, Neo4j, and my GNN Server).</li><li><strong>Partitions:</strong> Topics are split into buckets for speed and parallel processing.</li></ul><p>For my simulation, I’m using two topics to keep things organized:</p><ul><li><strong>Topic 1:</strong> For data going to my simple ledger (Postgres) and the GNN Inference Server.</li><li><strong>Topic 2:</strong> For data being structured and written to the graph database (Neo4j) to track relationships.</li></ul><p>Both my Producer and my Consumers are written in Golang It’s fast, has a simple syntax, and makes concurrent operations feel like a breeze.</p><p>The Producer is now happily loading the fake data and streaming transactions out. The Consumers (databases and the future GNN server) are waiting, ready to read and process that sweet, sweet data.</p><p><strong>Next up: The GNN Model itself!</strong> Wish me luck I’m going to need to figure out how to actually detect fraud now that I’ve built a ridiculously powerful data pipeline! 😅</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=76a9b82e3662" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From Data to Insights: An ETL Pipeline for LLM-Generated Reviews (Part6)]]></title>
            <link>https://medium.com/@aymenfkir23/from-data-to-insights-an-etl-pipeline-for-llm-generated-reviews-part6-7c53f8986fad?source=rss-b4e6f95a3a16------2</link>
            <guid isPermaLink="false">https://medium.com/p/7c53f8986fad</guid>
            <category><![CDATA[clean-code]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[best-practices]]></category>
            <dc:creator><![CDATA[Aymenfkir]]></dc:creator>
            <pubDate>Thu, 25 Sep 2025 08:15:57 GMT</pubDate>
            <atom:updated>2025-09-25T08:15:57.217Z</atom:updated>
            <content:encoded><![CDATA[<p>After optimizing the pipeline’s performance, the next logical step was to refine the codebase and repository structure. While some might argue this should have been a priority from the start, this project’s organic growth made it a necessary step to ensure long-term maintainability. This section details how I optimized the project’s structure by applying software engineering principles.</p><h3>Starting with the Go Client</h3><p>My initial Go client was a single file approaching <strong>500 lines of code</strong>. While functional, this made debugging difficult and collaboration challenging. To address this, I refactored the codebase to follow best practices, organizing it into a more modular structure</p><h4>Separation of Concerns</h4><ol><li><strong>Models</strong>: Defines all data structures and types.</li><li><strong>Extract</strong>: Handles data extraction from Supabase storage.</li><li><strong>Enrichment</strong>: Manages AI processing using Llama.cpp.</li><li><strong>Load</strong>: Deals with data merging and storage operations.</li><li><strong>Utils</strong>: Contains common utilities and helper functions.</li></ol><p>This separation ensures that each file has a single responsibility, making the code easier to fix, understand, and manage. After this refactoring, I containerized the Go client and added it to the Docker Compose file, establishing it as an independent service within my microservice architecture.</p><h3>Python ETL Pipeline</h3><p>Similarly, the Python ETL pipeline, originally a single file, was refactored into multiple files. This involved adding utility functions for data transformation and splitting the code into a modular design.</p><h4>Modular Design Principles</h4><ol><li><strong>Single Responsibility</strong>: Each module now handles a specific aspect of the ETL process.</li><li><strong>Separation of Concerns</strong>: There are clear boundaries between the <strong>Extract</strong>, <strong>Transform</strong>, and <strong>Load</strong> stages.</li><li><strong>Dependency Injection</strong>: Components receive their dependencies through constructors, improving testability.</li><li><strong>Configuration Management</strong>: Centralized settings with environment variable overrides provide flexibility.</li></ol><h3>Results</h3><p>By applying software engineering principles such as <strong>Single Responsibility</strong>, <strong>small functions</strong>, <strong>meaningful naming</strong>, and <strong>DRY</strong> (Don’t Repeat Yourself), I was able to make the codebase more readable and maintainable. This organizational step is crucial for the project’s future development and upkeep, allowing for easier feature additions and bug fixes.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7c53f8986fad" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From Data to Insights: An ETL Pipeline for LLM-Generated Reviews (Part5)]]></title>
            <link>https://medium.com/@aymenfkir23/from-data-to-insights-an-etl-pipeline-for-llm-generated-reviews-part5-601443f92b72?source=rss-b4e6f95a3a16------2</link>
            <guid isPermaLink="false">https://medium.com/p/601443f92b72</guid>
            <category><![CDATA[etl]]></category>
            <category><![CDATA[docker-compose]]></category>
            <dc:creator><![CDATA[Aymenfkir]]></dc:creator>
            <pubDate>Thu, 25 Sep 2025 08:15:19 GMT</pubDate>
            <atom:updated>2025-09-25T08:15:19.901Z</atom:updated>
            <content:encoded><![CDATA[<p>The latest update on the ETL pipeline journey is a masterclass in turning a fast-but-fussy local script into a robust, deployable system. After squeezing every drop of performance from the client and server code, it was time to address the most critical and often overlooked phase of any real-world project: <strong>deployment</strong>. 🐳</p><h3>The Big Docker</h3><p>I’ve been playing around with <strong>Docker</strong>, and it’s opened up a whole new world. This open-source platform makes it easy to package, deploy, and manage applications in <strong>containers</strong>, ensuring they run the same way on any machine.1 For this pipeline, it meant a complete shift in thinking, moving from a local-only setup to a portable, reproducible environment.</p><p>The server-side was the first to get the Docker treatment. I was lucky that <strong>Llama.cpp</strong> (the lean, mean LLM inference machine I’m using) comes with its own Dockerfile. All I had to do was install the CUDA toolkit, and boom, GPU-accelerated container.</p><p>Of course, it’s never that easy. My initial attempts to build the server from source inside a container hit a wall. Errors popped up about missing CUDA functions. It wasn’t that Docker couldn’t see the GPU, but that the shared libraries, those files ending in .so, weren&#39;t properly linked. The fix was a bit of bash scripting wizardry: copy those libraries into a dedicated lib folder and add it to the environment path so the system could find them during compilation.</p><p>note : i decided to use the already existed dockerfile provided by llama cpp , just for convenience , but still manged to make it run with my dockerfile</p><h3>The Two-Stage Build</h3><p>Containers can get chunky. The initial build, with all the compilers and dev tools, can be huge. That’s when I discovered <strong>multi-stage builds</strong>. This best practice is like a container diet plan.</p><p>Here’s the gist:</p><ol><li><strong>The “Devel” Stage:</strong> You create a first image (devel) that has everything you need to build your application, including compilers and source code.</li><li><strong>The “Runtime” Stage:</strong> You create a second, much smaller image (runtime) with only the bare essentials needed to run the final compiled binary. Then, you simply copy the binary from the first stage into the second.2</li></ol><p>By using this method for both the server and client images, I managed to <strong>reduce their size by a whopping 80%</strong>. That’s a massive win, saving a ton of disk space and making the images faster to push and pull.</p><h3>Orchestrating with Docker Compose</h3><p>Running two separate containers — the Go client and the Llama.cpp server — is a pain. This is where <strong>Docker Compose</strong> comes in to save the day.</p><p>Compose uses a single YAML file to define and manage multi-container applications.3 It lets you specify services and their parameters, so with one command, docker compose up, everything fires up.</p><p>But of course, the first time, my client couldn’t find the server. I had made a rookie mistake: the client was still looking for the server at http://localhost. The fix was a quick change to the URL, using the service name defined in the YAML file instead: http://servicename:port. Since I was smart enough to use environment variables from the start, this was a painless update that didn’t require a rebuild.</p><p>The final hurdle was a race condition. The client would start before the server was ready, throwing a “can’t find…” error. The solution? <strong>Health checks</strong> and <strong>dependencies</strong>. By adding a health check to the server and telling the client service to wait until the server was healthy, the whole system now comes up reliably and in the right order.</p><h3>The Final Push</h3><p>With these deployment hurdles cleared, the ETL pipeline is now a fully containerized, robust workflow. It starts by pulling data from a mock API, enriches it with LLM-generated reviews, and stores it in a “silver” bucket. The final “golden” bucket and some last-minute Go client features are all that remain.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=601443f92b72" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From Data to Insights: An ETL Pipeline for LLM-Generated Reviews (Part4)]]></title>
            <link>https://medium.com/@aymenfkir23/from-data-to-insights-an-etl-pipeline-for-llm-generated-reviews-part4-d3819226aace?source=rss-b4e6f95a3a16------2</link>
            <guid isPermaLink="false">https://medium.com/p/d3819226aace</guid>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[goroutines]]></category>
            <category><![CDATA[llama-cpp]]></category>
            <dc:creator><![CDATA[Aymenfkir]]></dc:creator>
            <pubDate>Thu, 25 Sep 2025 08:14:36 GMT</pubDate>
            <atom:updated>2025-09-25T08:14:36.034Z</atom:updated>
            <content:encoded><![CDATA[<p>Hey, my friend, I’m back with another update on the ETL pipeline for review generation. Last time, we managed to boost performance by 25% by switching from Ollama to the leaner, meaner Llama.cpp. That was a big win, reducing the processing time to a blazing-fast <strong>0.3 seconds per item</strong> on my low-spec setup (i5 12th gen and a GTX 1650). But for a dataset of a million rows, every millisecond counts. So, I went back to the drawing board for more optimizations.</p><h3>Client-Side Reinvention</h3><p>The server is what does the heavy lifting of generating the reviews, but the client is equally critical, especially when you’re processing a million records. Any delay on the client side, no matter how small, can pile up to days of extra processing time.</p><p>That’s why I made a crucial decision: <strong>I rewrote my client in Go</strong>.</p><p>Why Go? Go is one of the fastest languages around, and it was designed specifically for modern applications that need to handle high volumes of concurrent tasks. It’s the language behind giants like Docker and Kubernetes, so it’s built for speed and, most importantly, <strong>concurrency</strong>.</p><p>Concurrency is the ability to handle multiple tasks at the same time, and Go makes this incredibly easy with <strong>goroutines</strong>. Think of a goroutine as a lightweight thread of execution managed by the Go runtime. They’re cheap to create and consume minimal resources, making them perfect for my use case.</p><p>Imagine handling server responses, processing them, and saving the results to a list — all simultaneously. With goroutines, I can launch four concurrent requests to my Llama.cpp server, which in turn generates four responses at the same time. The result? My computer is now officially a little space heater, but the performance gains are worth it.</p><h3>Results</h3><p>This refactoring paid off big time. With the Go client, I’ve hit another milestone: a new processing time of <strong>0.2 seconds per item</strong>. That’s a <strong>33% increase</strong> in speed from the last update.</p><p>At this new rate, my pipeline can process a million records in just over 2.3 days. While this isn’t the sub-2-hour goal I initially aimed for, it’s a monumental leap from the estimated month it would have taken at the beginning of this project.</p><h3>Conclusion</h3><p>Although I haven’t reached my ultimate goal, I’m incredibly happy with where this project has landed. It started as a simple Python script on Kaggle and evolved into building my own server and a high-performance client in a language I had to learn from scratch.</p><p>It’s a powerful reminder of the journey. As they say, “Aim for the stars, and even if you miss, you’ll land among the stars.” In this case, I aimed for two hours and landed in a little over two days — a <strong>92% reduction</strong> in processing time from my initial estimate.</p><p>I’ve explored every viable optimization I could think of for my hardware. While I considered other options like NVIDIA NIM servers, they’re not compatible with my GPU. I could also strip out some elements from the request, like the item_id, to reduce generation time. However, this would make it incredibly difficult to verify the quality of the reviews and merge them back with the original data.</p><p>This project has been a masterclass in performance engineering. I’m finalizing the README and polishing the last few functions in the ETL pipeline, but the core work is done. I’m really enjoying Go, and I’d love any feedback on best practices for my new client.</p><p>Happy coding, everyone! 🍻</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d3819226aace" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>