<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Saheed Adebayo on Medium]]></title>
        <description><![CDATA[Stories by Saheed Adebayo on Medium]]></description>
        <link>https://medium.com/@adekunle.adebayo001?source=rss-03e8ec8a05b6------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*tB6jsFvoIaKjs4dF</url>
            <title>Stories by Saheed Adebayo on Medium</title>
            <link>https://medium.com/@adekunle.adebayo001?source=rss-03e8ec8a05b6------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 15:03:43 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@adekunle.adebayo001/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[I Built a Model That Gave Up Being Right to Never Be Wrong Where It Counts]]></title>
            <link>https://medium.com/@adekunle.adebayo001/i-built-a-model-that-gave-up-being-right-to-never-be-wrong-where-it-counts-074330e0d46f?source=rss-03e8ec8a05b6------2</link>
            <guid isPermaLink="false">https://medium.com/p/074330e0d46f</guid>
            <dc:creator><![CDATA[Saheed Adebayo]]></dc:creator>
            <pubDate>Sat, 05 Sep 2026 16:11:28 GMT</pubDate>
            <atom:updated>2026-09-05T16:11:28.837Z</atom:updated>
            <content:encoded><![CDATA[<p><em>My flood-prediction system traded a clean report card for a perfect safety record.</em></p><p>Fourteen floods hit my test data. My model caught all fourteen.</p><p>Its accuracy score dropped by 30 points to make that happen.</p><p>If you only looked at the accuracy number, you’d think I broke something. I didn’t. I built a flood-prediction system for Abuja, Nigeria, and along the way I learned that the metric everyone trusts most, accuracy, is exactly the wrong thing to optimize for when the disaster you’re trying to catch only happens 14 times out of 545 days. Chasing a high score would have meant building a model that stayed quiet through every single flood and still looked like a genius on paper.</p><p>So I threw the scorecard out. Here’s what happened when I stopped trying to be right, and started trying to never be wrong about the one thing that mattered.</p><h3>The Problem: Water Doesn’t Wait for a Warning</h3><p>Every rainy season, cities like Abuja face the same anxious question: will today be the day the drains overflow?</p><p>Floods aren’t random. They’re the product of two very different kinds of information colliding at the wrong moment.</p><ol><li><strong>The land itself.</strong> How steep a hillside is, which way water naturally flows, how much vegetation is around to soak it up, and how saturated the ground already is.</li><li><strong>The weather over time.</strong> Not just today’s rainfall, but yesterday’s, and the week before that, because ground that’s already soaked behaves completely differently from dry ground during a storm.</li></ol><p>Most flood-prediction tools traditionally look at only one of these. This project tried something more ambitious. It built a model that looks at both at once: a static picture of the terrain, and a moving picture of the weather, to see whether combining them predicts floods better than either one alone.</p><p>Let’s walk through how that was done in plain language, then look honestly at what the results actually showed, including the uncomfortable parts.</p><h3>Step One: Turning a Landscape Into Numbers</h3><p>Before any predicting can happen, a computer needs the terrain translated into something it can read. This is done by taking satellite and elevation data and slicing it into four separate “layers,” like transparent sheets stacked on top of a paper map.</p><ul><li><strong>Slope:</strong> How steep the ground is. Steep areas shed water fast; flat areas let it pool.</li><li><strong>Aspect:</strong> Which direction a slope faces, which affects how and where water drains.</li><li><strong>NDVI (vegetation index):</strong> How much green cover is present. Vegetation slows down runoff and helps soil absorb water; bare or paved land doesn’t.</li><li><strong>NDWI (moisture index):</strong> How much water the surface is already holding onto.</li></ul><p>Picture Abuja’s terrain as a 128-by-128 grid of tiny tiles, with each tile carrying four numbers, one for each of these layers. That’s the “spatial tensor” the model works with: essentially a very detailed, numerical version of a topographic map.</p><p>Here’s roughly what that looks like in code. Each layer gets computed from the elevation model, then squashed into a common 0-to-1 scale so the model doesn’t treat “360 degrees of aspect” as more important than “0.8 NDVI” just because the raw number is bigger:</p><pre># Slope &amp; Aspect derived from elevation gradients<br>dz_dy, dz_dx = np.gradient(dem)<br>slope = np.arctan(np.sqrt(dz_dx**2 + dz_dy**2)) * (180.0 / np.pi)<br>aspect = (270.0 - np.arctan2(dz_dy, -dz_dx) * (180.0 / np.pi)) % 360.0<br><br># Normalize every layer into the same [0, 1] range<br>scaler = MinMaxScaler()<br>slope_norm = scaler.fit_transform(slope)<br>aspect_norm = scaler.fit_transform(aspect)<br>ndvi_norm = scaler.fit_transform(ndvi)<br>ndwi_norm = scaler.fit_transform(ndwi)<br><br># Stack the four layers into one 128x128x4 &quot;image&quot;<br>tensor_stack = np.stack([slope_norm, aspect_norm, ndvi_norm, ndwi_norm], axis=-1)</pre><p>That last line is the key idea: four separate maps get stacked into a single object with a “depth” of four, the same way a normal photo stacks red, green, and blue into one image. The model doesn’t see four maps; it sees one four-channel image.</p><h3>Step Two: Turning Weather Into a Story With Memory</h3><p>Rain on its own doesn’t cause a flood. Rain on top of already-saturated ground does. So the model doesn’t just look at today’s rainfall. It looks at a 30-day rolling window of:</p><ul><li>Daily rainfall</li><li>Maximum and minimum temperature</li><li>Humidity</li><li>“Antecedent Precipitation Index” at 1, 7, and 14 days, which is essentially a running tally of how much water has fallen recently and acts as a stand-in for how saturated the ground probably is</li></ul><p>This is sometimes called the “sponge effect.” A sponge that’s already wet needs far less new water to start dripping than a dry one. By feeding the model a month’s worth of weather history instead of just one day, it can learn to recognize that build-up pattern before a flood hits, rather than just reacting to a single big rainstorm in isolation.</p><p>Across ten years of daily data (2015 to 2024), this sliding 30-day window produced thousands of overlapping weather “snapshots,” each one paired with a label: did a flood happen on that day, or not?</p><p>In code, the antecedent precipitation indices are just rolling sums, and the “sliding window” is exactly that, a loop that slices out 30 consecutive days at a time and pairs it with the flood label on the last day of that window:</p><pre># The &quot;sponge effect&quot;: running totals of recent rainfall<br>df[&#39;API_1d&#39;] = df[&#39;PRECTOTCORR&#39;]<br>df[&#39;API_7d&#39;] = df[&#39;PRECTOTCORR&#39;].rolling(window=7, min_periods=1).sum()<br>df[&#39;API_14d&#39;] = df[&#39;PRECTOTCORR&#39;].rolling(window=14, min_periods=1).sum()<br><br># Chronological sliding window: 30 days of weather -&gt; 1 label<br>X_seq, y_seq = [], []<br>for i in range(len(df_scaled) - lookback + 1):<br>    seq = df_scaled.iloc[i:i + lookback][feature_cols].values<br>    target = df_scaled.iloc[i + lookback - 1][&#39;Flood&#39;]<br>    X_seq.append(seq)<br>    y_seq.append(target)</pre><p>Each seq is one training example: a 30-day-by-7-feature table of weather leading up to a given day. The model never sees isolated days in this branch; it always sees the trailing month.</p><h3>Step Three: Three Competing Models</h3><p>With both types of data prepared, three separate models were built and compared, like three different specialists being asked to make the same call.</p><ul><li><strong>Model 1, the Map Reader (Baseline CNN).</strong> Looks only at the terrain layers. It never sees any weather data. It’s essentially answering: “Based purely on geography, how flood-prone is this location?”</li><li><strong>Model 2, the Weather Watcher (Baseline LSTM).</strong> Looks only at the 30-day weather sequence, with no idea what the terrain looks like. It’s answering: “Based purely on recent weather patterns, does this look like flood conditions?”</li><li><strong>Model 3, the Hybrid (Proposed CNN-LSTM).</strong> Looks at both at the same time, and lets the model decide how much weight to give each. This is the model the project is actually proposing as an improvement.</li></ul><p>Technically, the CNN (“Convolutional Neural Network”) is the type of model good at recognizing spatial patterns, the same family of models used in image recognition. The LSTM (“Long Short-Term Memory”) is a type of model built to remember sequences over time, which makes it well suited to weather history. The hybrid model runs both in parallel and merges their conclusions before making one final decision.</p><p>The hybrid architecture is the most interesting piece of code in the whole project, because you can see the “two specialists, one decision” structure directly in how the layers connect:</p><pre># Spatial branch: reads the four-layer terrain map<br>inp_spat = Input(shape=(128, 128, 4), name=&quot;Spatial_Input&quot;)<br>s = Conv2D(32, (3, 3), activation=&#39;relu&#39;, padding=&#39;same&#39;)(inp_spat)<br>s = MaxPooling2D((2, 2))(s)<br>s = Conv2D(64, (3, 3), activation=&#39;relu&#39;, padding=&#39;same&#39;)(s)<br>s = MaxPooling2D((2, 2))(s)<br>s_vec = Flatten(name=&quot;Spatial_Flatten&quot;)(s)<br><br># Temporal branch: reads the 30-day weather sequence<br>inp_temp = Input(shape=(30, 7), name=&quot;Temporal_Input&quot;)<br>t = LSTM(64, return_sequences=True)(inp_temp)<br>t = Dropout(0.20)(t)<br>t_vec = LSTM(32, return_sequences=False, name=&quot;Temporal_Summary&quot;)(t)<br><br># The two &quot;opinions&quot; are merged into one before the final decision<br>fused = Concatenate(name=&quot;Parallel_Fusion&quot;)([s_vec, t_vec])<br>x = Dense(64, activation=&#39;relu&#39;)(fused)<br>x = Dropout(0.30)(x)<br>out = Dense(1, activation=&#39;sigmoid&#39;, name=&quot;Output&quot;)(x)</pre><p>Notice there are two separate inputs (inp_spat and inp_temp) that never talk to each other until the Concatenate line. Up to that point, the map-reading half and the weather-reading half are working completely independently, each condensing their own data down into a short list of numbers. Only at the very end do those two summaries get glued together and handed to a final layer that makes the actual flood-or-no-flood call.</p><h3>Step Four: The Needle-in-a-Haystack Problem</h3><p>Here’s the part that trips up almost every real-world flood model, and it’s worth sitting with, because it explains everything that follows.</p><p>Out of 545 days in the test period, only 14 were actual flood days. That means if a model simply predicted “no flood, ever,” it would be right about 97% of the time and would still be completely useless, because it would miss every single flood.</p><p>This is called class imbalance, and it’s one of the most common traps in predictive modeling: a model can look impressively accurate on paper while being practically worthless at the one thing it was built to do.</p><p>To fight this, the project used class weighting, essentially telling the model that getting a flood day wrong is far more costly than getting a normal day wrong, so it should pay much more attention to those rare examples:</p><pre># Flood days are rare, so mistakes on them get penalized much harder<br>N_total = len(y_train)<br>N_flood = max(1, y_train.sum())<br>N_normal = N_total - N_flood<br><br>w0 = N_total / (2.0 * N_normal)   # weight for &quot;normal&quot; mistakes<br>w1 = N_total / (2.0 * N_flood)    # weight for &quot;flood&quot; mistakes<br>class_weights = {0: float(w0), 1: float(w1)}</pre><p>Because floods make up such a tiny slice of the data, w1 ends up dramatically larger than w0, essentially telling the model: &quot;missing one of these rare flood days should hurt your score far more than misjudging an ordinary day.&quot;</p><p>It also ran an automated “threshold sweep,” testing hundreds of different cutoff points for how confident the model needs to be before it calls something a flood, to find the setting that best balanced catching real floods against crying wolf too often:</p><pre>best_thresh, best_mcc = 0.5, -1.0<br><br>for th in np.linspace(0.001, 0.999, 500):<br>    val_metrics = evaluate_predictions(y_val, hybrid_val_probs, th)<br>    if val_metrics[&#39;mcc&#39;] &gt; best_mcc:<br>        best_mcc = val_metrics[&#39;mcc&#39;]<br>        best_thresh = th</pre><p>Instead of just using the default 0.5 cutoff (“more than 50% confident means yes”), this sweep tries 500 different cutoffs on validation data and keeps whichever one produces the healthiest overall balance, measured by the Matthews Correlation Coefficient (MCC), a metric built specifically to stay meaningful even when one class is rare. That search is how the Hybrid model ended up operating at a threshold of just 0.126, meaning it only needed to be about 13% confident before sounding the alarm, a deliberate choice to err toward caution.</p><h3>The Results: A Story Told in Trade-Offs</h3><p>Here’s how the three models performed on 545 unseen test days (531 normal, 14 flood days).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/578/1*qFuEVwLk-ClV8AsMcC7TxQ.png" /></figure><p>At first glance, this table looks like the “improved” model performed worse. Its accuracy dropped by nearly 30 points. But this is exactly the trap described above, and it’s where the real story is.</p><p>The Map Reader and Weather Watcher, on their own, essentially collapsed into the laziest possible strategy: predict “no flood” for everything. That’s why their accuracy looks high (531 out of 545 days genuinely were normal), but their recall is a flat 0%. They missed every single flood. A model that never rings the alarm isn’t a flood-warning system; it’s just quiet.</p><p>The Hybrid model, by contrast, was tuned to behave completely differently. Looking at its actual confusion matrix on the test set:</p><ul><li><strong>True Negatives: 366.</strong> Normal days correctly left alone.</li><li><strong>False Positives: 165.</strong> False alarms, days it flagged as risky that turned out fine.</li><li><strong>False Negatives: 0.</strong> Floods it missed.</li><li><strong>True Positives: 14.</strong> Floods it correctly caught, out of 14 total.</li></ul><p>That zero in “False Negatives” is the headline number. It didn’t miss a single real flood. The cost of that was a lot of false alarms (165 days flagged as risky that turned out calm), which is why its precision (7.82%) is low and its overall accuracy dropped.</p><h3>Why “Less Accurate” Can Still Mean “Better”</h3><p>This is the part that isn’t intuitive if you’re used to accuracy being the only number that matters, so it’s worth spelling out with an analogy.</p><p>Imagine a smoke detector. If you tuned it to only go off when it was 97% certain the building was actually on fire, it would rarely trigger falsely, and it would also fail to warn you about several real fires. If you tuned it to be extremely sensitive, it might go off sometimes when someone’s just burning toast, which is annoying, but nobody dies from a false alarm.</p><p>A flood-prediction system built for actual civil protection generally should behave more like the second detector. Missing a real flood (a false negative) can mean lost lives and property. A false alarm (a false positive) costs inconvenience: maybe a wasted precaution, a delayed commute, extra vigilance on a day that turns out fine. The Hybrid model was explicitly tuned toward this philosophy. It accepted a much higher false-alarm rate in exchange for never missing a real event in this test window.</p><p>Whether that trade-off is worth it is a judgment call, not a purely technical one. It depends on how costly false alarms actually are to the people receiving them, and how the system would be used operationally (does a false alarm just mean an SMS advisory, or does it trigger evacuations?).</p><h3>What the Training Process Revealed</h3><p>Watching how each model learned adds another layer to the story.</p><p>The Map Reader (CNN) struggled visibly. During training it briefly suffered what looks like a kind of learning collapse around epochs 5 and 6, and ultimately gave up and defaulted to predicting the majority class, the “always say no flood” shortcut. This suggests that terrain alone, without any weather context, simply isn’t enough signal to catch a rare, time-triggered event like a flood. Geography can tell you where floods are more likely in general, but not when.</p><p>The Weather Watcher (LSTM) converged much more smoothly and briefly touched 37.5% recall on validation data during training, a sign that weather sequences alone carry real predictive signal, but it still couldn’t translate that into catching floods reliably on the unseen test set.</p><p>The Hybrid model had the roughest-looking training curve of the three, with a large early loss spike as it adjusted to the heavy penalty for missing flood days. But that instability was actually the model being pushed, on purpose, away from the deceptive high-accuracy plateau and toward a version of itself that actually pays attention to flood days, eventually reaching 100% recall on validation data as well.</p><p>In other words, the worse-looking learning curve for the Hybrid model isn’t a sign of a broken model. It’s the visible cost of forcing a model to stop taking the easy way out.</p><h3>The Honest Takeaways</h3><p>A few things are worth stating plainly, because a responsible read of these results shouldn’t oversell them.</p><p><strong>The Hybrid model’s low precision (7.82%) is a real limitation.</strong> In practice, this would mean roughly 12 false alarms for every real flood correctly flagged. That’s usable for a cautionary early-warning tool, but not for something making high-stakes automated decisions without human review.</p><p><strong>Perfect recall on 14 flood events is a small sample.</strong> Catching all 14 out of 14 is a strong result, but with so few positive examples, a single missed or mislabeled event in a different test period could change the picture substantially. This kind of model deserves testing against more flood seasons before anyone treats its numbers as settled.</p><p><strong>The core finding held up:</strong> combining spatial and temporal data beats either one alone at actually detecting floods. Both baseline models caught zero floods. The Hybrid model caught all of them, at the cost of more false alarms. That’s a meaningful, demonstrable benefit of the “look at both the map and the weather” approach, even if the model still needs refinement before real-world deployment.</p><h3>Why This Matters Beyond One City</h3><p>The specific numbers here belong to one model, one place, and one dataset. But the underlying lesson generalizes to a lot of predictive systems being built for public safety, medicine, and disaster response: accuracy is often the wrong headline metric when the thing you’re trying to catch is rare. A model can be 97% accurate and still be catastrophically useless, simply because the 3% it gets wrong is the only 3% anyone cared about.</p><p>The more useful question, for flood prediction or any early-warning system, is almost never “how often is this model right?” It’s “when it matters most, does this model fail loudly, or does it fail silently?” A hybrid model that raises too many false alarms is a nuisance. A model that quietly says everything’s fine right before a flood is a disaster.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=074330e0d46f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[I Built a Token-Burning Multi-Agent System (and Fixed It)]]></title>
            <link>https://medium.com/@adekunle.adebayo001/i-built-a-token-burning-multi-agent-system-and-fixed-it-efb81fed5139?source=rss-03e8ec8a05b6------2</link>
            <guid isPermaLink="false">https://medium.com/p/efb81fed5139</guid>
            <dc:creator><![CDATA[Saheed Adebayo]]></dc:creator>
            <pubDate>Tue, 03 Feb 2026 20:40:13 GMT</pubDate>
            <atom:updated>2026-02-03T21:51:53.146Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Zap5UDiZGPUbIMiiVoRj9w.png" /></figure><p>A technical post-mortem on CrewAI, context amplification, and cost-efficient LLM architecture.</p><p>I recently decided to dive deeper into <strong>CrewAI</strong>, partly out of curiosity and partly to understand how much leverage agent-based abstractions really give compared to working directly with the <strong>OpenAI SDK</strong>.</p><p>Coming from a more “manual” SDK-first mindset, CrewAI felt immediately appealing. The moment I scaffolded a new project, I had agents, tasks, tools, memory, and orchestration wired together with almost no effort. Roles, goals, backstories, task dependencies all neatly structured. It felt like going from writing raw HTTP handlers to using a full-fledged web framework.</p><p>At first, this scaffolding felt like pure productivity. I could express intent at a higher level, move faster, and focus on <em>what</em> the system should do rather than <em>how</em> each model call was wired. Compared to the OpenAI SDK where every request, response, and control flow decision is explicit , CrewAI offered a powerful abstraction layer that made multi-agent systems feel approachable and even elegant.</p><p>And for a while, it was genuinely fun.</p><p>Then I started running real workloads.</p><p>What surprised me wasn’t a gradual increase in cost, but how quickly token usage spiked once the system moved beyond toy examples. A single run could burn through tokens at a rate that felt wildly disproportionate to the actual “amount of thinking” I expected the system to do. Nothing was obviously broken. The prompts weren’t absurd. The agents were doing exactly what I asked.</p><p>That’s when it became clear: the issue wasn’t the models, and it wasn’t even the prompts.</p><p>It was the <strong>architecture implied by the abstractions</strong>.</p><p>This post is a detailed post-mortem of that experience.</p><p>What CrewAI does exceptionally well, where its defaults can become dangerous at scale, how those tradeoffs differ from using the OpenAI SDK directly, and the architectural changes that ultimately turned a token-burning prototype into a predictable, production-grade pipeline.</p><p>If you’ve ever felt like your LLM costs “mysteriously” exploded after adding agents, tools, or orchestration layers, this is likely a story you’ll recognize.</p><h3>The Project: What I Was Actually Building</h3><p>Before digging into token behavior, it’s worth being explicit about the project itself.</p><p>The goal was to build a <strong>multi-agent stock research and selection pipeline</strong> focused on a given market sector. Conceptually, the system was simple:</p><ul><li>Identify <strong>trending companies</strong> in a sector</li><li>Research those companies using up-to-date information</li><li>Analyze growth potential and risks</li><li>Produce <strong>stock recommendations</strong> (what to buy, what to avoid)</li><li>Optionally notify the user via Pushover or Sendgrid and persist reports</li></ul><p>CrewAI made this deceptively easy to express.</p><p>I modeled the system using distinct agents, each with a clear responsibility:</p><ul><li>A <strong>trending company finder</strong> that searched the web for companies gaining attention</li><li>A <strong>financial researcher</strong> that performed deeper analysis on each company</li><li>A <strong>market analyst</strong> that synthesized findings into structured reports</li><li>A <strong>stock picker</strong> that made final buy/avoid decisions</li><li>A <strong>manager agent</strong> that coordinated the workflow</li></ul><p>Each step fed into the next, and outputs were written to markdown files for inspection and downstream use.</p><p>At a high level, the pipeline looked like this:</p><pre>Find trending companies<br>        ↓<br>Research each company<br>        ↓<br>Analyze market position<br>        ↓<br>Pick stocks to buy / avoid</pre><h3>Why I Looked at CrewAI in the First Place</h3><p>Compared to the OpenAI SDK, where every model call, response, and control flow decision is explicit, CrewAI offered a higher-level abstraction that made multi-agent systems feel approachable and even elegant.</p><p>And for a while, it was genuinely fun.</p><p>Until I ran real workloads.</p><p>That’s when I shockingly exhausted my tokens.</p><h3>CrewAI’s Strengths (Before Things Went Wrong)</h3><p>From a modeling perspective, this felt like a perfect use case for CrewAI.<br> The problem was well-defined, naturally decomposable, and sequential in nature. Each agent had a clear role, and the abstraction mapped cleanly to the mental model I already had from designing data pipelines.</p><p>In fact, this is exactly where CrewAI shines. It allows you to describe <em>intent</em> rather than wiring every model call by hand, as you would with the OpenAI SDK.</p><p>Unfortunately, that same convenience also hid some extremely expensive behavior.</p><p>Before diving into the failure, it’s important to say this clearly: <strong>CrewAI is powerful</strong>.</p><p>It gives you:</p><ul><li>Strong scaffolding out of the box</li><li>A clean mental model for agents and tasks</li><li>Declarative orchestration</li><li>Tooling and memory without boilerplate</li></ul><p>This abstraction is valuable. It lowers the barrier to entry for multi-agent systems dramatically.</p><p>The problem wasn’t CrewAI itself , it was how easily its abstractions can hide <strong>costly architectural behavior</strong>.</p><p>At first, token usage seemed normal.</p><p>Then workloads grew slightly:</p><ul><li>More realistic tasks</li><li>Web search enabled</li><li>Longer outputs requested</li></ul><p>Suddenly, token usage spiked in a way that felt <strong>disproportionate to the actual work being done</strong>.</p><p>Nothing was obviously broken.<br> The prompts looked fine.<br> The agents were doing exactly what I asked.</p><p>That’s when it became clear the issue wasn’t the models.</p><h3>My Mental Model (Coming from OpenAI SDK background)</h3><p>My initial assumptions were simple:</p><ul><li>Each agent thinks independently</li><li>Token cost roughly equals number of prompts</li><li>Tools are just helpers</li><li>Managers are just coordinators</li></ul><p>All of these assumptions broke down in multi-agent systems.</p><h3>What Was Actually Happening Under the Hood</h3><p>My crews looked like this:</p><pre>@CrewBase<br>class StockPickerCrew:<br>    agents: List[BaseAgent]<br>    tasks: List[Task]<br><br>    @agent<br>    def trending_company_finder(self) -&gt; Agent:<br>        return Agent(<br>            config=self.agents_config[&quot;trending_company_finder&quot;],<br>            tools=[SerperDevTool()],<br>            verbose=True<br>        )<br><br>    @agent<br>    def financial_researcher(self) -&gt; Agent:<br>        return Agent(<br>            config=self.agents_config[&quot;financial_researcher&quot;],<br>            tools=[SerperDevTool()],<br>            verbose=True<br>        )<br><br>    @agent<br>    def market_analyst(self) -&gt; Agent:<br>        return Agent(<br>            config=self.agents_config[&quot;market_analyst&quot;],<br>            tools=[SerperDevTool()],<br>            verbose=True<br>        )<br><br>    @agent<br>    def stock_picker(self) -&gt; Agent:<br>        return Agent(<br>            config=self.agents_config[&quot;stock_picker&quot;],<br>            verbose=True<br>        )<br><br>    @crew<br>    def crew(self) -&gt; Crew:<br>        manager = Agent(<br>            config=self.agents_config[&quot;manager&quot;],<br>            allow_delegation=True,<br>            verbose=True<br>        )<br><br>        return Crew(<br>            agents=self.agents,<br>            tasks=self.tasks,<br>            process=Process.hierarchical,<br>            manager_agent=manager,<br>            verbose=True<br>        )</pre><p>On its own, nothing here looks unreasonable:</p><ul><li>Each agent had a clearly defined role</li><li>Web search was enabled where fresh data was needed</li><li>A manager agent coordinated the workflow</li><li>Hierarchical execution promised better orchestration</li><li>Verbose logging helped during early development</li></ul><p>In fact, this setup is <em>exactly</em> what CrewAI’s abstractions encourage when you’re just getting started.</p><p>The problem wasn’t any single line of code.</p><p>It was the <strong>combined effect</strong> of:</p><ul><li>multiple agents browsing independently</li><li>hierarchical execution rereading context</li><li>verbose outputs being generated early</li><li>and all of that context being passed forward repeatedly</li></ul><p>Those costs only became visible once the system was run end-to-end.</p><p>The real problem was <strong>context re-consumption</strong>.</p><p>In hierarchical mode:</p><ul><li>Agent A produces output</li><li>The manager rereads Agent A</li><li>Agent B rereads Agent A</li><li>The manager rereads again</li><li>Agent C rereads everything</li></ul><blockquote><em>The same text is generated once but consumed many times.</em></blockquote><blockquote>Now add <strong>web search</strong>.</blockquote><p>Each search injects large result dumps into context, which are then:</p><ul><li>passed downstream</li><li>reread by every agent</li><li>reread again by the manager</li></ul><p>This led to an absolute context snowball which directly led to high token usage consumption.</p><pre>BAD (context snowball)<br><br>T1: Scout + Search<br>  context: ██████████<br><br>T2: Research + Search (reads T1)<br>  context: ████████████████████<br><br>T3: Analysis + Search (reads T2)<br>  context: ████████████████████████████<br><br>T4: Stock Picker (reads T3)<br>  context: ████████████████████████████████</pre><h3>Why Hierarchical Mode Made Everything Worse</h3><p>Hierarchical mode is powerful but expensive.</p><p>It shines when:</p><ul><li>tasks are dynamic</li><li>plans change</li><li>agents disagree</li><li>quality gates matter</li></ul><p>My project was not any of these. It was linear</p><pre>find → research → analyze → decide</pre><p>Hierarchical mode added rereads, delegation overhead, repeated reasoning passes, back and forth. Each operation was consuming extra tokens</p><p>…with no benefit.</p><h3>The Core Insight: Cost Is Dominated by Context, Not Generation</h3><p>This was the turning point.</p><p><strong>LLM cost is dominated by how often text is read, not how often it is written.</strong></p><p>Once verbose text exists early:</p><ul><li>every downstream step pays for it</li><li>every agent rereads it</li><li>every manager rereads it again</li></ul><p>This is <strong>context amplification </strong>and the real reason tokens disappear.</p><h3>The Architectural Fix (High-Level)</h3><p>The fix wasn’t prompt tuning.</p><p>It was architectural:</p><ul><li>One agent fetches external data</li><li>Early compression</li><li>Structured handoff</li><li>Sequential execution</li><li>Best models reserved for judgment</li></ul><h3><strong>Fixed Implementation (Token-Efficient)</strong></h3><pre>@CrewBase<br>class StockPickerBest:<br>    agents: List[BaseAgent]<br>    tasks: List[Task]<br><br>    @agent<br>    def scout(self) -&gt; Agent:<br>        # The ONLY agent allowed to browse<br>        return Agent(<br>            config=self.agents_config[&quot;scout&quot;],<br>            tools=[SerperDevTool()],<br>            verbose=False<br>        )<br><br>    @agent<br>    def compressor(self) -&gt; Agent:<br>        # Offline transformation only<br>        return Agent(<br>            config=self.agents_config[&quot;compressor&quot;],<br>            tools=[],<br>            verbose=False<br>        )<br><br>    @agent<br>    def decision_maker(self) -&gt; Agent:<br>        # Reasoning and judgment only<br>        return Agent(<br>            config=self.agents_config[&quot;decision_maker&quot;],<br>            tools=[],<br>            verbose=False<br>        )<br><br>    @crew<br>    def crew(self) -&gt; Crew:<br>        return Crew(<br>            agents=self.agents,<br>            tasks=self.tasks,<br>            process=Process.sequential,<br>            verbose=False<br>        )</pre><p><strong>What changed?</strong></p><ul><li>Exactly <strong>one browsing agent</strong></li><li>Web data frozen into a compact <strong>Evidence Pack</strong></li><li>Downstream agents operate <strong>offline</strong></li><li>Sequential execution (no manager rereads)</li><li>Early compression, late expansion</li><li>Reasoning isolated to final steps</li></ul><p>This wasn’t a micro-optimization.</p><p>It was the difference between:</p><ul><li><strong>context snowballing</strong> vs <strong>bounded context</strong></li><li>unpredictable cost vs predictable cost</li><li>prototypes that “feel fine” vs systems that scale</li></ul><p>The biggest savings didn’t come from better prompts or cheaper models,<br> they came from <strong>deciding which agent is allowed to expand context at all</strong>.</p><h3>The Evidence Pack Pattern</h3><p>Instead of passing prose, I introduced a strict handoff object.</p><pre>class EvidenceCompany(BaseModel):<br>    name: str<br>    ticker: str<br>    why_trending: str<br>    evidence: List[str]   # exactly 3 bullets<br>    sources: List[HttpUrl]<br><br>class EvidencePack(BaseModel):<br>    companies: List[EvidenceCompany]  # exactly 3</pre><p>This became the <strong>single source of truth</strong>.</p><p>No agent downstream was allowed to:</p><ul><li>browse again</li><li>introduce new facts</li><li>expand unnecessarily</li></ul><h3>Splitting Models by Task Type (Gemini + OpenAI)</h3><p>This unlocked massive savings.</p><pre>| Task            | Model  |<br>| --------------- | ------ |<br>| Web search      | Gemini |<br>| Summarization   | Gemini |<br>| Compression     | Gemini |<br>| Trade-offs      | OpenAI |<br>| Final decisions | OpenAI |</pre><p>Never ask your best model to ingest raw web data. It is an absolute waste of token usage</p><p><strong>The Final Architecture</strong></p><pre>[Scout (Gemini) + Search]<br>        |<br>        v<br>  EvidencePack (JSON)<br>        |<br>        v<br>[Compressor (Gemini)]<br>        |<br>        v<br> ResearchPack (JSON)<br>        |<br>        v<br>[Decision Maker (OpenAI)]<br>        |<br>        v<br>Final Picks</pre><p><strong>This is what context became:</strong></p><pre>T1: Scout + Search<br>  context: ████<br><br>T2: Compressor (offline)<br>  context: █████<br><br>T3: Decision Maker (offline)<br>  context: ██████</pre><h3>Sequential Execution</h3><pre>return Crew(<br>    agents=self.agents,<br>    tasks=self.tasks,<br>    process=Process.sequential,<br>    verbose=False<br>)</pre><p><strong>Only one browsing agent</strong></p><pre>@agent<br>def scout(self) -&gt; Agent:<br>    return Agent(<br>        config=self.agents_config[&quot;scout&quot;],<br>        tools=[SerperDevTool()],<br>        verbose=False<br>    )</pre><p><strong>Before vs After: Token Economics</strong></p><pre>| Aspect          | Before       | After        |<br>| --------------- | ------------ | ------------ |<br>| Execution mode  | Hierarchical | Sequential   |<br>| Browsing agents | 2–3          | 1            |<br>| Early output    | Long prose   | Compact JSON |<br>| Context growth  | Exponential  | Bounded      |<br>| Avg tokens/run  | 25k–70k+     | 3.6k–10k     |<br>| Predictability  | Low          | High         |</pre><h3><strong>Original vs Fixed: The Architectural Diff That Changed Everything</strong></h3><blockquote><strong><em>Original:</em></strong><em><br> Everyone fetches. Everyone rereads. Everyone expands.</em></blockquote><blockquote><strong><em>Fixed:</em></strong><em><br> One fetches. Everyone else transforms. One decides.</em></blockquote><p>That single shift is what took the system from a token-burning experiment to something production-viable.</p><h3>Key Lessons (Hard-Earned)</h3><ul><li>Architecture beats prompts</li><li>Schemas are cost controls</li><li>Hierarchical mode is powerful …and dangerous</li><li>Fetch once, reason last</li><li>Multi-agent systems amplify mistakes</li></ul><h3>Conclusion</h3><h3><strong>Abstractions Don’t Remove Cost — They Redistribute It</strong></h3><p>CrewAI did exactly what it promised: it made multi-agent systems easy to build.<br> What it didn’t do and realistically <em>can’t</em> do is make them cheap by default.</p><p>The core lesson from this experience wasn’t about a bug, a bad prompt, or even a flawed framework. It was about understanding that <strong>abstractions don’t remove cost; they redistribute it</strong>. CrewAI shifts complexity away from wiring model calls and toward orchestration, context flow, and implicit behavior. If you don’t reason about those layers explicitly, you end up paying for work the system is doing quietly on your behalf.</p><p>In contrast, working directly with the OpenAI SDK forces you to confront cost earlier. Every request is explicit. Every piece of context is consciously included. That friction can feel slower at first, but it also makes the economics of your system harder to ignore.</p><p>What ultimately fixed this system wasn’t choosing one tool over another. It was treating the multi-agent pipeline like any other production system: identifying the ingestion layer, enforcing strict interfaces, compressing early, and reserving expensive resources for the moments where judgment actually matters.</p><p>Once that shift happened, the system became predictable. Costs stabilized. Reasoning quality improved. And the architecture stopped fighting itself.</p><p>The takeaway isn’t “don’t use CrewAI.”<br> It’s <strong>use it deliberately</strong>.</p><p>If you’re building multi-agent systems, assume that every agent, every tool call, and every orchestration decision has a cost even when it’s hidden behind a friendly abstraction. Design for bounded context. Treat schemas as cost controls. And remember that the most expensive mistake isn’t calling a powerful model, it’s letting the same information be read over and over again without realizing it.</p><p>Or, more simply:</p><blockquote><em>Expand late.<br> Compress early.<br> Fetch once.<br> Reason last.</em></blockquote><p>That principle matters far more than which framework you choose.</p><p>#AIEngineering #MultiAgentSystems #CrewAI #LLM #SoftwareArchitecture #TokenEconomics #ProductionAI</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=efb81fed5139" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How I Broke (and Fixed) My First Modular ELT Pipeline Using Docker, dbt, and Superset]]></title>
            <link>https://medium.com/@adekunle.adebayo001/how-i-broke-and-fixed-my-first-modular-elt-pipeline-using-docker-dbt-and-superset-dedb34eb7ac5?source=rss-03e8ec8a05b6------2</link>
            <guid isPermaLink="false">https://medium.com/p/dedb34eb7ac5</guid>
            <dc:creator><![CDATA[Saheed Adebayo]]></dc:creator>
            <pubDate>Sun, 12 Oct 2025 01:52:18 GMT</pubDate>
            <atom:updated>2025-10-12T01:52:18.938Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>A hands-on story of debugging, containers, and growth from the Core Data Engineer Bootcamp.</blockquote><h3>The Beginning: “I just want to build something real.”</h3><p>When I started the <strong>CDE (Core Data Engineer) Bootcamp</strong>, I didn’t want to just follow tutorials — <br> I wanted to <strong>build</strong> a real, modular ELT pipeline.</p><p>One that could:</p><ul><li>Load CSV data automatically into a database</li><li>Transform it with <strong>dbt</strong></li><li>Visualise insights with <strong>Apache Superset</strong></li><li>And run fully in <strong>Docker containers</strong></li></ul><p>It sounded simple… until it wasn’t.<br> I broke things more times than I can count — but each time, I learned something new.</p><h3>The Architecture</h3><p>The pipeline followed this pattern:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OwJO31Ra4tdJwAPu1FvEqA.png" /></figure><blockquote><strong><em>Python (Extract &amp; Load)</em></strong><em> → </em><strong><em>Postgres (Store)</em></strong><em> → </em><strong><em>dbt (Transform)</em></strong><em> → </em><strong><em>Superset (Visualize)</em></strong></blockquote><p>Each component ran in its own Docker container, connected via a shared etl_network.</p><p>Every container had one clear job:</p><ul><li>Python loads the data</li><li>dbt transforms it</li><li>Both containers exit automatically once they’re done</li></ul><p>That design made my setup lightweight, modular, and easy to maintain.</p><h3>Extract &amp; Load with Python</h3><p>The <strong>Python container</strong> handled the extract and load step.<br> It read the sales CSV and pushed it into Postgres using pandas and SQLAlchemy.</p><pre>import os<br>import pandas as pd<br>from sqlalchemy import create_engine<br>from dotenv import load_dotenv<br><br>load_dotenv()<br><br>DB_HOST = os.getenv(&quot;DB_HOST&quot;)<br>DB_PORT = os.getenv(&quot;DB_PORT&quot;)<br>DB_NAME = os.getenv(&quot;DB_NAME&quot;)<br>DB_USER = os.getenv(&quot;DB_USER&quot;)<br>DB_PASS = os.getenv(&quot;DB_PASS&quot;)<br><br>engine = create_engine(<br>    f&quot;postgresql+psycopg2://{DB_USER}:{DB_PASS}@{DB_HOST}:{DB_PORT}/{DB_NAME}&quot;<br>)<br><br>def load_csv_to_postgres(csv_path):<br>    try:<br>        try:<br>            df = pd.read_csv(csv_path, encoding=&#39;utf-8&#39;)<br>        except UnicodeDecodeError:<br>            print(&quot;⚠️ UTF-8 failed, retrying with ISO-8859-1...&quot;)<br>            df = pd.read_csv(csv_path, encoding=&#39;ISO-8859-1&#39;)<br><br>        df.to_sql(&#39;sales_data&#39;, engine, if_exists=&#39;replace&#39;, index=False)<br>        print(f&quot;✅ Loaded {len(df)} rows into &#39;sales_data&#39;.&quot;)<br>    except Exception as e:<br>        print(f&quot;❌ Error loading data: {e}&quot;)<br><br>if __name__ == &quot;__main__&quot;:<br>    load_csv_to_postgres(&quot;/data/sales_data_sample.csv&quot;)</pre><p>Once the job finished, the Python container <strong>stopped automatically</strong>,<br> just like a real batch process in production.</p><p>I ran into an encoding issue here, after a brief research, I added a fallback encoding.</p><h3>PostgreSQL + pgAdmin — The Foundation</h3><p>My first big issue was container conflicts.</p><p>Docker refused to start, throwing:</p><blockquote>“Container name already in use.”</blockquote><p>Turns out I was hardcoding container_name across different compose files, so Docker tried to reuse names.</p><p><strong>Fix:</strong> I removed container_name and let Docker assign names dynamically.<br> Here’s a simplified version of my Postgres setup:</p><pre>version: &#39;3.9&#39;<br>services:<br>  postgres:<br>    image: postgres:15<br>    env_file: .env<br>    environment:<br>      POSTGRES_USER: ${DB_USER}<br>      POSTGRES_PASSWORD: ${DB_PASS}<br>      POSTGRES_DB: ${DB_NAME}<br>    ports:<br>      - &quot;${DB_PORT}:5432&quot;<br>    volumes:<br>      - postgres_data:/var/lib/postgresql/data<br>    networks:<br>      - elt_network<br><br>  pgadmin:<br>    image: dpage/pgadmin4<br>    env_file: .env<br>    environment:<br>      PGADMIN_DEFAULT_EMAIL: ${PGADMIN_DEFAULT_EMAIL}<br>      PGADMIN_DEFAULT_PASSWORD: ${PGADMIN_DEFAULT_PASSWORD}<br>    ports:<br>      - &quot;${PGADMIN_PORT}:80&quot;<br>    depends_on:<br>      - postgres<br>    networks:<br>      - elt_network<br><br>volumes:<br>  postgres_data:<br><br>networks:<br>  elt_network:<br>    external: true</pre><h3>dbt: Transform Layer</h3><p>dbt handled all my transformations in SQL models.</p><pre>-- models/staging_sales.sql<br>with raw as (<br>  select * from {{ source(&#39;staging&#39;, &#39;sales_data&#39;) }}<br>)<br>select<br>  ordernumber,<br>  customername,<br>  cast(totalamount as numeric) as total_amount,<br>  orderdate::date as order_date<br>from raw<br>where totalamount is not null</pre><p>Once dbt finished building, it <strong>automatically exited</strong>,<br> which was perfect for running transformations as part of a scheduled job.</p><p>I faced a funny challenge here. Apparently ending the SQL statement with a semi colon throws an error. Took me a while to figure the issue. But once I did, dbt ran without issues.</p><h3>Superset — and the Missing pip Saga 😅</h3><p>This one caught me off guard.</p><p>When I launched <strong>Superset</strong>, I needed to install a Postgres connector to read my transformed data.<br> But when I tried, I got this:</p><blockquote><em>pip: command not found</em></blockquote><p>After some digging, I discovered that the official Superset image doesn’t include pip.<br> So I built my own version by extending the base image.</p><pre>FROM apache/superset:latest<br><br>USER root<br>RUN apt-get update &amp;&amp; apt-get install -y python3-pip<br>RUN pip install psycopg2-binary<br>USER superset</pre><p>That tiny tweak fixed everything — I could now connect Superset to my dbt models and visualize results.</p><p>Here’s the sample dashboard I built:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XRhIvrr7_E9EeYjUBx_iPw.jpeg" /></figure><h3>Automating the Whole Thing</h3><p>I didn’t want to run everything manually, so I wrote a small <strong>Bash orchestrator</strong> to control all services. I ran everything manually, tested it, broke it, fixed it. When I was sure my code works as intended, I used the bash script below to start, restart, stop and check status of my containers or specific containers.</p><pre>#!/bin/bash<br>set -e<br><br># Define services and their Compose file paths<br>declare -A SERVICES<br>SERVICES=(<br>  [&quot;postgres&quot;]=&quot;postgres_pgAdmin/docker-compose.yml&quot;<br>  [&quot;elt&quot;]=&quot;elt_app/docker-compose.yml&quot;<br>  [&quot;dbt&quot;]=&quot;dbt/docker-compose.yml&quot;<br>  [&quot;superset&quot;]=&quot;superset/docker-compose.yml&quot;<br>)<br><br># Services in start order (dependencies first)<br>START_ORDER=(&quot;postgres&quot; &quot;elt&quot; &quot;dbt&quot; &quot;superset&quot;)<br># Services in stop order (dependents first)<br>STOP_ORDER=(&quot;superset&quot; &quot;dbt&quot; &quot;elt&quot; &quot;postgres&quot;)<br><br># Shared Docker network<br>NETWORK=&quot;etl_network&quot;<br><br># Create network if it doesn&#39;t exist<br>if ! docker network ls | grep -q &quot;$NETWORK&quot;; then<br>  echo &quot;Creating Docker network: $NETWORK&quot;<br>  docker network create &quot;$NETWORK&quot;<br>fi<br><br># Wait for Postgres to be ready<br>wait_for_postgres() {<br>  echo &quot;Waiting for Postgres to be healthy...&quot;<br>  until docker exec -it postgres pg_isready -U postgres &gt; /dev/null 2&gt;&amp;1; do<br>    sleep 2<br>  done<br>  echo &quot;Postgres is ready!&quot;<br>}<br><br># Filter services based on arguments (if provided)<br>filter_services() {<br>  local list=(&quot;$@&quot;)<br>  local filtered=()<br>  for svc in &quot;${list[@]}&quot;; do<br>    if [[ -n &quot;${SERVICES[$svc]}&quot; ]]; then<br>      filtered+=(&quot;$svc&quot;)<br>    else<br>      echo &quot;Warning: Unknown service &#39;$svc&#39;, skipping.&quot;<br>    fi<br>  done<br>  echo &quot;${filtered[@]}&quot;<br>}<br><br># Start services<br>start_services() {<br>  local target_services=(&quot;$@&quot;)<br>  if [ &quot;${#target_services[@]}&quot; -eq 0 ]; then<br>    target_services=(&quot;${START_ORDER[@]}&quot;)<br>  fi<br><br>  echo &quot;Starting services: ${target_services[*]}&quot;<br>  for service in &quot;${START_ORDER[@]}&quot;; do<br>    [[ ! &quot; ${target_services[*]} &quot; =~ &quot; ${service} &quot; ]] &amp;&amp; continue<br><br>    compose_file=${SERVICES[$service]}<br>    echo &quot;Starting $service from $compose_file...&quot;<br>    docker-compose -p etl_pipeline -f &quot;$compose_file&quot; up -d --no-recreate<br><br>    # Connect to shared network<br>    container_id=$(docker-compose -f &quot;$compose_file&quot; ps -q)<br>    docker network connect &quot;$NETWORK&quot; &quot;$container_id&quot; || true<br><br>    # Wait for Postgres if needed<br>    if [[ &quot;$service&quot; == &quot;postgres&quot; ]]; then<br>      wait_for_postgres<br>    elif [[ &quot;$service&quot; =~ ^(elt|dbt|superset)$ ]]; then<br>      wait_for_postgres<br>    fi<br><br>    echo &quot;$service started. Showing logs...&quot;<br>    docker-compose -f &quot;$compose_file&quot; logs -f --tail=20 &amp;<br>  done<br>  echo &quot;Requested services started!&quot;<br>}<br><br># Stop services<br>stop_services() {<br>  local target_services=(&quot;$@&quot;)<br>  if [ &quot;${#target_services[@]}&quot; -eq 0 ]; then<br>    target_services=(&quot;${STOP_ORDER[@]}&quot;)<br>  fi<br><br>  echo &quot;Stopping services: ${target_services[*]}&quot;<br>  for service in &quot;${STOP_ORDER[@]}&quot;; do<br>    [[ ! &quot; ${target_services[*]} &quot; =~ &quot; ${service} &quot; ]] &amp;&amp; continue<br><br>    compose_file=${SERVICES[$service]}<br>    echo &quot;Stopping $service from $compose_file...&quot;<br>    docker-compose -p etl_pipeline -f &quot;$compose_file&quot; down<br>    echo &quot;$service stopped.&quot;<br>  done<br>  echo &quot;Requested services stopped!&quot;<br>}<br><br># Check status<br>status_services() {<br>  local target_services=(&quot;$@&quot;)<br>  if [ &quot;${#target_services[@]}&quot; -eq 0 ]; then<br>    target_services=(&quot;${START_ORDER[@]}&quot;)<br>  fi<br><br>  echo &quot;Checking status for: ${target_services[*]}&quot;<br>  for service in &quot;${target_services[@]}&quot;; do<br>    compose_file=${SERVICES[$service]}<br>    container_id=$(docker-compose -f &quot;$compose_file&quot; ps -q)<br>    if [ -n &quot;$container_id&quot; ] &amp;&amp; [ &quot;$(docker inspect -f &#39;{{.State.Running}}&#39; $container_id)&quot; == &quot;true&quot; ]; then<br>      echo &quot;$service: running&quot;<br>    else<br>      echo &quot;$service: stopped&quot;<br>    fi<br>  done<br>}<br><br># Main<br>COMMAND=$1<br>shift || true<br>ARGS=(&quot;$@&quot;)<br>ARGS=($(filter_services &quot;${ARGS[@]}&quot;))<br><br>case &quot;$COMMAND&quot; in<br>  start)<br>    start_services &quot;${ARGS[@]}&quot;<br>    ;;<br>  stop)<br>    stop_services &quot;${ARGS[@]}&quot;<br>    ;;<br>  restart)<br>    stop_services &quot;${ARGS[@]}&quot;<br>    start_services &quot;${ARGS[@]}&quot;<br>    ;;<br>  status)<br>    status_services &quot;${ARGS[@]}&quot;<br>    ;;<br>  logs)<br>    if [ &quot;${#ARGS[@]}&quot; -eq 0 ]; then<br>      ARGS=(&quot;${START_ORDER[@]}&quot;)<br>    fi<br>    for service in &quot;${ARGS[@]}&quot;; do<br>      compose_file=${SERVICES[$service]}<br>      echo &quot;Tailing logs for $service...&quot;<br>      docker-compose -f &quot;$compose_file&quot; logs -f<br>    done<br>    ;;<br>  *)<br>    echo &quot;Usage: $0 {start|stop|restart|status|logs} [services...]&quot;<br>    echo &quot;Example: $0 start postgres elt&quot;<br>    exit 1<br>    ;;<br>esac</pre><pre>bash services.sh start<br>bash services.sh stop<br>bash services.sh status</pre><p>But I wanted more — I wanted it to <strong>run automatically every midnight</strong>.<br> So I added a simple <strong>cron job</strong>:</p><pre>0 0 * * * /home/user/projects/elt_pipeline/run_pipeline.sh &gt;&gt; /home/user/logs/elt_cron.log 2&gt;&amp;1</pre><p>Now, every day at midnight, the entire process runs:</p><ul><li>Postgres spins up</li><li>Python loads data</li><li>dbt transforms it</li><li>Superset refreshes dashboards</li></ul><p>Then, when everything’s done — the Python and dbt containers exit gracefully. No extra overhead. No wasted compute.</p><h3>Tradeoffs and Learnings</h3><ul><li>Using <strong>short-lived containers</strong> made the pipeline lightweight.</li><li><strong>Separate compose files</strong> improved modularity but added orchestration overhead.</li><li><strong>Custom Superset image</strong> added flexibility at the cost of extra build time.</li><li>And <strong>cron</strong> turned it from a project into a living, breathing workflow.</li></ul><p>The biggest lesson?</p><blockquote><em>“Data engineering isn’t just writing codes — it’s making systems run reliably and repeatably.”</em></blockquote><h3>Final Thoughts</h3><p>I won’t pretend it was smooth.<br> I fought with Docker conflicts, encoding errors, rebuilds, and missing dependencies.<br> But every failure taught me something valuable — about debugging, automation, and resilience.</p><p>When I finally saw my Superset dashboard light up with live, transformed data, it all felt worth it.</p><p>The <strong>CDE Bootcamp</strong> gave me the foundation.<br> <strong>Building (and breaking)</strong> this pipeline gave me confidence.<br> And fixing it — that gave me experience.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dedb34eb7ac5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ How a Nigerian Politician Accidentally Explained Docker]]></title>
            <link>https://medium.com/@adekunle.adebayo001/how-a-nigerian-politician-accidentally-explained-docker-3e2182769d3d?source=rss-03e8ec8a05b6------2</link>
            <guid isPermaLink="false">https://medium.com/p/3e2182769d3d</guid>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Saheed Adebayo]]></dc:creator>
            <pubDate>Fri, 19 Sep 2025 17:14:10 GMT</pubDate>
            <atom:updated>2025-09-19T17:14:10.626Z</atom:updated>
            <content:encoded><![CDATA[<p>🍲 How a Nigerian Politician Accidentally Explained Docker</p><p>The Story:</p><p>Once upon a time in Nigeria, there was a politician who loved Jollof rice. One day after eating in Abuja, he declared:</p><p>“From today, anywhere I go — Lagos, Kano, Sokoto, even London — my Jollof must taste exactly the same. No excuses!”</p><p>But he had a problem:</p><p>Lagos Jollof was smoky 🌶️</p><p>Kano Jollof lacked pepper</p><p>Sokoto Jollof had sand 😅</p><p>London Jollof came with basmati (abomination!)</p><p>The oga vexed: “Why is my Jollof inconsistent?!”<br>One smart adviser explained:</p><p>“Oga, the issue is not the rice, it is the environment. If you want consistency, we must package your entire kitchen — pot, stove, pepper, spices, even water — into a magic cooler and ship it everywhere.”</p><p>The oga smiled: “Ahh! Consistency at last.”<br>That magic cooler is Docker.</p><p>🔹 So, What is Docker Really?</p><p>Docker is an open platform that lets developers:</p><p>Build applications with all their dependencies.</p><p>Ship them anywhere (cloud, servers, laptops).</p><p>Run them consistently, regardless of the underlying environment.</p><p>Instead of saying “it works on my laptop but not on production”, Docker ensures it works the same everywhere.</p><p>🔹 Breaking It Down with the Jollof Metaphor</p><p>1. Dockerfile = Recipe Card</p><p>Just like Hilda Baci or our politician’s chef writes a step-by-step recipe (“boil water, add rice, add pepper”), developers write a Dockerfile.</p><p>It specifies everything needed: base OS, libraries, tools, configurations.</p><p>It’s the blueprint for the kitchen.</p><p>2. Docker Image = Sealed Jollof Batch</p><p>From the recipe, Docker creates a Docker Image.</p><p>Think of it as one frozen, sealed batch of Jollof.</p><p>It contains everything needed to “reheat” into a working environment.</p><p>Images are read-only and reusable.</p><p>3. Docker Container = Portable Kitchen (Magic Cooler)</p><p>When you run an image, it becomes a Container.</p><p>This is the magic cooler where the Jollof is served hot.</p><p>Containers are isolated but lightweight — they share the same “floor” (host operating system) but each has its own pot, stove, and spices.</p><p>You can run 10 containers side by side without them mixing flavours.</p><p>4. Docker Hub = Warehouse</p><p>The politician needed a place to store and distribute coolers. That’s Docker Hub.</p><p>It’s a public or private registry where images live.</p><p>Anyone can pull down an image and run a container from it.</p><p>🔹 Docker vs Virtual Machines</p><p>Here’s where many people get confused. Let’s stick with the Jollof metaphor:</p><p>Virtual Machines (VMs):<br>Imagine building a whole new restaurant for every meal.</p><p>New walls, fridge, stove, generator.</p><p>Very reliable, but heavy and slow.</p><p>Docker Containers:<br>Imagine carrying just your cooler with the kitchen setup inside.</p><p>Reuses the building floor (host OS).</p><p>Lightweight, fast, and efficient.</p><p>That’s why containers have become the preferred way to deploy apps at scale.</p><p>🔹 Why This Matters</p><p>For developers:</p><p>You can package an app once, and it runs anywhere (local machine, staging, production, cloud).</p><p>For teams:</p><p>Faster onboarding (just pull the Docker image, no need to “set up environment for 3 days”).</p><p>Consistency in testing and deployment.</p><p>For businesses:</p><p>Reduced infrastructure costs compared to heavy VMs.</p><p>Speed and reliability in deploying services globally.</p><p>💡 Final Takeaway</p><p>Docker is simply how our politician ensured his Jollof tasted the same everywhere.<br>👉 Dockerfile = Recipe<br>👉 Image = Frozen Jollof batch<br>👉 Container = Magic cooler kitchen<br>👉 Docker Hub = Warehouse<br>And the real power of Docker is this:<br>Build → Ship → Run, consistently, anywhere.<br>So next time someone asks you what Docker is, just smile and say:</p><p>“It’s how a Nigerian politician made sure his Jollof rice never lost its taste — from Abuja to London.”</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3e2182769d3d" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>