<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by MHAMDI Chiheb on Medium]]></title>
        <description><![CDATA[Stories by MHAMDI Chiheb on Medium]]></description>
        <link>https://medium.com/@chiheb.mhamdi?source=rss-148acfe082b4------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*ikMas2EAu2s4DmqOas_xRg.jpeg</url>
            <title>Stories by MHAMDI Chiheb on Medium</title>
            <link>https://medium.com/@chiheb.mhamdi?source=rss-148acfe082b4------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 03:08:16 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@chiheb.mhamdi/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[From ELIZA to MCP complete NLP AI stack]]></title>
            <link>https://medium.com/@chiheb.mhamdi/from-eliza-to-mcp-complete-nlp-ai-stack-fb76be568712?source=rss-148acfe082b4------2</link>
            <guid isPermaLink="false">https://medium.com/p/fb76be568712</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[MHAMDI Chiheb]]></dc:creator>
            <pubDate>Thu, 23 Jul 2026 14:20:29 GMT</pubDate>
            <atom:updated>2026-07-23T14:20:29.251Z</atom:updated>
            <content:encoded><![CDATA[<ul><li><em>Attention Changed Everything Then We Gave Models Memory, Tools, and a Protocol</em></li><li><em>The Long Road to Talking Machines: History of NLP → BERT vs GPT → RAG → Agents → MCP</em></li><li><em>How Language Models Grew Up: Tokens, Transformers, Vector Search, and Production AI</em></li><li>A <strong>history of NLP</strong> that actually explains <em>why</em> each era failed or won — not a name-drop timeline</li><li><strong>Attention</strong> and <strong>transformers</strong> in human language</li><li>A clear <strong>BERT vs GPT</strong> contrast: two philosophies that still shape every product choice</li><li><strong>AI fundamentals</strong> you need before frameworks: LLMs, tokens, embeddings, context windows</li><li><strong>Prompt engineering</strong> that goes beyond tips ,zero-shot, few-shot, chain-of-thought, and when each helps</li><li><strong>Vector databases</strong> and <strong>semantic search</strong> as the memory layer models never had</li><li><strong>RAG</strong> as the pattern that grounds generation in <em>your</em> documents</li><li><strong>LangChain</strong> as pre-built components that remove glue-code pain</li><li><strong>LangGraph</strong> as multi-step workflows and agents with real control flow</li><li><strong>MCP (Model Context Protocol)</strong> as the standard way to connect AI to external tools and data</li></ul><p><strong>One line to remember:</strong> Modern AI is not “a smarter autocomplete.” It is a <strong>stack : </strong>model + context + retrieval + tools + orchestration ,built on decades of trying to make machines understand language.</p><p><strong>Who this is for:</strong> engineers, data people, and builders who can use ChatGPT but want the <em>full story , </em>enough history to respect the hard parts, enough practice to ship.</p><blockquote>If you only learn tools, you will chase frameworks forever.<br>If you only learn history, you will sound smart and ship nothing.</blockquote><p>This article does both on purpose.</p><p>Every piece of the modern stack exists because an older approach <strong>hit a wall</strong>:</p><p><strong>Everyday analogy:</strong> Learning only ChatGPT tips is like learning to drive <strong>one car</strong>. Learning this stack is learning <strong>roads, traffic lights, maps, and how cars are built</strong> so the next model release does not erase your skill.</p><p>In 1950, Alan Turing asked a blunt question: can a machine <strong>imitate</strong> human conversation well enough that a judge cannot tell? The <strong>Turing Test</strong> was not a product roadmap — it was a provocation.</p><p>Early machine translation projects (Georgetown-IBM, 1954) demoed Russian→English on a tiny vocabulary. The press celebrated. Researchers quietly discovered the real problem: <strong>language is ambiguous</strong>, and meaning lives in <strong>context</strong>.</p><p><strong>Everyday analogy:</strong> A tourist phrasebook can order coffee. It cannot negotiate a lease.</p><p>Approach: write grammars, dictionaries, and if-then logic.</p><p>Famous milestone: <strong>ELIZA</strong> (1966, Joseph Weizenbaum) — a pattern-matching “therapist” that reflected your words back. People projected intelligence onto it. Weizenbaum was unsettled: humans trust fluent text too easily. That warning still applies to LLMs.</p><p>Also in this era: <strong>SHRDLU</strong> (Winograd) — understanding language in a tiny “blocks world.” It worked <em>inside</em> a toy universe. Outside, language exploded in complexity.</p><p><strong>Why it stalled:</strong> Language is not a closed rulebook. Idioms, sarcasm, typos, and culture break brittle systems. Maintaining rules became a full-time archaeology project.</p><p>A company builds a FAQ bot with 400 keyword rules.<br> Customer: “My payment didn’t go through but I got charged.”<br> Bot matches “payment” → sends billing FAQ.<br> Misses the real issue: <strong>double charge</strong>.</p><p>Rules are great until the world says something you did not anticipate — which is most of support.</p><p>Approach: learn probabilities from data. <strong>n-gram</strong> language models. Hidden Markov Models for speech and tagging. Early statistical machine translation.</p><p>Key mindset shift:</p><p><strong>Bag-of-words</strong> and <strong>TF-IDF</strong> powered search and classification. They treated documents as bags of word counts — order mostly ignored.</p><p><strong>Win:</strong> Spam filters, search ranking, part-of-speech tagging improved dramatically.<br><strong>Limit:</strong> “not good” and “good” can look similar as bags of words. Meaning needs <strong>order</strong> and <strong>relationship</strong>.</p><p><strong>Word2Vec</strong> (Mikolov et al., 2013) and later <strong>GloVe</strong> taught a shocking lesson: meaning can live in <strong>geometry</strong>.</p><ul><li>“king − man + woman ≈ queen”</li><li>Similar words cluster in vector space</li></ul><p>Then came <strong>sequence models</strong>: RNNs, LSTMs, GRUs — reading text left to right, carrying a hidden state like a short-term memory.</p><p><strong>Win:</strong> Sentiment, translation, and tagging jumped again.<br><strong>Limit:</strong> Long documents still suffered — information diluted across many steps. Training long RNNs was slow and hard to parallelize. The web wanted <strong>scale</strong>.</p><p><strong>Everyday analogy:</strong> RNN reading is like whispering a story person-to-person down a long line. By the end, the beginning is mush.</p><p>In neural machine translation, <strong>Bahdanau attention</strong> (2014) and related work let the decoder <strong>look back</strong> at relevant source words instead of trusting one compressed vector.</p><p>Then in 2017, Vaswani et al. published <strong>“Attention Is All You Need.”</strong> They removed recurrence as the backbone and made <strong>self-attention</strong> the engine. That paper is the hinge between “NLP research” and “everyone has a chatbot.”</p><ul><li><strong>BERT</strong> (2018): bidirectional encoder, masked language modeling — understanding king</li><li><strong>GPT</strong> lineage (2018→): generative pretraining — writing king</li><li><strong>T5, BART</strong>, and friends: more ways to cast every NLP task as text-in/text-out</li></ul><p>Transfer learning became the default: pretrain on massive text, then fine-tune (or later, just prompt).</p><p>GPT-3 showed that <strong>scale + prompting</strong> could do tasks with few or no labeled examples. Chat interfaces made the capability cultural. Then the industry learned the next walls:</p><ul><li>Models invent plausible falsehoods → <strong>RAG</strong></li><li>Apps need reusable plumbing → <strong>LangChain</strong></li><li>Real work needs loops and branches → <strong>agents / LangGraph</strong></li><li>Every tool integration is bespoke → <strong>MCP</strong></li></ul><p><strong>History in one breath:</strong><br> Rules → statistics → vectors → recurrence → <strong>attention/transformers</strong> → pretrained LLMs → grounded &amp; tool-using systems.</p><p>Old sequence models often crushed an entire sentence into one summary vector, then tried to generate from that. Long sentences lost details.</p><p><strong>Attention</strong> says: when producing or understanding a word, <strong>look at the other words and decide how much each matters</strong>.</p><p><strong>Everyday analogy:</strong> In a meeting, you do not give equal weight to every sentence. When someone says “the outage,” your brain flashes to “database,” “on-call,” “customer impact” — not the coffee order from minute one.</p><p>For each token, the model computes relationships to other tokens using three learned roles:</p><p>Roughly:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1016/1*g0v-sqTv07ly_ZuKgpmUEA.png" /></figure><p>You do not need to memorize the formula. Remember the story:</p><ol><li>Compare questions to labels (Q vs K)</li><li>Turn similarity into weights (softmax)</li><li>Mix information (weighted V)</li></ol><p><strong>Multi-head attention</strong> = several attention “specialists” in parallel — one head may track syntax, another coreference, another topic. The model learns the specialties from data; we name them after the fact.</p><ul><li><strong>Parallelism:</strong> self-attention over a sequence can be computed more in parallel than step-by-step recurrence → GPUs happy → scale possible</li><li><strong>Direct paths:</strong> distant words can influence each other in one step instead of through a long memory chain</li><li><strong>Interpretability (partial):</strong> attention weights are not perfect explanations, but they gave researchers a new lens</li></ul><p><strong>Pitfall:</strong> Attention is not magic understanding. It is a flexible routing mechanism trained to predict text. It can still confidently attend to the wrong things.</p><p>A <strong>transformer</strong> block typically stacks:</p><ol><li><strong>Multi-head self-attention</strong> — tokens talk to each other</li><li><strong>Feed-forward network</strong> — each token gets a nonlinear transform</li><li><strong>Residual connections + layer norm</strong> — training stability at depth</li></ol><p>Stack many blocks → deep model.</p><p>Original paper also used an <strong>encoder-decoder</strong> setup for translation:</p><ul><li><strong>Encoder:</strong> reads the source sentence (bidirectional context)</li><li><strong>Decoder:</strong> generates the target sentence (left-to-right, attending to encoder outputs)</li></ul><p>Attention alone does not know order. Transformers add <strong>positional encodings</strong> (sinusoidal in the original; learned or rotary variants later) so “dog bites man” ≠ “man bites dog.”</p><p>People say “transformer” to mean:</p><ul><li>the <strong>architecture family</strong>, and casually</li><li>the <strong>LLM</strong> built from it</li></ul><p>Strictly: transformers are the engine; GPT/BERT/Claude/Gemini-style models are vehicles built with that engine (plus enormous data, training recipes, and alignment).</p><p>A team tries to train a large LSTM for document classification. Weeks of tuning. Slow.<br> They switch to a transformer encoder. Same data, better accuracy, shorter wall-clock because training parallelizes.</p><p>The paper title was cheeky. The industry takeaway was practical: <strong>attention scaled</strong>.</p><p>This is not a fan war. It is an <strong>architecture and objective</strong> choice that still decides product design.</p><p>Mask 15% of tokens, force the model to use <strong>left and right</strong> context to fill gaps. That produces deep bidirectional representations — excellent for search ranking features, classification, entity extraction.</p><p><strong>Everyday analogy:</strong> A student who reads the <strong>whole paragraph</strong> before answering a fill-in-the-blank.</p><p>Predict the next token forever. To say word 50, you may only use words 1–49. That constraint creates a machine that can <strong>continue</strong> any prompt — the foundation of chat.</p><p><strong>Everyday analogy:</strong> A storyteller who must invent the next sentence without peeking at the ending.</p><p>Chat needs generation. Generation needs a causal decoder (or encoder-decoder used generatively). BERT can power pieces <em>inside</em> a system (retrievers, classifiers), but the face of “AI” became the generative model.</p><ul><li>Encoder models still matter in <strong>retrieval</strong>, <strong>ranking</strong>, and <strong>classification</strong> at lower cost</li><li>Decoder LLMs can also embed and classify (not always optimally)</li><li>Encoder-decoder models (T5, BART) remain strong for some seq2seq tasks</li><li>“BERT vs GPT” is the <strong>teaching contrast</strong>; production is a <strong>toolbox</strong></li></ul><p>Product asks for “AI search.” Team fine-tunes GPT to classify relevant docs expensive, slow, okay quality.<br> Better split: <strong>embedding model / encoder retriever</strong> finds candidates; <strong>generative LLM</strong> writes the answer with citations.</p><p>Two philosophies, one pipeline. That is mature design.</p><p>An LLM is trained to model:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/468/1*OBRy-YMDZOGrokjVrqiT2A.png" /></figure><p>In English: given the tokens so far, how probable is each next token?</p><p>Chat, coding, and “reasoning” are what you get when that prediction skill is scaled, steered with instructions, and wrapped in products.</p><p><strong>Critical honesty:</strong> Plausible ≠ true. The model optimizes for <strong>likely text</strong>, not verified world state — unless you ground it (RAG, tools, checks).</p><p>Models do not read characters the way humans do. A <strong>tokenizer</strong> splits text into tokens.</p><ul><li>&quot;unbelievable&quot; might be one token or several pieces (un, believ, able) depending on the tokenizer</li><li>Numbers, code, and other languages can tokenize inefficiently → <strong>cost and context</strong> surprises</li></ul><p><strong>Why you care in production:</strong></p><ul><li>Billing is often <strong>per token</strong></li><li>Context limits are in <strong>tokens</strong>, not words</li><li>Prompt length is a budget</li></ul><p><strong>Everyday analogy:</strong> Tokens are shipping boxes. Same furniture, different packing. More boxes → more truck space (context) and higher shipping cost.</p><p>An <strong>embedding</strong> maps text to a vector (a list of floats). Training pulls related meanings closer in that space.</p><p>Uses:</p><ul><li>semantic search</li><li>clustering support tickets</li><li>deduplicating near-copy documents</li><li>RAG retrieval</li></ul><p><strong>Everyday analogy:</strong> A library that shelves books by <strong>topic vibe</strong>, not only by title spelling. “How do I reset my password?” sits near “I forgot my login credentials.”</p><p>The <strong>context window</strong> is how many tokens the model can consider in one go (prompt + generation, with product-specific accounting).</p><p><strong>Long context does not replace good retrieval.</strong> A messy 200k-token dump can lose the one sentence that mattered. Attention is finite in practice; quality of context still wins.</p><p>Analyst pastes an entire policy PDF into chat: “Summarize obligations for EU customers.”<br> Model misses a clause on page 47 — buried in noise.<br> Better: chunk → embed → retrieve top sections → generate with citations.</p><p>Context is a desk. Retrieval is a librarian.</p><p>Prompt engineering is not spell-casting. It is <strong>interface design</strong> for a probabilistic next-token engine.</p><p>Give the task with <strong>no examples</strong>.</p><blockquote><em>“Classify this ticket as billing, technical, or account. Reply with one label only.”</em></blockquote><p><strong>Works when:</strong> the task is common in training data; instructions are crisp.<br><strong>Fails when:</strong> your labels are company-specific or edge cases dominate.</p><p>Show <strong>2–5 examples</strong> of input → output, then the real case.</p><p><strong>Works when:</strong> format and judgment are unusual; you need consistency.<br><strong>Fails when:</strong> examples are biased; the model copies superficial patterns; you blow the token budget.</p><p><strong>Everyday analogy:</strong> Zero-shot is “write a haiku.” Few-shot is showing three haiku your brand likes, then asking for another.</p><p>Ask the model to reason <strong>step by step</strong> before the final answer — or use styles that elicit intermediate reasoning.</p><p><strong>Works when:</strong> multi-step arithmetic, logic, policies with conditions.<br><strong>Caveats:</strong></p><ul><li>Longer, costlier outputs</li><li>“Reasoning” text can still be wrong — fluent steps ≠ proof</li><li>For some models/products, hidden reasoning and tool use change the pattern; the <em>idea</em> remains: <strong>decompose hard problems</strong></li></ul><p>Support labels: refund_partial, refund_full, chargeback_risk - not in generic internet speak.<br> Zero-shot accuracy: poor.<br> Five carefully chosen few-shot examples: suddenly usable.<br> Later: graduate from prompts to a classifier or fine-tune if volume justifies it.</p><p>Prompts are a ladder, not always the destination.</p><p>Production search often <strong>hybrid</strong>: keywords + vectors + reranking.</p><ol><li>You store embeddings + metadata (doc id, date, tenant, ACL)</li><li>At query time, embed the question</li><li>Find nearest neighbors (ANN indexes: HNSW, IVF, etc.)</li><li>Return top-k chunks for the LLM or UI</li></ol><p>Examples of the category: Pinecone, Weaviate, Qdrant, Milvus, pgvector, Chroma, cloud equivalents.</p><ul><li><strong>Chunking:</strong> too big → mixed topics; too small → missing context</li><li><strong>Metadata filters:</strong> tenant isolation, language, product version</li><li><strong>Embedding model choice:</strong> must match your language/domain</li><li><strong>Refresh pipeline:</strong> stale vectors = confident wrong answers</li><li><strong>Security:</strong> retrieval must respect permissions (or you leak)</li></ul><p><strong>Everyday analogy:</strong> A vector DB is a <strong>smart filing cabinet</strong>. RAG is calling a junior analyst who pulls three folders before answering the executive.</p><pre>User question → retrieve relevant chunks (vector / hybrid search) → stuff (or selectively add) them into the prompt → LLM generates answer grounded in those chunks → (optional) cite sources, verify, refuse if empty</pre><p>RAG attacks the core LLM weakness: <strong>parametric memory is blurry and outdated; your documents are not.</strong></p><ul><li>Internal wikis, policies, contracts</li><li>Product docs and runbooks</li><li>Customer-specific knowledge</li><li>Anything that must <strong>cite</strong> evidence</li><li>Poor docs in → poor answers out</li><li>Multi-hop questions may need <strong>agents</strong> (retrieve → read → retrieve again)</li><li>Fresh operational data may need <strong>tools/SQL</strong> more than static vectors</li></ul><p>HR chatbot invented parental leave length. Trust collapsed.<br> Team shipped RAG over the HR PDF + intranet pages with citations.<br> When retrieval returns nothing relevant, the bot says <strong>“I don’t know — ask HR.”</strong></p><p>That refusal path is senior engineering — not a softer model.</p><p>Without a framework, every demo reinvents:</p><ul><li>prompt templates</li><li>LLM client wrappers</li><li>document loaders</li><li>text splitters</li><li>vector store connectors</li><li>tool calling glue</li><li>output parsers</li></ul><p><strong>LangChain</strong> packages these as <strong>composable components</strong> so you spend time on product logic, not socket plumbing.</p><p><strong>Everyday analogy:</strong> You can forge your own nails. Or you can buy a toolbox and build the house.</p><p>Classic idea: connect steps in a <strong>chain</strong></p><p>loader → splitter → embedder → retriever → prompt → LLM → parser</p><p>Useful abstractions you will meet:</p><ul><li>It <strong>accelerates</strong> prototypes and standard patterns</li><li>It does not replace evaluation, observability, or security design</li><li>For complex control flow (loops, human approval, branching), you often graduate to <strong>LangGraph</strong> (same ecosystem, better fit)</li></ul><p><strong>Pitfall:</strong> Building a giant opaque chain you cannot debug. Prefer small, testable pieces and traced runs.</p><p>Two engineers build the same RAG demo.<br> Engineer A: 400 lines of ad hoc OpenAI + Pinecone glue.<br> Engineer B: LangChain loaders + retriever + prompt template in an afternoon — then spends the saved time on <strong>eval questions</strong> and ACL filters.</p><p>Speed is not the goal. <strong>Speed to a testable system</strong> is.</p><p>Real work is not always a straight line:</p><ul><li>retry a tool</li><li>ask a clarifying question</li><li>branch on confidence</li><li>loop until a checklist is done</li><li>wait for a human approval</li></ul><p><strong>LangGraph</strong> models your app as a <strong>graph</strong>: nodes (steps) + edges (transitions), including cycles.</p><p>An <strong>agent</strong> is typically:</p><blockquote><em>an LLM that </em><strong><em>decides</em></strong><em> the next action, uses </em><strong><em>tools</em></strong><em>, observes results, and repeats until a stop condition.</em></blockquote><p>LangGraph gives you a structured way to implement that loop without spaghetti while True.</p><p><strong>Everyday analogy:</strong> LangChain components are <strong>kitchen tools</strong>. LangGraph is the <strong>recipe with decision points</strong> — “if sauce is too thin, reduce; if guests arrive early, skip dessert prep.”</p><ul><li>Cap steps (infinite agent loops burn money)</li><li>Log every tool call</li><li>Separate <strong>read-only</strong> tools from <strong>write</strong> tools</li><li>Idempotency for actions that hit APIs</li><li>Evaluate trajectories, not only final answers</li></ul><p>Support agent can draft refunds.<br> Without graph control: model calls issue_refund twice under timeout retry → double refund.<br> With LangGraph: tool node + confirmation node + idempotency key + max two tool attempts.</p><p>Agents need <strong>seatbelts</strong>, not only brains.</p><p>Before a shared protocol, each AI app invented custom connectors:</p><ul><li>one plugin for GitHub</li><li>another for Slack</li><li>another for your warehouse</li><li>repeat for every host application</li></ul><p>That is an <strong>M × N</strong> explosion: M clients × N tools.</p><p>The <strong>Model Context Protocol (MCP)</strong> is an <strong>open standard</strong> for connecting AI applications to external <strong>tools</strong>, <strong>data sources</strong>, and prompt-like resources — often described as <strong>“USB-C for AI.”</strong></p><p>Introduced by Anthropic (2024) and adopted broadly across the ecosystem, it aims to make connectors <strong>portable</strong>: build a server once, reuse it from compatible hosts (IDEs, desktop assistants, agent runtimes).</p><p><strong>Everyday analogy:</strong> Before USB standards, every mouse had a weird plug. MCP is the shared plug shape so your “GitHub server” or “warehouse server” is not rewritten for each chat product.</p><ul><li>Expose <strong>governed</strong> enterprise tools instead of pasting secrets into prompts</li><li>Same connector usable from multiple agent products over time</li><li>Clearer security boundary: the <strong>server</strong> enforces authz; the model only sees allowed operations</li><li><strong>Authn/Authz:</strong> who can invoke run_sql?</li><li><strong>Least privilege:</strong> read tools vs write tools</li><li><strong>Audit logs:</strong> every tool call attributable</li><li><strong>Schema clarity:</strong> bad tool descriptions → wrong calls</li><li><strong>Human approval</strong> for irreversible actions</li><li><strong>Rate limits / backoff</strong> when tools hit APIs (your other article’s lessons apply)</li></ul><p>They complement; they do not replace each other.</p><p>Platform team builds one MCP server for “company search + ticket create.”<br> IDE assistant and internal chat agent both use it.<br> Before MCP thinking: two brittle plugins, two auth bugs, two audit gaps.<br> After: one server, one policy surface.</p><p>Walk the history forward into a modern request:</p><pre>User question → (optional) LangGraph workflow decides plan → Retriever hits vector DB (semantic search) → RAG prompt built with citations → LLM (transformer, GPT-style generator) answers → If tools needed, call via MCP servers / tool nodes → Graph loops until done or refusal</pre><p><strong>Junior path:</strong> jump straight to a framework tutorial.<br><strong>Senior path:</strong> know which layer owns which failure mode.</p><p>We started with machines that mirrored sentences (ELIZA), graduated to counting words, then to geometry of meaning, then to attention that let models weigh context at scale. BERT taught machines to <strong>read deeply</strong>. GPT taught them to <strong>continue endlessly</strong>. Neither alone owns the future.</p><p>The future is a <strong>system</strong>:</p><blockquote><strong><em>A transformer generates. Retrieval grounds. Prompts steer. Graphs control. Protocols connect.</em></strong></blockquote><p>Or in the lunch-table version:</p><blockquote><em>Junior you: “We use AI.”<br> Senior you: “We use a </em><strong><em>language model</em></strong><em> with a </em><strong><em>context budget</em></strong><em>, </em><strong><em>semantic memory</em></strong><em>, </em><strong><em>grounded generation</em></strong><em>, </em><strong><em>orchestrated steps</em></strong><em>, and </em><strong><em>standard tools</em></strong><em> — and we know which layer broke when the demo fails.”</em></blockquote><p>That is the real history lesson. Every era left you a piece of the stack. Your job is not to worship the newest logo it is to <strong>assemble the pieces with judgment</strong>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fb76be568712" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Talk About Pipeline Reliability in Your Performance Review]]></title>
            <link>https://medium.com/@chiheb.mhamdi/how-to-talk-about-pipeline-reliability-in-your-performance-review-ca5e43c1c921?source=rss-148acfe082b4------2</link>
            <guid isPermaLink="false">https://medium.com/p/ca5e43c1c921</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[MHAMDI Chiheb]]></dc:creator>
            <pubDate>Tue, 14 Jul 2026 11:31:37 GMT</pubDate>
            <atom:updated>2026-07-14T11:31:37.192Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Subtitle:</strong> Freshness SLOs, job failure rates, recovery time, and the language that makes data engineering look senior , not just busy.</p><ul><li><strong>Error budgets in practice</strong> for data pipelines — freshness, completeness, and when to ship risky changes</li><li><strong>DORA metrics translated</strong> for data teams (deployment frequency = pipeline releases; CFR = failed runs / bad publishes)</li><li><strong>Pipeline-native reliability:</strong> MTTR for backfills, MTBF for chronic flakiness, late data, schema drift</li><li><strong>How to present impact</strong> in performance reviews without listing every ticket</li><li><strong>Junior vs senior</strong> habits specific to batch, streaming, and hybrid stacks</li></ul><p><strong>One line to remember:</strong> A green Airflow task is not a promise , an <strong>SLO with an error budget</strong> is.</p><p><strong>Who this is for:</strong> data engineers, analytics engineers, and platform folks who own <strong>pipelines, warehouses, and SLAs to downstream teams , </strong>not generic product on-call (though the ideas rhyme).</p><p>In <a href="https://github.com/chiheb08/medium-content/blob/main/junior-to-senior-metrics/junior-to-senior-thinking-metrics.md">Part 1</a>, we mapped how thinking shifts from <strong>“my task is done”</strong> to <strong>“the system stays trustworthy.”</strong> For data teams, that shift has a twist:</p><blockquote><em>Your users often never open your repo. They open a </em><strong><em>dashboard</em></strong><em>, a </em><strong><em>ML feature store</em></strong><em>, or a </em><strong><em>finance spreadsheet</em></strong><em> , and assume the number is true.</em></blockquote><p><strong>Junior pipeline instinct:</strong> “The job succeeded.”<br><strong>Senior pipeline instinct:</strong> “Did the <strong>right data</strong> arrive <strong>on time</strong>, <strong>complete</strong>, and <strong>understood</strong> by consumers?”</p><p><strong>Real-life analogy:</strong> A bakery’s oven light says <strong>“done”</strong> (task success). Customers care if the <strong>bread arrived at the café by 7 a.m.</strong>, <strong>all loaves</strong>, <strong>not yesterday’s batch</strong> (SLO).</p><p><strong>7:55 a.m.</strong> — Finance opens the CEO revenue dashboard.<br><strong>7:56 a.m.</strong> — Slack: “Why does yesterday show <strong>$0</strong>?”<br> You check Airflow: <strong>all green</strong> ✅<br> You check the mart: <strong>empty partition</strong> for yesterday because upstream landed late and the job “succeeded” with <strong>zero rows</strong>.</p><p><strong>Junior:</strong> “But the task didn’t fail.”<br><strong>Senior:</strong> “The <strong>promise</strong> failed , and Finance does not care about our task color.”</p><p>Every pipeline makes an implicit promise. Seniors make it <strong>explicit</strong>.</p><p><strong>Defined:</strong> A <strong>measurable signal</strong> of pipeline health — the specific number you track (e.g. “minutes late,” “% jobs failed”).</p><p>Examples:</p><ul><li><strong>Freshness lag:</strong> now() - max(partition_date) for a daily mart</li><li><strong>Job success:</strong> % of scheduled runs succeeded without retry</li><li><strong>Data quality:</strong> % of checks passed (Great Expectations, dbt tests, Deequ)</li><li><strong>End-to-end latency:</strong> event time → warehouse availability (critical for streaming)</li></ul><p><strong>Defined:</strong> The <strong>target</strong> you commit to hit for that SLI over time (e.g. “on time 99% of days per month”).</p><p><strong>Example (daily revenue mart):</strong></p><p>Maybe: “Executive dashboard refreshed by 8 a.m. local <strong>or</strong> incident declared.”</p><p>You promise Product: <strong>“Sales mart ready by 6 a.m.”</strong><br> That is the <strong>SLO</strong>.<br> Legal’s contract with a partner might say <strong>“8 a.m. or penalty”</strong> — that is the <strong>SLA</strong>.<br> You aim for <strong>6</strong> so missing by a bit still beats <strong>8</strong>.</p><p><strong>Everyday analogy:</strong> You set your alarm <strong>30 minutes early</strong> so traffic does not make you late for the interview.</p><p>An <strong>error budget</strong> is how much unreliability your SLO allows before you are “out of policy.”</p><p>That slack is not “free vacation.” It is <strong>currency</strong> for:</p><ul><li>planned risky migrations,</li><li>vendor outages,</li><li>holiday traffic spikes,</li><li>intentional late loads when <strong>upstream</strong> was wrong and you refused to publish garbage.</li></ul><p><strong>Real-life analogy:</strong> A monthly phone data plan with <strong>1% roaming buffer</strong> — you can use it for a real trip, not to stream movies daily and act shocked at the bill.</p><p>For a <strong>binary</strong> daily SLO (on-time vs late):</p><p>$$ \text{budget remaining} \approx 1 — \frac{\text{missed days}}{\text{allowed missed days}} $$</p><p>For <strong>time-based</strong> SLO (minutes late):</p><p>$$ \text{burn} = \frac{\text{total lateness minutes this month}}{\text{monthly lateness allowance}} $$</p><p><em>In plain English:</em> are we <strong>ahead</strong> or <strong>behind</strong> on our monthly “oops allowance”?</p><p><strong>Senior move:</strong> Log <strong>why</strong> budget burned in the incident channel — “late because Stripe export slipped” vs “late because we had no alert.”</p><p>Black Friday week. Marketing runs a <strong>risky</strong> catalog sync. It might slip freshness by <strong>2 hours</strong> — but revenue depends on it.<br><strong>Senior:</strong> “We have <strong>40% error budget</strong> left. This spend is <strong>worth it</strong> — I’ll post in #data-incidents and watch the partition alerts.”</p><p>Same week, someone wants a <strong>cosmetic refactor</strong> of a stable mart.<br><strong>Senior:</strong> “Budget is thin — defer to January.”</p><p>Error budgets are useless without <strong>policy</strong>. A practical playbook:</p><ul><li>Ship refactors, partition changes, cost optimizations</li><li>Experiment with new orchestration features</li><li>Accept small known risks with rollback plans</li><li>Freeze <strong>non-essential</strong> schema changes</li><li>Increase monitoring granularity (partition-level freshness)</li><li>Pre-mortem on upcoming heavy days (Black Friday, month-end close)</li><li><strong>Change freeze</strong> on critical paths</li><li>Daily standup on recovery only</li><li>Escalate upstream SLAs with evidence (“we missed 4/4 days because source landed at 11:00”)</li></ul><blockquote><em>“We have </em><strong><em>18% error budget left</em></strong><em> on revenue mart freshness. I recommend we </em><strong><em>defer</em></strong><em> the dimension table refactor until next sprint and ship only the </em><strong><em>null-rate fix</em></strong><em> that stops bad joins.”</em></blockquote><p>That is not bureaucracy. That is <strong>risk management in plain language</strong>.</p><p>Your PM wants both <strong>new metrics</strong> and <strong>zero risk</strong> this week.<br><strong>Junior:</strong> silent stress + weekend work.<br><strong>Senior:</strong> “Freshness SLO has <strong>12% budget</strong> left. Pick <strong>one</strong> schema change; the other waits. Here’s the chart.”</p><p>Numbers turn arguments into <strong>choices</strong>.</p><p><a href="https://dora.dev/">DORA metrics</a> were framed for <strong>application delivery</strong>. Data platforms map cleanly if you translate nouns:</p><p><strong>Real-life analogy:</strong> DORA for data is not “how fast we edit SQL.” It is <strong>how often we safely change the factory line</strong> without shipping defective batches.</p><p><strong>High deployment frequency</strong> for data teams does <strong>not</strong> mean “run full refresh hourly because we can.”</p><p>It means:</p><ul><li><strong>Small, incremental</strong> dbt model changes merged often</li><li><strong>Feature flags</strong> for new marts (_v2 tables, shadow pipelines)</li><li><strong>CI</strong> that runs tests on every PR</li><li><strong>Idempotent</strong> jobs — safe to re-run</li></ul><p>Be consistent in your metrics:</p><ul><li>dbt model/tag deploy to prod target</li><li>Airflow DAG bundle promotion</li><li>Spark JAR + config rollout</li><li>Streaming job savepoint redeploy</li></ul><p><strong>Old way:</strong> one <strong>big</strong> migration every 6 weeks — Friday 11 p.m., pizza, prayer.<br><strong>New way:</strong> ship <strong>one dbt model</strong> per day with tests; shadow table _v2 for a week, then swap.</p><p>Deploy frequency goes <strong>up</strong>. Bad surprises go <strong>down</strong>.</p><p><strong>Everyday analogy:</strong> Packing for a trip <strong>a little each night</strong> beats one panic suitcase at 4 a.m.</p><p><strong>CFR for data</strong> should include failures that <strong>matter downstream</strong>, not only task exceptions:</p><p>Rough form (same as Part 1):</p><p>$$ \text{CFR} = \frac{\text{failed or harmful pipeline changes}}{\text{total pipeline changes}} \times 100% $$</p><p><strong>Real-life analogy:</strong> A food plant tracks <strong>recalls</strong>, not “conveyor belt stopped for 30 seconds with no bad shipment.”</p><ul><li><strong>Contract tests</strong> on upstream APIs and landing zones</li><li><strong>Impact analysis</strong> — which dashboards break if this column moves?</li><li><strong>Canary datasets</strong> — publish to _beta schema first</li><li><strong>Row-level comparisons</strong> old vs new before swap</li></ul><p>You merge a dbt model rename: revenue → total_revenue.<br> dbt run: <strong>green</strong>.<br> Looker dashboard: <strong>broken</strong> all morning because nobody updated the explore.<br> Finance used <strong>cached wrong</strong> numbers until 2 p.m.</p><p><strong>CFR event?</strong> Yes — if your policy says <strong>consumer harm</strong> counts.<br><strong>Lesson:</strong> deploy includes <strong>downstream comms</strong>, not only SQL.</p><p>Measure: <strong>merge to prod truth</strong>.</p><p>Includes: CI, review queue, deployment window, <strong>backfill duration</strong> if historical recompute is required.</p><p><strong>Senior pain point:</strong> A “one-line fix” that requires <strong>30-day backfill</strong> is not a small change — say so in planning.</p><p>For pipelines, recovery might mean:</p><ul><li><strong>Rerun</strong> from failed task with correct params</li><li><strong>Point-in-time backfill</strong> for bad partitions</li><li><strong>Rollback</strong> dbt version + invalidate caches</li><li><strong>Communicate</strong> “do not use table X until 14:00”</li></ul><p><strong>MTTR clock starts</strong> when consumers <strong>could</strong> be harmed — not when you noticed a red square for fun.</p><p><strong>9:00</strong> — bad join doubles revenue in the mart.<br><strong>9:05</strong> — senior disables the mart view, pins message in Slack: <strong>“Use yesterday until 11:00.”</strong><br><strong>10:30</strong> — partition backfill completes, tests pass, view re-enabled.</p><p><strong>Wrong data exposure:</strong> ~1.5 hours — that is what <strong>MTTR</strong> measures, not “when dbt turned green again.”</p><p><strong>MTBF</strong> asks: how long between <strong>meaningful</strong> failures?</p><p>Chronic patterns:</p><ul><li>Same <strong>upstream file</strong> late every Monday</li><li><strong>OOM</strong> on day 31 of every month (partition explosion)</li><li><strong>Schema drift</strong> after vendor “silent” column add</li><li><strong>Race</strong> between two DAGs writing same table</li></ul><p><strong>Senior move:</strong> Track <strong>repeat incidents</strong> in postmortems — if the same root cause appears twice, the fix was cosmetic.</p><p>Every Monday, the <strong>vendor CSV</strong> lands at 10:00 instead of 6:00. Someone re-runs the DAG. Green by noon.<br><strong>Six months</strong> of this — nobody asks <strong>why Monday</strong>.</p><p><strong>Everyday analogy:</strong> Your bus is <strong>late every Monday</strong> — buying a faster alarm clock does not fix the bus.</p><p>One screen is enough if it answers <strong>“are we trustworthy right now?”</strong></p><p><strong>Real-life analogy:</strong> Air traffic control does not only watch “planes took off.” They watch <strong>delays, near-misses, and runway capacity</strong> together.</p><p>Monday standup: someone shares <strong>one</strong> dashboard — freshness lag, SLO %, CFR last week.<br><strong>No</strong> scrolling through 40 Grafana tabs.<br> When finance asks “can we trust revenue?”, the answer is <strong>one number</strong>: <strong>99.2% on-time, budget green</strong>.</p><p><strong>Everyday analogy:</strong> Doctor checks <strong>blood pressure + heart rate + temperature</strong> — not 50 lab reports every visit.</p><p>Managers are not impressed by <strong>47 JIRA tickets closed</strong>. They remember <strong>trust, risk removed, and money/time saved</strong>.</p><p><strong>Engineer A:</strong> “Closed 52 tickets, maintained DAGs.”<br><strong>Engineer B:</strong> “Revenue mart <strong>99.4% on-time</strong> (target 99%). Cut bad publishes from <strong>8% → 2%</strong>. Saved <strong>~$4k/month</strong> cluster cost.”</p><p>Both worked hard. Only one story survives <strong>budget season</strong>.</p><ul><li><strong>Situation:</strong> “Finance close depends on mart X by 06:00.”</li><li><strong>Task:</strong> “SLO breaching 2×/week; root cause unclear.”</li><li><strong>Action:</strong> “Partition-level freshness alerts, upstream SLA doc, idempotent backfill.”</li><li><strong>Result:</strong> “<strong>99.2% on-time</strong> over 90 days; <strong>one</strong> breach vs nine prior quarter.”</li></ul><blockquote><em>“I chose to </em><strong><em>miss freshness by 2 hours</em></strong><em> once rather than publish </em><strong><em>unreconciled</em></strong><em> Stripe data — preserved correctness, spent error budget, notified stakeholders.”</em></blockquote><p>That shows <strong>judgment</strong>, not heroics.</p><ul><li>1–2 <strong>SLO charts</strong> (before/after)</li><li><strong>One incident</strong> where you improved MTTR or MTBF</li><li><strong>One delivery win</strong> (lead time or CFR)</li><li><strong>One business line</strong> — analyst hours saved, infra cost, decision delay avoided</li><li><strong>One thing you would do next quarter</strong> tied to error budget policy</li></ul><p>Copy into your notes and fill honestly:</p><p><strong>Weekly habit (15 min):</strong></p><ol><li>Check <strong>error budget</strong> on tier-1 SLIs</li><li>Check <strong>CFR</strong> on last 5 prod changes</li><li>Any <strong>repeat</strong> failure pattern? (MTBF)</li><li>One <strong>consumer</strong> check-in — do numbers match their reality?</li></ol><p>Friday <strong>3:00 p.m.</strong>, calendar block: <strong>pipeline health</strong>.<br> Open one page: SLO %, CFR, any repeat incident.<br> Takes <strong>15 minutes</strong> — catches the “always late Monday file” before it becomes <strong>quarterly fire drill</strong>.</p><p><strong>Everyday analogy:</strong> Weekly <strong>weigh-in</strong> on a diet , small check beats discovering in June you never lost the weight.</p><p>Junior you asked: <strong>“Did the job succeed?”</strong><br> Senior you asks: <strong>“Did we keep our data promise — and did we spend our error budget wisely?”</strong></p><p>For <strong>data pipelines</strong>, that means:</p><ul><li><strong>SLOs</strong> consumers can repeat in a meeting</li><li><strong>Error budgets</strong> that change when you ship risky work</li><li><strong>DORA-style discipline</strong> on how often you change prod and how often it hurts</li><li><strong>MTTR / MTBF</strong> that reflect <strong>bad data exposure</strong>, not only red tasks</li><li><strong>Performance reviews</strong> that cite <strong>outcomes</strong>, not ticket counts</li></ul><p>You do not need a perfect data mesh slide deck. You need <strong>one critical mart</strong> with a real SLO, a tracked budget, and a story you can tell without apologizing.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ca5e43c1c921" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Junior to senior metrics , what you need to LEARN!]]></title>
            <link>https://medium.com/@chiheb.mhamdi/junior-to-senior-metrics-what-you-need-to-learn-53bf9e4103b9?source=rss-148acfe082b4------2</link>
            <guid isPermaLink="false">https://medium.com/p/53bf9e4103b9</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[MHAMDI Chiheb]]></dc:creator>
            <pubDate>Sun, 12 Jul 2026 20:42:41 GMT</pubDate>
            <atom:updated>2026-07-12T20:42:41.563Z</atom:updated>
            <content:encoded><![CDATA[<ul><li><em>First Thinking Ideas That Should Change as You Move From Junior to Senior</em></li><li><em>Junior Engineers Ship Features. Senior Engineers Ship Trust.</em></li><li><em>The Scoreboard Changes: SLAs, DORA, ROI, and the Mindset Shift Nobody Prints in the Onboarding Doc</em></li><li>A clear <strong>junior vs senior mindset shift</strong>: from “did my code work?” to “did the <em>system</em> stay trustworthy?”</li><li>Plain definitions of <strong>SLA, SLO, error rates, MTTR, MTBF</strong> — and when each one actually matters</li><li><strong>DORA metrics</strong> (deployment frequency, change failure rate) without treating them like a leaderboard</li><li><strong>Business metrics</strong> (ROI, utilization, on-time delivery) — and why seniors stop worshipping utilization alone</li><li><strong>Security metrics</strong> (time to detect, patching cadence) — risk as a clock, not a checkbox</li><li>A <strong>one-page mental model</strong> you can reuse in 1:1s, incident reviews, and planning meetings</li></ul><p>Early in your career, feedback is simple:</p><ul><li>Did the feature ship?</li><li>Did the tests pass?</li><li>Did the reviewer approve the PR?</li></ul><p>Those are <strong>necessary</strong>. They are not <strong>sufficient</strong> and nobody sends a calendar invite when the scoreboard quietly changes.</p><p>Somewhere between “junior” and “senior,” you are expected to care about different questions:</p><ul><li>Did we <strong>break trust</strong> with users or other teams?</li><li>Did we <strong>recover quickly</strong> when something went wrong?</li><li>Did we <strong>move safely</strong>, not just move often?</li><li>Did this work <strong>pay for itself</strong> — in money, time, or risk removed?</li><li>Did we <strong>leave the system more secure</strong> than we found it?</li></ul><p>This article is a map of that shift through the metrics that start showing up in dashboards, incident reviews, and leadership slides. Not so you can memorize acronyms, but so you can <strong>think like the job is bigger than your branch</strong>.</p><p>None of this means juniors are “wrong.” It means <strong>the job’s definition of good expands</strong>. The metrics below are how organizations make that expansion visible.</p><p><strong>Junior frame:</strong> “I merged it; QA passed; we’re good.”</p><p><strong>Senior frame:</strong> “What happens at 2× traffic, during a deploy, when a dependency hiccups, or when someone runs yesterday’s job twice?”</p><p>Reliability metrics are not about fear. They are about <strong>honest promises</strong> — to users, to sales, to finance, and to the on-call engineer who inherits your design at 3 a.m.</p><p><strong>Bridge:</strong> Before SLAs and SLOs, remember — <strong>stability is a feature</strong> with compound interest. Every silent failure you ignore becomes someone else’s emergency later.</p><p>These two get swapped in meetings like user_id and User Id. They are related - but not the same thing.</p><p>An <strong>SLA</strong> is a <strong>contractual promise</strong> to a customer (or internal customer). It usually includes:</p><ul><li>what you measure (availability, latency, support response),</li><li>the threshold (e.g. 99.9% uptime per month),</li><li>and <strong>consequences</strong> if you miss (credits, penalties, escalations).</li></ul><p><strong>Real-life analogy:</strong> A pizza shop promises <strong>“30 minutes or it’s free.”</strong> That is an SLA — customer-facing, with teeth.</p><p>An <strong>SLO</strong> is an <strong>internal target</strong> your team sets <strong>before</strong> you promise it externally. It should be <strong>stricter</strong> than the SLA — a buffer zone.</p><p><strong>Example:</strong></p><ul><li><strong>SLO:</strong> 99.95% availability (what engineering tries to hit)</li><li><strong>SLA:</strong> 99.9% availability (what legal puts in the contract)</li></ul><p>If your SLO is 99.9% monthly, you have a small <strong>error budget</strong> — allowed unreliability before you breach the objective. Seniors ask:</p><ul><li>“Should we spend budget on <strong>this risky launch</strong>, or save it for holiday traffic?”</li><li>“Are we burning budget on <strong>known debt</strong>?”</li></ul><p><strong>Pitfall:</strong> Publishing an SLA without an SLO is like signing a lease without checking your bank balance.</p><p><strong>Error rate</strong> = how often requests, jobs, or transactions fail — usually as a <strong>percentage</strong> over a window.</p><p>Examples:</p><ul><li>HTTP <strong>5xx rate</strong> = 0.2% of requests</li><li>Pipeline <strong>failure rate</strong> = 3 failed runs per 1,000</li><li>Data job <strong>null-rate spikes</strong> treated as quality errors</li></ul><p><strong>Real-life analogy:</strong> A hospital tracks <strong>complication rates</strong> — not to shame surgeons, but to find <strong>systemic</strong> issues (scheduling, handoffs, equipment). A single bad day matters less than a <strong>pattern</strong>.</p><ul><li>Split <strong>client errors (4xx)</strong> vs <strong>server errors (5xx)</strong> — different owners, different fixes.</li><li>Track <strong>error budget burn rate</strong> after deploys.</li><li>Ask <strong>“errors per what?”</strong> — per request, per dollar, per customer segment?</li></ul><p>Note: DORA also uses <strong>MTTR</strong> for <em>failed changes</em> — how fast you restore after a bad deploy. Same spirit: <strong>recovery speed</strong>.</p><p><strong>Real-life analogy:</strong> Fire drills are not about pretending fires never happen. They are about <strong>everyone knowing where the extinguishers are</strong>.</p><ul><li><strong>Runbooks</strong> that match real failures (not fantasy docs)</li><li><strong>Feature flags</strong> and <strong>canaries</strong></li><li><strong>Observability</strong> — logs, traces, metrics with <strong>deployment markers</strong></li><li><strong>Clear incident roles</strong> — commander, comms, scribe</li><li><strong>Blameless postmortems</strong> that produce <strong>one concrete follow-up</strong></li></ul><p><strong>Pitfall:</strong> Teams celebrate “zero incidents” while <strong>MTTR is secretly terrible</strong> — the first real outage becomes a multi-hour legend for the wrong reasons.</p><p><strong>MTBF (Mean Time Between Failures)</strong> = average time a system runs <strong>without a failure</strong> (for repairable systems).</p><p>High MTBF → failures are <strong>rare</strong>.<br> Low MTBF → you are living in <strong>chronic pain</strong> — even if each fix is fast.</p><p><strong>Senior move:</strong> Stop optimizing only firefighting (MTTR) while ignoring <strong>why the fire started again</strong> (MTBF).</p><p><strong>DORA metrics</strong> (DevOps Research and Assessment) describe <strong>how teams deliver software</strong> — popularized by the <em>Accelerate</em> research. Four core metrics:</p><ol><li><strong>Deployment frequency</strong> — how often you deploy to production</li><li><strong>Lead time for changes</strong> — commit → production</li><li><strong>Change failure rate</strong> — % of changes causing failure</li><li><strong>MTTR</strong> — recovery time after failure</li></ol><p><strong>Real-life analogy:</strong> DORA is not “who’s fastest.” It is “can you <strong>jog reliably</strong> without tripping every third step?”</p><p><strong>Deployment frequency</strong> = how often you successfully release to production (hourly, daily, weekly…).</p><p><strong>Why seniors like higher frequency (when done right):</strong></p><ul><li>Smaller diffs → easier reviews and debugging</li><li>Faster feedback from real users</li><li>Less “release day trauma”</li><li>Deploys that are <strong>config-only theater</strong> (not real change)</li><li>“Deploy” meaning <strong>restart</strong> without validation</li><li>Frequency up, <strong>CFR</strong> also up — that is not maturity, that is ** roulette**</li></ul><p><strong>Senior question:</strong> “What <strong>percentage</strong> of deploys are low-risk vs high-risk changes — and do we treat them differently?”</p><p>Rough form:</p><p><em>In plain English:</em> of everything we shipped, how much <strong>broke prod</strong>?</p><p><strong>Pitfall:</strong> Gaming CFR by defining “failure” too narrowly (ignore customer-impacting bugs that did not trigger rollback).</p><p>At some point, someone asks: <strong>“Was this worth it?”</strong></p><p>Not morally — <strong>economically</strong>. Seniors translate engineering work into language finance and product already speak.</p><p>This is not selling out. It is <strong>prioritization with lights on</strong>.</p><p><strong>ROI (Return on Investment)</strong> = financial gain relative to cost.</p><p>Simple form:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/832/1*o8xg8YylGuboOTaSARO3Ew.png" /></figure><p><em>In plain English:</em> for every dollar spent, how many dollars came back (or how much cost was removed)?</p><p><strong>Real-life analogy:</strong> Renovating a kitchen increases home value <strong>only if</strong> you plan to sell or use it for years — not every upgrade pays back.</p><p><strong>Pitfall:</strong> ROI spreadsheets that ignore <strong>maintenance cost</strong>, <strong>on-call burden</strong>, and <strong>security exposure</strong>.</p><p><strong>Resource utilization</strong> = how much of available capacity is used — CPU, memory, disk, <strong>or human hours</strong>.</p><p><strong>Real-life analogy:</strong> A highway at <strong>100% capacity</strong> is not efficient — it is a <strong>traffic jam</strong>. Engineers want <strong>headroom</strong>, like lanes and on-ramps.</p><ul><li><strong>Too low</strong> — wasted money (but maybe fine for burst workloads)</li><li><strong>Too high</strong> — fragile under load; <strong>everything becomes urgent</strong></li><li><strong>Senior goal</strong> — predictable performance with <strong>autoscaling</strong> or <strong>queueing</strong> strategies, not bragging about redlines</li></ul><p><strong>Data/engineering angle:</strong> Batch pipelines at 100% cluster utilization sound great until <strong>one late partition</strong> delays everything downstream.</p><p><strong>On-time delivery rate</strong> = percentage of milestones, sprint commitments, or project phases completed by the <strong>original</strong> deadline.</p><ul><li><strong>Definition of done</strong> includes ops, docs, monitoring — not just merged code</li><li><strong>Buffer</strong> for unknowns — especially integrations and compliance</li><li><strong>Leading indicators</strong> — blocked tickets, scope creep, test flakiness — not only the final date</li></ul><p><strong>Pitfall:</strong> Gaming on-time rate by <strong>moving deadlines</strong> silently or shrinking scope without telling stakeholders.</p><p>Security is not a phase at the end. It is <strong>exposure over time</strong>:</p><ul><li>how long vulnerabilities exist,</li><li>how long until you notice abuse,</li><li>how fast you patch when the world moves.</li></ul><p>Seniors think in <strong>timers</strong>, not checkboxes.</p><p>TTD is often worse than time to fix — because until you detect, <strong>nothing else starts</strong>.</p><p><strong>Real-life analogy:</strong> A water leak behind the wall — the damage isn’t the pipe breaking; it’s the <strong>weeks until you see the stain</strong>.</p><ul><li>Centralized logs with <strong>retention</strong> that matches attacker dwell time</li><li><strong>Anomaly detection</strong> on auth and data access</li><li><strong>Canary credentials</strong> and honeypots (where appropriate)</li><li>Regular <strong>tabletop exercises</strong> — “what would we see if…?”</li></ul><p><strong>Pair TTD with MTTR:</strong> Detect fast <strong>and</strong> recover fast — both show up in incident timelines and customer trust.</p><p><strong>Patching cadence</strong> = how regularly the team applies <strong>security patches</strong> — OS, dependencies, libraries, infrastructure images.</p><p>Measure it as:</p><ul><li><strong>average time</strong> from critical CVE publish → patch in production,</li><li>or <strong>% systems</strong> within policy compliance at any moment.</li></ul><p><strong>Real-life analogy:</strong> Dental cleaning every six months beats <strong>one emergency root canal</strong> after years of skipping flossing.</p><ul><li><strong>Golden images</strong> rebuilt regularly</li><li><strong>Dependency pinning</strong> with automated update PRs</li><li><strong>Policy tiers:</strong> critical (48h), high (1 week), medium (scheduled)</li><li><strong>Exception process</strong> with expiry dates — no permanent waivers</li></ul><p><strong>Pitfall:</strong> Measuring “patches applied” without measuring <strong>exposure window</strong> — 100 patches late is not hygiene.</p><p>You do not need seventeen dashboards. You need <strong>one story</strong>:</p><p><strong>Read it like a senior:</strong></p><ol><li><strong>Ship often</strong> (deployment frequency) but watch <strong>CFR</strong> — speed without safety is debt.</li><li><strong>SLOs / error rates</strong> tell you if users feel pain.</li><li><strong>MTTR</strong> is your apology speed; <strong>MTBF</strong> is whether you keep apologizing for the same thing.</li><li><strong>ROI</strong> and <strong>on-time delivery</strong> anchor engineering to outcomes — not just activity.</li><li><strong>Utilization</strong> explains fragility — too tight, and SLOs crack under load.</li><li><strong>TTD</strong> and <strong>patching cadence</strong> are security’s MTTR/MTBF — exposure time kills.</li></ol><p><strong>Real-life analogy:</strong> Running a restaurant — ticket count (junior) vs <strong>repeat customers, health inspections, food waste, staff burnout, and profit</strong> (senior).</p><p>If you are growing toward senior impact, pick <strong>one metric per area</strong> and make it honest:</p><p>The mindset shift is not cynicism. It is <strong>ownership at a longer time horizon</strong>.</p><p>Junior you proved you could <strong>build</strong>. Senior you proves the organization can <strong>trust what you built</strong> — through metrics that reflect reality, not comfort.</p><p><strong>Feedback welcome:</strong> if you want a follow-up on <strong>error budgets in practice</strong>, <strong>DORA metrics for data teams</strong>, or <strong>how to present these in performance reviews</strong>, say which scenario matches your world on-call, data pipelines, or product engineering.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=53bf9e4103b9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[GPU architecture for LLMs , simply explained.]]></title>
            <link>https://medium.com/@chiheb.mhamdi/gpu-architecture-for-llms-simply-explained-c693b0f282f4?source=rss-148acfe082b4------2</link>
            <guid isPermaLink="false">https://medium.com/p/c693b0f282f4</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[MHAMDI Chiheb]]></dc:creator>
            <pubDate>Wed, 08 Jul 2026 10:31:52 GMT</pubDate>
            <atom:updated>2026-07-08T10:31:52.792Z</atom:updated>
            <content:encoded><![CDATA[<p>This is a complete, beginner-friendly guide to understanding how GPUs behave in the LLM world.</p><p>When running LLMs, performance is mostly controlled by three things:</p><ol><li><strong>VRAM capacity</strong> — Can the model and KV cache fit?</li><li><strong>Memory bandwidth</strong> — How fast can the GPU read/write model data each token?</li><li><strong>Compute throughput</strong> — How fast can the GPU do matrix math when data is available?</li></ol><p>A lot of “mystery slowness” is simply this:</p><ul><li><strong>Prefill</strong> (reading long prompts) is often <strong>compute-heavy</strong>.</li><li><strong>Decode</strong> (generating token by token) is often <strong>memory-bandwidth-heavy</strong>.</li></ul><p>So yes, two GPUs with enough VRAM to hold the same model can still differ greatly in token speed.</p><p>If you are new to LLMs, this section is the foundation.</p><p>A model is a very large math function that converts input text into next-token predictions.</p><p>It learns this behavior during training by adjusting many internal numbers.</p><p>Parameters are those internal numbers.</p><p>Think of parameters like millions or billions of tiny knobs in a giant sound mixer:</p><ul><li>each knob affects output a little bit</li><li>together they define the model’s behavior</li></ul><p>When we say:</p><ul><li><strong>7B model</strong> → about 7 billion parameters</li><li><strong>13B model</strong> → about 13 billion parameters</li><li><strong>70B model</strong> → about 70 billion parameters</li></ul><p>more parameters usually mean more expressive power, but also more memory and compute cost.</p><p>Quantization is a compression technique for those parameter values.</p><p>Instead of storing each value with high precision (many bits), we store it with lower precision (fewer bits).</p><p>This makes the model smaller and cheaper to run.</p><p>Simple view:</p><ul><li>higher precision = clearer photo, larger file</li><li>lower precision = slightly less detail, much smaller file</li></ul><p>For example, <strong>Q4</strong> roughly means 4-bit style quantized representation (exact storage varies by implementation).</p><p>At planning level, use this simple formula:</p><pre>Model weight size (bytes) ≈ Number of parameters × Bytes per parameter</pre><p>Examples:</p><pre>13e9 × 2 = 26e9 bytes ≈ 26 GB</pre><pre>13e9 × 0.7 ≈ 9.1e9 bytes ≈ 9.1 GB</pre><p>So quantization can dramatically reduce required VRAM for weights.</p><p>“Model size” usually refers to <strong>weights only</strong>.</p><p>But runtime memory needs are bigger because you also need:</p><ul><li>KV cache</li><li>temporary activations/buffers</li><li>framework overhead</li></ul><p>That is why a model that “fits on paper” can still fail in production if context length and concurrency are high.</p><p>Imagine an AI factory:</p><ul><li><strong>VRAM</strong> = warehouse size (how much stuff you can store)</li><li><strong>Memory bandwidth</strong> = road width to/from warehouse (how fast you move stuff)</li><li><strong>GPU cores / tensor cores</strong> = workers doing the math</li><li><strong>PCIe/NVLink</strong> = highways between GPU and CPU or GPU to GPU</li></ul><p>You need all of them balanced.</p><p>Big warehouse + tiny road = workers waiting for materials. Huge worker team + small warehouse = not enough model/context fits.</p><p>At runtime, VRAM is not just “the model”. It includes:</p><ol><li><strong>Model weights</strong> (largest static block)</li><li><strong>KV cache</strong> (grows with context and concurrent requests)</li><li><strong>Activations / temporary buffers</strong></li><li><strong>Framework overhead</strong> (allocator, CUDA context, fragmentation)</li><li><strong>Batching buffers</strong></li></ol><p>This is why a model that “should fit” on paper may still OOM in real workloads.</p><pre>Total VRAM needed ≈ Weights + KV Cache + Runtime Overhead</pre><p>Quantization reduces precision to save memory.</p><p>A common rough assumption for Q4 is:</p><ul><li>~0.5 to 0.8 bytes per parameter depending on format/tooling</li><li>many people use ~0.7 bytes/parameter as a rough planning number</li></ul><p>For a model with N parameters:</p><pre>Weight size ≈ N × bytes_per_param</pre><p>Example for 13B parameters at ~0.7 bytes:</p><pre>13e9 × 0.7 ≈ 9.1 GB of weights</pre><p>Great , you fit in more GPUs.</p><p>But quantization does <strong>not</strong> remove memory traffic needs. You still read lots of weight data every token.</p><p>A practical rule of thumb used in engineering discussions:</p><p>Using the same 13B Q4 example:</p><ul><li>weight size ~9.1 GB</li><li>effective read per token ~18 GB</li></ul><p>Now time per token lower bound from memory bandwidth is roughly:</p><pre>ms_per_token_theoretical ≈ (GB_per_token / GBps) × 1000</pre><p>If GB/token ≈ 18:</p><ul><li>960 GB/s → ~19 ms/token</li><li>504 GB/s → ~36 ms/token</li><li>256 GB/s → ~70 ms/token</li></ul><p>That alone is ~2.8× spread — same model fit, very different speed.</p><p>This is why “I have enough VRAM” is only half the story.</p><p>LLM inference has two different phases:</p><p>You feed the full input context (maybe 2k, 8k, 32k tokens).</p><ul><li>Work is dense and parallel.</li><li>Usually more compute utilization.</li><li>Throughput (tokens/sec) can look strong.</li></ul><p>Now model generates <strong>one token at a time</strong> repeatedly.</p><ul><li>Work becomes sequential across time.</li><li>Often memory-bound.</li><li>Latency per token dominates user experience.</li></ul><p>Users judge chat by “how quickly tokens appear” (decode speed), not by how fast your prefill benchmark looked in slides.</p><p>For transformer attention, past keys/values are stored so model doesn’t recompute everything.</p><p>This <strong>KV cache</strong> grows with:</p><ul><li>context length</li><li>batch/concurrent requests</li><li>layers / heads / hidden size</li><li>dtype</li><li>Long contexts and many simultaneous chats can consume more memory than expected.</li><li>OOM or aggressive eviction can suddenly crush latency.</li></ul><p>“It works with 1 user, gets unstable with 8 users.”<br> Usually not magic. KV cache pressure + scheduling + memory fragmentation.</p><p>These 3 are related but not identical.</p><ul><li><strong>Latency</strong>: time for one user’s response (or first token)</li><li><strong>Throughput</strong>: total tokens/sec across all users</li><li><strong>Concurrency</strong>: how many active requests at once</li></ul><p>You can optimize one while hurting another.</p><p>Example:</p><ul><li>Big batch improves throughput</li><li>But may increase first-token latency for interactive chat</li></ul><p>For chat products, first-token latency and stability often matter more than peak benchmark throughput.</p><p>Modern inference servers use dynamic/continuous batching.</p><p>Think of a bus system:</p><ul><li>If you wait to fill the bus, efficiency rises (throughput)</li><li>But first passenger waits longer (latency)</li></ul><p>Good schedulers try to balance this in real time.</p><p>When tuning LLM serving, don’t ask only:</p><p>“How many tokens/sec can we get?”</p><p>Also ask:</p><p>“What happens to p95 first-token latency with real user traffic?”</p><p>Two models can both be “13B” and still behave differently due to:</p><ul><li>architecture differences</li><li>attention implementation</li><li>quantization backend</li><li>context length configuration</li><li>serving engine optimizations</li></ul><p>So capacity planning must use <strong>measured</strong> results, not just parameter count.</p><p>Multi-GPU can help with larger models or more throughput, but introduces communication costs.</p><p>Key links:</p><ul><li><strong>PCIe</strong>: common, slower than on-package memory bandwidth</li><li><strong>NVLink / high-speed interconnect</strong>: much faster GPU-GPU transfer</li></ul><p>If tensor/model parallelism requires frequent cross-GPU sync, interconnect becomes a bottleneck.</p><p>Analogy:</p><ul><li>One kitchen with one team (single GPU) can be very fast if recipe fits.</li><li>Splitting one dish across 4 kitchens needs runners between kitchens (communication overhead).</li></ul><p>GPU is central, but the full system can bottleneck elsewhere:</p><ul><li>tokenizer on CPU</li><li>request orchestration in API/gateway</li><li>model load from disk</li><li>cross-node network hops</li><li>queueing delays</li></ul><p>Classic failure mode:</p><p>GPU utilization looks low, but users still wait. Why? Data pipeline and scheduling overhead upstream.</p><p>If you run LLM services, watch these continuously:</p><ol><li><strong>Time to first token (TTFT)</strong> p50/p95</li><li><strong>Decode tokens/sec</strong> per model</li><li><strong>Request queue depth</strong> and waiting time</li><li><strong>GPU memory used / free</strong> and OOM events</li><li><strong>GPU utilization</strong> (compute) vs memory utilization</li><li><strong>Batch size distribution</strong></li><li><strong>Gateway/API timeout rates</strong></li><li><strong>Error rates by phase</strong> (prefill, decode, tool calls)</li></ol><p>If you only monitor “GPU utilization %”, you’ll miss many real user problems.</p><p>Common chain:</p><ol><li>Prompt sizes increase + multiple uploads/chat sessions</li><li>KV cache grows</li><li>Batch/scheduler queues requests</li><li>decode gets memory-bound</li><li>gateway/api hits timeout before completion</li><li>users retry, increasing load (feedback loop)</li></ol><p>This is a traffic jam, not one bug.</p><p>Don’t start with “largest model we can fit.” Start with SLOs.</p><ol><li><strong>Define target SLOs:</strong></li></ol><ul><li>TTFT p95</li><li>response completion p95</li><li>max concurrent active users</li></ul><p>2<strong>. Benchmark realistic prompts and output lengths.</strong></p><p>3.<strong> Test mixed traffic (short + long prompts).</strong></p><p>4. <strong>Add 30–40% headroom for spikes.</strong></p><p>5.<strong> Re-test after every model/context change.</strong></p><p>Model or context changes can invalidate old sizing immediately.</p><p>No. More VRAM means bigger fit. Speed often comes from bandwidth and scheduling.</p><p>Not always. You can be efficiently overloaded with terrible p95 latency.</p><p>It helps fit and cost. It doesn’t remove all memory movement constraints.</p><p>No. Separate prefill, decode, TTFT, throughput, and tail latency.</p><ul><li><strong>VRAM</strong> = pantry size</li><li><strong>Bandwidth</strong> = width of kitchen counter where ingredients move</li><li><strong>Compute cores</strong> = chefs</li><li><strong>KV cache</strong> = prepared ingredients kept on side tables</li><li><strong>Batching</strong> = cooking many orders together</li></ul><p>If your side tables (KV cache) are full and counter is narrow (bandwidth), adding more chefs (compute) won’t fix dinner service.</p><p>Assume:</p><ul><li>model effective movement per generated token: ~18 GB</li><li>GPU A bandwidth: 900 GB/s</li><li>GPU B bandwidth: 300 GB/s</li></ul><p>Lower-bound token times:</p><ul><li>A: 18/900 × 1000 ≈ 20 ms/token</li><li>B: 18/300 × 1000 ≈ 60 ms/token</li></ul><p>If each answer is 120 tokens:</p><ul><li>A decode time floor ≈ 2.4 s</li><li>B decode time floor ≈ 7.2 s</li></ul><p>Before any extra overhead.</p><p>Same model. Very different user experience.</p><p><strong>Remember:</strong> More parameters = usually smarter, but heavier (more VRAM + compute).</p><p><strong>Remember:</strong> Quantization helps fit the model. It does not remove all memory traffic at inference time.</p><h3>Closing</h3><p>LLM infrastructure feels mysterious until you reduce it to movement, math, and queues.</p><p>The key is to stop asking only:</p><p>“Can this GPU run the model?”</p><p>and start asking:</p><p>“At my target concurrency and context length, can this system move data fast enough to keep latency stable?”</p><p>Once you ask that question, architecture decisions become much clearer.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c693b0f282f4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What production teaches you about deploying ai platforms]]></title>
            <link>https://medium.com/@chiheb.mhamdi/what-production-teaches-you-about-deploying-ai-platforms-76eda912f3f2?source=rss-148acfe082b4------2</link>
            <guid isPermaLink="false">https://medium.com/p/76eda912f3f2</guid>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[MHAMDI Chiheb]]></dc:creator>
            <pubDate>Sun, 05 Jul 2026 20:18:30 GMT</pubDate>
            <atom:updated>2026-07-05T20:18:49.585Z</atom:updated>
            <content:encoded><![CDATA[<p>If you have ever shipped a modern application stack to a real cluster-a web UI, an API, background workers, a search engine, object storage, and a large language model behind a gateway-you have probably lived a version of this story. The problems rarely show up in a hello-world tutorial. They show up on a Tuesday afternoon when someone uploads a 200-page PDF while three colleagues ask questions in chat and a fourth deletes an old document.</p><p>This article is a field report. Not tied to one product or one company. Just the patterns that repeat when teams move from “it works on my laptop” to “it must work for everyone, every day.”</p><p>For each problem: what happened in plain language, why juniors (and honestly, seniors under pressure) walk into it, and what to do instead.</p><p>Most deployment guides optimize for the <strong>happy path</strong>:</p><ol><li>Install components.</li><li>Run one test request.</li><li>Declare victory.</li></ol><p>Production optimizes for the <strong>unhappy path at scale</strong>:</p><ul><li>Two users do the same action at once.</li><li>A background job fails halfway.</li><li>A proxy times out while the server is still working.</li><li>A model needs the internet in an environment that has no internet.</li></ul><p>The gap between those two worlds is where careers learn fast.</p><p><strong>Analogy:</strong> Building a bridge from a picture of a bridge. The picture shows cars crossing one at a time on a sunny day. Nobody drew the rush hour, the snowplow, or the truck that is too heavy.</p><p>Users click “upload” or “send.” The browser calls an API path like /api/something/.... The gateway (nginx, ingress, route) forwards the request-but <strong>does not rewrite the path correctly</strong>. The API receives a URL it does not recognize. Result: 404 Not Found even though &quot;the app is up.&quot;</p><p>Separately, large uploads fail or hang because the proxy’s <strong>body size limit</strong> or <strong>timeout</strong> is lower than the application’s limit. The API would have accepted the file; the gateway gave up first.</p><ul><li>They test with curl directly against the API pod, <strong>bypassing the gateway</strong>. It works. They assume production is the same.</li><li>They treat nginx/ingress as “just routing,” not as part of the application contract.</li><li>They raise timeouts in one layer (app) but not in the outer layer (load balancer / cloud route).</li><li><strong>Always test through the same URL users use</strong> (public hostname, TLS, full path).</li><li>Draw one diagram: Browser → Gateway → App. Write the <strong>exact path</strong> at each hop.</li><li>Align three numbers everywhere: max upload size, read timeout, idle timeout (gateway, app, cloud route).</li><li>For streaming responses (chat tokens), disable response buffering on the gateway. A buffer that waits for “the full response” kills streaming.</li></ul><p><strong>Analogy:</strong> The restaurant kitchen is open, but guests are knocking on the loading dock door. Same building, wrong entrance.</p><p>Background jobs (file processing, indexing, deletion, sync) share <strong>one worker pool</strong> and <strong>one queue consumer</strong>. Heavy jobs (index a document) run beside urgent small jobs (delete a file). Deletes pile up. Uploads pile up. The UI shows “processing” or “deleting” for hours.</p><p>This is not a mystery bug. It is <strong>queue starvation</strong>: one lane of traffic, ambulance and school bus in the same line.</p><ul><li>Default docker-compose or Helm chart uses <strong>one generic worker</strong>. It works for one user.</li><li>Redis queue depth is invisible until someone asks “why is it slow?”</li><li>“Async” sounds like magic-teams forget that <strong>someone still has to execute the work</strong>, and capacity is finite.</li><li><strong>Separate queues by job type</strong>: ingestion, deletion, light sync, heavy batch.</li><li><strong>Dedicated workers</strong> for destructive or latency-sensitive paths (e.g. delete).</li><li>Monitor <strong>queue length</strong>, not only “worker is running.”</li><li>Load-test with 5–10 people doing different actions at once-not one person clicking slowly.</li></ul><p><strong>Analogy:</strong> One cashier for prescriptions, returns, and lottery tickets. Everyone is polite. The line still goes around the block.</p><p>A periodic scheduler (Celery Beat, cron sidecar, etc.) tries to write its state file to a <strong>read-only directory</strong> in the container. It crashes or silently fails. Periodic cleanup never runs. Retries never run. Disk or queues fill up over days.</p><ul><li>Images run as non-root (good for security) but manifests still assume /var/lib/... is writable.</li><li>Beat/cron is deployed last and tested never-”we’ll check later.”</li><li>Logs show permission errors once; nobody connects them to stuck files or growing Redis keys.</li><li>Give schedulers a <strong>writable path</strong> (/tmp, emptyDir volume) explicitly in the manifest.</li><li>Health-check the scheduler pod the same way you health-check the API.</li><li>After deploy, verify <strong>one scheduled job actually fired</strong> (log line, metric, or test row in DB).</li></ul><p><strong>Analogy:</strong> Hiring a night-shift cleaner and locking the supply closet. The shift is on the roster; the floor stays dirty.</p><p>Redis serves sessions, Celery broker, rate limits, locks-sometimes all on <strong>one small instance</strong> with <strong>aggressive memory limits</strong> and <strong>eviction policies</strong> meant for cache. Under load, memory fills. Keys disappear. Tasks vanish or duplicate. Sessions behave oddly.</p><p>Teams also run diagnostics on the <strong>wrong Redis logical database</strong> (db0 vs db15). Queue length shows zero while work is backed up elsewhere.</p><ul><li>Tutorial Redis: one port, no password, unlimited memory.</li><li>maxmemory 400mb looks fine until file metadata and task payloads add up.</li><li>“Redis is fast” becomes “Redis needs no design.”</li><li>Document <strong>which Redis DB</strong> each subsystem uses.</li><li>Size Redis for <strong>broker + peak queue</strong>, not only session cache.</li><li>Avoid eviction policies that drop <strong>queue keys</strong> under pressure.</li><li>Script LLEN (or equivalent) on critical queues; run it during incident, not after.</li></ul><p><strong>Analogy:</strong> Using your desk drawer as inbox, filing cabinet, and lunch fridge. It works until Tuesday.</p><p>The vector/search service (OpenSearch, Elasticsearch, etc.) is misconfigured: a <strong>one-character typo</strong> in config ( single-nod instead of single-node), or <strong>heap larger than container memory</strong>. The pod enters a restart loop (OOM kill). Uploads &quot;succeed&quot; to object storage but <strong>indexing fails</strong>. Chat may still work for small files; large-document Q&amp;A breaks. Status fields show FAILED or stuck PROCESSING.</p><ul><li>Search is “phase 2”; uploads are tested before search health is verified.</li><li>Cluster health is checked once at install, not continuously.</li><li>Logs are split across API, worker, and search-nobody correlates them.</li><li>Gate deploy on <strong>cluster health green/yellow</strong>, not only TCP port open.</li><li>API init containers should wait for <strong>search readiness</strong>, not only “port listening.”</li><li>Alert on pod <strong>restart count</strong> for stateful search nodes.</li><li>Treat FAILED rows in metadata DB as a first-class metric.</li></ul><p><strong>Analogy:</strong> Books arrive at the warehouse, but the catalog department is closed. Shelves look empty even though boxes are stacked in the back.</p><p>At query or index time, the embedding or LLM layer tries to <strong>download weights from the public internet</strong>. Production is air-gapped or egress-restricted. Error: cannot reach hub, file not in cache. Indexing fails; retrieval fails; uploads never complete.</p><p>Health endpoint still returns 200 because the <strong>process is alive</strong>, not because the <strong>model is loaded</strong>.</p><ul><li>Dev laptop has Hugging Face cache populated from months of experiments.</li><li>HF_HUB_OFFLINE and volume mounts are &quot;prod hardening&quot; items skipped for speed.</li><li>Admin UI selects the biggest shiny model without matching hardware or offline cache.</li><li><strong>Pre-load models</strong> to shared storage; mount read-only into inference pods.</li><li>Set offline flags in prod; fail deploy if model path is empty.</li><li>Smoke-test <strong>/embed or equivalent</strong> after deploy, not only /health.</li><li>Match model size to <strong>GPU/RAM</strong>; an 8B embedding model on a 4Gi pod is a plan, not a deployment.</li></ul><p><strong>Analogy:</strong> Sending a chef to cook a recipe that says “ingredients: download from cloud.” Kitchen has no Wi‑Fi.</p><p>Chat uses <strong>streaming</strong> or multi-step tool calls. Each step waits on the LLM gateway. Default HTTP read timeout (60s) is shorter than model latency under load. Proxy returns 504; user refreshes; answer appears in history-because the backend finished <strong>after</strong> the client gave up.</p><ul><li>Timeouts copied from REST CRUD examples, not from LLM reality.</li><li>Streaming not configured end-to-end (gateway buffers SSE).</li><li>Load test is one user, one short question.</li><li>Raise read timeouts on <strong>gateway, route, and HTTP client</strong> together (e.g. 5–30 minutes for long streams, with care).</li><li>Test <strong>six concurrent streams</strong>, not one.</li><li>Put a <strong>gateway</strong> (LiteLLM-style) in front of raw inference for routing, retries, and observability-don’t let every service speak a different dialect to the GPU layer.</li></ul><p><strong>Analogy:</strong> Hanging up the phone at 60 seconds while the other person is still talking. They finish the sentence to an empty line.</p><p>Delete is implemented as: mark row DELETING → enqueue async cleanup → remove from search index and object storage → delete row. Any step fails; row stays DELETING. UI looks broken. Retries make it worse if workers are starved.</p><ul><li>Happy path tested: delete one small file once.</li><li>No dashboard for <strong>stuck status counts</strong> in the database.</li><li>Delete queue shares workers with heavy indexing.</li><li>Dedicated delete workers and queues.</li><li>Metrics: COUNT(*) WHERE status = &#39;DELETING&#39; and age of oldest row.</li><li>Idempotent cleanup jobs and alerts when delete queue age &gt; N minutes.</li></ul><p><strong>Analogy:</strong> Trash marked “taken out” still sitting in the hallway because the truck never came.</p><p>PDF pipeline fails on specific files (e.g. JBIG2-compressed scans). Base image lacks a system dependency ( jbig2dec, tesseract, etc.). Small test PDFs pass; one real customer document fails. Error looks like &quot;application bug,&quot; root cause is <strong>Dockerfile</strong>.</p><ul><li>CI tests sample files, not production corpus.</li><li>“It worked in dev” with a fatter local image.</li><li>Assume Python package alone handles all formats.</li><li>Collect <strong>real failed files</strong> (redacted) into a regression set.</li><li>Extend image FROM upstream:tag + apt-get install missing tools.</li><li>Log <strong>parser dependency errors</strong> clearly at ingest time.</li></ul><p><strong>Analogy:</strong> A car wash that handles dust but not mud because nobody installed the underbody spray.</p><p>Each component passed its individual smoke test. Nobody tested <strong>combined</strong> load: chat + upload + delete + search. CPU requests sum above node capacity; workers throttle; GPUs serialize; everything feels like a crash.</p><ul><li>Sizing docs read but treated as “future problem.”</li><li>Horizontal scale only on the API, not on workers or inference.</li><li>No single diagram of <strong>who calls whom under load</strong>.</li><li>Run a <strong>30-minute multi-user script</strong>: N people chat, M upload, K delete.</li><li>Watch queue depth, pod restarts, GPU utilization, p95 latency <strong>during</strong> the test.</li><li>Scale the bottleneck you measure, not the one you guess.</li></ul><p><strong>Analogy:</strong> Every room passed fire inspection alone; nobody checked what happens when 500 people are in the building at once.</p><p>Login issues send juniors to grep the wrong layer: password in Postgres, session in Redis, token in browser cookie, signing key in Kubernetes secret. psql without host connects to local socket and shows &quot;role does not exist.&quot; Hours lost.</p><ul><li>Auth taught as “JWT tutorial,” not as <strong>distributed state</strong>.</li><li>Shell into wrong pod; env vars differ from laptop .env.</li><li>One-page map: <strong>where credentials live, where session lives, where permissions live</strong>.</li><li>Standard debug recipe: validate token → check session store → check user row → check ownership query.</li><li>Always connect to DB with <strong>explicit host</strong> inside the cluster.</li></ul><p><strong>Analogy:</strong> Looking for your keys in the jacket while they are in the car. Same person, wrong pocket.</p><p>Small file works in chat; large file fails after delete. Team assumes “upload broken.” Actually <strong>two pipelines</strong> exist: full text in prompt vs search-index retrieval. Large files need indexing; indexing needs search + embeddings. Delete leaves <strong>conversation history</strong> that still references old filenames; model emits raw tool markup when tool routing breaks.</p><ul><li>“File upload” and “question answering” taught as one feature.</li><li>FAILED on file row interpreted as &quot;unusable everywhere,&quot; but small-file path may still use object storage directly.</li><li>Document <strong>which path</strong> each file size takes.</li><li>After delete, test in a <strong>new conversation</strong>.</li><li>Separate alerts for <strong>ingest failure</strong> vs <strong>chat failure</strong>.</li></ul><p><strong>Analogy:</strong> Two doors into the library-express lane for pamphlets, catalog desk for encyclopedias. Complaining the express lane won’t fit an encyclopedia is the wrong complaint.</p><p>Print this. Check boxes in order.</p><ol><li><strong>User path test</strong> — Hit the public URL, not internal service DNS.</li><li><strong>Gateway alignment</strong> — Paths, upload size, timeouts, streaming headers.</li><li><strong>Background capacity</strong> — Queue depths visible; workers split by job type.</li><li><strong>Scheduler writable</strong> — Beat/cron actually ran once post-deploy.</li><li><strong>Search health</strong> — Cluster green; no restart loops; sample index + query.</li><li><strong>Models offline-ready</strong> — Weights on disk; embed smoke test passes.</li><li><strong>LLM path under load</strong> — N concurrent chats; timeouts aligned.</li><li><strong>Delete path</strong> — No growing DELETING (or equivalent) population.</li><li><strong>Real files</strong> — At least one nasty PDF from the wild in CI.</li><li><strong>Six-user soak</strong> — Mixed actions for 30 minutes; watch restarts and queues.</li></ol><p>Production failures are usually <strong>boring</strong>:</p><ul><li>Wrong path at the proxy.</li><li>Not enough workers.</li><li>One dependency restarted.</li><li>Timeout too short.</li><li>Disk or memory too small.</li><li>Async job never consumed.</li></ul><p>They are rarely “the framework is broken.” They are almost always <strong>integration and capacity</strong> under constraints you did not simulate.</p><p>The tutorial showed you how to start the orchestra. Production asks whether the violins can still be heard when the drums, the choir, and the fire alarm go off at the same time.</p><p>If you are early in your career and this list feels overwhelming-good. That means you are thinking at the right altitude. You do not need to memorize every tool. You need habits:</p><ul><li>Test the <strong>full path</strong> users take.</li><li>Measure <strong>queues and stuck states</strong>, not only HTTP 200.</li><li>Draw <strong>one diagram</strong> before you change timeouts or replicas.</li><li>Run <strong>small multi-user tests</strong> before the big demo.</li></ul><p>The goal is not zero incidents. The goal is incidents that are <strong>boring, short, and diagnosable</strong> -because you already know which hallway the problem lives in.´</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=76eda912f3f2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Apache Iceberg for Data Engineers]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/@chiheb.mhamdi/apache-iceberg-for-data-engineers-f8dc92cd3cb2?source=rss-148acfe082b4------2"><img src="https://cdn-images-1.medium.com/max/1566/1*51AbGgec5PPLHdjJgtrrLQ.png" width="1566"></a></p><p class="medium-feed-snippet">Iceberg is how modern lakehouses make S3/HDFS behave more like a database while staying cheap and open.</p><p class="medium-feed-link"><a href="https://medium.com/@chiheb.mhamdi/apache-iceberg-for-data-engineers-f8dc92cd3cb2?source=rss-148acfe082b4------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://medium.com/@chiheb.mhamdi/apache-iceberg-for-data-engineers-f8dc92cd3cb2?source=rss-148acfe082b4------2</link>
            <guid isPermaLink="false">https://medium.com/p/f8dc92cd3cb2</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-visualization]]></category>
            <dc:creator><![CDATA[MHAMDI Chiheb]]></dc:creator>
            <pubDate>Wed, 24 Jun 2026 14:38:58 GMT</pubDate>
            <atom:updated>2026-06-24T14:38:58.781Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Data modeling concepts]]></title>
            <link>https://medium.com/@chiheb.mhamdi/data-modeling-concepts-f5b606a0d59d?source=rss-148acfe082b4------2</link>
            <guid isPermaLink="false">https://medium.com/p/f5b606a0d59d</guid>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[MHAMDI Chiheb]]></dc:creator>
            <pubDate>Tue, 23 Jun 2026 13:20:38 GMT</pubDate>
            <atom:updated>2026-06-23T13:20:38.780Z</atom:updated>
            <content:encoded><![CDATA[<p>If you understand fact tables, dimensions, grain, and slowly changing dimensions, you can read almost any warehouse, dbt project, or BI dashboard and know <em>why</em> it was built that way.</p><h3>1. Fact Tables</h3><p><strong>Analogy:</strong> A cash register receipt.</p><p>Each line records <em>what happened</em>: you bought 2 coffees for $6.00 on Tuesday. The receipt doesn’t explain what “coffee” means in detail — it just records the event and the numbers.</p><p><strong>One sentence: </strong>A fact table stores measurable business events — usually numbers (amounts, counts, quantities) linked to keys that point to dimension tables.</p><p><strong>Tiny example:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YXYwronBeNkKe4GnJ5nfSw.png" /></figure><p>Each row = one business event at a defined level of detail.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0qxL60IV_UFdUqUtEN7PxQ.png" /></figure><p><strong>Facts hold metrics. Dimensions hold context.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Pwlsv1kM1_wAcvH--mcXAQ.png" /></figure><p><strong>Types of fact tables:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Xw-qzNS17xdq3Y4oNJF5CA.png" /></figure><p><strong>Why surrogate keys? </strong>Use customer_key = 1042 instead of natural ID email@example.com. Natural IDs change, merge, or contain PII. Surrogate keys are stable join targets.</p><p><strong>What breaks:</strong></p><ul><li>Wrong grain — two rows for the same event → double-counted revenue in dashboards.</li><li>Nullable foreign keys — customer_key = NULL silently drops rows from joins or creates orphan facts.</li><li>Mixing grains — one fact table with both order-level and line-item-level rows. Aggregations lie.</li><li>Huge wide facts — 200 columns, most unused; scans get expensive.</li></ul><p><strong>How teams fix it:</strong></p><ul><li>Document grain in the model (schema.yml in dbt: grain: one row per order line).</li><li>Add data tests: unique, not_null on grain columns.</li><li>Split facts when grains differ (fct_orders vs fct_order_lines).</li><li>Use partitioning on date_key in BigQuery/Snowflake for pruning.</li></ul><p><strong>Observability:</strong> Row count anomalies, null FK rate, revenue reconciliation vs source system.</p><p><strong>Common questions:</strong></p><ul><li>“What is a fact table?”</li><li>“What’s the difference between additive and semi-additive measures?”</li><li>“Design a fact table for e-commerce orders.”</li></ul><p><strong>Strong answer:</strong> “A fact table stores business events at a declared grain. Keys point to dimensions; columns hold numeric measures. I’d define grain first — one row per order line — then choose surrogate keys and partition by date for query performance.”</p><p><strong>Follow-up:</strong> “Degenerate dimension?” — A descriptive attribute (like order_id) stored in the fact because it doesn&#39;t warrant its own dimension table.</p><h3>Hands-On</h3><pre>-- Transaction fact: one row per order line<br>CREATE TABLE fct_order_lines (<br>    order_line_key    BIGINT PRIMARY KEY,<br>    order_id          VARCHAR(50),      -- degenerate dimension<br>    order_date_key    INT NOT NULL,     -- FK → dim_date<br>    customer_key      INT NOT NULL,     -- FK → dim_customer<br>    product_key       INT NOT NULL,     -- FK → dim_product<br>  store_key         INT NOT NULL,     -- FK → dim_store<br>    quantity          INT NOT NULL,<br>    unit_price        DECIMAL(10,2),<br>    line_revenue      DECIMAL(12,2)     -- additive measure<br>);</pre><h3>2. Dimension Tables</h3><p><strong>Analogy:</strong> The labels on a filing cabinet drawer.</p><p>The receipt says you bought product #7. The drawer label explains: <em>Product #7 = Medium Latte, Beverages, $3.00 list price.</em></p><p><strong>One sentence:</strong> A dimension table stores descriptive attributes that give context to facts — who, what, where, when.</p><p><strong>Tiny example — </strong><strong>dim_product</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7IHiCYEbZIUyrg0vQnnGVg.png" /></figure><p>Facts store product_key = 7. Analysts filter WHERE category = &#39;Beverages&#39; without touching the fact table&#39;s raw IDs.</p><p><strong>Common dimensions (the “who, what, where, when”):</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fTb4ksDNxNxRz5-7JY4bzg.png" /></figure><p><strong>Conformed dimensions: </strong>the same dim_date or dim_product used across multiple fact tables. This is how you compare sales vs returns vs inventory consistently.</p><p><strong>Role-playing dimensions:</strong> one physical dim_date joined multiple times with aliases:</p><pre>SELECT ...<br>FROM fct_orders f<br>JOIN dim_date d_order  ON f.order_date_key  = d_order.date_key<br>JOIN dim_date d_ship   ON f.ship_date_key   = d_ship.date_key</pre><p><strong>What breaks:</strong></p><ul><li>Dimension bloat — wide tables with 150 attributes nobody queries.</li><li>Inconsistent naming — cust_id in one mart, customer_key in another.</li><li>No conformed dimension — each team builds their own dim_product; metrics don&#39;t reconcile.</li><li>PII in dimensions — emails and phone numbers copied everywhere without masking.</li></ul><p><strong>How teams fix it:</strong></p><ul><li>Conformed dimension layer in dbt (models/marts/core/dim_customer.sql).</li><li>Column-level governance — tag PII, mask in BI tools.</li><li>Attribute pruning — core dims stay lean; optional attrs in extension tables or snowflaked sub-dims.</li></ul><p><strong>Common questions:</strong></p><ul><li>“What is a conformed dimension?”</li><li>“What is a junk dimension?”</li><li>“Why use surrogate keys instead of natural keys?”</li></ul><p><strong>Follow-up:</strong> “Bridge table?” — Used for many-to-many (e.g., a patient with multiple diagnoses). Facts link to bridge; bridge links to dimension.</p><h3>Hands-On</h3><pre>CREATE TABLE dim_product (<br>    product_key     INT PRIMARY KEY,   -- surrogate key<br>    product_id      VARCHAR(50),       -- natural key from source<br>    product_name    VARCHAR(200),<br>    category        VARCHAR(100),<br>    subcategory     VARCHAR(100),<br>    brand           VARCHAR(100),<br>    list_price      DECIMAL(10,2),<br>    is_active       BOOLEAN,<br>    effective_from  DATE,              -- used with SCD Type 2<br>    effective_to    DATE,<br>    is_current      BOOLEAN<br>);</pre><h3>3. Grain</h3><p><strong>Analogy: </strong>The zoom level on a map.</p><p>“One dot = one building” is different from “one dot = one city.” If you mix zoom levels on the same map, distances and counts make no sense.</p><p><strong>One sentence:</strong> Grain is the exact level of detail of one row in a table — what one row <em>means</em>.</p><p><strong>Tiny examples:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*J01O51jbair57hbHqhwcug.png" /></figure><p>If grain is wrong, every dashboard built on top is wrong.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*h5F700LTUH5gkGIVlLIM_A.png" /></figure><p><strong>Declare grain before you design columns.</strong></p><p>Good grain statement:</p><blockquote><em>fct_orders has one row per </em><em>order_id. Measures are order-level totals.</em></blockquote><p>Bad (implicit) grain:</p><blockquote><em>A table with orders and line items mixed together because “it’s all order data.”</em></blockquote><p><strong>Grain vs aggregation:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HAmKiW52DlpcxjQAO1PM9A.png" /></figure><p><strong>Rolling up:</strong> You can aggregate fine grain → coarse grain. You cannot disaggregate coarse → fine without guessing.</p><p><strong>What breaks:</strong></p><ul><li>Fan-out joins — joining an order-level fact to a line-level dimension (or vice versa) multiplies rows and inflates metrics.</li><li>Late-arriving facts — yesterday’s orders land today; daily rollup already ran → numbers shift.</li><li>Duplicate grain keys — two rows for same order_line_id → revenue counted twice.</li></ul><p><strong>How teams fix it:</strong></p><ul><li>Enforce unique on grain columns in dbt/SQL tests.</li><li>Document grain in model metadata and PR templates.</li><li>Separate marts at different grains; don’t force one table to serve all use cases.</li><li>Reconciliation queries: SUM(fact.revenue) = SUM(source.revenue) per day.</li></ul><p><strong>Interview red flag: </strong>“We have one big table for everything.” That usually means grain was never decided</p><p><strong>Common questions:</strong></p><ul><li>“What is grain?”</li><li>“How would you handle order-level and line-level metrics?”</li><li>“What happens if you join facts at different grains?”</li></ul><p><strong>Strong answer:</strong> “I’d define grain explicitly, build separate facts if needed, and use allocation logic only when business rules require splitting order-level costs to lines.”</p><p><strong>Follow-up:</strong> “Chasm trap?” — Two fact tables at different grains joined through a shared dimension can produce incorrect results. Use bridge tables or aligned grains.</p><h3>Hands-On</h3><pre># dbt schema.yml — document grain explicitly<br>models:<br>  - name: fct_order_lines<br>    description: &quot;One row per order line item&quot;<br>    columns:<br>      - name: order_line_id<br>        tests:<br>          - unique<br>          - not_null</pre><pre>-- Fan-out trap example (DON&#39;T do this)<br>-- fct_orders: grain = order (1 row per order)<br>-- fct_order_lines: grain = line (N rows per order)<br>-- Joining them on order_id without care → N duplicate order rows<br><br>-- Correct: pick one grain per query path, or use subqueries/aggregates<br>SELECT<br>    l.product_key,<br>    SUM(l.line_revenue) AS revenue<br>FROM fct_order_lines l<br>GROUP BY 1;</pre><h3>4. Slowly Changing Dimensions (SCD)</h3><p><strong>Analogy:</strong> Your friend’s address book.</p><p>When someone moves, you can:</p><ul><li><strong>Scratch out the old address and write the new one</strong> (history lost).</li><li><strong>Add a new entry and mark the old one “former”</strong> (history kept).</li><li><strong>Keep the old entry but add a “current city” note</strong> (partial history).</li></ul><p>Dimensions change slowly — a customer moves cities, a product gets renamed, a store rebrands. SCD is the rulebook for how your warehouse handles those changes.</p><p><strong>One sentence:</strong> Slowly Changing Dimensions (SCDs) define how dimension updates are stored when attribute values change over time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*f2Kix7nT5nPcfxvOhvK0Dg.png" /></figure><p><strong>the workhorse (most common in production):</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wnel0zk1yFmIrW48jiNe3g.png" /></figure><ul><li>Same customer_id, new customer_key per version.</li><li>Facts keep the key valid at transaction time — historical orders still show Paris.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*q3p6IrDQOu5RC2LfIY0r1A.png" /></figure><p><strong>Choosing a type:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/850/1*BjmpWnQOB3egmFNG1Wl28A.png" /></figure><p><strong>What breaks:</strong></p><ul><li>Type 2 explosion — every tiny attribute change creates a new row; dimension tables grow fast.</li><li>Late-arriving dimension changes — fact loaded before dimension update → wrong join or orphan key.</li><li>Retroactive corrections — source fixes a mistake from 6 months ago; Type 2 pipeline must backfill carefully.</li><li>Multiple concurrent versions — two rows with is_current = true due to buggy merge logic.</li></ul><p><strong>How teams fix it:</strong></p><ul><li>Hash diff / merge in dbt (dbt snapshot or custom SCD2 macro) — only new row when tracked columns change.</li><li>Limit Type 2 to attributes that matter for analytics (city, segment) — Type 1 for typos.</li><li>Late-arriving dimension handling — default row (Unknown), or reprocess facts after dim catch-up.</li><li>Tests: exactly one is_current = true per natural key.</li></ul><p><strong>dbt snapshots </strong>— built-in SCD Type 2 for sources that change:</p><pre># Tracks changes on check_cols; adds dbt_valid_from / dbt_valid_to<br>dbt snapshot --select snapshot_customers</pre><p><strong>Observability:</strong> SCD row growth rate, orphan FK count, snapshot lag vs source.</p><p><strong>Common questions:</strong></p><ul><li>“Explain SCD Type 1 vs Type 2.”</li><li>“A customer moves cities. How do you model it?”</li><li>“How do you preserve historical reporting after a dimension change?”</li></ul><p><strong>Walkthrough (customer moves city):</strong></p><ol><li>Identify natural key: customer_id.</li><li>Use Type 2: close old row (effective_to = yesterday, is_current = false).</li><li>Insert new row with new customer_key, updated city, is_current = true.</li><li>Existing facts keep old customer_key → reports by city at time of sale stay correct.</li></ol><p><strong>Follow-ups:</strong></p><ul><li>“Type 1 vs 2 trade-off?” — Type 1 simpler; Type 2 correct for historical analysis but more storage and pipeline complexity.</li><li>“What if product category changes?” — Type 2 if “revenue by category <em>as it was</em>” matters; Type 1 if only current category matters.</li></ul><h3>Hands-On</h3><pre>-- SCD Type 2 merge pattern (simplified)<br>MERGE INTO dim_customer AS target<br>USING staging_customer AS source<br>ON target.customer_id = source.customer_id AND target.is_current = true<br>WHEN MATCHED AND (<br>    target.city       IS DISTINCT FROM source.city OR<br>    target.segment    IS DISTINCT FROM source.segment<br>) THEN UPDATE SET<br>    effective_to = CURRENT_DATE - 1,<br>    is_current   = false<br>;<br><br>INSERT INTO dim_customer (customer_key, customer_id, city, segment,<br>                          effective_from, effective_to, is_current)<br>SELECT<br>    NEXTVAL(&#39;customer_key_seq&#39;),<br>    source.customer_id,<br>    source.city,<br>    source.segment,<br>    CURRENT_DATE,<br>    DATE &#39;9999-12-31&#39;,<br>    true<br>FROM staging_customer source<br>WHERE NOT EXISTS (<br>    SELECT 1 FROM dim_customer d<br>    WHERE d.customer_id = source.customer_id AND d.is_current = true<br>       AND d.city = source.city AND d.segment = source.segment<br>);</pre><pre>-- dbt snapshot output columns (Type 2)<br>-- dbt_scd_id, dbt_valid_from, dbt_valid_to, dbt_updated_at</pre><h3>5. Star Schema vs Snowflake Schema</h3><p><strong>Analogy:</strong> Organizing a store display.</p><ul><li><strong>Star:</strong> one big product tag with everything on it: name, category, brand. Quick to read.</li><li><strong>Snowflake: </strong>product tag points to a category tag, which points to a department tag. More organized folders, more steps to read.</li></ul><p><strong>One sentence:</strong> A star schema denormalizes dimensions (flat tables); a snowflake schema normalizes dimensions into hierarchies of linked tables.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5TOWuulFxgwhm7_D-3ff1A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dM5s6PrbHVoG6rfJR-agQw.png" /></figure><p><strong>Example — product hierarchy:</strong></p><p>Star (dim_product):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1014/1*vgxEQz-U6AznoI-9Eamfjw.png" /></figure><p>Snowflake:</p><ul><li>dim_product → category_key</li><li>dim_category → department_key</li><li>dim_department</li></ul><p><strong>Kimball vs Inmon (brief context):</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1nwrZFl_xjwvJ6vQPfYWEA.png" /></figure><p>Most modern cloud warehouses + dbt teams lean Kimball / star for marts because analyst productivity and query performance matter more than saving dimension storage.</p><p><strong>What breaks with snowflake:</strong></p><ul><li>Join explosion — BI tool generates 5-way joins; query planner struggles; p95 latency spikes.</li><li>Role-playing confusion — analysts don’t know which dim_region path to use.</li><li>Partial updates — category renamed in dim_category but stale copy in flat export somewhere.</li></ul><p><strong>What breaks with star:</strong></p><ul><li>Wide dimensions — 80 columns, most NULL; hard to maintain.</li><li>Redundant storage — category repeated on 10M product rows (usually cheap in columnar stores).</li><li>Inconsistent hierarchy — same category string spelled two ways in flat column.</li></ul><p><strong>How teams decide (practical rule):</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*a9kE7uc-EUpcy0L-7wHyvA.png" /></figure><p><strong>Modern compromise:</strong> Star schema in the <strong>mart layer</strong>; optional normalized staging/intermediate models upstream. Analysts see stars; engineers maintain clean normalized staging.</p><p><strong>Columnar warehouses</strong> (Snowflake, BigQuery) compress repeated dimension values well — star’s “redundancy” is often cheaper than snowflake’s extra joins.</p><p><strong>Common questions:</strong></p><ul><li>“Star schema vs snowflake schema?”</li><li>“Which would you use for a sales data mart?”</li><li>“What are the trade-offs?”</li></ul><p><strong>Strong answer:</strong> “Default to star for marts — fewer joins, simpler for BI, good performance on columnar warehouses. I’d snowflake only when a shared hierarchy must be maintained in one place or source is already normalized and churn is high. Often staging is normalized, marts are star.”</p><p><strong>Follow-ups:</strong></p><ul><li>“Galaxy schema?” — Multiple facts sharing dimensions (constellation). Common in real warehouses.</li><li>“Wide table / OBT?” — One Big Table (all dims denormalized into fact). Fast for ML/feature stores; painful for consistency.</li></ul><p><strong>Design exercise:</strong> “Model retail sales.”</p><ol><li>Grain: fct_sales = one row per transaction line.</li><li>Dimensions: date, store, product, customer (star — flat attrs).</li><li>SCD Type 2 on dim_customer city/segment; Type 1 on name typos.</li><li>Conformed dim_date across sales and returns facts.</li></ol><h3>Hands-On</h3><pre>-- Star schema query (simple path)<br>SELECT<br>    d.year,<br>    s.region,<br>    p.category,<br>    SUM(f.revenue) AS total_revenue<br>FROM fct_sales f<br>JOIN dim_date    d ON f.date_key    = d.date_key<br>JOIN dim_store   s ON f.store_key   = s.store_key<br>JOIN dim_product p ON f.product_key = p.product_key<br>WHERE d.year = 2024<br>GROUP BY 1, 2, 3;</pre><pre>-- Snowflake schema query (extra joins)<br>SELECT<br>    d.year,<br>    r.region_name,<br>    c.category_name,<br>    SUM(f.revenue) AS total_revenue<br>FROM fct_sales f<br>JOIN dim_date      d ON f.date_key      = d.date_key<br>JOIN dim_store     s ON f.store_key     = s.store_key<br>JOIN dim_region    r ON s.region_key    = r.region_key<br>JOIN dim_product   p ON f.product_key   = p.product_key<br>JOIN dim_category  c ON p.category_key  = c.category_key<br>WHERE d.year = 2024<br>GROUP BY 1, 2, 3;</pre><h3>6. How It All Fits Together</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3vgobI6oPZZmstAdY8tj4w.png" /></figure><p><strong>End-to-end checklist for any new mart:</strong></p><ol><li><strong>Define grain</strong> — what does one row mean?</li><li><strong>Identify facts </strong>— what events produce measurable rows?</li><li><strong>Identify dimensions </strong>— what context do analysts need?</li><li><strong>Choose SCD policy</strong> per dimension attribute (Type 1 vs 2).</li><li><strong>Choose star vs snowflake</strong> — default star in marts.</li><li><strong>Add tests</strong> — unique grain, not null keys, reconciliation to source.</li><li><strong>Document</strong> — grain statement, SCD policy, ownership.</li></ol><h3>Conclusion:</h3><p>Data modeling is not about drawing pretty diagrams — it is about making business questions answerable without lying.</p><p>You covered the building blocks:</p><ul><li>Fact tables record what happened and how much.</li><li>Dimension tables explain who, what, where, and when.</li><li>Grain is the contract that prevents double-counting and broken joins.</li><li>SCDs define how you handle change without rewriting history.</li><li>Star vs snowflake is a trade-off between simplicity and normalization — and in most modern warehouses, star wins in the mart layer.</li></ul><p>These concepts sit on top of the fundamentals you learned earlier (databases, caching, APIs). Before you jump into Spark, Flink, or lakehouse buzzwords, get comfortable here. Every pipeline you build eventually lands in tables with a grain. Every dashboard metric is only as honest as the fact table beneath it. Every “why don’t our numbers match Finance?” incident traces back to grain, SCD policy, or a fan-out join someone didn’t catch.</p><p>Learn grain before you learn Spark. Learn dimensions before you learn dbt macros.</p><p>The tools change constantly. Kimball’s dimensional modeling ideas have survived three decades of technology churn because they match how humans ask questions: <em>“How much did we sell, by region, by month, compared to last year?”</em> That question is a fact, dimensions, and a time attribute — not a novel about Kafka.</p><p>Revisit this lesson when you design your first mart, debug a reconciliation mismatch, or walk into a system design interview. The patterns compound. The next topics — ETL pipelines, warehouses, and streaming — all assume you know what you’re modeling and why.</p><p>Keep learning. Get the model right first; the pipeline is the easy part once the shape of truth is clear.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f5b606a0d59d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Core System Design Foundations for data engineers]]></title>
            <link>https://medium.com/@chiheb.mhamdi/core-system-design-foundations-for-data-engineers-32947fbb1bd7?source=rss-148acfe082b4------2</link>
            <guid isPermaLink="false">https://medium.com/p/32947fbb1bd7</guid>
            <category><![CDATA[system-design-interview]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[MHAMDI Chiheb]]></dc:creator>
            <pubDate>Mon, 22 Jun 2026 16:52:49 GMT</pubDate>
            <atom:updated>2026-06-22T16:52:49.624Z</atom:updated>
            <content:encoded><![CDATA[<p>Scalable data systems stay stateless, cache aggressively, choose consistency trade-offs consciously, decouple with queues, pick the right database for the job, and expose clear APIs. These six ideas show up in almost every system design interview and every production data platform. Master them first — everything else (Kafka pipelines, lakehouses, CDC) builds on top.</p><h3>1. Statelessness</h3><p>Analogy: A food truck with no memory.</p><p>Every customer orders from scratch. The truck doesn’t remember what you ordered yesterday. If the truck breaks down, another identical truck can take its place , no special knowledge required.</p><p>A stateless server treats every request as independent and stores no session data in memory between requests.</p><p>Tiny example: You hit GET /users/42 twice. The server doesn&#39;t &quot;remember&quot; you from the first call. It looks up user 42 from a database (or cache) both times.</p><p>==&gt; Any server can handle any request. That’s the point.</p><h4>Stateful vs stateless:</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*k4kw3dEFrN5oxA0vd_IWBQ.png" /></figure><p>Where state actually lives in stateless systems:</p><ul><li>Database — user profiles, orders, pipeline metadata</li><li>Cache (Redis) — sessions, rate-limit counters, hot keys</li><li>Client — JWT tokens, cookies with opaque session IDs</li><li>Object storage — files, parquet datasets, model artifacts</li></ul><p><strong>Why data engineering cares:</strong> Spark executors, Flink task managers, and Airflow workers are designed to be replaceable. If a worker dies mid-job, the orchestrator reschedules the task. State lives in the metastore, checkpoint, or object storage — not in the worker’s head.</p><p><strong>What breaks:</strong></p><ul><li>Sticky sessions by accident — load balancer pins user to one pod; that pod becomes a hotspot.</li><li>Local disk as “temporary” state — pod restarts, data vanishes. Seen in poorly designed batch jobs writing to /tmp.</li><li>In-memory dedup — works on one instance; duplicates appear when you scale to ten.</li></ul><p><strong>How teams fix it:</strong></p><ul><li>Externalize all durable state (S3, HDFS, warehouse tables).</li><li>Use idempotency keys on writes so retries are safe.</li><li>Design pipelines so any worker can replay from a checkpoint.</li></ul><p><strong>Observability:</strong> Track per-instance memory growth. If one pod’s RAM climbs while others stay flat, you probably leaked state into the process.</p><p><strong>Common questions:</strong></p><ul><li>“Why should API servers be stateless?”</li><li>“Where would you store user session data?”</li><li>“How does statelessness help during deployments?”</li></ul><p>Strong answer structure:</p><ol><li>Stateless servers scale horizontally , add/remove instances without migration.</li><li>Rolling deploys and auto-healing work because any instance can serve any request.</li><li>State moves to shared stores with defined consistency and backup policies.</li></ol><p><strong>Follow-up:</strong> “Is Kafka stateless?” — Brokers are stateful (they hold partitions on disk), but consumers achieve resilience via consumer groups and offset commits stored in Kafka or an external store. Nuance matters.</p><p>At 10x scale: Shard state by tenant or key range. Stateless compute + partitioned state is the default pattern.</p><h3>Hands-On</h3><pre># BAD: in-memory session store (does not scale)<br>sessions = {}<br><br>def get_cart(user_id):<br>    return sessions.get(user_id, [])<br><br># GOOD: external store<br>def get_cart(user_id):<br>    return redis.get(f&quot;cart:{user_id}&quot;) or []</pre><h3>2. Caching</h3><p><strong>Analogy: </strong>Keeping frequently used tools on your desk instead of walking to the warehouse every time.</p><p>The warehouse (database) has everything, but the trip is slow. Your desk (cache) holds what you need right now — but it might be slightly out of date.</p><p>One sentence: Caching stores copies of data closer to where it’s needed so reads are faster, trading freshness for speed.</p><p>Tiny example: A dashboard shows “total users: 1,000,042.” That number is cached for 60 seconds instead of running SELECT COUNT(*) on a billion-row table every page load.</p><p><strong>Three common cache layers:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ywUXPsYKDQiYSm1Rq3Nj3w.png" /></figure><p><strong>Cache patterns:</strong></p><ul><li>Cache-aside: app checks cache; on miss, reads DB and populates cache.</li><li>Write-through : write to cache and DB together.</li><li>Write-behind : write to cache first, flush to DB asynchronously (fast, riskier).</li></ul><p><strong>Common questions:</strong></p><ul><li>“Design a caching layer for a read-heavy API.”</li><li>“How do you keep cache and database consistent?”</li><li>“When would you NOT cache?”</li></ul><p><strong>When NOT to cache:</strong></p><ul><li>Financial balances requiring strong consistency.</li><li>Rapidly changing data (live bidding, stock ticks).</li><li>Low-traffic endpoints where cache overhead exceeds benefit.</li></ul><h3>Hands-On</h3><pre>import json<br>import redis<br><br>r = redis.Redis()<br><br>def get_daily_revenue(date: str) -&gt; float:<br>    key = f&quot;revenue:{date}&quot;<br>    cached = r.get(key)<br>    if cached:<br>        return float(cached)  # cache hit<br><br>    # cache miss — expensive query<br>    revenue = db.query(&quot;SELECT SUM(amount) FROM orders WHERE date = %s&quot;, date)<br>    r.setex(key, 3600, str(revenue))  # TTL 1 hour<br>    return revenue</pre><pre># CDN / browser caching via HTTP headers<br>Cache-Control: public, max-age=300, stale-while-revalidate=60<br>ETag: &quot;abc123&quot;</pre><h3>3. CAP Theorem</h3><p><strong>Analogy:</strong> A group chat during a network outage.</p><ul><li>Consistency — everyone sees the same messages in the same order.</li><li>Availability — everyone can always send and read messages.</li><li>Partition tolerance — the system keeps working even when some people lose connection to each other.</li></ul><p>When the network splits the group, you can’t guarantee <em>both</em> perfect consistency and instant availability. You pick.</p><p><strong>One sentence:</strong> In a distributed system, when nodes can’t talk to each other (a partition), you must choose between strong consistency and availability.</p><p><strong>Tiny example:</strong> Two data centers lose connectivity. Do you block writes until they reconnect (CP), or accept writes on both sides and reconcile later (AP)?</p><p>CAP in plain terms:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sT0G5p_ZIpErWKQ7A2xO2g.png" /></figure><p><strong>Important nuance:</strong> CAP applies during a partition. When the network is healthy, you can have both C and A.</p><p><strong>Common questions:</strong></p><ul><li>“What is CAP? Is it still relevant?”</li><li>“Is your system CP or AP?”</li><li>“How do you handle consistency across regions?”</li></ul><p><strong>Strong answer:</strong> “We choose per feature. Payment ledger = strong consistency (CP behavior). Product catalog = available reads with seconds of lag (AP behavior). Pipelines use at-least-once delivery with idempotent sinks.”</p><p><strong>Follow-ups:</strong></p><ul><li>“What’s eventual consistency?” — Replicas converge over time; reads may be stale briefly.</li><li>“What’s linearizability?” — Strongest single-object consistency; reads appear instantaneous and global.</li></ul><p><strong>At 10x scale:</strong> Per-region active-active with conflict resolution, or active-passive with failover and defined data loss window.</p><h3>Hands-On</h3><pre>-- PostgreSQL: read-your-writes via primary<br>SET SESSION CHARACTERISTICS AS TRANSACTION READ WRITE;<br>SELECT * FROM orders WHERE id = 8812;<br><br>-- Cassandra: tunable consistency per query<br>-- QUORUM = majority of replicas must agree<br>SELECT * FROM events WHERE user_id = 42;<br>-- consistency level set at driver: LOCAL_QUORUM</pre><h3>4. Message Queues</h3><p><strong>Analogy:</strong> A post office between you and a busy friend.</p><p>You drop a letter in the mailbox (producer). You don’t wait for your friend to read it. The post office holds it until they’re ready (consumer). If your friend is sick, letters pile up safely — they catch up later.</p><p><strong>One sentence:</strong> Message queues let services communicate asynchronously by buffering messages between senders and receivers.</p><p><strong>Tiny example:</strong> A user uploads a video. The API puts {&quot;video_id&quot;: 99} on a queue and returns 202 Accepted. A worker transcodes the video in the background.</p><p><strong>Core concepts:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/964/1*JxBwB1tn-ZPg1XDSocgoxQ.png" /></figure><p><strong>Kafka vs SQS (when teams pick what):</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kAGoifF8cio6ZuvxO2kBaQ.png" /></figure><p><strong>Delivery guarantees:</strong></p><ul><li>At-most-once — may lose messages; no duplicates.</li><li>At-least-once — may duplicate; none lost (most common; pair with idempotency).</li><li>Exactly-once — hardest; Kafka transactions + idempotent producers + dedup sinks.</li></ul><p><strong>Async workflow pattern:</strong></p><pre>Event → Queue → Worker → Downstream queue → Aggregator → Warehouse table</pre><p>Each stage scales independently. Backpressure shows up as growing queue lag.</p><p><strong>What breaks:</strong></p><ul><li>Poison messages — bad payload crashes consumer in a loop.</li><li>Consumer lag — processing slower than arrival rate; data gets stale.</li><li>Ordering assumptions — scale to 10 partitions; global order is gone.</li><li>Duplicate processing — consumer crashes after work but before ack.</li></ul><p><strong>How teams fix it:</strong></p><ul><li>DLQ + alert on DLQ depth.</li><li>Idempotent consumers — INSERT ... ON CONFLICT, dedup keys.</li><li>Backpressure — pause polling, scale consumers, or shed load.</li><li>Schema registry — enforce Avro/Protobuf contracts on Kafka topics.</li></ul><p><strong>Data engineering patterns:</strong></p><ul><li>CDC — Debezium publishes row changes to Kafka; Flink/Spark consumes.</li><li>Medallion architecture — bronze (raw events) → silver (cleaned) → gold (aggregates), each stage a topic or table.</li><li>Late data — watermark policies in stream processors.</li></ul><p><strong>Metrics:</strong> consumer lag, publish rate, processing latency p99, DLQ rate, rebalance frequency.</p><p><strong>Common questions:</strong></p><ul><li>“Kafka vs RabbitMQ vs SQS?”</li><li>“How do you handle failed messages?”</li><li>“Design an event-driven order pipeline.”</li></ul><p>Design walkthrough (order placed):</p><ol><li>API writes order to DB, publishes OrderPlaced event.</li><li>Inventory service consumes, reserves stock.</li><li>Payment service consumes, charges card.</li><li>Fulfillment service consumes, creates shipment.</li><li>Each service owns its data; queue provides decoupling and burst absorption.</li></ol><p><strong>Follow-ups:</strong></p><ul><li>“How do you guarantee order?” — single partition keyed by order_id.</li><li>“How do you replay?” — Kafka: new consumer group from earliest. SQS: not built-in; re-publish from event store.</li></ul><p><strong>At 100x scale:</strong> Tiered storage in Kafka, compacted topics for changelog, separate clusters for operational vs analytics traffic.</p><h3>Hands-On:</h3><pre># Producer pseudocode (Kafka)<br>producer.send(<br>    topic=&quot;orders&quot;,<br>    key=order_id,          # same key → same partition → ordering<br>    value={&quot;event&quot;: &quot;OrderPlaced&quot;, &quot;order_id&quot;: order_id},<br>    headers={&quot;trace_id&quot;: trace_id},<br>)<br><br># Consumer with idempotency<br>def handle(msg):<br>    if already_processed(msg.id):<br>        return<br>    process(msg)<br>    mark_processed(msg.id)<br>    commit_offset()</pre><h3>5. API Design</h3><p><strong>Analogy:</strong> A restaurant menu with clear names, prices, and rules.</p><p>You don’t guess how to order. The menu (API contract) says what’s available, what you need to provide, and what you get back. If the menu changes, regulars appreciate a note (“v2: spicy option renamed”).</p><p><strong>One sentence:</strong> APIs are formal contracts that let systems exchange data reliably , design, versioning, and consistency determine how safe changes are over time.</p><p><strong>Tiny example:</strong></p><pre>GET /api/v1/users/42        → 200 { &quot;id&quot;: 42, &quot;name&quot;: &quot;Ada&quot; }<br>POST /api/v1/orders         → 201 { &quot;order_id&quot;: 8812 }</pre><p><strong>REST vs GraphQL:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KLZaO5w5ql93LrobO3UytQ.png" /></figure><p><strong>Versioning strategies:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Wqtmfrw_VkM_y38wnlN-YQ.png" /></figure><p><strong>Contracts (why data teams care):</strong></p><ul><li>OpenAPI / Swagger — REST schema, generates docs and clients.</li><li>Protobuf / Avro — binary schemas for gRPC and Kafka.</li><li>JSON Schema — validate payloads at the edge.</li></ul><p>Breaking change = removing a field, changing a type, or renaming without backward compatibility. Non-breaking = adding optional fields.</p><p><strong>HTTP semantics that matter:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/610/1*jHW-qhS2z4inxKsA1vQmzQ.png" /></figure><p><strong>What breaks:</strong></p><ul><li>Undocumented breaking changes — mobile app crashes in production.</li><li>No rate limits — one client takes down the API.</li><li>Chatty GraphQL — N+1 queries without DataLoader.</li><li>Giant payloads — returning 10 MB JSON when client needed one field.</li></ul><p><strong>How teams fix it:</strong></p><ul><li>API gateway — auth, rate limiting, request validation.</li><li>Deprecation policy — sunset headers, migration guides, minimum support window.</li><li>Contract testing — Pact, schema diff in CI.</li><li>Idempotency-Key header on POST for safe retries.</li></ul><p><strong>Data engineering APIs:</strong></p><ul><li>Ingestion endpoints (POST /events) with batch support and backpressure (429 + Retry-After).</li><li>Export endpoints with async jobs (202 + polling or webhook).</li><li>Query APIs over warehouse with cost guards (row limits, query timeouts).</li></ul><p><strong>Observability:</strong> request rate, error rate, latency histograms per route, payload size, schema validation failures.</p><p><strong>Common questions:</strong></p><ul><li>“Design a REST API for a URL shortener.”</li><li>“REST vs GraphQL — when would you pick each?”</li><li>“How do you version APIs without breaking clients?”</li></ul><p><strong>URL shortener API sketch:</strong></p><pre>POST   /v1/links          { &quot;url&quot;: &quot;...&quot;, &quot;custom_slug&quot;: &quot;optional&quot; }<br>GET    /v1/links/{slug}   → redirect or metadata<br>GET    /v1/links/{id}/stats?from=&amp;to=<br>DELETE /v1/links/{id}</pre><p>Requirements → estimate QPS → stateless API + Redis cache hot slugs + Postgres durable store + async click aggregation via queue.</p><p><strong>Follow-ups:</strong></p><ul><li>“How handle auth?” — API keys for partners, OAuth for users, mTLS service-to-service.</li><li>“How document?” — OpenAPI spec published; changelog per version.</li></ul><p><strong>At 10x scale:</strong> GraphQL federation or BFF per client, edge caching for GET, async exports for heavy queries.</p><h3>Hands-On</h3><pre># OpenAPI snippet<br>paths:<br>  /v1/users/{id}:<br>    get:<br>      parameters:<br>        - name: id<br>          in: path<br>          required: true<br>          schema:<br>            type: integer<br>      responses:<br>        &#39;200&#39;:<br>          description: User found<br>          content:<br>            application/json:<br>              schema:<br>                $ref: &#39;#/components/schemas/User&#39;<br>        &#39;404&#39;:<br>          description: User not found</pre><pre># GraphQL — client specifies fields<br>query {<br>  user(id: 42) {<br>    name<br>    orders(last: 5) {<br>      id<br>      total<br>    }<br>  }<br>}</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=32947fbb1bd7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why Your Fast Query Suddenly Got Slow: Storage & Retrieval Made Simple]]></title>
            <link>https://medium.com/@chiheb.mhamdi/why-your-fast-query-suddenly-got-slow-storage-retrieval-made-simple-1eae5b875f83?source=rss-148acfe082b4------2</link>
            <guid isPermaLink="false">https://medium.com/p/1eae5b875f83</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[database]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <dc:creator><![CDATA[MHAMDI Chiheb]]></dc:creator>
            <pubDate>Tue, 14 Apr 2026 17:04:40 GMT</pubDate>
            <atom:updated>2026-04-14T17:04:40.098Z</atom:updated>
            <content:encoded><![CDATA[<p>Your app feels fast. Then one day a tiny feature lands, traffic rises, and one endpoint goes from 90ms to 1.4s.</p><p>You check CPU. Fine.<br>You check memory. Fine.<br>You blame the framework. Wrong layer.</p><p>Most of the time, this is a storage layout problem:<br>• the database reads too many pages,<br>• compaction is rewriting too much data,<br>• or your query shape no longer matches your indexes.</p><p>“What is my database actually doing on disk when I run this query?”</p><p>In this article: <br><strong>1.Quick idea in 30 seconds<br>2.A simple running example<br>3.Log-structured storage (LSM) in plain English<br>4.How B-tree lookups work<br>5.Why WAL exists (and why crashes don’t ruin everything)<br>6.LSM vs B-tree: which one should you use?<br>7.Indexes without confusion<br>8.OLTP vs OLAP (why one database can struggle to do both)<br>9.Row vs column storage<br>10.What to check first when performance drops<br>11.Final takeaway</strong></p><p><strong>1.Quick idea in 30 seconds</strong></p><p>Databases are just trade-offs:<br>• optimize for fast writes now, pay cleanup later;<br>• optimize for fast reads, pay more during writes;<br>• optimize for single-row transactions, or optimize for large analytics scans.</p><p>There is no perfect engine. There is only a good fit for your workload.</p><p><strong>2.A simple running example</strong></p><p>Imagine one table:</p><p>orders : order_id, user_id, amount, country, created_at, status, …70 more columns , And two queries:</p><p>• App query (OLTP):<br> `SELECT status FROM orders WHERE order_id = 91231`</p><p>• Analytics query (OLAP):<br> `SELECT country, SUM(amount) FROM orders WHERE created_at &gt;= … GROUP BY country`</p><p>These two queries need different storage behavior. Most performance problems happen when we treat them as if they are the same.</p><p><strong>3.Log-structured storage (LSM) in plain English</strong></p><p>Think of LSM like this:</p><p>• New writes go to memory first (fast).<br>• Memory flushes to sorted files on disk.<br>• Background compaction merges files and removes old versions.</p><p>Why teams like it:<br>• high write throughput.<br>• sequential disk behavior.<br>• good fit for heavy ingest pipelines.</p><p>What surprises teams:<br>• compaction spikes can hurt latency.<br>• reads may touch several files (read amplification).<br>• data can be rewritten multiple times (write amplification).</p><p>Simple translation:<br>• great when writes are intense.<br>• needs active tuning and monitoring.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Vq97JLqdy6bZcMHAN4k1zg.jpeg" /></figure><p><strong>4.How B-tree lookups work</strong></p><p>A B-tree is like a book index:<br>• start at the root.<br>• follow pointers by key range.<br>• land on the leaf page with your row.</p><p>This is why point lookups can be very efficient.</p><p>What can hurt:<br>• random insert patterns that trigger more page splits.<br>• flush/checkpoint behavior causing tail latency bumps.</p><p>Simple translation:<br>• often a strong fit for classic transactional systems with many small reads/writes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*m6xXDbUh5loHHAseJn4QgQ.jpeg" /></figure><p><strong>5.Why WAL exists (and why crashes don’t ruin everything)</strong></p><p>WAL = Write-Ahead Log.</p><p>Rule:<br>• Record the change in the log first.<br>• Then apply/update pages.</p><p>If the server crashes mid-update, replaying WAL restores consistency.</p><p>Simple translation:<br>• WAL is your crash safety net.<br>• Turning durability settings off for speed can make benchmarks look good and production recoveries painful.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PY477ksdUyiHK8EztWswcg.jpeg" /></figure><p><strong>6.LSM vs B-tree: which one should you use?</strong></p><p>Use this practical lens:</p><p>Choose LSM-like when:<br>• writes dominate.<br>• ingest speed matters.<br>• you can monitor and tune compaction.</p><p>Choose B-tree-like when:<br>• predictable point reads matter.<br>• OLTP behavior dominates.<br>• simpler operational read paths are preferred.</p><p>This is not religion. Measure with your real queries and data distribution.</p><p><strong>7.Indexes without confusion</strong></p><p>An index is a shortcut for reads, paid for by writes.</p><p>Three practical reminders:</p><p>• Composite index order matters<br> `(user_id, created_at)` helps query by `user_id` first; not always by `created_at` alone.</p><p>• Too many indexes hurt writes<br> Every insert/update must maintain each relevant index.</p><p>• Use EXPLAIN before guessing<br> It tells you if the planner is scanning, sorting, or using your intended index.</p><p><strong>8.OLTP vs OLAP (why one database can struggle to do both)</strong></p><p>OLTP:<br>• many small, fast, concurrent operations,<br>• millisecond-level user expectations.</p><p>OLAP:<br>• fewer but very heavy scans and aggregations,<br>• throughput over single-query latency.</p><p>Running big BI queries on the same primary OLTP path often creates lag and contention.</p><p>Simple translation:<br>• isolate heavy analytics workloads whenever possible.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6CaZvuDxxTKM-NerQBS6rw.png" /></figure><p><strong>9.Row vs column storage</strong></p><p>Row storage keeps all columns of one row together.<br>• great for fetching one record.</p><p>Column storage keeps values by column.<br>• great for analytics over a few columns (`SUM`, `AVG`, groupings),<br>• often compresses better.</p><p>For our `orders` example:<br>• app query by `order_id` likes row-oriented behavior.<br>• aggregate by `country` and `amount` loves column-oriented behavior.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UGziRgI9znZMKCdOpVtW1Q.jpeg" /></figure><p><strong>10.What to check first when performance drops</strong></p><p>Use this quick checklist:</p><p>• Run `EXPLAIN` on the slow query.<br>• Check if the query is doing a scan instead of an index seek.<br>• Check storage-specific background work (compaction/checkpoint).<br>• Check whether OLAP traffic is competing with OLTP.<br>• Check if data volume changed enough to invalidate old assumptions.</p><p>If you only watch CPU and ignore I/O patterns, you will miss the real bottleneck.</p><p><strong>11.Final takeaway</strong></p><p>Storage and retrieval is not “database internals trivia.”<br>It is the reason your API is fast or slow in production.</p><p>If you remember one line from this article, make it this:</p><p>Performance is usually the cost of reading or rewriting bytes , not the cost of writing SQL.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1eae5b875f83" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Streaming in Data Engineering: What Breaks in Production
(and How to Fix It)]]></title>
            <link>https://medium.com/@chiheb.mhamdi/streaming-in-data-engineering-what-breaks-in-production-and-how-to-fix-it-39e831521e5e?source=rss-148acfe082b4------2</link>
            <guid isPermaLink="false">https://medium.com/p/39e831521e5e</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[MHAMDI Chiheb]]></dc:creator>
            <pubDate>Fri, 10 Apr 2026 14:34:03 GMT</pubDate>
            <atom:updated>2026-04-10T14:34:03.109Z</atom:updated>
            <content:encoded><![CDATA[<p>Streaming systems look clean in architecture diagrams: events in, insights out, alerts in real time. Production is different. Once traffic increases and failures start, streaming pipelines face hard realities: late events, duplicate processing, schema drift, hot partitions, and replay chaos after incidents. Most teams do not fail because they chose the wrong tool. They fail because they treated streaming as a throughput problem instead of a correctness-and-recovery problem.This article breaks down streaming in data engineering through a production-first lens inspired by <strong>Designing Data-Intensive Applications</strong>. We cover how event streams, time semantics, joins, delivery guarantees, and operational controls behave under real load, and how to design pipelines that stay correct when systems are messy. If your team wants low latency and trustworthy data, this is the playbook.</p><h4><strong>In this article:</strong></h4><ol><li><strong>Why Most Streaming Designs Fail After Launch</strong></li><li><strong>What DDIA Means by Stream Processing</strong></li><li><strong>Brokers, Logs, and the Recovery Story</strong></li><li><strong>CDC and Event Sourcing: Two Ways to Treat Change as Data</strong></li><li><strong>Time Is the Hard Problem: Event Time, Windows, and Late Data</strong></li><li><strong>Joining Streams Without Lying to Yourself</strong></li><li><strong>Delivery Guarantees: What Exactly-Once Actually Means</strong></li><li><strong>Scale Realities: Partitioning, Backpressure, and Lag</strong></li><li><strong>Batch and Stream Are Partners, Not Rivals</strong></li><li><strong>Production Checklist Before You Call It Real-Time</strong></li><li><strong>Final Takeaway</strong></li></ol><h4>1. Why Most Streaming Designs Fail After Launch:</h4><p>A lot of teams implement streaming as if the architecture diagram is the system. It is not. The diagram is the intention. Production behavior is the truth. This is why Designing Data-Intensive Applications (DDIA) still feels fresh: it does not sell streaming as a feature. It treats streaming as a reliability discipline around logs, ordering, state, time, and failure recovery.</p><p>In real systems, failures are not exotic:<br>• a mobile client sends old events after reconnecting,<br>• a consumer restarts and replays data,<br>• a schema evolves and one service lags behind,<br>• one tenant key receives 80% of traffic and creates a hot partition,<br>• a slow downstream sink introduces lag that quietly destroys freshness.</p><p>If your architecture does not model these behaviors, you do not have a streaming system yet. You have a hopeful one.</p><h4>2. What DDIA Means by Stream Processing:</h4><p>DDIA frames stream processing as computation over unbounded input. Data keeps arriving and results must be updated continuously.</p><p>That changes the shape of engineering work:<br>• state must be persisted and recoverable,<br>• processing must tolerate reordering and duplicates,<br>• time semantics must be explicit,<br>• output correctness must be verified continuously, not once.</p><p>A useful mental model is:<br>event log → stateful processing → materialized serving view</p><p>That model is powerful because each stage can be tested and replayed. Replayability is the difference between “we can debug this incident” and “we are guessing.”</p><h4>3. Brokers, Logs, and the Recovery Story</h4><p>DDIA differentiates between traditional queue brokers and log-based brokers. This is not an implementation detail; it affects what recovery means.</p><p>Queue-oriented systems are often great for distributing tasks quickly.<br>Log-oriented systems are often better when you need durable history, reprocessing, and multiple independent consumers.</p><p>In data engineering, history matters:<br>• rebuilding a feature store,<br>• correcting a bug in enrichment logic,<br>• reconstructing a materialized view,<br>• auditing business events after a production incident.</p><p>If your transport does not preserve a useful replay model, recovery becomes manual and expensive.</p><h4>4. CDC and Event Sourcing: Two Ways to Treat Change as Data</h4><p>DDIA’s chapter on streams and databases highlights two patterns that age well in production.</p><p>1) Change Data Capture (CDC)<br>CDC emits database changes as events. It reduces the classic dual-write race: app writes DB successfully, then fails writing the event (or the opposite).</p><p>In practice, CDC is highly effective for:<br>• syncing OLTP changes into analytics pipelines,<br>• updating search indexes,<br>• feeding downstream microservices.</p><p>But CDC is not free. Lag becomes a first-class metric. If downstream consumers stall, your “real-time” system becomes a delayed one.</p><p>2) Event Sourcing<br>Event sourcing treats events as the source of truth and reconstructs state from event history.</p><p>Benefits:<br>• strong audit trail,<br>• clear temporal model,<br>• deterministic rebuilds.</p><p>Costs:<br>• schema evolution is harder over long event histories,<br>• correction flows require compensating events, not row mutation,<br>• replay cost grows unless you invest in snapshots and compaction strategies.</p><h4>5. Time Is the Hard Problem: Event Time, Windows, and Late Data</h4><p>Many streaming bugs are not throughput bugs. They are time bugs.</p><p>DDIA makes a crucial distinction:<br>• event time: when something happened in the real world,<br>• processing time: when your pipeline handled it.</p><p>Under ideal conditions, they are close. Under retry storms, regional outages, or offline clients, they diverge significantly.</p><p>If your business metric depends on event time but your pipeline aggregates on processing time, your results can be confidently wrong.</p><p>Window choices matter:<br>• tumbling windows are simple and predictable,<br>• hopping/sliding windows improve responsiveness but raise state/compute cost,<br>• session windows reflect user behavior but are harder to tune under load.</p><p>Practical safeguards:<br>• define watermark strategy up front,<br>• set explicit late data policy,<br>• monitor late-event ratios as a correctness signal.</p><h4>6. Joining Streams Without Lying to Yourself</h4><p>Joins are where elegance meets operational friction.</p><p>Stream-stream joins require buffering both sides and managing out-of-order arrivals. The wider the window, the larger the state and the memory pressure.</p><p>Stream-table joins are often more practical for enrichment workflows, but stale reference data can introduce silent drift.</p><p>Minimum discipline for joins:<br>• explicit join windows,<br>• state size limits and alerts,<br>• skew testing on high-cardinality and hotspot keys,<br>• replay-based validation with known-good datasets.</p><p>If you cannot explain what happens when one side of the join is delayed by 15 minutes, the join is not production-ready.</p><h4>7. Delivery Guarantees: What Exactly-Once Actually Means</h4><p>Teams often over-trust “exactly-once” claims.In many implementations, exactly-once applies within a specific processing engine and its internal state handling. End-to-end pipelines still cross independent systems: brokers, state stores, databases, caches, external APIs.That boundary matters because failures occur at boundaries.</p><p>What works in practice:<br>• design for at-least-once delivery.<br>• make sinks idempotent.<br>• use deterministic dedup keys with bounded retention.<br>• rehearse replay and rollback procedures.</p><p>DDIA’s transactions chapter mindset applies here: unknown commit outcomes and retries are normal. Engineer for safe repetition.</p><h4>8. Scale Realities: Partitioning, Backpressure, and Lag</h4><p>Throughput and correctness are connected.</p><p>Partition strategy defines:<br>• parallelism potential.<br>• ordering scope.<br>• skew risk.</p><p>A bad partition key can collapse cluster parallelism. One hot key then drives tail latency, lag, and checkpoint delays.Backpressure is another common blind spot. One slow sink can cause a chain reaction:<br>upstream queues grow → consumer lag rises → freshness degrades → on-call noise increases.</p><p>Observe what matters:<br>• consumer lag by partition,<br>• end-to-end event-time latency,<br>• dropped/late event counts,<br>• deserialization errors and schema mismatch rates,<br>• checkpoint duration and recovery times.</p><p>CPU-only monitoring cannot tell you whether your business numbers are correct.</p><h4>9. Batch and Stream Are Partners, Not Rivals</h4><p>DDIA positions batch and stream as complementary models.</p><p>Use stream for freshness and incremental updates.<br>Use batch for backfills, full recomputation, and reconciliation.</p><p>This hybrid posture is resilient:<br>• stream paths deliver low-latency value,<br>• batch paths repair drift and recover from logic bugs.</p><p>Architecturally, this avoids false choices and gives teams an operational safety net.</p><h4>10. Production Checklist Before You Call It Real-Time</h4><p>Before rollout, confirm these basics:</p><p>• Schema governance: compatibility checks and rollout discipline are automated.<br>• Time semantics: event-time logic, watermarks, and late-data policy are explicit.<br>• Idempotency: all critical sinks are safe under retries and replay.<br>• Replayability: deterministic reprocessing is documented and tested.<br>• Partition hygiene: hotspot risks are measured with realistic traffic.<br>• Backpressure controls: buffering, autoscaling, and circuit-breaker policies are defined.<br>• Correctness observability: data quality and lag SLOs are instrumented.</p><p>If two or more items are weak, reduce scope and harden foundations first.</p><h4>11. Final Takeaway</h4><p>The DDIA-inspired approach to streaming is straightforward:<br>build systems that remain correct when reality is messy. That means choosing replayable architectures, respecting time semantics, treating schemas as contracts, and designing for failure before launch.A streaming system is mature when your team can answer one question calmly during an incident:<br>“Can we recover this pipeline deterministically?” If yes, you are operating engineering. If no, you are operating hope.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=39e831521e5e" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>