<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Everything is MindGame on Medium]]></title>
        <description><![CDATA[Stories by Everything is MindGame on Medium]]></description>
        <link>https://medium.com/@everythingismindgame?source=rss-6e9afd5faba4------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*qtDl591aZqU0NOPo6834lA.png</url>
            <title>Stories by Everything is MindGame on Medium</title>
            <link>https://medium.com/@everythingismindgame?source=rss-6e9afd5faba4------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 21:26:26 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@everythingismindgame/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Language Models: From N-Grams to Transformers and Modern LLMs]]></title>
            <link>https://medium.com/@everythingismindgame/language-models-from-n-grams-to-transformers-and-modern-llms-602f877bbb6d?source=rss-6e9afd5faba4------2</link>
            <guid isPermaLink="false">https://medium.com/p/602f877bbb6d</guid>
            <category><![CDATA[transformers]]></category>
            <dc:creator><![CDATA[Everything is MindGame]]></dc:creator>
            <pubDate>Thu, 01 Oct 2026 17:36:28 GMT</pubDate>
            <atom:updated>2026-10-01T17:36:28.674Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tFyBVgo3Cic0jC-cVPqWqg.jpeg" /></figure><p>Language is full of patterns. When we read the phrase:</p><blockquote><em>“I am going to _”</em></blockquote><p>we can naturally guess possible next words such as <em>school</em>, <em>office</em>, <em>home</em>, or <em>market</em>. A language model is software trained to learn such statistical and semantic patterns in text, then use them to estimate what token is most likely to come next.</p><p>For example, a model may assign probabilities like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/907/1*QnFCqJRCViJQtcDWqf7lqg.png" /></figure><p>The model does not “know” the answer with certainty. Instead, it produces a probability distribution over its vocabulary, then selects or samples a token according to a decoding strategy.</p><p>This article explains the evolution from simple <strong>n-gram language models</strong> to Transformer-based <strong>Large Language Models (LLMs)</strong> such as GPT, Llama, Claude, Gemma, Phi, and Qwen.</p><h3>1. N-Gram Language Models</h3><p>An <strong>n-gram</strong> is a sequence of N consecutive words or tokens.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/912/1*RTEhBi7DKm13Y0QUNphYtg.png" /></figure><p>A bigram model estimates the next word using only one previous word:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/157/1*sR4SrhxPS6H8MgwlR9itFg.png" /></figure><p>A trigram model considers the previous two words:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/216/1*2iRwZn8FMbIaBoz0LFUwvw.png" /></figure><p>For example, after:</p><blockquote><em>“I like machine _”</em></blockquote><p>a trigram model may predict <em>learning</em> with a high probability because it has observed “machine learning” many times in its training data.</p><h3>The context problem</h3><p>N-gram models have a major limitation: they only use a fixed, short window of previous text.</p><p>Consider:</p><blockquote><em>“The orange is _”</em></blockquote><p>Possible completions could be:</p><ul><li>“ripe,” if <em>orange</em> refers to a fruit</li><li>“bright,” if <em>orange</em> refers to a color</li><li>“on the table,” if we are describing location</li></ul><p>The words “orange is” alone may not contain enough context to determine the intended meaning. Relevant information could have appeared much earlier in the paragraph.</p><h3>Why not simply use a very large N?</h3><p>At first, it might seem that using N=100N = 100N=100 or even N=1000N = 1000N=1000 would solve the context problem. In practice, it creates another problem: <strong>data sparsity</strong>.</p><p>A specific long phrase such as:</p><blockquote><em>“I like machine learning because it helps automate repetitive tasks”</em></blockquote><p>may occur very rarely — or never — in the training data. Even if shorter fragments are common, the exact long sequence is not.</p><p>As N increases:</p><ul><li>Exact sequences become rarer.</li><li>The model needs vastly more training data.</li><li>Storage requirements grow.</li><li>The model struggles to generalize to new wording.</li></ul><p>N-gram models are useful for simple baselines, spelling correction, autocomplete, and some speech-processing tasks, but they cannot reliably capture long-range meaning.</p><h3>2. What Is a Large Language Model?</h3><p>A <strong>Large Language Model</strong>, or LLM, is a neural-network-based language model trained on enormous collections of text and code. Like n-gram models, it predicts tokens. Unlike n-gram models, it can use a much larger context window and learn complex relationships between words, concepts, syntax, code, and documents.</p><p>A token is not always a complete word. Depending on the tokenizer, a token may be:</p><ul><li>A word, such as computer</li><li>Part of a word, such as trans and former</li><li>Punctuation, such as .</li><li>Whitespace or formatting symbols</li><li>Code fragments, such as function, =&gt;, or {</li></ul><p>For example, the word <em>unbelievable</em> might be represented as one token in one tokenizer and multiple subword tokens in another.</p><h3>Examples of LLMs</h3><p>Some widely used model families include:</p><p>CategoryExamplesOpen-weight modelsLlama, Gemma, Phi, Qwen, MistralProprietary modelsGPT, Claude, Gemini</p><p>The term “open source” is often used loosely in AI discussions. A more precise phrase is usually <strong>open-weight</strong>: the trained model weights may be available, while the training data, full training pipeline, or all licensing rights may not be fully open.</p><p>Meta’s Llama 3 family, for example, includes 8B and 70B parameter models and uses an autoregressive Transformer architecture.meta+1</p><h3>Why are LLMs powerful?</h3><p>LLMs can perform tasks that were traditionally handled by separate systems:</p><ul><li>Text completion and writing assistance</li><li>Translation</li><li>Summarization</li><li>Question answering</li><li>Code generation and debugging</li><li>Classification and extraction</li><li>Conversational interfaces</li><li>Document analysis</li></ul><p>This flexibility comes from pretraining: the model learns general language patterns before being adapted for specific use cases through instruction tuning, fine-tuning, retrieval, tools, or application-level prompting.</p><h3>3. The Challenges of Training LLMs</h3><p>Building an LLM is difficult because it requires much more than a large neural network.</p><h3>Training data</h3><p>A useful model needs a massive and diverse training corpus, often including:</p><ul><li>Books and articles</li><li>Websites</li><li>Documentation</li><li>Source code</li><li>Educational material</li><li>Multilingual text</li><li>Curated instruction-response examples</li></ul><p>The data must also be filtered for duplication, quality, safety concerns, privacy risks, and licensing constraints.</p><h3>Compute and parallelism</h3><p>Modern LLMs can contain billions or hundreds of billions of parameters. Training them requires large clusters of GPUs or specialized accelerators.</p><p>This creates distributed-systems challenges familiar to backend engineers:</p><ul><li>Splitting model computation across many devices</li><li>Synchronizing gradients efficiently</li><li>Handling node failures</li><li>Managing memory limits</li><li>Reducing network communication overhead</li><li>Keeping accelerators highly utilized</li></ul><p>Common approaches include data parallelism, tensor parallelism, pipeline parallelism, and optimizer-state sharding.</p><h3>Hallucinations</h3><p>LLMs generate text that is statistically plausible, not text that is automatically verified as true. A model can confidently produce incorrect facts, fake citations, broken code, or invented API behavior. This is commonly called <strong>hallucination</strong>.</p><p>For production systems, treat an LLM as a probabilistic component rather than an authoritative database. Important mitigations include:</p><ul><li>Retrieval-Augmented Generation (RAG)</li><li>Tool calls to trusted systems</li><li>Schema validation</li><li>Human review for high-stakes workflows</li><li>Citations and source grounding</li><li>Automated tests for generated code</li><li>Guardrails and output constraints</li></ul><h3>4. Transformers: The Architecture Behind LLMs</h3><p>Most modern LLMs are built using the <strong>Transformer</strong> architecture. The original Transformer introduced a design based on attention mechanisms and feed-forward layers, without recurrent neural networks or convolutions.arxiv+1</p><p>The core idea is simple:</p><blockquote><em>Each token can determine which other tokens are most relevant when building its contextual meaning.</em></blockquote><p>The original Transformer was designed as an <strong>encoder-decoder</strong> architecture, especially useful for sequence-to-sequence tasks such as translation:</p><blockquote><em>English: “How are you?”<br>French: “Comment allez-vous ?”</em></blockquote><p>The encoder reads the input sequence and produces contextual representations. The decoder generates the target sequence one token at a time while using both the encoder output and previously generated tokens.<a href="https://arxiv.org/html/1706.03762v7">arxiv</a></p><p>However, an important distinction is needed:</p><ul><li><strong>Encoder-only Transformers</strong> are common in understanding tasks, such as classification or embedding generation.</li><li><strong>Encoder-decoder Transformers</strong> are common in translation and text-to-text generation.</li><li><strong>Decoder-only Transformers</strong> are the most common architecture for autoregressive chat and completion LLMs, including the GPT-style and Llama-style families. Meta describes Llama 3 as a decoder-only Transformer architecture.meta+1</li></ul><h3>5. How a Transformer LLM Generates Text</h3><p>Let us use this input:</p><blockquote><em>“The brown dog sat next to the red fox.”</em></blockquote><p>A modern decoder-only LLM processes this prompt through multiple Transformer blocks, then predicts the next token.</p><h3>Step 1: Tokenization</h3><p>First, the input text is split into tokens.</p><p>Conceptually, the sentence may become:</p><pre>text[The] [brown] [dog] [sat] [next] [to] [the] [red] [fox] [.]</pre><p>In reality, the exact tokens depend on the tokenizer. A word might be divided into multiple pieces.</p><h3>Step 2: Token embeddings</h3><p>Every token ID is mapped to a learned vector called an <strong>embedding</strong>.</p><p>For illustration only:</p><pre>textdog → [0.10, 0.21, 0.05, 0.12, 0.07, ...]<br>fox →     [0.34, 0.11, 0.66, 0.08, 0.42, ...]</pre><p>These vectors have many dimensions — often hundreds or thousands — not five as in a toy example.</p><p>Initially, a token embedding represents broad learned properties. The vector for dog may reflect relationships to animals, pets, barking, and similar concepts. At this stage, it does not yet fully represent what <em>dog</em> means in this particular sentence.</p><h3>Step 3: Position information</h3><p>Attention alone does not inherently understand word order. Without positional information, these two sentences contain the same words:</p><blockquote><em>“Dog bites man.”<br>“Man bites dog.”</em></blockquote><p>But their meanings are obviously different.</p><p>The model therefore adds position information to each token representation. Depending on the architecture, this may use learned positional embeddings or methods such as rotary positional embeddings.</p><p>Conceptually:</p><p>Input representation=Token embedding+Position information\text{Input representation} = \text{Token embedding} + \text{Position information}Input representation=Token embedding+Position information</p><p>Now the model can distinguish the first the from the later the, and understand the order of brown dog versus red fox.</p><h3>Step 4: Self-attention</h3><p>Self-attention allows each token to determine which earlier tokens matter most for interpreting itself.</p><p>For the token dog, relevant tokens may include:</p><ul><li>brown, which describes the dog</li><li>sat, which tells us what the dog did</li><li>next to, which defines a relationship</li><li>fox, which identifies the other object in that relationship</li></ul><p>A simplified conceptual view could look like this:</p><pre>The brown dog sat next to the red fox.<br>    └─────┘ └─┘<br>   descriptor action</pre><p>In an autoregressive LLM, the model uses <strong>causal masking</strong>. This means a token can attend to previous tokens, but not future tokens. For example, while processing dog, the model must not look ahead at fox, because that would leak future information during next-token prediction.</p><p>The attention values are learned dynamically. The model does not permanently reserve “70% attention for adjectives” or “20% for verbs.” The weights depend on the prompt, the layer, and the attention head.</p><h3>Multi-head attention</h3><p>Transformers use multiple attention heads in parallel. Different heads can learn different types of relationships, such as:</p><ul><li>A noun and its adjective</li><li>A verb and its subject</li><li>Pronoun references</li><li>Long-distance dependencies</li><li>Punctuation and formatting patterns</li><li>Code syntax relationships</li><li>Positional or structural patterns</li></ul><p>The original Transformer architecture uses multi-head self-attention followed by position-wise feed-forward layers.<a href="https://arxiv.org/html/1706.03762v7">arxiv</a></p><p>A head is not manually assigned a role like “object-description head.” Such patterns may emerge during training, and they are not always cleanly interpretable.</p><h3>Step 5: Feed-forward network</h3><p>After attention combines information from relevant tokens, each token representation passes through a feed-forward neural network, often called an MLP block.</p><p>This is usually one of the largest parameter-heavy components in a Transformer block. It transforms each token’s contextual representation independently after attention has mixed information across tokens.</p><p>A simplified Transformer block includes:</p><pre>textInput token representations<br>          ↓<br>Masked multi-head self-attention<br>          ↓<br>Residual connection + normalization<br>          ↓<br>Feed-forward network / MLP<br>          ↓<br>Residual connection + normalization<br>          ↓<br>Updated token representations</pre><p>The model repeats these blocks many times. Lower layers may capture local syntax and token patterns, while deeper layers can represent more abstract semantic, structural, or task-relevant information.</p><h3>6. From Embeddings to the Next Token</h3><p>After the final Transformer layer, the model has an updated representation for each token position. To generate the next token, it uses the final representation at the last position.</p><p>Suppose the prompt is:</p><blockquote><em>“The brown dog sat next to the red fox”</em></blockquote><p>The model may estimate the next token distribution as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/905/1*Ul_lOz0DW1cJu64SaomA4A.png" /></figure><p>Internally, the model first produces raw scores called <strong>logits</strong>. A softmax function converts them into probabilities:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/241/1*7usSfbFkO_WotMHOwGxLXA.png" /></figure><p>where zi is the logit for token i.</p><p>The model then chooses one token, appends it to the context, and predicts again:</p><pre>textPrompt: The brown dog sat next to the red fox<br>Next token: and<br><br>New prompt: The brown dog sat next to the red fox and</pre><p>This continues until the model reaches a stopping condition, such as an end-of-sequence token, a configured maximum-token limit, or an application-defined stop sequence.</p><h3>Does it recompute everything from scratch?</h3><p>Conceptually, each newly generated token requires another forward pass through the model. But efficient inference systems generally use a <strong>KV cache</strong>.</p><p>The cache stores the key and value tensors from previous tokens, so the model does not need to recompute them at every generation step. This is crucial for fast chat and code-completion experiences.</p><p>The main consequence is that generation is still sequential:</p><pre>Generate token 1→Generate token 2→Generate token 3</pre><p>Unlike training, which can process many positions in parallel, autoregressive decoding produces tokens one at a time.</p><h3>7. Strategies for Choosing the Next Token</h3><p>The model’s probability distribution does not force a single decoding behavior. Applications can choose tokens in different ways.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/922/1*iRRKgrCzm0VrBgsgu4V4yw.png" /></figure><h3>Greedy decoding</h3><p>Greedy decoding selects the token with the highest probability:</p><pre>textand → 32%<br>in  → 25%<br>on  → 18%</pre><p>The model chooses and.</p><p>This is deterministic: the same input and settings produce the same output. It can be useful for constrained outputs, extraction, and predictable code-completion workflows, but it may generate bland or repetitive text.</p><h3>Top-K sampling</h3><p>Top-K sampling keeps only the k highest-probability tokens and samples from them.</p><p>If K=3K = 3K=3, the model may sample from:</p><pre>textand, in, on</pre><p>All lower-probability tokens are discarded.</p><h3>Top-P sampling</h3><p>Top-P sampling, also called <strong>nucleus sampling</strong>, chooses from the smallest set of tokens whose cumulative probability reaches a threshold PPP.</p><p>For instance, with P=0.75P = 0.75P=0.75:</p><pre>textand → 32%<br>in  → 25%<br>on  → 18%<br>--------------<br>Total = 75%</pre><p>The model samples from and, in, and on.</p><p>Top-P does <strong>not</strong> discard high-probability tokens. It usually discards the low-probability tail, which helps avoid sampling extremely unlikely tokens while allowing the candidate set to adapt to the confidence of the model.</p><h3>Temperature</h3><p>Temperature controls randomness before sampling.</p><ul><li>Lower temperature, such as 0.1–0.3: more deterministic</li><li>Moderate temperature, such as 0.6–0.9: balanced</li><li>Higher temperature, such as 1.0 or above: more varied and risky</li></ul><p>For a production backend workflow, lower temperature is usually preferable when output must be consistent, machine-readable, or safe to validate. Higher temperature is more useful for brainstorming, creative writing, or generating multiple alternative approaches.</p><h3>Final Thoughts</h3><p>Language models began with simple frequency-based systems that looked at a few nearby words. N-gram models were useful, but their fixed context window and inability to generalize limited what they could do.</p><p>Transformers changed this by allowing a model to dynamically connect tokens across a much larger context. Through token embeddings, positional information, self-attention, feed-forward networks, and repeated layers, LLMs build increasingly contextual representations of language.</p><p>At their core, though, modern LLMs still perform a deceptively simple task:</p><blockquote><em>Given the tokens so far, predict the next token.</em></blockquote><p>The impressive behavior — writing code, translating languages, answering questions, summarizing documents, and carrying on conversations — emerges from scaling that prediction task across vast data, large neural networks, and powerful distributed training infrastructure.</p><p>For more Machine Learning Topics:</p><p><a href="https://code-with-amitk.github.io/Machine%20Learning/">https://code-with-amitk.github.io/Machine%20Learning/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=602f877bbb6d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From KNN to HNSW: How Vector Search Scales for Semantic Search]]></title>
            <link>https://medium.com/@everythingismindgame/from-knn-to-hnsw-how-vector-search-scales-for-semantic-search-fdb715353b33?source=rss-6e9afd5faba4------2</link>
            <guid isPermaLink="false">https://medium.com/p/fdb715353b33</guid>
            <category><![CDATA[colbert]]></category>
            <category><![CDATA[ann]]></category>
            <category><![CDATA[knn-algorithm]]></category>
            <category><![CDATA[hnsw]]></category>
            <dc:creator><![CDATA[Everything is MindGame]]></dc:creator>
            <pubDate>Sun, 27 Sep 2026 15:43:57 GMT</pubDate>
            <atom:updated>2026-09-27T15:43:57.558Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OCueoGHmnzOjF12qL5oOVw.png" /></figure><p>Semantic search and RAG systems often represent text as <strong>embedding<br>vectors</strong> and then search for documents whose vectors are closest to a<br>query vector.</p><p>The basic idea is simple: convert the query and documents into vectors,<br>then find the nearest vectors. The challenge appears when the number of<br>documents becomes very large.</p><p>This article explains <strong>KNN, ANN, NSW and HNSW</strong>, followed by three<br>important encoder architectures: <strong>Bi-Encoder, Cross-Encoder and<br>ColBERT</strong>.</p><blockquote><strong><em>Source for KNN, ANN, NSW and HNSW:</em></strong><em> </em><a href="https://code-with-amitk.github.io/Machine%20Learning/ML_and_Types_Algorithms.html#algorithms"><em>Machine Learning Types &amp;<br>Algorithms</em></a></blockquote><h3>1. KNN — K Nearest Neighbors</h3><p><strong>KNN performs nearest-neighbor search with O(n) complexity.</strong></p><p>In semantic search, the query is converted into an embedding and<br>compared with the embeddings of the stored documents.</p><p>For example:</p><pre>User Query<br>    |<br>    v<br>Query Embedding<br>    |<br>    +---- compare with Document 1 embedding<br>    +---- compare with Document 2 embedding<br>    +---- compare with Document 3 embedding<br>    ...<br>    +---- compare with Document 1 Billion embedding</pre><p>If there are <strong>1 billion document embeddings</strong>, KNN may need to perform<br>comparisons against all 1 billion entries.</p><p>Therefore, although KNN is straightforward, it does not scale well for<br>very large vector collections.</p><p>The page describes keyword search and semantic search as using KNN-style<br>nearest-neighbor retrieval in a RAG retriever, and notes that production<br>retrievers use ANN when the collection becomes very large.</p><h3>2. ANN —Approximate Nearest Neighbors</h3><p><strong>ANN (Approximate Nearest Neighbors)</strong> reduces the amount of searching<br>required.</p><p>Unlike exact KNN, ANN does <strong>not guarantee that the absolute closest<br>document will be returned</strong>. Instead, it attempts to find a vector that<br>is very close to the query vector.</p><p>The trade-off is:</p><pre>KNN<br>Exact search<br>   |<br>   +--&gt; potentially very large number of comparisons<br>   |<br>   +--&gt; O(n)<br><br>ANN<br>Approximate search<br>   |<br>   +--&gt; fewer comparisons<br>   |<br>   +--&gt; much faster for large collections</pre><p>ANN is therefore useful when the vector database contains a very large<br>number of embeddings and an exhaustive comparison is too expensive.</p><p>Two ANN approaches described on the page are:</p><ol><li><strong>Navigable Small World (NSW)</strong></li><li><strong>Hierarchical Navigable Small World (HNSW)</strong></li></ol><h3>3. NSW —Navigable Small World</h3><p>NSW represents the documents as a <strong>proximity graph</strong>.</p><p>The basic idea is:</p><pre>Document embedding<br>       |<br>       v<br>      Node<br>       |<br>       +---- connected to nearby nodes<br>       |<br>       v<br>Proximity Graph</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/852/0*DjeETdJyTvT76yu7.png" /></figure><p>Each document embedding becomes a node in the graph, and nodes are<br>connected to nearby embeddings.</p><h3>How NSW search works</h3><ol><li>Every document is represented by an embedding in embedding space.</li><li>A proximity graph is created where each document embedding is<br>represented as a node.</li><li>Each node is connected to its nearest neighbors.</li><li>When a user query arrives, the query is converted into an embedding.</li><li>A candidate node is selected from the graph.</li><li>The algorithm examines the candidate’s nearby nodes.</li><li>If a neighbor is closer to the query vector, that neighbor becomes<br>the new candidate.</li><li>The process continues until no neighboring node is closer than the<br>current candidate.</li><li>The candidate vector is returned.</li><li>With a small modification, the approach can return multiple closest<br>vectors.</li></ol><p>An important point is that the <strong>proximity graph can be computed before<br>a user query arrives</strong>.</p><p><em>Figure: Navigable Small World — — image from the source page.</em></p><h3>4. HNSW —Hierarchical Navigable Small World</h3><p>HNSW extends the NSW idea by creating <strong>multiple hierarchical proximity<br>graphs</strong>.</p><p>Instead of searching the complete graph immediately, the search starts<br>with a much smaller graph and progressively moves toward the detailed<br>graph.</p><p>The page describes three layers:</p><pre>Layer 3<br>  |<br>  |  small graph<br>  v<br>Layer 2<br>  |<br>  |  intermediate graph<br>  v<br>Layer 1<br>  |<br>  |  complete graph<br>  v<br>Final nearest vectors</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*qDC7nZbdacpvpUrq.png" /></figure><h3>HNSW layers</h3><p><strong>Layer 1</strong></p><p>Contains all nodes and the complete proximity graph.</p><p><strong>Layer 2</strong></p><p>Contains a smaller subset of vectors and provides intermediate<br>navigation.</p><p><strong>Layer 3</strong></p><p>Contains an even smaller subset of vectors and provides fast navigation<br>at the highest level.</p><p>The source page illustrates this using approximately:</p><ul><li>Layer 1 → 1,000 vectors</li><li>Layer 2 → 100 vectors</li><li>Layer 3 → 10 vectors</li></ul><h3>How HNSW search works</h3><ol><li>Build the proximity graph containing all vectors.</li><li>Create smaller graphs by selecting subsets of vectors.</li><li>Start the search in the highest/smallest layer.</li><li>Find a candidate close to the query vector.</li><li>Move down to the next layer.</li><li>Refine the candidate using the larger graph.</li><li>Continue until reaching the complete graph.</li><li>Return the closest candidates found.</li></ol><p><em>Figure: Hierarchical Navigable Small World — — image from the source<br>page.</em></p><p>The page describes HNSW as <strong>O(log n)</strong> and highlights that it is very<br>fast compared with KNN.</p><h3>KNN vs ANN vs NSW vs HNSW</h3><p>Approach Main idea Search characteristic</p><p>KNN Compare query with Exact, O(n)<br>stored vectors</p><p>ANN Find approximately Faster approximate<br>nearest vectors search</p><p>NSW Navigate a proximity Graph-based ANN<br>graph</p><p>The progression can be viewed as:</p><pre>KNN<br> |<br> | Exact but expensive at large scale<br> v<br>ANN<br> |<br> | Approximate nearest-neighbor search<br> v<br>NSW<br> |<br> | Proximity graph<br> v<br>HNSW<br> |<br> | Hierarchical proximity graphs<br> v<br>Fast large-scale vector search</pre><h3>Encoders</h3><p>In Information Retrieval and Neural Search architectures, <strong>encoders</strong><br>generate vector representations that can be used for retrieval.</p><p>Bi-Encoders, Cross-Encoders and ColBERT use different strategies for<br>representing queries and documents and calculating their relevance.</p><h3>1. Bi-Encoder</h3><p>A <strong>Bi-Encoder</strong> encodes the prompt and documents separately.</p><p>The document embeddings can be calculated in advance and stored in a<br>vector database. When a prompt arrives, its embedding is generated and<br>searched against the stored document embeddings.</p><pre>Prompt                    Documents<br>  \/                        \/<br>|---------------------------------|<br>|       Embedding Model           |<br>|---------------------------------|<br>  \/                        \/<br>[0,5,0,2,1,3]         [1,4,1,5,7,9,9]<br>  \/                        \/<br>|-------------------------------------|<br>|            Bi-Encoder               |<br>| vector DB uses ANN to search/compare|<br>| prompt and document vectors         |<br>|-------------------------------------|</pre><p>The important advantage is that <strong>document embeddings can be<br>precomputed</strong>.</p><p>A typical retrieval flow is:</p><pre>Documents<br>    |<br>    v<br>Embedding Model<br>    |<br>    v<br>Document Vectors<br>    |<br>    v<br>Vector Database<br>    |<br>    |       Query<br>    |         |<br>    |         v<br>    |   Query Embedding<br>    |         |<br>    +---------+<br>              |<br>              v<br>        ANN Search<br>              |<br>              v<br>       Relevant Documents</pre><p>This makes the Bi-Encoder approach suitable for large-scale retrieval<br>systems.</p><h3>2. Cross-Encoder</h3><p>A <strong>Cross-Encoder</strong> processes the prompt and document together.</p><p>The prompt is combined with each candidate document:</p><pre>Prompt + Document<br>       |<br>       v<br>|---------------------------|<br>|       Cross Encoder       |<br>|---------------------------|<br>       |<br>       v<br> Relevancy Score</pre><p>The embedding model produces a <strong>relevancy score</strong> for the<br>prompt-document pair. Documents with higher relevance scores can then be<br>selected.</p><h3>Advantages</h3><ol><li>It can provide very high-quality relevance matching.</li><li>It can be used to improve or rerank results produced by other<br>retrieval techniques.</li></ol><h3>Disadvantages</h3><ol><li>If there are billions of documents, every document cannot<br>practically be paired with the prompt for scoring. The naive<br>approach has <strong>O(n)</strong> candidate evaluations.</li><li>Document preprocessing cannot be used in the same way as a<br>Bi-Encoder because the relevance calculation depends on both the<br>prompt and the document together.</li></ol><p>This makes Cross-Encoders particularly useful when applied to a<br><strong>smaller candidate set</strong>, such as documents already retrieved by an<br>initial search stage.</p><h3>3. ColBERT —Contextualized Late Interaction over BERT</h3><p>ColBERT uses a different strategy.</p><p>Instead of producing one embedding vector for the entire document,<br>ColBERT generates <strong>an embedding vector for each token</strong>.</p><p>The same approach is applied to the prompt.</p><pre>Document<br>    |<br>    v<br>|--------------------|<br>| Embedding Model    |<br>|--------------------|<br>    |<br>    v<br>[1,2,4,6,9,2]        // Token 1<br>[7,2,4,8,3,6]        // Token 2<br>[9,5,1,4,5,9]        // Token 3<br>[3,4,1,6,3,1]        // Token 4<br><br>Prompt<br>    |<br>    v<br>|--------------------|<br>| Embedding Model    |<br>|--------------------|<br>    |<br>    v<br>[3,4,6,0,1,4]        // Token 1<br>[5,6,9,3,1,3]        // Token 2</pre><p>The token-level vectors from the prompt can then be compared against<br>token-level vectors from the documents to determine similarity.</p><h3>Advantages</h3><ol><li>It combines the scalability benefits of separately encoded<br>representations with richer token-level matching.</li><li>It can be faster than a full Cross-Encoder and can support real-time<br>search scenarios.</li></ol><h3>Disadvantage</h3><p>The number of vectors that need to be stored is much higher because a<br>document produces multiple vectors instead of a single vector.</p><h3>Putting It All Together</h3><p>A modern neural retrieval pipeline can combine these ideas:</p><pre>DOCUMENTS<br>                    |<br>                    v<br>              Encoder Model<br>                    |<br>        +-----------+-----------+<br>        |                       |<br>   Bi-Encoder                ColBERT<br>        |                       |<br>   Document vectors       Token vectors<br>        |                       |<br>        +-----------+-----------+<br>                    |<br>                    v<br>              Vector Storage<br>                    |<br>                    v<br>              ANN / HNSW<br>                    |<br>                    v<br>           Candidate Documents<br>                    |<br>                    v<br>             Cross-Encoder<br>             / Reranker<br>                    |<br>                    v<br>             Final Results</pre><p>The key distinction is:</p><ul><li><strong>KNN</strong> → compare against everything.</li><li><strong>ANN</strong> → search approximately to reduce the search cost.</li><li><strong>NSW</strong> → use a proximity graph for navigation.</li><li><strong>HNSW</strong> → use multiple proximity-graph layers for faster<br>navigation.</li><li><strong>Bi-Encoder</strong> → encode query and document independently; document<br>vectors can be precomputed.</li><li><strong>Cross-Encoder</strong> → encode the query-document pair together for<br>relevance scoring.</li><li><strong>ColBERT</strong> → retain multiple token-level vectors and perform late<br>interaction.</li></ul><p>Together, these techniques explain an important part of modern<br><strong>semantic search and RAG retrieval</strong>: embedding generation, scalable<br>vector search, and relevance refinement can be separated into different<br>stages.</p><h3>Source</h3><p>KNN, ANN, NSW and HNSW content and the two diagrams in this article are<br>based on:</p><p><a href="https://code-with-amitk.github.io/Machine%20Learning/ML_and_Types_Algorithms.html#algorithms">Machine Learning Types &amp; Algorithms — -<br>code-with-amitk</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fdb715353b33" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Beyond GET and SET: How to Design a Distributed Redis Cache at Scale]]></title>
            <link>https://medium.com/@everythingismindgame/beyond-get-and-set-how-to-design-a-distributed-redis-cache-at-scale-b468ca3e1133?source=rss-6e9afd5faba4------2</link>
            <guid isPermaLink="false">https://medium.com/p/b468ca3e1133</guid>
            <category><![CDATA[redis]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <dc:creator><![CDATA[Everything is MindGame]]></dc:creator>
            <pubDate>Tue, 22 Sep 2026 08:28:11 GMT</pubDate>
            <atom:updated>2026-09-22T08:30:28.534Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CGJpugpYeA47W0BDW7UPZQ.png" /></figure><p>Caching is one of the simplest ideas in system design:</p><blockquote><em>Keep frequently accessed data closer to the application so that we don’t have to repeatedly access the database.</em></blockquote><p>But designing a cache for <strong>millions of requests per second</strong> introduces several interesting problems:</p><ul><li>How is data distributed across Redis nodes?</li><li>What happens when a Redis node crashes?</li><li>How do we handle a hot tenant or hot key?</li><li>What happens when a cache entry expires?</li><li>How do we maintain low p99 latency?</li><li>What happens when an application crashes after writing to Redis?</li></ul><p>This article walks through the design of a <strong>distributed Redis cache</strong>, from basic GET/SET operations to sharding, fault tolerance, consistency, memory efficiency, and latency.</p><h3>1. What Is a Cache?</h3><p>A cache stores frequently accessed data in a faster storage layer, typically <strong>RAM</strong>, instead of repeatedly fetching it from a database.</p><p>For example, suppose an application needs session information for every HTTP request.</p><p>Instead of querying the database every time:</p><pre>Application → Database → Session information</pre><p>we can store the session in Redis:</p><pre>Key:   abc123<br>Value: user_id=A123, role=admin, expires=10:30</pre><p>The request path becomes:</p><pre>Application → Redis → Session information</pre><p>RAM access is significantly faster than disk-based database access, making Redis useful for high-throughput workloads.</p><p>A cache commonly exposes a simple API:</p><pre>SET(key, value)<br>GET(key)<br>DELETE(key)</pre><p>The application generally treats the database as the <strong>source of truth</strong> and Redis as a temporary, performance-oriented copy.</p><h3>2. Why Do We Need a Distributed Cache?</h3><p>A single Redis server has finite CPU, memory, and network capacity.</p><p>Suppose the cache needs hundreds of terabytes of storage. Putting everything on one machine is impractical.</p><p>We therefore distribute the data across multiple Redis nodes:</p><pre>                    Application<br>                         |<br>                  Redis Client<br>                         |<br>          +--------------+--------------+<br>          |              |              |<br>       Shard 0         Shard 1        Shard 2<br>       Redis            Redis           Redis</pre><p>This is what <strong>distributed cache</strong> means in this context.</p><p>It does <strong>not necessarily mean that Redis in Region 1 continuously synchronizes its cache with Redis in Region 2</strong>.</p><p>A common architecture is:</p><pre>Region 1                         Region 2<br>Application                     Application<br>     |                                |<br> Redis Cluster                    Redis Cluster<br>     |                                |<br> Database                        Database</pre><p>Each region can maintain its own cache for the traffic it serves.</p><p>The database or another authoritative backend remains the source of truth.</p><p>If a cache entry is missing, the application can retrieve it from the database and populate the local cache again.</p><h3>3. Requirements</h3><p>Before designing the system, define the requirements.</p><h3>Functional requirements</h3><p>The cache should support:</p><pre>SET(key, value)<br>GET(key)<br>DELETE(key)</pre><p>It should also support:</p><ul><li>TTL / expiration</li><li>Cache eviction when memory is full</li><li>Cache refresh or invalidation</li></ul><h3>Non-functional requirements</h3><p>The important characteristics are:</p><p><strong>Low latency</strong></p><p>For example, target approximately 1 ms cache operation latency at the application level, depending on the deployment and workload.</p><p><strong>High throughput</strong></p><p>The cache should support very large numbers of operations per second.</p><p><strong>Fault tolerance</strong></p><p>Failure of a cache node should not bring down the application.</p><p><strong>Memory efficiency</strong></p><p>RAM is expensive, so unnecessary duplication and oversized values should be avoided.</p><p><strong>Eventual consistency</strong></p><p>The cache is generally allowed to temporarily differ from the database because the database is the source of truth.</p><h3>4. High-Level Architecture</h3><p>A simplified architecture looks like this:</p><pre>                    +----------------+<br>                    |  Application   |<br>                    +-------+--------+<br>                            |<br>                     Redis Client<br>                            |<br>                    +-------+--------+<br>                    | Redis Cluster  |<br>                    +-------+--------+<br>                     /       |       \<br>                    /        |        \<br>               +------+  +------+  +------+<br>               |Shard0|  |Shard1|  |Shard2|<br>               +------+  +------+  +------+<br>                  |         |         |<br>                 RAM       RAM       RAM</pre><p>The important component here is the <strong>Redis client library</strong>.</p><p>The application doesn’t necessarily need to know which Redis node owns a particular key.</p><p>The client calculates the appropriate shard and sends the request to that Redis node.</p><h3>5. SET: Writing Data to Redis</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*grSfuJcGAWDaiv1kuslH8A.png" /></figure><p>Consider:</p><pre>SET(abc123, user/session data)</pre><p>The request path is approximately:</p><pre>Application<br>     |<br>Redis Client<br>     |<br>Hash(key)<br>     |<br>Determine Redis shard<br>     |<br>Redis Pod<br>     |<br>RAM<br>     |<br>ACK</pre><p>The Redis client uses the key to determine where the data belongs.</p><p>For example:</p><pre>hash(&quot;abc123&quot;) → Shard 0</pre><p>The client sends:</p><pre>SET abc123 &lt;value&gt;</pre><p>to Shard 0.</p><p>Redis stores the value in memory and returns an acknowledgement.</p><h3>6. Choosing a Good Sharding Key</h3><p>Sharding strategy is one of the most important parts of a distributed cache.</p><p>Suppose our application is multi-tenant.</p><p>We could construct a key like:</p><pre>tenantA:customer123<br>tenantA:customer456<br>tenantB:customer789</pre><p>The tenant can become part of the hashing strategy.</p><p>The advantage is that related traffic can be routed predictably.</p><p>But this introduces another problem:</p><h3>Hotspots</h3><p>Imagine:</p><pre>Tenant A → 50x normal traffic<br>Tenant B → normal traffic<br>Tenant C → normal traffic</pre><p>If all Tenant A keys map to the same shard:</p><pre>Shard 0 → Tenant A → 50x traffic<br>Shard 1 → Tenant B<br>Shard 2 → Tenant C</pre><p>Shard 0 becomes a <strong>hot shard</strong>.</p><p>Adding capacity elsewhere doesn’t necessarily solve the problem.</p><p>Possible solutions include:</p><ul><li>Better key distribution</li><li>Splitting a large tenant across multiple shards</li><li>Adding more shards</li><li>Using a hashing strategy that distributes keys more evenly</li><li>Autoscaling based on workload</li></ul><p>The important lesson is:</p><blockquote><strong><em>More Redis nodes do not automatically solve a hotspot if the sharding strategy keeps sending traffic to one node.</em></strong></blockquote><h3>7. What Happens When a Redis Node Fails?</h3><p>Consider:</p><pre>Shard 0<br>   |<br>  CRASH</pre><p>If the cache has no redundancy, requests mapped to that shard will fail.</p><p>A common solution is to maintain replicas:</p><pre>             Shard 0<br>            /       \<br>       Primary     Replica</pre><p>If the primary fails, a replica can take over.</p><p>However, changing the set of nodes can cause keys to move between nodes.</p><p>A naive approach could cause a large amount of <strong>rehashing and data movement</strong>.</p><p>This is where techniques such as <strong>consistent hashing</strong> can be useful.</p><p>Instead of remapping almost every key when a node is added or removed, consistent hashing attempts to limit the number of keys that need to move.</p><h3>8. Application Failure and Idempotency</h3><p>Consider an application writing:</p><pre>Application → Redis</pre><p>The sequence could be:</p><pre>App<br> |<br> | SET(key, value)<br> v<br>Redis<br> |<br> | writes to RAM<br> |<br> X<br>App crashes</pre><p>Redis successfully stored the value, but the application crashed before receiving the ACK.</p><p>From the application’s perspective:</p><pre>Did SET succeed?</pre><p>It doesn’t know.</p><p>After restarting, the application may retry:</p><pre>Application-2<br>     |<br>     | SET(key, value)<br>     v<br>Redis</pre><p>This is where <strong>idempotency</strong> is valuable.</p><p>If performing the same operation multiple times results in the same final state, retrying is safe.</p><p>For example:</p><pre>SET(user123, session-data)</pre><p>can safely be repeated if overwriting the same value is acceptable.</p><h3>9. GET: Reading from Redis</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zUzP16Fjv2MlQBoq9F9UCw.png" /></figure><p>Now consider:</p><pre>GET(abc123)</pre><p>The request follows a similar path:</p><pre>Application<br>     |<br>Redis Client<br>     |<br>Hash(key)<br>     |<br>Shard 0<br>     |<br>Check RAM</pre><h3>Cache hit</h3><p>If the key exists:</p><pre>Redis → Value → Application</pre><p>No database query is required.</p><h3>Cache miss</h3><p>If the key doesn’t exist:</p><pre>Application<br>     |<br>     +----&gt; Redis → MISS<br>     |<br>     +----&gt; Database → Value<br>                         |<br>                         v<br>                       Redis</pre><p>The application retrieves the value from the database and populates Redis:</p><pre>SET(key, value)</pre><p>The next request can then be served from the cache.</p><p>This pattern is commonly called <strong>cache-aside</strong>.</p><h3>10. TTL and Expiration</h3><p>Cache data is usually not supposed to live forever.</p><p>For example:</p><pre>SET abc123 session-data EX 3600</pre><p>The entry expires after one hour.</p><p>TTL is useful because:</p><ul><li>stale data eventually disappears</li><li>memory can be reclaimed</li><li>temporary data doesn’t require explicit cleanup forever</li></ul><p>After expiration:</p><pre>GET abc123<br>       |<br>       v<br>     MISS<br>       |<br>       v<br>   Database<br>       |<br>       v<br>   SET in Redis</pre><p>The database remains the source of truth.</p><h3>11. DELETE and Cache Invalidation</h3><p>Suppose a user’s role changes in the database:</p><pre>Database:<br>user123 → role=user</pre><p>The cache might still contain:</p><pre>user123 → role=admin</pre><p>The application can invalidate the cache:</p><pre>DELETE(user123)</pre><p>The flow becomes:</p><pre>Application<br>     |<br>Redis Client<br>     |<br>Redis Shard<br>     |<br>Delete cache entry<br>     |<br>ACK</pre><p>The next GET becomes a cache miss, causing the application to fetch the latest value from the database and repopulate Redis.</p><p>This illustrates a fundamental cache principle:</p><blockquote><strong><em>Caching improves performance, but cache invalidation determines whether cached data remains useful and sufficiently fresh.</em></strong></blockquote><h3>12. Keeping p99 Latency Low</h3><p>Average latency is not enough for a high-scale system.</p><p>We care about <strong>p99 latency</strong>.</p><p>If p99 = 1 ms, it means roughly:</p><blockquote><em>99% of requests complete within 1 ms.</em></blockquote><p>The request path is:</p><pre>Application<br>     |<br>Redis Client<br>     |<br>Network<br>     |<br>Redis Pod<br>     |<br>RAM<br>     |<br>Network<br>     |<br>Application</pre><p>Several factors influence latency:</p><h3>Keep application and Redis close</h3><p>Place them in the same region and, where appropriate, the same availability zone.</p><p>This reduces network latency.</p><h3>Use persistent connections</h3><p>Repeatedly establishing connections adds overhead.</p><p>Connection pooling allows applications to reuse existing connections.</p><h3>Avoid expensive Redis operations</h3><p>Operations involving huge datasets or very large values can increase latency.</p><p>For example, avoid using expensive broad key scans as a normal request-path operation.</p><h3>Keep values reasonably small</h3><p>Large values consume memory and increase network transfer time.</p><h3>13. Memory Efficiency</h3><p>Redis stores more than just:</p><pre>key → value</pre><p>There is also internal metadata and data-structure overhead.</p><p>Memory usage can therefore be thought of approximately as:</p><pre>Memory =<br>    Key<br>  + Value<br>  + Object metadata<br>  + Hash-table overhead<br>  + Allocator overhead/fragmentation</pre><p>At large scale, small inefficiencies become expensive.</p><p>For example:</p><pre>Bad:<br>&quot;customer_session_information_for_customer_12345&quot;</pre><pre>Better:<br>&quot;cs:12345&quot;</pre><p>Similarly, avoid unnecessarily duplicating large values.</p><p>TTL and eviction policies are also important because cache memory is finite.</p><p>When memory reaches its limit, the cache needs an eviction strategy to determine which entries can be removed.</p><h3>14. Eventual Consistency</h3><p>A cache is usually <strong>not the authoritative source of data</strong>.</p><p>Consider:</p><pre>Database<br>   |<br>   | source of truth<br>   |<br> Redis<br>   |<br>   | cached copy<br>   |<br>Application</pre><p>Suppose the database changes:</p><pre>DB: role = admin → user</pre><p>but Redis still contains:</p><pre>role = admin</pre><p>There is temporarily a consistency gap.</p><p>Once the cache is invalidated or refreshed:</p><pre>Redis → role = user</pre><p>the systems converge.</p><p>Therefore, many caching architectures accept <strong>eventual consistency</strong> in exchange for much lower latency and higher throughput.</p><h3>15. Putting It All Together</h3><p>A production-oriented distributed cache looks conceptually like this:</p><pre>                    +------------------+<br>                    |   Application    |<br>                    +--------+---------+<br>                             |<br>                      Redis Client<br>                             |<br>                   Hash / Routing Logic<br>                             |<br>          +------------------+------------------+<br>          |                  |                  |<br>      +---+---+          +---+---+          +---+---+<br>      |Shard 0|          |Shard 1|          |Shard 2|<br>      |Primary|          |Primary|          |Primary|<br>      +---+---+          +---+---+          +---+---+<br>          |                  |                  |<br>      Replica             Replica             Replica<br>          |                  |                  |<br>         RAM                RAM                RAM</pre><p>The main request flows are:</p><h3>SET</h3><pre>Application<br> → Redis Client<br> → Hash(key)<br> → Shard<br> → RAM<br> → ACK</pre><h3>GET — Cache Hit</h3><pre>Application<br> → Redis Client<br> → Shard<br> → RAM<br> → Value</pre><h3>GET — Cache Miss</h3><pre>Application<br> → Redis<br> → MISS<br> → Database<br> → SET Redis<br> → Return value</pre><h3>DELETE</h3><pre>Application<br> → Redis Client<br> → Shard<br> → Delete/Invalidate<br> → ACK</pre><h3>16. Key Design Lessons</h3><p>Designing a distributed cache is much more than putting Redis in front of a database.</p><p>The important questions are:</p><ol><li><strong>What is the source of truth?</strong><br>Usually the database, not the cache.</li><li><strong>How is data distributed?</strong><br>Sharding determines scalability and can also create hotspots.</li><li><strong>What happens when a node fails?</strong><br>Replication and failover provide fault tolerance.</li><li><strong>How is stale data handled?</strong><br>TTL, invalidation, and refresh strategies are required.</li><li><strong>How do we maintain low p99 latency?</strong><br>Network placement, persistent connections, small values, and efficient operations matter.</li><li><strong>What happens when memory is full?</strong><br>Eviction policies determine which cached data is removed.</li><li><strong>Can operations be safely retried?</strong><br>Idempotency becomes important when failures occur between the operation and acknowledgement.</li></ol><h3>Conclusion</h3><p>A cache starts with a simple idea:</p><pre>Frequently accessed data → RAM</pre><p>But at scale, the design becomes a distributed systems problem.</p><p>You need to reason about <strong>sharding, hotspots, replication, failures, retries, TTLs, eviction, consistency, memory overhead, and p99 latency</strong>.</p><p>The most important mental model is:</p><pre>Database = Source of Truth</pre><pre>Redis = Fast, Distributed, Temporary Copy</pre><p>Once this distinction is clear, many caching decisions become easier to reason about.</p><p>The goal of the cache is not to replace the database.</p><p><strong>The goal is to prevent the database from becoming the bottleneck for data that can be served much faster from memory.</strong></p><h3>More System Design Topics</h3><p><a href="https://github.com/code-with-amitk/Code-examples/tree/master/System-Design">https://github.com/code-with-amitk/Code-examples/tree/master/System-Design</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b468ca3e1133" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Google Drive/Dropbox System Design which Sank many in Interviews]]></title>
            <link>https://medium.com/@everythingismindgame/google-drive-dropbox-system-design-which-sank-many-in-interviews-e6be8150b097?source=rss-6e9afd5faba4------2</link>
            <guid isPermaLink="false">https://medium.com/p/e6be8150b097</guid>
            <category><![CDATA[system-design-interview]]></category>
            <category><![CDATA[google-drive]]></category>
            <dc:creator><![CDATA[Everything is MindGame]]></dc:creator>
            <pubDate>Tue, 15 Sep 2026 11:55:41 GMT</pubDate>
            <atom:updated>2026-09-15T11:55:41.986Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/350/1*sRmw5yj5iR9-53fD87GtFA.png" /></figure><h3>Requirements</h3><p><strong>Functional</strong></p><ol><li>Create(Upload/PUT) new file on google drive, save this on cloud</li><li>Read(GET) file when cerate on cloud shared with this user. Sharing/authorization userA(editor), userB(reader only)</li><li>Sync(GET) Updated file. User-A changes a file, User-B want to sync the updated file on its Google Driver Desktop</li><li>Once login to goggle drive, sync all files in workspace.</li><li>Concurrent Edit</li></ol><p><strong>Non-functional</strong>:</p><ol><li>Highly available</li><li>Eventual consistent: Means replicas may temporarily return different versions but eventually converge. What is SLA to be eventual consistent(5 sec or 30 sec)?</li><li>Offline edit. User is offline, he edits the file, Once User comes online information should go on drive.</li><li>Deduplication: Same file upload(PUT) 2 times, should only be stored once on DB</li></ol><p><strong>Extended</strong>:</p><ol><li>Snaphot of data: System should support snapshotting of the data, so that users can go back to any version of the files.</li></ol><h3>HLD</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fsJO2P66u_ClMoTHJmDWXQ.png" /></figure><h3>Create(Upload/PUT) new file</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vq9PcXuMqVIOHWG2PFTltg.png" /></figure><h3>Non-Functional</h3><h4>a. High Availability</h4><ul><li>Metadata Database replication</li><li>multipods</li></ul><h4>b. Resumable Upload</h4><ul><li>Client sends chunk0, Object store will send HTTP-ACK back to client.</li><li>Client crashes before sending chunk1</li><li>Once client restarts it checks which chunks have ACK and will not send those again</li></ul><h4>b. Resumable Download</h4><ul><li>Server sends chunk0 to client, client sends ACK and client crashes</li><li>Client restarts, checks for chunks from server, but on server file is updated.</li><li>Server will 1st send the prev version which client was having. Once ver1 is sent completely to client, then server will send ver2</li></ul><pre>Server               Client<br>    --chunk1,2,3--&gt; Version=1 downloaded<br>    --version=2---&gt; Now new version is downloaded</pre><p><strong>Download using HTTP Range Requests</strong></p><ul><li>if client downloads 10G(of 50G file) from server and crashes.</li><li>Client can send range to server to download 11G onwards</li></ul><pre>GET &lt;presigned-upload-URL&gt;<br>Range: bytes=10G-</pre><h4>c. Fault Tolerance (failover to other Pod, DB Redundancy)</h4><p>if some pods crash system still continue working without downtime. server(pod1) writes the meta data to shared DB, if pod1 crashes pod2 will pick from that state and continue work</p><h4>d. Eventual Consistent</h4><ul><li>Ask from interviewer. What’s agreed SLA for eventual consistency (3 or 30 sec)?</li><li>Replicas may temporarily return different versions but eventually converge</li></ul><pre>file1 → version 4<br>file2 → version 5</pre><ul><li>Files will be versioned, once clientApp finds locally metadata DB has version=4 and version=5 from server, it will sync</li></ul><h4>e. Idempotency / Deduplication</h4><ul><li>ClientApp sends chunk-0, chunk-1. Server recieved chunk-0, chunk-1 and sends ACK1, ACK2. ClientApp recieves ACK1 &amp; network failure happened, ClientApp did not recieve ACK2.</li><li>Network resumes and clientApp sends chunk-1 again.</li><li>In order for server to not create 2 copies of chunk-1, Server need to maintain a idempotency key which is(file_id + version + chunk_number). if same key is found duplicate is rejected.</li></ul><h3>2. GET. Read file from server</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*63Qnic8mn6OoT8JFAtz8NQ.png" /></figure><h3>3. Sync the updated File</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EEW5hty0zH9w-3g9BmScIQ.png" /></figure><pre>v1: chunk-0 → A<br>    chunk-1 → B<br>    chunk-2 → C<br><br>v2: chunk-0 → A<br>    chunk-1 → D &lt;&lt;&lt; Changed<br>    chunk-2 → C</pre><ul><li>user changes bytes chunk1©. ClientApp will send Delta:(file_id, version, chunk_id, start_offset, end_offset, delta_object_pointer) to server</li></ul><p><a href="https://github.com/code-with-amitk/Code-examples/tree/master/System-Design/Scalable/Dropbox_or_GoogleDrive">Design on github</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e6be8150b097" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From Raw Firewall Logs to Answers: Building a RAG Pipeline for Network Security]]></title>
            <link>https://medium.com/@everythingismindgame/from-raw-firewall-logs-to-answers-building-a-rag-pipeline-for-network-security-b31d625f081d?source=rss-6e9afd5faba4------2</link>
            <guid isPermaLink="false">https://medium.com/p/b31d625f081d</guid>
            <category><![CDATA[retrieval-augmented-gen]]></category>
            <category><![CDATA[rags]]></category>
            <dc:creator><![CDATA[Everything is MindGame]]></dc:creator>
            <pubDate>Mon, 14 Sep 2026 15:34:55 GMT</pubDate>
            <atom:updated>2026-09-14T15:34:55.073Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2LblxvpA0I-TFM46GMi2yQ.png" /></figure><p>Network and security systems generate enormous amounts of logs.</p><p>VPN connections, firewall decisions, authentication events, IP addresses, ports, protocols, and timestamps can quickly become difficult to analyze manually.</p><p>What if we could simply ask:</p><blockquote><strong><em>“Why was this VPN connection rejected?”</em></strong></blockquote><p>or</p><blockquote><strong><em>“Show me suspicious firewall activity from this user.”</em></strong></blockquote><p>and get an answer from the logs?</p><p>That is the idea behind this <strong>RAG (Retrieval-Augmented Generation) pipeline for VPN and firewall logs</strong>.</p><h3>The Problem</h3><p>Traditional log analysis usually involves:</p><ol><li>Searching large log files.</li><li>Filtering timestamps, IPs, users, and ports.</li><li>Correlating multiple events.</li><li>Manually interpreting the results.</li></ol><p>LLMs are good at understanding natural language, but they don’t inherently know the contents of our latest firewall logs.</p><p>This is where RAG helps.</p><h3>RAG + Security Logs</h3><p>The pipeline follows the standard RAG pattern:</p><pre>VPN / Firewall Logs<br>        ↓<br>   Load &amp; Prepare<br>        ↓<br>      Chunking<br>        ↓<br>    Embeddings<br>        ↓<br>   Vector Store<br>        ↓<br>   User Question<br>        ↓<br>    Retrieval<br>        ↓<br> Relevant Log Context<br>        ↓<br>       LLM<br>        ↓<br>    Natural Language Answer</pre><p>Instead of asking the LLM to memorize the logs, we retrieve the relevant log entries and provide them as context.</p><h3>Step 1: Prepare the Logs</h3><p>The first step is to ingest the VPN/firewall log data.</p><p>Logs contain information such as:</p><ul><li>Source and destination IP</li><li>User/session information</li><li>Ports and protocols</li><li>Connection status</li><li>Firewall actions</li><li>Timestamps</li><li>VPN events</li></ul><p>The raw logs are transformed into documents that can be processed by the RAG pipeline.</p><h3>Step 2: Create Embeddings</h3><p>Each document/chunk is converted into an <strong>embedding</strong> — a numerical representation of its meaning.</p><p>This allows the system to search by semantic meaning rather than only exact keywords.</p><p>For example, a query such as:</p><blockquote><em>“Find failed VPN connections”</em></blockquote><p>can retrieve relevant records even when the log doesn’t literally contain the phrase <em>failed VPN connections</em>.</p><h3>Step 3: Store the Vectors</h3><p>The generated embeddings are stored in a vector database/store.</p><p>Conceptually:</p><pre>Log Document<br>     ↓<br> Text Chunk<br>     ↓<br> Embedding Vector<br>     ↓<br> Vector Store</pre><p>When a user asks a question, the question is also converted into an embedding and compared with stored vectors.</p><h3>Step 4: Retrieve Relevant Logs</h3><p>Suppose the user asks:</p><blockquote><strong><em>“Were there any suspicious login attempts?”</em></strong></blockquote><p>The retriever searches the vector store and returns the most relevant log chunks.</p><p>Only those relevant records are passed to the LLM.</p><p>This is the key advantage of RAG:</p><p><strong>The LLM doesn’t need the entire log dataset.</strong></p><p>It receives a focused context containing the evidence needed to answer the question.</p><h3>Step 5: Generate the Answer</h3><p>The retrieved logs are combined with the user’s question and sent to the LLM.</p><p>The LLM can then transform raw technical records into a human-readable response.</p><p>For example:</p><pre>User:<br>Why was the VPN connection rejected?</pre><pre>RAG:<br>The connection from 10.20.15.21 was rejected by the<br>firewall because the requested destination port was<br>not permitted by the active security policy.</pre><p>The answer is based on retrieved log evidence rather than the model simply guessing.</p><h3>Why RAG Is Useful for Security Operations</h3><p>This approach can make security logs much easier to investigate.</p><p>Instead of writing complex queries for every investigation, analysts can ask questions in natural language:</p><ul><li>“Show failed VPN connections.”</li><li>“Which users generated unusual activity?”</li><li>“Why was this connection blocked?”</li><li>“Find traffic to this IP address.”</li><li>“Show suspicious activity during the last hour.”</li><li>“What firewall rule caused this request to be denied?”</li></ul><p>RAG effectively provides a <strong>natural-language interface over security data</strong>.</p><h3>Important Limitation</h3><p>RAG is not a replacement for traditional security analytics.</p><p>For production systems, structured querying, SIEM correlation, detection rules, authentication, authorization, and audit mechanisms are still essential.</p><p>RAG works best as an <strong>investigation and analysis layer</strong> on top of existing security infrastructure.</p><h3>The Bigger Picture</h3><p>This small project demonstrates an important pattern:</p><pre>Enterprise Data<br>      ↓<br> Retrieval<br>      ↓<br> Relevant Context<br>      ↓<br>      LLM<br>      ↓<br> Natural Language Investigation</pre><p>The same architecture can be extended beyond VPN and firewall logs to:</p><ul><li>Application logs</li><li>Kubernetes logs</li><li>Authentication logs</li><li>Network telemetry</li><li>Incident reports</li><li>SOC alerts</li><li>Runbooks</li><li>Troubleshooting documentation</li></ul><p>The real value of RAG is not simply “chatting with documents.”</p><p>It is about connecting an LLM to <strong>current, private, domain-specific data</strong> and allowing users to reason over that information using natural language.</p><h3>Conclusion</h3><p>A VPN/firewall log dataset may look like nothing more than lines of technical events.</p><p>With RAG, those events become searchable knowledge.</p><p><strong>Logs → Embeddings → Retrieval → Context → LLM → Security Insights</strong></p><p>That simple pipeline is a powerful foundation for building AI-assisted security operations and network troubleshooting tools.</p><p><strong>Code:</strong> <a href="https://github.com/code-with-amitk/Code-examples/tree/master/Machine_Learning/RAG_Pipeline/Code/RAG_Pipeline_VPN_Firewall_logs?utm_source=chatgpt.com">GitHub — RAG Pipeline for VPN &amp; Firewall Logs</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b31d625f081d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Multi-Agent AI Systems with LangChain: Building an Orchestrator That Delegates Work]]></title>
            <link>https://medium.com/@everythingismindgame/multi-agent-ai-systems-with-langchain-building-an-orchestrator-that-delegates-work-15399a812998?source=rss-6e9afd5faba4------2</link>
            <guid isPermaLink="false">https://medium.com/p/15399a812998</guid>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[langchain]]></category>
            <dc:creator><![CDATA[Everything is MindGame]]></dc:creator>
            <pubDate>Fri, 11 Sep 2026 13:59:51 GMT</pubDate>
            <atom:updated>2026-09-11T13:59:51.218Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5KWY08u7f0P4qiCeY-_Syg.png" /></figure><p>Large AI tasks are often not a single problem.</p><p>Consider a stock-market analysis system. To produce a useful report, an AI system may need to:</p><ul><li>Collect information from the web</li><li>Analyze financial data</li><li>Correlate information</li><li>Generate a report</li><li>Send the report to a user</li></ul><p>A single agent can potentially perform all these steps, but it becomes increasingly difficult to control, specialize, test, and scale.</p><p>A <strong>multi-agent system</strong> solves this by dividing the work among multiple specialized agents.</p><p>In this article, we will build a simple multi-agent system using <strong>LangChain and Ollama</strong>, based on the implementation in my GitHub repository.</p><h3>1. What Is a Multi-Agent System?</h3><p>A multi-agent system consists of multiple AI agents that cooperate to solve a larger problem.</p><p>Instead of:</p><pre>User<br>  |<br>  v<br>Single Agent<br>  |<br>  +-- Search<br>  +-- Analyze<br>  +-- Calculate<br>  +-- Generate Report</pre><p>we can create specialized agents:</p><pre>                         User<br>                           |<br>                           v<br>                    +--------------+<br>                    | Main Agent   |<br>                    | Orchestrator |<br>                    +------+-------+<br>                           |<br>              +------------+------------+<br>              |                         |<br>              v                         v<br>       +-------------+           +-------------+<br>       | Math Agent  |           | Search Agent|<br>       +-------------+           +-------------+<br>              |                         |<br>          square()                  search()</pre><p>Each subagent has a specific responsibility.</p><p>The main agent decides <strong>which subagent should handle the request</strong>.</p><p>This is commonly called the <strong>orchestrator pattern</strong> or <strong>supervisor pattern</strong>. Modern LangChain documentation also demonstrates the pattern of wrapping specialized agents as tools that a supervisor can invoke.</p><h3>2. Why Use Multiple Agents?</h3><p>The main reason is <strong>specialization</strong>.</p><p>Suppose we build an enterprise AI assistant.</p><p>One agent might specialize in:</p><pre>Database queries</pre><p>Another:</p><pre>Security analysis</pre><p>Another:</p><pre>Documentation search</pre><p>Another:</p><pre>Report generation</pre><p>The orchestrator does not need to know how each specialist performs its task.</p><p>It only needs to know:</p><pre>&quot;I have a database question → call database agent.&quot;</pre><pre>&quot;I have a security question → call security agent.&quot;</pre><pre>&quot;I need documentation → call documentation agent.&quot;</pre><p>This creates a clean separation of responsibilities.</p><h3>Single agent</h3><pre>                    LLM<br>                     |<br>       +-------------+-------------+<br>       |             |             |<br>    Database       Search       Security<br>       |             |             |<br>       +-------------+-------------+</pre><p>The agent has to reason about everything.</p><h3>Multi-agent</h3><pre>                      Main Agent<br>                    /     |      \<br>                   /      |       \<br>                  v       v        v<br>             DB Agent  Search   Security<br>                        Agent      Agent</pre><p>Each agent has a much smaller responsibility.</p><h3>3. Example: Two Specialized Agents</h3><p>The example in the repository intentionally uses a very simple problem:</p><pre>Calculate the square of a number<br>Calculate the square root of a number</pre><p>This is useful because it makes the architecture easy to understand without introducing unnecessary complexity.</p><p>We first create two tools.</p><pre>@tool<br>def square_root(x: float) -&gt; float:<br>    &quot;&quot;&quot;Calculate the square root of a number&quot;&quot;&quot;<br>    return x ** 0.5<br></pre><pre>@tool<br>def square(x: float) -&gt; float:<br>    &quot;&quot;&quot;Calculate the square of a number&quot;&quot;&quot;<br>    return x ** 2</pre><p>Now we create two specialized agents.</p><pre>subagent_square_root = create_agent(<br>    model=llm,<br>    tools=[square_root]<br>)</pre><pre>subagent_square = create_agent(<br>    model=llm,<br>    tools=[square]<br>)</pre><p>The first agent knows about:</p><pre>square_root()</pre><p>The second agent knows about:</p><pre>square()</pre><p>They are therefore specialized agents.</p><h3>4. The Important Part: Turning Agents into Tools</h3><p>Now comes the key idea.</p><p>The main agent does not directly interact with the subagents.</p><p>Instead, each subagent is exposed to the main agent as a <strong>tool</strong>.</p><p>For example:</p><pre>@tool<br>def call_subagent_square_root(x: float) -&gt; float:<br>    &quot;&quot;&quot;Call subagent 1 in order to calculate<br>    the square root of a number&quot;&quot;&quot;</pre><pre>    response = subagent_square_root.invoke(<br>        {<br>            &quot;messages&quot;: [<br>                HumanMessage(<br>                    content=f&quot;Calculate the square root of {x}&quot;<br>                )<br>            ]<br>        }<br>    )</pre><pre>    return response[&quot;messages&quot;][-1].content</pre><p>Similarly:</p><pre>@tool<br>def call_subagent_square(x: float) -&gt; float:<br>    &quot;&quot;&quot;Call subagent 2 in order to calculate<br>    the square of a number&quot;&quot;&quot;</pre><pre>    response = subagent_square.invoke(<br>        {<br>            &quot;messages&quot;: [<br>                HumanMessage(<br>                    content=f&quot;Calculate the square of {x}&quot;<br>                )<br>            ]<br>        }<br>    )</pre><pre>    return response[&quot;messages&quot;][-1].content</pre><p>This creates an important abstraction:</p><pre>Main Agent<br>     |<br>     +-- call_subagent_square_root()<br>     |<br>     +-- call_subagent_square()</pre><p>From the main agent’s perspective, these are simply tools.</p><p>It does not need to understand the internal implementation of the subagents.</p><h3>5. Creating the Orchestrator</h3><p>Now we create the main agent:</p><pre>main_agent = create_agent(<br>    model=llm,<br>    tools=[<br>        call_subagent_square_root,<br>        call_subagent_square<br>    ],<br>    system_prompt=<br>        &quot;You are a helpful assistant who can call &quot;<br>        &quot;subagents to calculate the square root or &quot;<br>        &quot;square of a number.&quot;<br>)</pre><p>The architecture now looks like:</p><pre>                         User<br>                           |<br>                           |<br>                           v<br>                  +----------------+<br>                  |   Main Agent    |<br>                  |  Orchestrator   |<br>                  +-------+--------+<br>                          |<br>                 Tool selection<br>                    /          \<br>                   /            \<br>                  v              v<br>        +----------------+  +----------------+<br>        | Square Root    |  | Square         |<br>        | Subagent       |  | Subagent       |<br>        +-------+--------+  +-------+--------+<br>                |                  |<br>                v                  v<br>          square_root()        square()</pre><p>The LLM is responsible for deciding which tool to call.</p><p>This is the fundamental mechanism behind this example.</p><h3>6. What Happens When the User Asks a Question?</h3><p>Suppose the user asks:</p><pre>What is the square root of 456?</pre><p>The request goes to the main agent.</p><pre>response = main_agent.invoke(<br>    {<br>        &quot;messages&quot;: [<br>            HumanMessage(<br>                content=&quot;What is the square root of 456?&quot;<br>            )<br>        ]<br>    }<br>)</pre><p>The main agent recognizes that this is a square-root operation.</p><p>It therefore generates a tool call:</p><pre>call_subagent_square_root(x=456)</pre><p>The subagent then receives:</p><pre>Calculate the square root of 456</pre><p>It has access to:</p><pre>square_root()</pre><p>and returns approximately:</p><pre>21.3541565</pre><p>The result is returned to the main agent.</p><p>Finally, the main agent produces the user-facing answer:</p><pre>The square root of 456 is approximately 21.35.</pre><p>The complete flow is:</p><pre>User<br> |<br> | &quot;What is the square root of 456?&quot;<br> v<br>Main Agent<br> |<br> | selects call_subagent_square_root<br> v<br>Square Root Subagent<br> |<br> | invokes square_root(456)<br> v<br>21.3541565<br> |<br> v<br>Main Agent<br> |<br> v<br>&quot;The square root of 456 is approximately 21.35.&quot;</pre><p>The execution output in the repository shows this exact delegation: the main agent produces a call_subagent_square_root tool call, receives the tool result, and then generates the final response.</p><h3>7. Where Does the LLM Fit?</h3><p>The example uses Ollama:</p><pre>llm = ChatOllama(<br>    model=&quot;llama3.2:latest&quot;,<br>    base_url=&quot;http://localhost:11434&quot;,<br>    temperature=0,<br>)</pre><p>The same LLM is used by the main agent and the subagents.</p><p>Conceptually:</p><pre>                Ollama / Llama<br>                     |<br>          +----------+----------+<br>          |                     |<br>          v                     v<br>     Main Agent            Subagents<br>     Orchestrator          Specialized</pre><p>There is no requirement that every agent use the same model.</p><p>A production architecture could use:</p><pre>Main Agent       → powerful reasoning model</pre><pre>Security Agent   → security-specialized model</pre><pre>Coding Agent     → coding model</pre><pre>Simple Agent     → smaller/cheaper model</pre><p>This is one of the potential advantages of multi-agent architectures: <strong>different workloads can have different models, tools, prompts, and resource requirements.</strong></p><h3>8. Multi-Agent Does Not Mean Multiple LLMs</h3><p>An important distinction:</p><p><strong>Multi-agent ≠ multiple LLM models.</strong></p><p>We can have:</p><pre>Model A<br>  |<br>  +-- Agent 1<br>  +-- Agent 2<br>  +-- Agent 3</pre><p>or:</p><pre>Model A → Agent 1<br>Model B → Agent 2<br>Model C → Agent 3</pre><p>An agent is better understood as a combination of:</p><pre>Agent =<br>    Model<br>  + Instructions<br>  + Tools<br>  + State/Context<br>  + Decision-making</pre><p>Therefore, several agents can use the same underlying LLM while having completely different responsibilities.</p><h3>9. Why Not Just Give All Tools to One Agent?</h3><p>A natural question is:</p><blockquote><em>Why create multiple agents? Why not give </em><em>square() and </em><em>square_root() directly to one agent?</em></blockquote><p>For this tiny example, that would actually be simpler.</p><p>The value of multi-agent architecture becomes clearer as the system grows.</p><p>Imagine an enterprise assistant:</p><pre>                         Orchestrator<br>                              |<br>          +-------------------+-------------------+<br>          |                   |                   |<br>          v                   v                   v<br>    Database Agent       Security Agent      Research Agent<br>          |                   |                   |<br>       PostgreSQL          SIEM/Security       Web/MCP<br>                              |<br>                              v<br>                       Report Generator</pre><p>Each agent can have:</p><ul><li>Its own tools</li><li>Its own system prompt</li><li>Its own model</li><li>Its own permissions</li><li>Its own domain knowledge</li><li>Its own error handling</li><li>Its own evaluation criteria</li></ul><p>This is much easier to reason about than one enormous agent containing dozens of unrelated capabilities.</p><h3>10. Multi-Agent vs. Sequential Workflow</h3><p>These two ideas are related but different.</p><h3>Sequential workflow</h3><p>The execution path is predefined:</p><pre>Step 1<br>  ↓<br>Step 2<br>  ↓<br>Step 3<br>  ↓<br>Step 4</pre><p>For example:</p><pre>Retrieve documents<br>      ↓<br>Analyze documents<br>      ↓<br>Generate report<br>      ↓<br>Send email</pre><p>The application determines the sequence.</p><h3>Orchestrated multi-agent system</h3><p>The main agent decides what should happen:</p><pre>                     Orchestrator<br>                    /     |      \<br>                   /      |       \<br>               Agent A  Agent B  Agent C</pre><p>The LLM can determine which specialist is appropriate based on the user’s request.</p><p>Therefore:</p><pre>Workflow = predefined control flow</pre><pre>Multi-Agent =<br>LLM-driven delegation between specialized agents</pre><p>In real systems, both approaches can be combined.</p><h3>11. Production Architecture</h3><p>The simple mathematical example can be extended into an enterprise system:</p><pre>User<br>                           |<br>                           v<br>                 +-------------------+<br>                 | Orchestrator      |<br>                 | Agent             |<br>                 +---------+---------+<br>                           |<br>          +----------------+----------------+<br>          |                |                |<br>          v                v                v<br>    +-----------+    +-----------+    +-----------+<br>    | Research  |    | Security  |    | Database  |<br>    | Agent     |    | Agent     |    | Agent     |<br>    +-----+-----+    +-----+-----+    +-----+-----+<br>          |                |                |<br>          v                v                v<br>       Web/MCP          SIEM/APIs       PostgreSQL<br>          |                |                |<br>          +----------------+----------------+<br>                           |<br>                           v<br>                  +----------------+<br>                  | Final Response |<br>                  | / Report Agent |<br>                  +----------------+</pre><p>This architecture is particularly useful when each domain has different tools and security boundaries.</p><p>For example:</p><pre>Research Agent<br>    → web search<br>    → document retrieval<br>    → RAG</pre><pre>Security Agent<br>    → SIEM<br>    → vulnerability database<br>    → security APIs</pre><pre>Database Agent<br>    → SQL<br>    → analytics</pre><pre>Report Agent<br>    → summarize<br>    → format<br>    → email</pre><p>The orchestrator becomes the coordination layer.</p><h3>12. Key Takeaways</h3><p>The example demonstrates a very important AI architecture pattern with surprisingly little code.</p><h3>1. Specialize agents</h3><p>Give each agent a focused responsibility.</p><pre>Agent → one domain / capability</pre><h3>2. Wrap subagents as tools</h3><p>The orchestrator can treat an entire agent as a tool:</p><pre>Main Agent<br>    |<br>    +-- Subagent Tool</pre><h3>3. Let the orchestrator delegate</h3><p>The main LLM determines which specialist should handle the request.</p><h3>4. Keep the interfaces simple</h3><p>The orchestrator does not need to know how the subagent works internally.</p><p>It only needs:</p><pre>Tool name<br>Description<br>Arguments<br>Result</pre><h3>5. Start simple</h3><p>The square/square-root example is intentionally small.</p><p>The same architecture can evolve into:</p><pre>AI Orchestrator<br>                         |<br>        +----------------+----------------+<br>        |                |                |<br>    Research          Coding          Security<br>      Agent            Agent            Agent<br>        |                |                |<br>       MCP             GitHub          Security APIs<br>       RAG             Tools           SIEM</pre><h3>Conclusion</h3><p>A multi-agent system is fundamentally about <strong>decomposition and delegation</strong>.</p><p>Instead of building one large agent that understands every domain, we create specialized agents and allow an orchestrator to delegate work to them.</p><p>The LangChain implementation demonstrates this pattern clearly:</p><pre>User<br>  ↓<br>Main Agent<br>  ↓<br>Tool representing a subagent<br>  ↓<br>Specialized Subagent<br>  ↓<br>Domain-specific Tool<br>  ↓<br>Result<br>  ↓<br>Main Agent<br>  ↓<br>User</pre><p>The most important architectural idea is therefore not “multiple LLMs.”</p><p>It is:</p><blockquote><strong><em>An agent can itself become a capability that another agent can invoke.</em></strong></blockquote><p>Once this concept is understood, much larger systems become easier to design — from research assistants and coding systems to enterprise security and AI-powered automation platforms.</p><p>The complete example is available in the <a href="https://github.com/code-with-amitk/Code-examples/tree/master/Machine_Learning/Langchain/MultiAgent_System">MultiAgent_System repository</a>.</p><p>Enjoyed this article? Give it a few claps 👏 and follow for more beginner-friendly AI and LangChain tutorials!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=15399a812998" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[MCP the Layer Between LLMs and Real-World Tools]]></title>
            <link>https://medium.com/@everythingismindgame/mcp-the-layer-between-llms-and-real-world-tools-48f025bae469?source=rss-6e9afd5faba4------2</link>
            <guid isPermaLink="false">https://medium.com/p/48f025bae469</guid>
            <category><![CDATA[mcp-server]]></category>
            <dc:creator><![CDATA[Everything is MindGame]]></dc:creator>
            <pubDate>Fri, 11 Sep 2026 12:01:13 GMT</pubDate>
            <atom:updated>2026-09-11T12:01:13.908Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JeijuZUQArJR0Z-dM3AW_g.png" /></figure><h3>MCP Is the Missing Layer Between LLMs and Real-World Tools</h3><p>Large Language Models are remarkably good at understanding and generating text. But an LLM by itself has an important limitation: <strong>it cannot directly access the systems around it</strong>.</p><p>It cannot automatically search the web, read your company’s internal documentation, query a database, inspect a Git repository, or invoke an internal API.</p><p>Traditionally, developers solved this by writing custom integrations between every AI application and every external service.</p><p>That approach does not scale.</p><p><strong>Model Context Protocol (MCP)</strong> provides a standardized way for AI applications to discover and use external capabilities such as <strong>tools, resources, and prompts</strong>. The MCP Python SDK and the LangChain MCP adapters make it possible to build this architecture with relatively little code.</p><p>In this article, I will walk through a small but complete example:</p><blockquote><strong><em>An Ollama-powered LangChain agent connects to an MCP server that exposes web search, a system-design knowledge resource, and an MCP prompt.</em></strong></blockquote><p>The complete implementation is available in my GitHub repository:</p><p><a href="https://github.com/code-with-amitk/Code-examples/blob/master/Machine_Learning/Langchain/MCPServer_MCPClient.adoc?utm_source=chatgpt.com">MCPServer_MCPClient.adoc on GitHub</a></p><h3>1. First: What Problem Are We Solving?</h3><p>Imagine asking an AI agent:</p><blockquote><strong><em>“Tell me about the Box.com system design.”</em></strong></blockquote><p>Suppose the agent has access to my system-design website.</p><p>The agent should ideally:</p><ol><li>Understand the user’s question.</li><li>Check the available knowledge resource.</li><li>Find relevant information.</li><li>If the answer is not available there, perform a web search.</li><li>Combine the retrieved information.</li><li>Generate the final answer.</li></ol><p>The important point is that the LLM itself does not need to know <strong>how</strong> to access the website or perform a web search.</p><p>Instead, those capabilities are exposed by an <strong>MCP server</strong>.</p><p>The architecture looks like this:</p><pre>User<br>                          |<br>                          | &quot;Tell me about Box.com<br>                          |  system design&quot;<br>                          v<br>                +----------------------+<br>                |   LangChain Agent    |<br>                |                      |<br>                |      Ollama LLM      |<br>                +----------+-----------+<br>                           |<br>                           | MCP Client<br>                           v<br>                +----------------------+<br>                |      MCP Server      |<br>                |                      |<br>                |  Tools               |<br>                |   └─ search_web      |<br>                |                      |<br>                |  Resources           |<br>                |   └─ System Design   |<br>                |                      |<br>                |  Prompts             |<br>                |   └─ Agent behavior  |<br>                +----------+-----------+<br>                           |<br>             +-------------+-------------+<br>             |                           |<br>             v                           v<br>      System Design Website        Tavily Web Search</pre><p>This separation is the key idea.</p><p>The <strong>LLM reasons</strong>.</p><p>The <strong>MCP server provides capabilities</strong>.</p><p>The <strong>MCP client connects the two</strong>.</p><h3>2. Building the MCP Server</h3><p>The server starts with FastMCP:</p><pre>from mcp.server.fastmcp import FastMCP</pre><pre>mcp = FastMCP(&quot;mcp_server&quot;)</pre><p>FastMCP provides a convenient Python interface for creating an MCP server.</p><p>Instead of manually implementing protocol messages, we can describe our capabilities using decorators.</p><p>For example:</p><pre>@mcp.tool()<br>def search_web(query: str) -&gt; Dict[str, Any]:<br>    ...</pre><p>The decorator tells the MCP framework:</p><blockquote><em>“Expose this Python function as an MCP tool.”</em></blockquote><p>This is one of the most important concepts in MCP.</p><h3>MCP Tool</h3><p>A <strong>tool</strong> represents an action that the AI application can invoke.</p><p>Examples could include:</p><pre>search_web()<br>query_database()<br>create_ticket()<br>send_email()<br>execute_command()<br>get_customer()</pre><p>In this example, the tool is:</p><pre>@mcp.tool()<br>def search_web(query: str) -&gt; Dict[str, Any]:<br>    &quot;&quot;&quot;Search the web for information&quot;&quot;&quot;</pre><pre>    results = tavily_client.search(query)</pre><pre>    return results</pre><p>The implementation is simple.</p><p>The MCP server receives a search query and forwards it to Tavily.</p><p>Conceptually:</p><pre>LLM<br> |<br> | &quot;I need more information&quot;<br> v<br>MCP Client<br> |<br> | invoke search_web(&quot;Box.com architecture&quot;)<br> v<br>MCP Server<br> |<br> v<br>Tavily<br> |<br> v<br>Search Results<br> |<br> v<br>MCP Client<br> |<br> v<br>LLM</pre><p>The interesting part is that the LLM does <strong>not</strong> need to know anything about Tavily’s API.</p><p>That implementation detail belongs inside the MCP server.</p><p>This creates a clean abstraction boundary.</p><h3>3. MCP Resources Are Different from Tools</h3><p>The second capability exposed by the server is a <strong>resource</strong>.</p><p>The code defines:</p><pre>@mcp.resource(&quot;https://code-with-amitk.github.io/&quot;)<br>def get_my_system_design_page():<br>    ...</pre><p>The resource retrieves the HTML from my system-design website:</p><pre>url = &quot;https://code-with-amitk.github.io/&quot;</pre><pre>resp = get(url)</pre><pre>return resp.text</pre><p>This is conceptually different from a tool.</p><p>A useful mental model is:</p><pre>Tool      → &quot;Do something&quot;<br>Resource  → &quot;Give me information&quot;</pre><p>For example:</p><pre>Tool:<br>search_web(&quot;Kafka architecture&quot;)</pre><pre>Resource:<br>https://my-company.com/documentation</pre><p>The tool performs an operation.</p><p>The resource provides context/data.</p><p>This distinction becomes particularly useful when building larger AI systems.</p><p>For example, an enterprise MCP server could expose:</p><pre>Resources<br> ├── Engineering documentation<br> ├── API documentation<br> ├── Architecture documents<br> └── Product specifications</pre><pre>Tools<br> ├── Search Jira<br> ├── Create Jira ticket<br> ├── Query database<br> └── Deploy service</pre><p>The AI agent can then combine <strong>knowledge</strong> and <strong>actions</strong>.</p><h3>4. MCP Prompts: Controlling Agent Behavior</h3><p>The third primitive in the example is a prompt:</p><pre>@mcp.prompt()<br>def prompt():<br>    ...</pre><p>The prompt tells the agent how it should behave.</p><p>The example says, in effect:</p><pre>You are a helpful assistant that answers<br>questions about System Design.</pre><pre>You can use:<br>- search_web<br>- files/resources</pre><pre>You may use multiple calls.<br>You may ask clarifying questions.</pre><p>This is interesting because the MCP server is not merely exposing APIs.</p><p>It can also provide <strong>instructions for how the capabilities should be used</strong>.</p><p>Think of it as:</p><pre>Tool      = capability<br>Resource  = knowledge<br>Prompt    = behavioral instruction</pre><p>This gives us three different building blocks:</p><pre>MCP Server<br>                     |<br>        +------------+------------+<br>        |            |            |<br>        v            v            v<br>      Tools      Resources      Prompts<br>        |            |            |<br>     Actions       Data        Behavior</pre><h3>5. Starting the MCP Server</h3><p>At the end of the server:</p><pre>if __name__ == &quot;__main__&quot;:<br>    mcp.run(transport=&quot;stdio&quot;)</pre><p>The important part here is:</p><pre>transport=&quot;stdio&quot;</pre><p>The server communicates through standard input/output.</p><p>That means the client can launch the MCP server as a local process and communicate with it through its stdin/stdout streams.</p><p>This is particularly convenient for local development.</p><p>The resulting architecture is:</p><pre>LangChain Application<br>       |<br>       | spawn process<br>       v<br> python local_mcp_server.py<br>       |<br>       +---- stdin/stdout ----+</pre><p>The MCP server therefore does not necessarily need to be exposed through an HTTP endpoint.</p><p>It can run as a local process.</p><p>This is one reason MCP is useful for developer tools and local AI assistants.</p><h3>6. Now Build the MCP Client</h3><p>The client uses LangChain’s MCP adapter:</p><pre>from langchain_mcp_adapters.client import MultiServerMCPClient</pre><p>Then it creates the MCP client:</p><pre>mcp_client = MultiServerMCPClient(<br>    {<br>        &quot;local_server&quot;: {<br>            &quot;transport&quot;: &quot;stdio&quot;,<br>            &quot;command&quot;: &quot;python&quot;,<br>            &quot;args&quot;: [&quot;local_mcp_server.py&quot;],<br>        }<br>    }<br>)</pre><p>There are several important things happening here.</p><p>The client knows:</p><pre>Server name:<br>    local_server</pre><pre>Transport:<br>    stdio</pre><pre>Command:<br>    python</pre><pre>Script:<br>    local_mcp_server.py</pre><p>So the client can effectively start:</p><pre>python local_mcp_server.py</pre><p>and communicate with the process.</p><h3>7. Discovering MCP Tools</h3><p>The next important statement is:</p><pre>tools = await mcp_client.get_tools()</pre><p>This asks the MCP server:</p><blockquote><em>“What tools do you provide?”</em></blockquote><p>The server exposes:</p><pre>search_web</pre><p>The client receives the tool definition and makes it available to LangChain.</p><p>This is powerful because the client does not have to hard-code every tool implementation.</p><p>Conceptually:</p><pre>MCP Server<br>    |<br>    | Tool discovery<br>    v<br>MCP Client<br>    |<br>    | get_tools()<br>    v<br>LangChain Agent</pre><p>The agent can now reason about whether it needs to invoke search_web.</p><h3>8. Discovering Resources</h3><p>The code also calls:</p><pre>resources = await mcp_client.get_resources(&quot;local_server&quot;)</pre><p>This retrieves resources exposed by the server.</p><p>In our example, the server exposes:</p><pre>https://code-with-amitk.github.io/</pre><p>The resource provides the agent with access to the system-design knowledge hosted there.</p><p>So we now have:</p><pre>MCP Client<br>   |<br>   +---- Tools<br>   |       |<br>   |       +-- search_web<br>   |<br>   +---- Resources<br>           |<br>           +-- System Design Website</pre><p>This separation becomes extremely useful as an MCP server grows.</p><h3>9. Retrieving the MCP Prompt</h3><p>The client also retrieves the server-side prompt:</p><pre>prompt = await mcp_client.get_prompt(<br>    &quot;local_server&quot;,<br>    &quot;prompt&quot;<br>)</pre><pre>prompt = prompt[0].content</pre><p>Now the client has the instructions supplied by the MCP server.</p><p>These instructions are subsequently passed to the LangChain agent as its system prompt.</p><p>This creates an important relationship:</p><pre>MCP Server<br>     |<br>     | prompt<br>     v<br>MCP Client<br>     |<br>     | system_prompt<br>     v<br>LangChain Agent</pre><p>The server therefore communicates not only capabilities, but also guidance about how those capabilities should be used.</p><h3>10. Connecting the LLM</h3><p>The example uses Ollama:</p><pre>from langchain_ollama import ChatOllama</pre><p>and creates:</p><pre>llm = ChatOllama(<br>    model=&quot;llama3.2:latest&quot;,<br>    base_url=&quot;http://localhost:11434&quot;,<br>    temperature=0,<br>)</pre><p>This means the LLM is running locally through Ollama.</p><p>The important architectural point is that <strong>MCP is independent of the model</strong>.</p><p>We could replace Ollama with another model provider without changing the MCP server.</p><p>For example:</p><pre>MCP Server<br>                     |<br>             +-------+-------+<br>             |               |<br>          Ollama          Cloud LLM<br>             |               |<br>             +-------+-------+<br>                     |<br>                  Agent</pre><p>MCP provides the integration layer; the LLM remains a replaceable component.</p><h3>11. Connecting Everything with a LangChain Agent</h3><p>The agent is created using:</p><pre>agent = create_agent(<br>    model=llm,<br>    tools=tools,<br>    system_prompt=prompt<br>)</pre><p>Now we have finally connected the three major pieces:</p><pre>              +----------------+<br>              |     Ollama     |<br>              |      LLM       |<br>              +-------+--------+<br>                      |<br>                      v<br>              +---------------+<br>              |    Agent      |<br>              +-------+-------+<br>                      |<br>             MCP-discovered tools<br>                      |<br>                      v<br>              +---------------+<br>              |   MCP Server  |<br>              +---------------+</pre><p>The agent receives:</p><pre>HumanMessage(<br>    content=&quot;Tell me about the box.com system design&quot;<br>)</pre><p>and executes:</p><pre>response = await agent.ainvoke(...)</pre><p>At this point, the agent can decide how to answer the question using the capabilities it has been given.</p><h3>12. What Actually Happens During a Request?</h3><p>This is the most important part of understanding the example.</p><p>Suppose the user asks:</p><blockquote><strong><em>“Tell me about the Box.com system design.”</em></strong></blockquote><p>The high-level flow is:</p><pre>1. User<br>      |<br>      v<br>2. LangChain Agent<br>      |<br>      v<br>3. LLM analyzes the question<br>      |<br>      +----------------------+<br>      |                      |<br>      v                      v<br>4. Resource lookup       Web Search<br>      |                      |<br>      v                      v<br>System Design            Tavily<br>knowledge                  API<br>      |                      |<br>      +----------+-----------+<br>                 |<br>                 v<br>5. Retrieved information<br>                 |<br>                 v<br>6. LLM generates answer<br>                 |<br>                 v<br>7. User</pre><p>The important idea is that <strong>the LLM decides when it needs external information</strong>, while the MCP server provides standardized mechanisms to obtain it.</p><p>This is fundamentally different from simply putting a huge amount of information into the model’s context window.</p><h3>13. Why MCP Instead of Direct APIs?</h3><p>Without MCP, an application might look like this:</p><pre>AI Application<br>   |<br>   +---- Tavily API<br>   |<br>   +---- Jira API<br>   |<br>   +---- GitHub API<br>   |<br>   +---- Database API<br>   |<br>   +---- Internal Documentation API</pre><p>Every AI application needs custom integration code.</p><p>With MCP:</p><pre>MCP<br>                  |<br>        +---------+---------+<br>        |         |         |<br>      Search    GitHub     Jira<br>        |         |         |<br>      Tool      Tool       Tool</pre><p>The AI application interacts with the MCP interface rather than implementing every external integration itself.</p><p>This is the real value of MCP:</p><blockquote><strong><em>It standardizes the boundary between an AI application and the tools/data it needs to use.</em></strong></blockquote><p>MCP is therefore less about “another way to call an API” and more about creating a <strong>common capability layer for AI applications</strong>.</p><p>The official MCP SDK describes the protocol as a way to connect applications with context and external capabilities through standardized interfaces.</p><h3>14. MCP Is More Than Function Calling</h3><p>It is tempting to think:</p><pre>MCP = function calling</pre><p>But that is too simplistic.</p><p>Function calling is primarily about allowing a model to request execution of a function.</p><p>MCP provides a broader protocol for exposing capabilities.</p><p>In this example alone we have:</p><pre>MCP<br> |<br> +-- Tools<br> |     └── search_web()<br> |<br> +-- Resources<br> |     └── system-design website<br> |<br> +-- Prompts<br>       └── agent instructions</pre><p>Modern MCP implementations also support additional protocol capabilities and transports, depending on the SDK/version.</p><p>Therefore, a better mental model is:</p><blockquote><strong><em>MCP is a standardized capability layer between AI applications and external systems.</em></strong></blockquote><h3>15. Why This Architecture Matters for Enterprise AI</h3><p>The simple example can be extended significantly.</p><p>Imagine an enterprise MCP server:</p><pre>AI Agent<br>                       |<br>                       v<br>                 MCP Interface<br>                       |<br>        +--------------+--------------+<br>        |              |              |<br>        v              v              v<br>     Company        GitHub          Jira<br>      Docs           API             API<br>        |              |              |<br>        +--------------+--------------+<br>                       |<br>                    Database</pre><p>The MCP server could expose:</p><h3>Resources</h3><pre>Architecture documents<br>Engineering documentation<br>Product specifications<br>Runbooks<br>API documentation</pre><h3>Tools</h3><pre>Search Jira<br>Create Jira ticket<br>Search GitHub<br>Query production metrics<br>Run diagnostics<br>Deploy a service</pre><h3>Prompts</h3><pre>Incident investigation<br>Code review<br>Architecture review<br>Security analysis<br>Production troubleshooting</pre><p>Now the same architecture can support multiple AI clients.</p><p>That is where MCP becomes particularly interesting for enterprise AI platforms.</p><h3>16. One Important Lesson From This Example</h3><p>There is a subtle but important architectural principle here.</p><p><strong>Do not put everything inside the LLM.</strong></p><p>The LLM should primarily handle:</p><pre>Reasoning<br>Planning<br>Understanding<br>Generation<br>Decision making</pre><p>External systems should handle:</p><pre>Data access<br>Search<br>Authentication<br>Execution<br>Business logic<br>State<br>Actions</pre><p>MCP provides a clean boundary between these responsibilities.</p><pre>AI Layer<br>        +----------------+<br>        |      LLM       |<br>        |     Agent      |<br>        |    Reasoning   |<br>        +-------+--------+<br>                |<br>              MCP<br>                |<br>        +-------+--------+<br>        | External World |<br>        |                |<br>        | APIs           |<br>        | Databases      |<br>        | Files          |<br>        | Search         |<br>        | Enterprise     |<br>        | Systems        |<br>        +----------------+</pre><p>This separation makes agentic systems easier to extend and reason about.</p><h3>Conclusion</h3><p>The MCP example presented here is intentionally small, but it demonstrates the architecture behind a much larger class of AI systems.</p><p>We created:</p><pre>MCP Server<br> ├── Tool       → Tavily web search<br> ├── Resource   → System-design website<br> └── Prompt     → Agent instructions</pre><p>Then the MCP client:</p><pre>1. Starts the MCP server<br>2. Discovers tools<br>3. Discovers resources<br>4. Retrieves the prompt<br>5. Connects everything to a LangChain agent<br>6. Uses Ollama as the LLM<br>7. Processes the user&#39;s question</pre><p>The most important takeaway is not the individual Python APIs.</p><p>It is the architecture:</p><blockquote><strong><em>The LLM does not need to know how every external system works. It needs a standardized way to discover and use the capabilities provided by those systems.</em></strong></blockquote><p>That is the problem MCP addresses.</p><p>And once you understand MCP as a <strong>capability layer rather than simply a function-calling mechanism</strong>, it becomes much easier to see how it can be used to build AI agents that interact with real enterprise systems.</p><p>The complete source code is available here:</p><p><a href="https://github.com/code-with-amitk/Code-examples/blob/master/Machine_Learning/Langchain/MCPServer_MCPClient.adoc?utm_source=chatgpt.com">View the complete MCP Server and Client implementation</a></p><h3>A simple mental model to remember</h3><pre>User<br>               |<br>               v<br>          AI Agent / LLM<br>               |<br>               | MCP Client<br>               v<br>          +----------+<br>          | MCP      |<br>          | Server   |<br>          +----------+<br>           /    |    \<br>          /     |     \<br>       Tools Resources Prompts<br>         |       |       |<br>       APIs     Data   Behavior</pre><p><strong>LLM = intelligence</strong></p><p><strong>MCP = capability bridge</strong></p><p><strong>Tools = actions</strong></p><p><strong>Resources = information</strong></p><p><strong>Prompts = instructions</strong></p><p>Together, these components provide a clean foundation for building practical agentic AI systems.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=48f025bae469" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Give Your AI Agent a Memory — Then Watch Everything It Does with LangSmith]]></title>
            <link>https://medium.com/@everythingismindgame/give-your-ai-agent-a-memory-then-watch-everything-it-does-with-langsmith-9b9bad5f8322?source=rss-6e9afd5faba4------2</link>
            <guid isPermaLink="false">https://medium.com/p/9b9bad5f8322</guid>
            <category><![CDATA[langgraph]]></category>
            <category><![CDATA[langchain]]></category>
            <category><![CDATA[langsmith]]></category>
            <dc:creator><![CDATA[Everything is MindGame]]></dc:creator>
            <pubDate>Tue, 08 Sep 2026 16:03:15 GMT</pubDate>
            <atom:updated>2026-09-08T16:03:15.435Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ShX5PKOjbTSw_2Hu-W8laQ.png" /></figure><p>Building an AI agent that can answer a question is relatively easy.</p><p>Building one that can <strong>remember what you told it</strong>, use tools when necessary, and give you visibility into <strong>what happened behind the scenes</strong> is a different story.</p><p>Consider this conversation:</p><blockquote><strong><em>User:</em></strong><em> My favorite color is Green.<br></em><strong><em>AI:</em></strong><em> Nice! Green is a wonderful color.</em></blockquote><blockquote>A few moments later…</blockquote><blockquote><strong><em>User:</em></strong><em> What’s my favorite color?<br></em><strong><em>AI:</em></strong><em> Your favorite color is Green!</em></blockquote><p>For a human, this is trivial.</p><p>For an AI agent, however, something needs to maintain the conversation state.</p><p>And once your agent starts calling tools, APIs, search engines, databases, or multiple LLMs, another problem appears:</p><p><strong>How do you know what the agent actually did?</strong></p><p>This is where two important concepts become extremely useful:</p><ul><li>🧠 <strong>Memory persistence</strong> — allowing an agent to remember previous interactions.</li><li>🔍 <strong>Observability with LangSmith</strong> — allowing developers to inspect and trace agent execution.</li></ul><p>In this article, we’ll walk through both concepts using a local <strong>Ollama + Llama 3.2</strong> agent and see how they fit together.</p><h3>The Architecture</h3><p>At a high level, our setup looks like this:</p><pre>                    ┌─────────────────────┐<br>                    │       User          │<br>                    └──────────┬──────────┘<br>                               │<br>                               ▼<br>                    ┌─────────────────────┐<br>                    │    AI Agent         │<br>                    │  LangChain Agent    │<br>                    └──────────┬──────────┘<br>                               │<br>                 ┌─────────────┴─────────────┐<br>                 │                           │<br>                 ▼                           ▼<br>        ┌─────────────────┐        ┌──────────────────┐<br>        │ Conversation    │        │     Tools        │<br>        │ Memory          │        │                  │<br>        │ InMemorySaver   │        │ Tavily Web Search│<br>        └─────────────────┘        └──────────────────┘<br>                 │                           │<br>                 └─────────────┬─────────────┘<br>                               │<br>                               ▼<br>                    ┌─────────────────────┐<br>                    │     Ollama          │<br>                    │     Llama 3.2       │<br>                    └─────────────────────┘<br>                               │<br>                               ▼<br>                    ┌─────────────────────┐<br>                    │     LangSmith       │<br>                    │  Traces &amp; Debugging │<br>                    └─────────────────────┘</pre><p>The two code examples demonstrate different parts of this architecture.</p><p>Let’s break them down.</p><h3>1. Running an LLM Locally with Ollama</h3><p>The examples use ChatOllama to communicate with a locally running Llama model.</p><pre>from langchain_ollama import ChatOllama</pre><pre>llm = ChatOllama(<br>    model=&quot;llama3.2:latest&quot;,<br>    base_url=&quot;http://localhost:11434&quot;,<br>    temperature=0,<br>)</pre><p>There are three important parameters here.</p><h3>model</h3><pre>model=&quot;llama3.2:latest&quot;</pre><p>This tells LangChain which Ollama model should process the request.</p><p>The advantage of using Ollama is that the model can run locally rather than sending every prompt to a hosted LLM API.</p><h3>base_url</h3><pre>base_url=&quot;http://localhost:11434&quot;</pre><p>Ollama exposes its local API through port 11434.</p><p>Therefore, LangChain knows where to send the requests.</p><h3>temperature</h3><pre>temperature=0</pre><p>A temperature of 0 generally makes the model&#39;s output more deterministic.</p><p>For agent workflows, deterministic behavior can be useful because we usually want predictable tool selection and responses.</p><h3>2. Creating the Agent</h3><p>The next step is creating an agent.</p><pre>from langchain.agents import create_agent</pre><pre>agent = create_agent(<br>    model=llm<br>)</pre><p>The agent becomes the component responsible for interacting with the language model.</p><p>In the second example, we also provide tools:</p><pre>agent = create_agent(<br>    model=llm,<br>    tools=[tool1],<br>    system_prompt=&quot;You are an intelligent knowlegable agent&quot;<br>)</pre><p>Now the agent has something more interesting to work with.</p><p>It isn’t just an LLM.</p><p>It can also invoke a tool.</p><h3>3. Giving the Agent a Web Search Tool</h3><p>The second example defines a custom tool using LangChain’s @tool decorator.</p><pre>from langchain.tools import tool</pre><pre>@tool(&quot;web_search&quot;, description=&quot;Search the web for information&quot;)<br>def tool1(query: str) -&gt; Dict[str, Any]:<br>    tavily = TavilyClient()<br>    return tavily.search(query=query)</pre><p>This is an important concept in agent development.</p><p>We’re essentially telling the LLM:</p><blockquote><em>“You have access to a capability called </em><em>web_search. Use it when you need external information.&quot;</em></blockquote><p>The function itself uses Tavily:</p><pre>tavily = TavilyClient()<br>return tavily.search(query=query)</pre><p>The model doesn’t directly perform the search.</p><p>Instead, the agent decides whether to call the tool, and the tool performs the actual operation.</p><p>The overall flow becomes:</p><pre>User Question<br>      │<br>      ▼<br>     LLM<br>      │<br>      │ &quot;I need external information&quot;<br>      ▼<br>  Web Search Tool<br>      │<br>      ▼<br>   Tavily API<br>      │<br>      ▼<br> Search Results<br>      │<br>      ▼<br>     LLM<br>      │<br>      ▼<br> Final Answer</pre><p>This is one of the fundamental patterns behind modern AI agents.</p><h3>4. The Problem: LLMs Don’t Automatically Remember Everything</h3><p>Now let’s look at the memory example.</p><p>The code creates an InMemorySaver:</p><pre>from langgraph.checkpoint.memory import InMemorySaver</pre><pre>agent = create_agent(<br>    model=llm,<br>    checkpointer=InMemorySaver()<br>)</pre><p>This is where the interesting part begins.</p><p>A normal LLM invocation can be thought of as:</p><pre>Question → LLM → Answer</pre><p>Once the request is finished, the model doesn’t magically know what happened in previous interactions.</p><p>We therefore need some mechanism to maintain state.</p><p>That’s where the checkpointer comes in.</p><h3>5. InMemorySaver — Giving the Agent a Memory</h3><p>The code uses:</p><pre>checkpointer=InMemorySaver()</pre><p>The checkpointer stores the agent’s state in memory.</p><p>This is particularly useful for demonstrations, development, and experiments.</p><p>But there is an important limitation:</p><p><strong>It’s in-memory storage.</strong></p><p>If your application restarts, you should not treat this as durable production storage.</p><p>For production systems, you would typically use a persistent storage mechanism appropriate to your architecture.</p><p>But for demonstrating conversational memory, InMemorySaver is wonderfully simple.</p><h3>6. The Most Important Line: thread_id</h3><p>Now look at this:</p><pre>config = {<br>    &quot;configurable&quot;: {<br>        &quot;thread_id&quot;: &quot;1&quot;<br>    }<br>}</pre><p>This small piece of configuration is extremely important.</p><p>The thread_id identifies the conversation.</p><p>Think about it like a conversation ID.</p><p>For example:</p><pre>Thread 1<br>────────────<br>User → My favorite color is Green<br>AI   → Great!</pre><pre>User → What&#39;s my favorite color?<br>AI   → Green</pre><p>Another user could have:</p><pre>Thread 2<br>────────────<br>User → My favorite color is Blue<br>AI   → Blue</pre><p>The two conversations remain separate.</p><p>That’s why the same agent can serve multiple conversations while maintaining independent state.</p><h3>7. First Conversation</h3><p>The example sends:</p><pre>question = HumanMessage(<br>    content=&quot;I am X my favorite color is Green&quot;<br>)</pre><p>Then invokes the agent:</p><pre>response = agent.invoke(<br>    {&quot;messages&quot;: [question]},<br>    config,<br>)</pre><p>Notice that config contains:</p><pre>&quot;thread_id&quot;: &quot;1&quot;</pre><p>The agent therefore associates this interaction with thread 1.</p><p>The response contains the conversation messages, including the user’s message and the AI’s response.</p><h3>8. Asking the Agent Later</h3><p>Now comes the interesting part.</p><p>The second question is:</p><pre>question2 = HumanMessage(<br>    content=&quot;What&#39;s my favorite color?&quot;<br>)</pre><p>And we invoke the agent again:</p><pre>response2 = agent.invoke(<br>    {&quot;messages&quot;: [question2]},<br>    config,<br>)</pre><p>The same configuration is used:</p><pre>config = {<br>    &quot;configurable&quot;: {<br>        &quot;thread_id&quot;: &quot;1&quot;<br>    }<br>}</pre><p>Because the conversation belongs to the same thread, the agent can access the previously stored conversation state.</p><p>The result is:</p><pre>Your favorite color is Green!</pre><p>That’s the essence of conversational memory in this example.</p><h3>9. Why thread_id Matters So Much</h3><p>Imagine you’re building a customer-support chatbot.</p><p>You might have:</p><pre>Customer A → thread_id = &quot;customer-A&quot;<br>Customer B → thread_id = &quot;customer-B&quot;<br>Customer C → thread_id = &quot;customer-C&quot;</pre><p>Now each customer gets an independent conversation history.</p><pre>AI Agent<br>                │<br>       ┌────────┼────────┐<br>       ▼        ▼        ▼<br>   Customer A Customer B Customer C<br>      │           │          │<br>   Thread A    Thread B   Thread C</pre><p>Without some form of conversation/session identifier, maintaining independent conversational state becomes much harder.</p><h3>10. But Memory Isn’t Enough</h3><p>At this point we have an agent that can remember.</p><p>But imagine something goes wrong.</p><p>The user asks:</p><blockquote><em>“Who is the Chief Minister of Tamil Nadu?”</em></blockquote><p>The agent decides to use the web-search tool.</p><p>Now several things happen:</p><pre>User Question<br>     ↓<br>Agent<br>     ↓<br>LLM decides to call web_search<br>     ↓<br>Tavily<br>     ↓<br>Search results<br>     ↓<br>LLM<br>     ↓<br>Final answer</pre><p>If the final answer is wrong, slow, or unexpectedly expensive, how do we debug it?</p><p>Printing the final response isn’t enough.</p><p>This is where <strong>observability</strong> becomes important.</p><h3>11. Enter LangSmith 🔍</h3><p>The second example integrates the agent with LangSmith.</p><p>The required environment variables include:</p><pre>os.environ[&quot;LANGSMITH_TRACING&quot;] = &quot;true&quot;<br>os.environ[&quot;LANGSMITH_API_KEY&quot;] = &quot;YYY&quot;<br>os.environ[&quot;LANGSMITH_ENDPOINT&quot;] = &quot;https://langsmith-endpoint&quot;<br>os.environ[&quot;LANGSMITH_PROJECT&quot;] = &quot;local-ollama-agent&quot;</pre><p>Once tracing is enabled, the LangChain execution can be observed in LangSmith.</p><p>The important variable is:</p><pre>LANGSMITH_TRACING=true</pre><p>This tells the LangChain/LangSmith integration that execution tracing should be enabled.</p><h3>12. Why Tracing Is Better Than print()</h3><p>A common debugging approach is:</p><pre>print(response)</pre><p>That’s useful.</p><p>But it doesn’t tell you the entire story.</p><p>Consider an agent execution:</p><pre>Question<br>   │<br>   ├── LLM call<br>   │<br>   ├── Tool decision<br>   │<br>   ├── Web search<br>   │<br>   ├── Tool response<br>   │<br>   ├── Another LLM call<br>   │<br>   └── Final response</pre><p>A single print() generally gives you the final output.</p><p>Tracing gives you visibility into the execution itself.</p><p>The example specifically highlights the ability to inspect structured information such as parameters, execution timing, and which calls were made.</p><p>That’s incredibly useful when developing AI agents.</p><h3>13. What Can We Learn From a Trace?</h3><p>Suppose an agent responds slowly.</p><p>Without tracing, you might think:</p><blockquote><em>“The LLM is slow.”</em></blockquote><p>With tracing, you may discover:</p><pre>Total execution: 8 seconds</pre><pre>LLM call              1.5 sec<br>Web search             5.2 sec<br>Final LLM call         1.3 sec</pre><p>Suddenly the problem is obvious.</p><p>The LLM wasn’t the main bottleneck.</p><p>The web-search tool was.</p><p>Similarly, if the agent produces an unexpected answer, tracing can help answer questions such as:</p><ul><li>Which model was called?</li><li>What prompt was sent?</li><li>Which tool was selected?</li><li>What arguments were passed to the tool?</li><li>What did the tool return?</li><li>How long did each step take?</li><li>What happened before the final response?</li></ul><p>This is the difference between <strong>seeing the answer</strong> and <strong>understanding the execution</strong>.</p><h3>14. Putting Memory and Observability Together</h3><p>Now let’s combine both ideas.</p><p>A useful production-style mental model is:</p><pre>                         ┌───────────────┐<br>                         │     User      │<br>                         └───────┬───────┘<br>                                 │<br>                                 ▼<br>                      ┌────────────────────┐<br>                      │     AI Agent       │<br>                      └─────────┬──────────┘<br>                                │<br>                 ┌──────────────┼──────────────┐<br>                 │              │              │<br>                 ▼              ▼              ▼<br>             Memory          LLM            Tools<br>          InMemorySaver     Ollama          Tavily<br>                 │              │              │<br>                 └──────────────┼──────────────┘<br>                                │<br>                                ▼<br>                         ┌─────────────┐<br>                         │ LangSmith   │<br>                         │   Tracing   │<br>                         └─────────────┘</pre><p>Each component has a different responsibility.</p><h3>🧠 Memory</h3><p>Answers:</p><blockquote><em>“What happened earlier in this conversation?”</em></blockquote><h3>🤖 LLM</h3><p>Answers:</p><blockquote><em>“How should I reason about the user’s request?”</em></blockquote><h3>🛠️ Tools</h3><p>Answer:</p><blockquote><em>“What external capabilities can I use?”</em></blockquote><h3>🔍 LangSmith</h3><p>Answers:</p><blockquote><em>“What exactly happened during execution?”</em></blockquote><p>This separation is extremely useful when designing agentic applications.</p><h3>15. The Complete Conceptual Flow</h3><p>Let’s imagine a user starts a conversation.</p><h3>Step 1 — User provides information</h3><pre>&quot;My name is Amit and my favorite color is Green.&quot;</pre><p>The agent processes the message and stores the state associated with:</p><pre>thread_id = 1</pre><h3>Step 2 — User asks another question</h3><pre>&quot;What&#39;s my favorite color?&quot;</pre><p>The agent retrieves the relevant conversation state.</p><p>It can then answer:</p><pre>&quot;Your favorite color is Green.&quot;</pre><h3>Step 3 — User asks a knowledge question</h3><pre>&quot;Who is the Chief Minister of Tamil Nadu?&quot;</pre><p>The agent may determine that it needs external information.</p><p>It can invoke:</p><pre>web_search</pre><h3>Step 4 — Tool executes</h3><p>Tavily performs the search and returns results.</p><h3>Step 5 — LLM generates the answer</h3><p>The model processes the tool results and produces the final response.</p><h3>Step 6 — LangSmith records the execution</h3><p>The complete execution can be inspected through tracing.</p><p>Now the developer has visibility into the entire workflow.</p><h3>16. A Note About Secrets</h3><p>The examples show API keys directly through environment variables:</p><pre>os.environ[&quot;TAVILY_API_KEY&quot;] = &quot;XXX&quot;<br>os.environ[&quot;LANGSMITH_API_KEY&quot;] = &quot;YYY&quot;</pre><p>That’s convenient for a quick demonstration.</p><p>However, <strong>don’t hard-code real API keys in source code</strong>, especially if the repository is public.</p><p>A better approach is to use environment variables from your shell or a .env file that isn&#39;t committed to Git.</p><p>For example:</p><pre>export TAVILY_API_KEY=&quot;...&quot;<br>export LANGSMITH_API_KEY=&quot;...&quot;<br>export LANGSMITH_TRACING=&quot;true&quot;<br>export LANGSMITH_PROJECT=&quot;local-ollama-agent&quot;</pre><p>Then your Python code can simply read those variables.</p><h3>17. Why This Architecture Matters</h3><p>The interesting lesson isn’t really about InMemorySaver or LangSmith individually.</p><p>It’s about how AI applications are evolving.</p><p>A basic chatbot looks like:</p><pre>User → LLM → Response</pre><p>An AI agent looks more like:</p><pre>┌── Memory<br>                   │<br>User → Agent → LLM ├── Tools<br>                   │<br>                   └── State<br>                        │<br>                        ▼<br>                    Tracing</pre><p>As applications become more complex, the ability to <strong>remember state</strong> and <strong>observe execution</strong> becomes just as important as the model itself.</p><h3>18. Memory vs. Persistence</h3><p>One important distinction is worth making.</p><p>The example uses:</p><pre>InMemorySaver()</pre><p>This provides memory while the application is running.</p><p>But “memory” doesn’t necessarily mean durable persistence.</p><p>For example:</p><pre>Application starts<br>      ↓<br>Thread 1 created<br>      ↓<br>Conversation stored in memory<br>      ↓<br>Application restarts<br>      ↓<br>Memory is gone</pre><p>For a production application, you generally need persistent storage if conversations must survive application restarts.</p><p>So think of the example as demonstrating the <strong>concept and API pattern</strong>, rather than a complete production persistence architecture.</p><h3>19. A Simple Mental Model</h3><p>If you’re new to LangChain/LangGraph agents, remember these four concepts:</p><h3>1️⃣ Agent</h3><p>The orchestrator.</p><pre>agent = create_agent(...)</pre><h3>2️⃣ Model</h3><p>The reasoning engine.</p><pre>llm = ChatOllama(...)</pre><h3>3️⃣ Memory</h3><p>The conversation state.</p><pre>checkpointer=InMemorySaver()</pre><h3>4️⃣ Observability</h3><p>The execution visibility.</p><pre>LANGSMITH_TRACING=true</pre><p>Together:</p><pre>Agent<br> ├── Model<br> ├── Memory<br> ├── Tools<br> └── Observability</pre><p>That’s a powerful foundation for building real-world AI applications.</p><h3>Conclusion</h3><p>Building an AI agent isn’t just about getting an LLM to generate a response.</p><p>A useful agent needs <strong>context</strong>.</p><p>A reliable agent needs <strong>state</strong>.</p><p>A tool-using agent needs <strong>orchestration</strong>.</p><p>And a production-grade agent needs <strong>observability</strong>.</p><p>The two examples we explored demonstrate these ideas using a lightweight local stack:</p><pre>Ollama<br>   +<br>Llama 3.2<br>   +<br>LangChain Agent<br>   +<br>InMemorySaver<br>   +<br>Tavily<br>   +<br>LangSmith</pre><p>The memory example shows how an agent can maintain conversation state using a thread_id and InMemorySaver.</p><p>The LangSmith example shows how the same general agent architecture can be instrumented so that developers can inspect execution details rather than relying only on print() statements.</p><p>And that’s the bigger lesson:</p><blockquote><strong><em>Don’t just build an AI agent that can answer. Build one that can remember, use tools, and explain what happened.</em></strong></blockquote><p>That’s when an LLM starts becoming a real application component rather than just a text-generation API.</p><h3>🚀 What’s Next?</h3><p>Once you’ve understood these building blocks, there are several natural extensions:</p><ul><li>Replace InMemorySaver with a persistent database.</li><li>Support multiple users and conversation threads.</li><li>Add more tools such as databases, APIs, or calculators.</li><li>Add human-in-the-loop approval.</li><li>Stream agent responses to the UI.</li><li>Use LangSmith traces to monitor production agents.</li><li>Measure latency and token consumption.</li><li>Add evaluation datasets to test agent quality automatically.</li></ul><p>The basic agent is only the beginning.</p><p><strong>Memory gives your agent context. Tools give it capabilities. Observability gives you control.</strong></p><p>And together, these three pieces form the foundation of a much more reliable agentic AI system.</p><h3>Source Code</h3><p>The examples discussed in this article are available in the author’s GitHub repository:</p><p><a href="https://github.com/code-with-amitk/Code-examples/blob/master/Machine_Learning/Langchain/Memory_Persistence/README.adoc?utm_source=chatgpt.com">LangChain Memory Persistence example</a></p><p><a href="https://github.com/code-with-amitk/Code-examples/blob/master/Machine_Learning/Langsmith/Code/AI_Agent_Sending_Log_to_Langsmith.adoc?utm_source=chatgpt.com">LangSmith Agent Logging example</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9b9bad5f8322" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Empowering AI Agents: How to Give LangChain Superpowers with Custom Tools]]></title>
            <link>https://medium.com/@everythingismindgame/empowering-ai-agents-how-to-give-langchain-superpowers-with-custom-tools-32037de322a9?source=rss-6e9afd5faba4------2</link>
            <guid isPermaLink="false">https://medium.com/p/32037de322a9</guid>
            <category><![CDATA[langchain]]></category>
            <category><![CDATA[generative-ai-tools]]></category>
            <dc:creator><![CDATA[Everything is MindGame]]></dc:creator>
            <pubDate>Mon, 07 Sep 2026 14:29:24 GMT</pubDate>
            <atom:updated>2026-09-07T14:29:24.451Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kIMvPKHaoHTiE_CI4n3idg.png" /></figure><p>Large Language Models (LLMs) like GPT-4 or Claude are incredible at writing essays, summarizing documents, and answering general knowledge questions. However, they hit a wall when it comes to some critical tasks. Especially when you downloaded a Agent(eg: llama) on your laptop and trying to use it. One of blocker is:</p><ol><li><strong>Accessing real-time, up-to-date web information</strong> (LLMs are restricted by their training cutoff dates).</li></ol><p>So, how do we bridge this gap? The answer lies in <strong>LangChain Tools</strong>.</p><p>In this article, we’ll explore two fundamental examples of custom and built-in tools in LangChain:</p><ul><li><strong>Web Search Tool (</strong><strong>web_search.adoc)</strong>: Giving an agent live access to the internet using DuckDuckGo.</li></ul><h3>What is a LangChain?</h3><p>Langchain is a framework for creating Agents, which can interact with LLM models, Tools and ask LLMs to perform certain tasks. In real sense Agents can be thought of as Masters and LLMs as slaves responding to queries doing tasks asked by Agents.</p><h3>What is a Tool in LangChain?</h3><p>Think of an LLM as the “brain” of an AI agent. While the brain can reason and plan, it needs hands and instruments to interact with the external world. In LangChain, <strong>Tools</strong> are functions or utilities that an agent can invoke to perform specific actions — such as searching Google, querying a database, or running a Python snippet.</p><p>When a user asks a question, the agent determines:</p><ol><li><em>Do I know this answer off the top of my head?</em></li><li><em>If not, which Tool should I use to get the correct answer?</em></li></ol><h3>Integrating Live Web Search (web_search.adoc)</h3><p>What happens when an user asks about current events, such as today’s weather or recent news? Because model knowledge is frozen in time, we need a web search tool.</p><h3>The Code</h3><p>Using the built-in DuckDuckGoSearchRun tool, we can give our LLM real-time internet access without needing expensive API keys:</p><pre>from langchain_community.tools import DuckDuckGoSearchRun<br>from langchain.agents import initialize_agent, AgentType<br>from langchain_openai import ChatOpenAI<br><br># 1. Instantiate the built-in DuckDuckGo search tool<br>search_tool = DuckDuckGoSearchRun()<br><br># 2. Initialize our LLM (e.g., GPT-3.5 or GPT-4)<br>llm = ChatOpenAI(temperature=0, model=&quot;gpt-3.5-turbo&quot;)<br><br># 3. Create our tools list<br>tools = [<br>    search_tool<br>]<br><br># 4. Initialize an Agent equipped with the search tool<br>agent = initialize_agent(<br>    tools=tools,<br>    llm=llm,<br>    agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION,<br>    verbose=True<br>)<br><br># 5. Ask a real-time question<br>response = agent.run(&quot;What were the top news headlines today?&quot;)<br>print(response)</pre><h3>How It Works (Step-by-Step)</h3><ol><li><strong>DuckDuckGoSearchRun</strong>: LangChain provides pre-packaged integrations with popular search engines. DuckDuckGo allows instant web queries without requiring registration or secret keys.</li><li><strong>Autonomous Tool Selection</strong>: When asked about recent news, the LLM realizes its pre-trained memory is insufficient. It formulates a search query, passes it to DuckDuckGo, parses the search results, and synthesizes a concise, readable summary back to the user.</li><li><strong>verbose=True</strong>: Setting verbose=True prints the agent&#39;s inner monologue (Thought, Action, Action Input, Observation), giving you a front-row seat to how the AI decides to search the web.</li></ol><h3>Bringing It Together: The Power of Multi-Tool Agents</h3><p>The true magic of LangChain happens when you combine tools! Imagine equipping a single agent with <strong>with</strong> multiple toolscalculate_custom_square_root tool and the DuckDuckGoSearchRun tool:</p><pre>tools = [calculate_custom_square_root, search_tool]<br>agent = initialize_agent(tools=tools, llm=llm, agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION)</pre><h3>Key Takeaways</h3><ul><li><strong>Extensibility</strong>: Tools turn LLMs from simple text generators into actionable software agents capable of executing code and searching the live web.</li><li><strong>Precision</strong>: Custom tools prevent hallucinations by handing specialized tasks (like math or database lookups) to dedicated Python code.</li><li><strong>Simplicity</strong>: LangChain makes defining custom tools as simple as writing standard Python functions with clear docstrings.</li></ul><p>Enjoyed this article? Give it a few claps 👏 and follow for more beginner-friendly AI and LangChain tutorials!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=32037de322a9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Demystifying Distributed Systems: Sharding, 2PC, and the Saga Pattern at Scale]]></title>
            <link>https://medium.com/@everythingismindgame/demystifying-distributed-systems-sharding-2pc-and-the-saga-pattern-at-scale-c346a06c15a8?source=rss-6e9afd5faba4------2</link>
            <guid isPermaLink="false">https://medium.com/p/c346a06c15a8</guid>
            <category><![CDATA[database]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[saga]]></category>
            <dc:creator><![CDATA[Everything is MindGame]]></dc:creator>
            <pubDate>Thu, 03 Sep 2026 03:53:50 GMT</pubDate>
            <atom:updated>2026-09-03T03:53:50.864Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jSQUagHRdxAeCGagdYs4cw.png" /></figure><h3>Transaction?</h3><p>A transaction is a way for an application to group several reads and writes(INSERT INTO, SELECT x FROM) together into 1 logical unit.<br>Its upto Application to decide whether it wants every read/write to be written directly into DB or group several statements as 1 transaction</p><pre>Application --INSERT INTO--&gt; |DB Manager| --&gt; | DB |</pre><h3>Modes of Transactions</h3><p><strong>AutoCommit (Default Mode)</strong></p><p><strong>Every INSERT is a seperate transaction</strong></p><ul><li>Application does not call any dbmgr API, its default mode. Every read/write is a seperate transaction</li><li>That means every insert is directly written into DB<br>When you connect to the database and execute an INSERT command, the driver immediately sends a hidden BEGIN before the command and a hidden COMMIT immediately after it finishes. Every single command you run is its own, isolated, permanent transaction</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*_Qu_mBmfJRZWUxdBlYhOYA.png" /></figure><p><strong>Explicit Mode</strong></p><p><strong>x number of operations are grouped as 1 transaction</strong></p><ul><li>Application want dbmgr, to group x reads/writes into 1 transaction<br>To group multiple operations(INSERT, DELETE etc) into one logical unit, the application must override the default behavior by sending specific SQL commands(db.begin_transaction()) before executing the data manipulation statements.</li><li><strong>Commit:</strong><br> The application sends multiple data packets (e.g., INSERT, UPDATE, DELETE statements). The DBMgr writes these changes to a Temporary log (Write-Ahead Log) rather than immediately writing to DB, once application sends Commit, these are written to DB.<br> The application sends a final COMMIT to save everything DB</li><li><strong>Rollback(error occured):</strong><br> Application sends Rollback to discard all changes. Once the database Rollback, it deletes entry from Temporary(WAL log)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/726/1*qZsV8QW7Mgv0rJBV5RJZKg.png" /></figure><h3>Why Transaction is needed?</h3><ol><li>Atomicity:<br>- if Application is writing 20KB json document to DB. After 10KB node crashes<br>- User will see partially updated values which is Wrong. if write is done as part of transaction into Atomic DB, partial write is discarded and db moves to prev state</li><li>Safety Gurantees</li></ol><h3>Distributed Transaction</h3><p><strong>What:</strong> A database operation that updates data across 2 or more network seperated databases<br><strong>Features of Distributed Transaction</strong></p><ol><li>All Commit or All Abort: Either all nodes commit or all nodes abort.</li><li>ACID: Distributed transaction should follow ACID</li></ol><h3>Methods to Achieve Distributed Transactions</h3><h3>1. Two Phase Commit (2PC) (Also called XA Transaction)</h3><p>External transaction coordinator (e.g., Zookeeper) manages the commit process across multiple databases.</p><p>Consider a <strong>Example of a pizza ordering system:</strong><br> Where the order service acts as the coordinator, which generates a globally unique transaction ID<br> kitchen service and transaction service are participants.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/738/1*YCviJGPZc3w_y3usOMBvpw.png" /></figure><p><strong>Steps</strong></p><ol><li>Coordinator generates globally unique transactionId and sends to microservices. Now-on-wards microservices will use same id for communication.</li><li>Coordinator sends Prepare message(with transactionId) asking microservices whether they can commit or not?</li><li>When microservices recieves prepare, it makes sure that it can commit to DB at any cost.<br>- Even if microservices crashes After sending Yes. Then after recovery it has to commit the transaction.<br>- There is no turning Back.</li><li>Coordinator recieves response from all nodes. This is called COMMIT POINT</li><li>Coordinator writes its decision to Transaction log. so that if any microervice responds with No, whole commit can be moved to prev state.</li><li>Phase-2: Coordinator Send Message to commit. microervice Commit ie write to DB else //if any one says no Phase-2: Send Abort Message to all microervice via coordinator</li></ol><p>DOUBT/UNCETAIN STATE:</p><ul><li>Coordinator sends prepare, microervice voted Yes, and Coordinator crashes.</li><li>Now microervices are in Doubt state. microervices will wait forever for Coordinator to send Phase-2 message(Commit or Abort).</li></ul><p>HEURISTICS DECISIONS:<br>Microervices can communicate &amp; decide amongst themseleves once did not hear from Coordinator for particular time.</p><h3>Problems with 2PC</h3><ol><li><strong>Not supported by modern technologies</strong> Eg: NoSQL DB does not support. Apache Kafka, Cassandra, RabbitMQ does not support.</li><li><strong>2PC is form of Synchronous RPC:</strong> Kitchen microservice is waiting, holding locks If coordinator crashes → Kitchen blocks indefinitely and even cannot write other transactions<br> Solution: Saga Design Pattern</li></ol><h3>2. Saga Design Pattern</h3><p><strong>Dictionary Meaning of Saga:</strong><br> A long, epic story or narrative detailing a series of adventures, events, and twists over time. Just like an epic story has multiple chapters that unfold sequentially — where a plot twist or failure in chapter three might force the characters to backtrack or undo what they did in chapter one<br> Saga design pattern is carried over multiple microservices, where each microservice does its own local DB transaction, listen on message &amp; also compensate or revert the DB write if something goes wrong.</p><p><strong>Where Saga can be used?</strong></p><ol><li>Distributed database updates(ie Distributed Transactions)</li><li>Anywhere in long-running business process or workflow that spans multiple isolated systems that cannot use standard ACID transactions.<br>Examples:</li></ol><ul><li>E-Commerce Checkout &amp; Fulfillment: Many microsservices, many databases</li><li>Travel Booking: Flight, Hotel, Car Rental all are separate systems</li><li>Loan Application: Credit Check, Risk Assessment, Document Verification. All are separate systems</li></ul><p><strong>Concept of Saga</strong></p><ul><li>A Saga is an architectural pattern for managing distributed transactions across multiple independent microservices without locking data globally.</li><li>Microservice 1 performs a DB/Shard write to its database and publishes an event to Kafka, Microservice 2 consume this event to execute the next local write(on its DB).</li><li>If any step down the line fails, the saga sequentially triggers compensating transactions (undo actions) via event messages to maintain eventual consistency across the system</li></ul><p><strong>Twitter System Example</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DMLU3YQ-hcnT2GEdq4tPPQ.png" /></figure><ul><li>(Steps 1–4)<br> Rather than consulting a central coordinator, the Recvr Pod independently computes the destination shard using local CPU memory (hash(userId) % num of Shards).<br> This eliminates network bottlenecks and allows any stateless pod to process the incoming request identically without external lookups</li><li>(Steps 5–6)<br> The pod directly writes the tweet to Shard1 (Shard_01) and receives an acknowledgment</li><li>(Steps 7–8)<br> Instead of Pod publishes a TweetCreated event to Kafka and immediately returns a 200 OK response to the client.</li><li>(Steps 9–11)<br> Consumer Pod picks up the event from Kafka to execute the next phase of the workflow. It determines the target shard (either via local hash calculation or by reading metadata directly from the event payload) and writes the data to the Downstream Timeline Shard (Shard_02).</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c346a06c15a8" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>