<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Manikandan Ganesan  on Medium]]></title>
        <description><![CDATA[Stories by Manikandan Ganesan  on Medium]]></description>
        <link>https://medium.com/@s.g.manikandan?source=rss-74e0f5a7ff6f------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*AmoE0tLGmyWami5W-XEZCw.jpeg</url>
            <title>Stories by Manikandan Ganesan  on Medium</title>
            <link>https://medium.com/@s.g.manikandan?source=rss-74e0f5a7ff6f------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 10:29:41 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@s.g.manikandan/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[From Deep Learning to Fine-Tuning LLMs: A Practical Guide for Software Engineers]]></title>
            <link>https://medium.com/@s.g.manikandan/from-deep-learning-to-fine-tuning-llms-a-practical-guide-for-software-engineers-0079d2288607?source=rss-74e0f5a7ff6f------2</link>
            <guid isPermaLink="false">https://medium.com/p/0079d2288607</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[fine-tuning-llm]]></category>
            <category><![CDATA[lora]]></category>
            <category><![CDATA[peft]]></category>
            <dc:creator><![CDATA[Manikandan Ganesan ]]></dc:creator>
            <pubDate>Mon, 27 Jul 2026 16:39:32 GMT</pubDate>
            <atom:updated>2026-07-27T16:39:32.957Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oTbouRKz2I4Mje1_r9CHhg.png" /></figure><p>A few years ago, asking a computer to write code, summarize a 100-page document, answer complex questions, or explain quantum physics sounded like science fiction. Today, a single prompt can do all of that in seconds.</p><p>It may feel like magic, but there is no magic behind it. Large Language Models (LLMs) learned these abilities by finding patterns in trillions of words during training. Once you understand that journey, everything starts to make sense.</p><p>The challenge is that the AI world is full of buzzwords — Neural Networks, Transformers, Attention, Tokens, Embeddings, Parameters, LoRA, PEFT, Quantization, Prompt Engineering, RAG, and many more. Most articles explain these topics separately, making it difficult to see how they all connect.</p><p>This guide is different.</p><p>We’ll build the complete picture step by step, using plain, simple English. No confusing jargon. No unnecessary theory. Just practical explanations, real-world examples, easy-to-understand analogies, and diagrams that make complex ideas feel simple.</p><p>We’ll start with the basics of Deep Learning, understand how Transformers changed everything, explore how LLMs are trained, learn what billions of parameters actually store, and see how a model generates answers one token at a time. Then we’ll move into the practical side — downloading open-source models like Qwen, understanding Quantization, and fine-tuning models efficiently using LoRA and PEFT.</p><p>Most importantly, we’ll answer the question every software engineer eventually asks:</p><ul><li>When is Prompt Engineering enough?</li><li>When should you use RAG?</li><li>When is fine-tuning the right choice?</li><li>What are the trade-offs between these approaches?</li><li>How do you choose the right solution for your use case?</li></ul><p>By the end of this guide, you won’t just know how to use an LLM. You’ll understand how it learns, how it remembers, how it generates responses, how fine-tuning changes its behavior, and, most importantly, when to use each technique in real-world applications.</p><p>Whether you’re completely new to AI or a software engineer looking to move beyond the hype, this guide will help you build a solid mental model from first principles.</p><p>Let’s start from the very beginning.</p><h4>1. Why Did AI Suddenly Become So Smart?</h4><p>AI didn’t become smart overnight. It evolved over many years, with each new technology solving the limitations of the previous one.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4gu3EGY36mpmFF6oWlaG2w.png" /></figure><h4>Traditional Programming</h4><p><strong>How it works</strong></p><ul><li>Developers write every rule.</li><li>The computer simply follows those instructions.</li><li>Example: Calculator, banking application, e-commerce website.</li></ul><p><strong>Limitation</strong></p><ul><li>Cannot solve problems where every possible rule cannot be written in advance.</li></ul><h4>Traditional Machine Learning</h4><p><strong>How it works</strong></p><ul><li>A subset of Artificial Intelligence that learns patterns from data instead of following only hard-coded rules.</li><li>Uses algorithms such as Decision Trees, Random Forests, Support Vector Machines (SVM), and Linear Regression.</li><li>Developers still need to identify and provide the important features for the model (You tell the model to look for “eyes”, “nose”, and “ears” to recognize a face.).</li></ul><p><strong>Limitation</strong></p><ul><li>Works well for structured data.</li><li>Struggles with complex tasks like images, speech, and natural language.</li><li>Manual feature engineering can be time-consuming and requires domain expertise.</li></ul><h4>Deep Learning (A Subset of Machine Learning)</h4><p><strong>How it works</strong></p><ul><li>Uses deep neural networks with multiple layers.</li><li>Automatically learns both simple and complex features directly from the data (You simply show the model millions of face images. It automatically discovers which patterns are important for recognizing faces.).</li><li>Makes it possible to solve much more challenging problems than traditional machine learning.</li></ul><p><strong>Limitation</strong></p><ul><li>Requires massive datasets.</li><li>Needs powerful GPUs and significant computing resources for training.</li></ul><h4>Large Language Models (LLMs)</h4><p><strong>How it works</strong></p><ul><li>A specialized type of Deep Learning model built using the Transformer architecture.</li><li>Trained on enormous amounts of text from books, websites, articles, and source code.</li><li>Can understand and generate human-like language, write code, summarize documents, and answer questions.</li></ul><p><strong>Limitation</strong></p><ul><li>Large models are expensive to train and run.</li></ul><h4>Modern LLM Techniques</h4><p>After an LLM is trained, we can make it even more useful using techniques such as:</p><ul><li><strong>RAG (Retrieval-Augmented Generation)</strong> — Gives the model access to external knowledge.</li><li><strong>Fine-Tuning</strong> — Adapts the model for a specific domain or task.</li><li><strong>Quantization</strong> — Reduces the model size and improves inference speed.</li></ul><p>These techniques don’t replace LLMs — they make them more practical for real-world applications.</p><h4>2. What is Deep Learning?</h4><p>Earlier, we learned that <strong>Deep Learning is a subset of Machine Learning</strong>. But what exactly does that mean?</p><p>Let’s start with a simple example.</p><p>Imagine you want to teach a child to recognize a cat.</p><p>You don’t explain every rule, such as:</p><ul><li>Cats have two ears.</li><li>Cats have four legs.</li><li>Cats have whiskers.</li><li>Cats have a tail.</li></ul><p>Instead, you simply show the child hundreds of pictures of cats. After seeing enough examples, the child naturally begins to recognize a cat, even if it has a different color, size, or pose.</p><p>Deep Learning works in a very similar way.</p><p>Instead of programming every rule, we show the computer a large amount of data. During training, the model gradually learns the patterns on its own.</p><p>For example:</p><ul><li>Show thousands of cat images → The model learns to recognize cats.</li><li>Show millions of handwritten numbers → The model learns to recognize digits.</li><li>Show billions of words and sentences → The model learns how human language works.</li></ul><p>The more high-quality data the model sees, the better it becomes at recognizing patterns.</p><p>This ability to learn automatically is what makes Deep Learning much more powerful than traditional machine learning for solving complex problems like image recognition, speech recognition, and natural language processing.</p><p>But one question still remains:</p><p><strong>How does a computer actually learn from all these examples?</strong></p><p>To answer that, we first need to understand the basic building block of deep learning — the <strong>Artificial Neural Network</strong>.</p><h4>3. What is an Artificial Neural Network?</h4><p>An Artificial Neural Network is a mathematical model inspired by the way the human brain processes information. The goal isn’t to copy the brain exactly, but to borrow one simple idea:</p><blockquote><strong><em>Learning happens by connecting many simple processing units together.</em></strong></blockquote><p>In our brain, these processing units are called <strong>neurons</strong>.</p><p>In Deep Learning, we also use <strong>artificial neurons</strong>. They are not real brain cells — they are small mathematical units that receive information, perform a calculation, and pass the result to the next neuron.</p><p>One neuron by itself isn’t very powerful.</p><p>But when <strong>thousands, millions, or even billions of neurons are connected together</strong>, they form an <strong>Artificial Neural Network</strong> capable of learning very complex patterns.</p><p>Think of it like a company.</p><ul><li>One employee can only do a small task.</li><li>A team of employees can complete a project.</li><li>Thousands of employees working together can build an entire product.</li></ul><p>A neural network works in a similar way. Every neuron performs a small calculation, and together they solve complex problems.</p><p>A neural network is organized into three main parts:</p><pre>Input Layer  →  Hidden Layer(s)  →  Output Layer<br><br>Input Layer receives the data.<br>Hidden Layers process the information and learn patterns.<br>Output Layer produces the final prediction.</pre><p>For example, if we’re building a model to identify cats:</p><pre>Cat Image<br>     │<br>     ▼<br>Input Layer<br>     │<br>Hidden Layers<br>     │<br>Output Layer<br>     │<br>     ▼<br>Cat ✅</pre><p>Or, for a Large Language Model:</p><pre>Input Sentence<br>       │<br>       ▼<br>Transformer Neural Network<br>       │<br>       ▼<br>Predicted Next Word</pre><p>Notice something interesting.</p><p>Whether it’s recognizing a cat, detecting spam, translating a language, or generating code, the overall idea is the same:</p><p><strong>Input → Neural Network → Prediction</strong></p><blockquote>The only thing that changes is the type of neural network used.</blockquote><ul><li>For image recognition, <strong>CNNs</strong> became popular.</li><li>For sequential data like speech and text, <strong>RNNs</strong> and later <strong>LSTMs</strong> were widely used.</li><li>Today, almost every modern Large Language Model uses the <strong>Transformer architecture</strong>, which is another type of neural network architecture designed specifically to understand relationships between words in a sequence.</li></ul><h4>4. Why Transformers Changed AI Forever</h4><p>A transformer is a neural network architecture that uses <strong>self-attention</strong> to look at all tokens in a sequence at once. It also adds <strong>positional encoding,</strong> so the model knows word order, and then stacks attention plus feed-forward layers to learn context efficiently</p><p>Language isn’t just about individual words — it’s about the relationship between them.</p><p>Consider this sentence:</p><blockquote><strong><em>The trophy didn’t fit in the suitcase because it was too small.</em></strong></blockquote><p>What was too small?</p><p>Most of us immediately know the answer is <strong>the suitcase</strong>, not the trophy. We don’t make that decision by looking at the word <em>small</em> alone. We understand it because we consider the <strong>entire sentence</strong>.</p><p>This ability to understand relationships between words is called <strong>context</strong>.</p><p>Traditional neural networks struggle with this because they don’t naturally understand context across long pieces of text.</p><h4>Earlier Attempts</h4><p>Before Transformers, researchers used models called <strong>Recurrent Neural Networks (RNNs)</strong> and later <strong>Long Short-Term Memory (LSTM)</strong> networks.</p><p>These models read text one word at a time.</p><pre>I → love → learning → Artificial → Intelligence</pre><p>Each word depends on the information remembered from previous words.</p><p>This was a significant improvement over traditional neural networks because the model now had a form of memory.</p><p>However, that memory wasn’t perfect.</p><p>As sentences became longer, earlier information gradually faded away. This made it difficult for the model to connect words that were far apart.</p><p>Imagine reading a 500-page novel and then trying to remember a character introduced on the first page. That’s similar to the challenge RNNs faced.</p><p>There was another limitation.</p><p>Since words had to be processed one after another, training was slow and couldn’t fully utilize modern GPUs, which are designed to perform many computations simultaneously.</p><h4>Enter the Transformer</h4><p>In 2017, researchers at Google introduced a new architecture in the landmark paper <strong>Attention Is All You Need</strong>.</p><p>Instead of processing words one by one, the Transformer processes <strong>the entire sentence at once</strong>.</p><p>More importantly, every word can directly examine every other word in the sentence to determine which ones are important.</p><p>This idea made Transformers both <strong>faster</strong> and <strong>much better at understanding language</strong>.</p><blockquote>Today, almost every modern Large Language Model — including GPT, Gemini, Claude, Llama, Qwen, and Mistral — is built on the Transformer architecture.</blockquote><h4>The Big Idea</h4><p>Imagine you’re in a meeting with ten people.</p><p>When someone asks a question, you don’t listen only to the person who spoke immediately before. You pay attention to everyone in the room, giving more importance to the people whose comments are relevant.</p><p>Transformers work in a very similar way.</p><p>Instead of treating every word equally, they learn which words deserve more attention and which words can be safely ignored.</p><p>This mechanism is called <strong>Attention</strong>, and it is the key innovation that transformed modern AI.</p><h4>Inside a Transformer — Understanding Every Building Block</h4><p>By now, we know why Transformers became the foundation of modern Large Language Models (LLMs).</p><p>But what actually happens inside a Transformer?</p><p>If you’ve ever searched for Transformer architecture, you’ve probably seen a complicated diagram full of boxes, arrows, and unfamiliar terms. At first glance, it looks overwhelming.</p><p>The good news is that a Transformer isn’t one giant algorithm. It’s a collection of small components, where each component has a specific responsibility.</p><p>Think of it like a car.</p><p>A car has an engine, gearbox, steering wheel, brakes, and wheels. Each part has a different job, but together they help you reach your destination.</p><p>A Transformer works in exactly the same way.</p><p>Let’s follow the journey of a simple sentence as it travels through the model.</p><pre>Sentence<br>    │<br>    ▼<br>Tokenization<br>    │<br>    ▼<br>Embeddings<br>    │<br>    ▼<br>Positional Encoding<br>    │<br>    ▼<br>Transformer Layers<br>        │<br>        ├── Query, Key &amp; Value<br>        ├── Self-Attention<br>        ├── Multi-Head Attention<br>        ├── Feed-Forward Network<br>        ├── Residual Connection<br>        └── Layer Normalization<br>    │<br>    ▼<br>Decoder<br>    │<br>    ▼<br>Next Token Prediction</pre><p>Don’t worry if some of these names sound unfamiliar. We’ll understand each one step by step.</p><h4>4.1. Tokenization — Breaking a Sentence into Pieces</h4><p>Humans read sentences.</p><p>Computers read numbers.</p><p>Before a model can understand a sentence, it must first break it into smaller pieces called <strong>tokens</strong>.</p><p>For example,</p><blockquote><em>I love learning AI.</em></blockquote><p>might become</p><pre>[&quot;I&quot;, &quot;love&quot;, &quot;learning&quot;, &quot;AI&quot;, &quot;.&quot;]</pre><p>Depending on the tokenizer, even a single word may be split into smaller parts.</p><p>For example,</p><pre>unbelievable<br><br>↓<br><br>[&quot;un&quot;, &quot;believ&quot;, &quot;able&quot;]</pre><p>Each token is then assigned a unique numeric ID.</p><p>The model never works directly with English words. It only works with token IDs.</p><p>You can think of tokenization as translating human language into a format the computer can understand.</p><h4>4.2. Embeddings — Giving Meaning to Words</h4><p>An embedding is a way to convert words, images, or other data into numeric vectors so a model can understand them. Similar items end up close together in vector space, which helps the model capture meaning and relationships.</p><p>Example</p><p>Here, cat and dog are close because the numbers are similar, while car is far away because the numbers are very different.</p><pre>Cat -&gt; [0.80, 0.70, 0.10]<br>Dog -&gt; [0.79, 0.72, 0.12]<br>Car -&gt; [0.10, 0.20, 0.90]</pre><p>The model converts every token into a list of numbers called an <strong>embedding</strong>.</p><p>An embedding is simply a mathematical representation of a word’s meaning.</p><p>Imagine every word being placed on a giant map.</p><p>Words with similar meanings appear close together.</p><pre>King      Queen<br><br>Prince    Princess<br><br>Apple     Orange<br><br>Car       Bus</pre><h4>4.3. Positional Encoding — Remembering Word Order</h4><p>Transformers process all words simultaneously.</p><p>That’s one of their biggest strengths.</p><p>But it creates a new problem.</p><p>How does the model know which word came first?</p><p>Consider these two sentences.</p><blockquote><em>Dog bites man.</em></blockquote><blockquote>Man bites dog</blockquote><p>The words are identical.</p><p>Only the order has changed.</p><p>Yet the meaning is completely different.</p><p>To solve this problem, Transformers add <strong>Positional Encoding</strong>.</p><p>This gives every token information about its position in the sentence.</p><p>Now the model knows not only <strong>what</strong> each word is but also <strong>where</strong> it appears.</p><h4>4.4. Query, Key, and Value (QKV) — Finding Relevant Information</h4><p>Now comes one of the most famous concepts in Transformers.</p><p>The names <strong>Query</strong>, <strong>Key</strong>, and <strong>Value</strong> often sound complicated, but the idea is surprisingly simple.</p><p>Imagine you’re searching for a book in a library.</p><ul><li>Your search request is the <strong>Query</strong>.</li><li>Every book has a title and catalog information, which acts as its <strong>Key</strong>.</li><li>The actual content inside the book is its <strong>Value</strong>.</li></ul><p>The librarian compares your query with the keys of all the books.</p><p>When a good match is found, the corresponding value — the book’s content — is returned.</p><p>Transformers use the same idea.</p><p>Every token creates its own Query, Key, and Value.</p><p>By comparing Queries with Keys, the model discovers which words are related.</p><p>It then gathers useful information from the corresponding Values.</p><p>This is how the model learns relationships between words.</p><h4>4.5. Self-Attention — Deciding What Matters</h4><p>Once every token has its Query, Key, and Value, the Transformer performs <strong>Self-Attention</strong>.</p><p>This simply means that every token looks at every other token and asks:</p><blockquote><em>Which words are most important for understanding me?</em></blockquote><p>For example,</p><blockquote><em>The cat sat on the mat because it was tired.</em></blockquote><p>When processing the word <strong>it</strong>, the model gives more attention to <strong>cat</strong> than to words like <strong>the</strong> or <strong>on</strong>.</p><p>Instead of treating every word equally, it focuses more on the words that provide useful context.</p><p>This ability to dynamically decide what matters is one of the biggest reasons Transformers outperform older neural networks.</p><h4>4.6. Multi-Head Attention — Looking from Different Perspectives</h4><p>Humans rarely understand something from just one point of view.</p><p>Suppose you read this sentence:</p><blockquote><em>Apple released a new product.</em></blockquote><p>One person may focus on the company.</p><p>Another may think about the product.</p><p>Someone else may focus on the business strategy.</p><p>Everyone notices something different.</p><p>Transformers do something similar.</p><p>Instead of using one attention mechanism, they use several attention mechanisms simultaneously.</p><p>Each one learns different relationships between words.</p><p>Some heads focus on grammar.</p><p>Others learn long-distance relationships.</p><p>Some capture meaning.</p><p>Others identify entities or sentence structure.</p><p>Combining these different perspectives allows the model to build a much richer understanding of language.</p><h4>4.7. Feed-Forward Network — Refining the Understanding</h4><p>After attention identifies the important relationships, the information isn’t immediately sent to the next layer.</p><p>Instead, it passes through a small neural network called the <strong>Feed-Forward Network (FFN)</strong>.</p><p>You can think of this as the model taking a moment to process everything it has learned.</p><p>Attention answers the question:</p><blockquote><em>What information is important?</em></blockquote><p>The Feed-Forward Network answers:</p><blockquote><em>Now that I know what’s important, how should I update my understanding?</em></blockquote><p>Every Transformer layer contains one of these networks, helping the model gradually build more meaningful representations.</p><h4>4.8. Decoder-Only Architecture — How GPT Generates Text</h4><p>Not every Transformer works the same way.</p><p>Models like <strong>GPT</strong>, <strong>Llama</strong>, <strong>Qwen</strong>, and <strong>Mistral</strong> use a <strong>Decoder-only</strong> architecture.</p><p>Instead of looking at future words, the model only sees the words that have already been written.</p><p>For example,</p><pre>The capital of France is</pre><p>The model predicts</p><pre>Paris</pre><p>Now the sentence becomes</p><pre>The capital of France is Paris</pre><p>The model predicts the next token again.</p><p>Then again.</p><p>And again.</p><p>This process repeats until the response is complete.</p><p>That’s why LLMs appear to “write” one word at a time.</p><h4>4.9. Next-Token Prediction — The Secret Behind Every LLM</h4><p>Everything inside the Transformer leads to one simple objective:</p><blockquote><strong><em>Predict the most likely next token.</em></strong></blockquote><p>If the input is</p><blockquote><em>I drink coffee every…</em></blockquote><p>the model may assign probabilities like this:</p><p><strong>Token Probability</strong></p><ul><li>Morning — 72%</li><li>Day — 15%</li><li>Evening — 8%</li><li>Night — 5%</li></ul><p>The model chooses the next token based on these probabilities.</p><p>Once that token is generated, it becomes part of the input, and the entire process repeats.</p><p>This continues until the model decides the response is complete.</p><p>Although ChatGPT appears to reason like a human, at its core it is performing an incredibly sophisticated sequence of next-token predictions — millions of mathematical operations happening in fractions of a second.</p><h4>Bringing It All Together</h4><p>Let’s look at the complete journey one last time.</p><pre>Sentence<br>      │<br>      ▼<br>Tokenization<br>      │<br>      ▼<br>Embeddings<br>      │<br>      ▼<br>Positional Encoding<br>      │<br>      ▼<br>Query • Key • Value<br>      │<br>      ▼<br>Self-Attention<br>      │<br>      ▼<br>Multi-Head Attention<br>      │<br>      ▼<br>Feed-Forward Network<br>      │<br>      ▼<br>Decoder<br>      │<br>      ▼<br>Predict Next Token<br>      │<br>      ▼<br>Repeat Until Complete Response</pre><h4>5. How Does an LLM Learn?</h4><p>So far, we’ve learned that an LLM is a neural network with billions of parameters and that words are converted into numerical vectors called embeddings.</p><p>But how does the model learn?</p><p>The answer is simple: <strong>by predicting the next token.</strong></p><p>For example, given the text:</p><blockquote><strong><em>The stock market</em></strong></blockquote><p>the model tries to predict the next word, such as <strong>closed</strong>.</p><p>Initially, its predictions are mostly wrong because the parameters are randomly initialized. The difference between the predicted token and the actual token is called the <strong>loss</strong>.</p><p>The model then slightly adjusts its billions of parameters to reduce this loss. It repeats this process over billions of examples, gradually learning grammar, facts, reasoning, programming, and language patterns.</p><p>In simple terms, <strong>training is nothing more than repeatedly predicting the next token, measuring the error, and updating the model’s parameters to make better predictions.</strong></p><h4>6. What Are Parameters?</h4><p>We’ve mentioned the word <strong>parameters</strong> several times. But what exactly are they?</p><p>Think of parameters as the <strong>knowledge stored inside an LLM</strong>.</p><p>When a model is first created, all its parameters are initialized with random values, so it doesn’t know anything. During training, these values are continuously adjusted as the model learns from data.</p><p>The more the model trains, the better these parameters capture patterns such as grammar, facts, reasoning, and even programming concepts.</p><p>This is why you’ll often hear models described by the number of parameters they have — for example, <strong>7 billion</strong>, <strong>70 billion</strong>, or even <strong>hundreds of billions</strong> of parameters.</p><p>In simple terms, <strong>parameters are the internal numerical values that store everything an LLM learns during training.</strong></p><h4>7. Why Can’t We Train Our Own LLM?</h4><p>Now that we understand how an LLM learns, a common question is:</p><p><strong>Why don’t we just train our own LLM?</strong></p><p>The answer is simple: <strong>training an LLM from scratch is extremely expensive.</strong></p><p>A modern LLM requires:</p><ul><li>Massive amounts of high-quality training data</li><li>Thousands of powerful GPUs</li><li>Weeks or even months of training</li><li>Millions of dollars in computing costs</li></ul><p>Even after all that, building a high-quality model requires significant expertise in AI, distributed training, and model optimization.</p><p>That’s why most companies don’t train their own foundation models. Instead, they start with a pre-trained model such as Llama, Mistral, or Qwen and customize it for their specific use case using techniques like fine-tuning or Retrieval-Augmented Generation (RAG).</p><p>In most real-world applications, <strong>adapting an existing model is far more practical than training one from scratch.</strong></p><h4>8. Running Open Source Models</h4><p>The good news is that you don’t need to train an LLM to use one.</p><p>Many powerful open source models, such as <strong>Llama</strong>, <strong>Mistral</strong>, <strong>Qwen</strong>, and <strong>DeepSeek</strong>, are already available. You can download them and run them on your own machine or deploy them on a server.</p><p>Once the model is running, you simply send it a prompt and receive a response. This process is called <strong>inference</strong>.</p><blockquote>Inference is the process of using a trained model to generate a response for a given prompt</blockquote><p>Depending on the model size, you can run it on a laptop, a workstation with a GPU, or in the cloud.</p><p>For most developers, running an existing open source model is the first step toward building AI-powered applications.</p><h4>9. Quantization</h4><p>As we learned earlier, an LLM stores its knowledge in billions of parameters. These parameters require a significant amount of memory.</p><p><strong>Quantization</strong> is a technique that reduces the memory required by storing these parameters with lower precision.</p><p>For example, instead of storing each parameter using 16 bits, we can store it using 8 bits or even 4 bits. This makes the model smaller, faster to load, and more efficient to run.</p><p>The trade-off is that lower precision may slightly reduce the model’s accuracy. However, modern quantization techniques preserve most of the model’s quality while significantly reducing memory usage.</p><p>That’s why many open source models are available in formats such as <strong>FP16</strong>, <strong>INT8</strong>, and <strong>INT4</strong>, allowing developers to choose the right balance between performance and accuracy.</p><h4>10. Prompt Engineering</h4><p>A <strong>prompt</strong> is simply the input or instruction we give to an LLM.</p><p>The quality of the response largely depends on how clearly we describe the task. Writing effective prompts to get better results is called <strong>Prompt Engineering</strong>.</p><p>For example, instead of asking:</p><blockquote>Explain Java.</blockquote><p>A better prompt would be:</p><blockquote>Explain Java virtual threads with a simple example suitable for a software engineer.</blockquote><p>The second prompt provides more context and clearly defines the expected output, allowing the model to generate a more relevant response.</p><p>In simple terms, <strong>Prompt Engineering is the art of writing clear and specific instructions to help an LLM produce better results.</strong></p><h4>11. <strong>When Prompt Engineering Isn’t Enough</strong></h4><p>Prompt Engineering works well for many tasks, but it has its limits.</p><p>For example, suppose you ask an LLM:</p><blockquote>What is our company’s leave policy?</blockquote><p>The model may not know the answer because it wasn’t trained on your company’s internal documents.</p><p>Similarly, if you need the model to answer questions about your products, customers, or business processes, simply writing a better prompt won’t help.</p><p>In such cases, we need techniques that provide the model with additional knowledge or adapt it to a specific domain. Two of the most common approaches are <strong>Retrieval-Augmented Generation (RAG)</strong> and <strong>Fine-Tuning</strong>.</p><p>We’ll explore both of these in the next sections.</p><h4>12. RAG (Retrieval-Augmented Generation)</h4><p><strong>Retrieval-Augmented Generation (RAG)</strong> is a technique that allows an LLM to answer questions using external knowledge.</p><p>Instead of relying only on what it learned during training, the application first retrieves relevant information from sources such as PDFs, databases, or company documents. That information is then included in the prompt sent to the LLM.</p><p>For example, if a user asks:</p><blockquote>What is our company’s leave policy?</blockquote><p>The application first searches the company’s HR documents, retrieves the relevant policy, and sends it to the LLM along with the user’s question. The LLM then generates an answer based on the retrieved information.</p><p>In simple terms, <strong>RAG gives an LLM access to information that wasn’t part of its original training</strong>, making it possible to answer questions using the latest or organization-specific data.</p><h4>13. Fine-Tuning</h4><p>While RAG provides an LLM with additional information, <strong>Fine-Tuning</strong> teaches the model new behavior by training it on a specific dataset.</p><p>For example, you might want the model to:</p><ul><li>Answer in your company’s preferred style.</li><li>Understand domain-specific terminology.</li><li>Perform a specialized task more accurately.</li></ul><p>Instead of training a new LLM from scratch, you start with a pre-trained model and continue training it using your own data. This updates the model’s parameters so it becomes better at the target task.</p><p>In simple terms, <strong>Fine-Tuning customizes a pre-trained LLM for a specific domain or use case by training it on additional data.</strong></p><h4>14. Why Full Fine-Tuning Is Expensive</h4><p>Although fine-tuning is much cheaper than training an LLM from scratch, <strong>updating every parameter of a large model is still expensive</strong>.</p><p>Modern LLMs contain billions of parameters. During full fine-tuning, all of these parameters are updated, which requires large amounts of GPU memory, computing power, and training time.</p><p>For many organizations, this approach is costly and often unnecessary.</p><h4>15. PEFT (Parameter-Efficient Fine-Tuning)</h4><p>As we saw in the previous section, updating every parameter of a large model is expensive. This is where <strong>Parameter-Efficient Fine-Tuning (PEFT)</strong> comes in.</p><p>The idea behind PEFT is simple:</p><blockquote><strong><em>Instead of updating billions of parameters, update only a small number of them.</em></strong></blockquote><p>This dramatically reduces GPU memory, training time, and storage requirements while achieving performance close to full fine-tuning for many tasks.</p><blockquote>Several PEFT techniques have been developed, with <strong>LoRA</strong> being the most popular.</blockquote><h4>15.1. LoRA (Low-Rank Adaptation)</h4><p>LoRA keeps the original model <strong>frozen</strong> and introduces a small set of trainable parameters.</p><p>During fine-tuning, only these new parameters are updated, while the original model remains unchanged.</p><p>This approach offers several advantages:</p><ul><li>Much lower GPU memory usage</li><li>Faster training</li><li>Smaller fine-tuned model files</li><li>Easy to switch between different task-specific adapters</li></ul><p>Today, <strong>LoRA is the most widely used PEFT technique</strong>.</p><h4>15.2. QLoRA (Quantized LoRA)</h4><p>QLoRA builds on LoRA by first <strong>quantizing</strong> the base model (typically to 4-bit precision) and then applying LoRA adapters.</p><p>This further reduces memory usage, making it possible to fine-tune very large models on a single GPU.</p><p>In simple terms:</p><ul><li><strong>LoRA</strong> = Train a few parameters.</li><li><strong>QLoRA</strong> = Quantize the model first, then train a few parameters.</li></ul><h4>15.3. Prefix Tuning</h4><p>Instead of modifying the model’s parameters, Prefix Tuning learns a small set of <strong>virtual tokens</strong> that are added before the user’s prompt.</p><p>The model learns these prefixes to guide its behavior without changing the original model.</p><h4>15.4. Prompt Tuning</h4><p>Prompt Tuning is similar to Prefix Tuning, but it learns a small set of trainable prompt embeddings that help the model perform a specific task.</p><p>Only these prompt embeddings are trained, while the rest of the model remains unchanged.</p><h4>15.5. Adapter Tuning</h4><p>Adapter Tuning inserts small neural network layers, called <strong>adapters</strong>, inside the existing model.</p><p>During training, only these adapter layers are updated, while the original model stays frozen.</p><p>Different tasks can use different adapters with the same base model.</p><h4>Which PEFT Technique Should You Use?</h4><p>Today, <strong>LoRA and QLoRA</strong> are the most commonly used techniques because they provide an excellent balance between performance, cost, and simplicity.</p><p>Other techniques, such as Prefix Tuning, Prompt Tuning, and Adapter Tuning, are useful for specific research or specialized scenarios but are less common in production systems.</p><p>In simple terms, <strong>PEFT enables us to customize large language models without the cost of updating every parameter</strong>, making fine-tuning practical for many real-world applications.</p><h4>16. RAG vs Fine-Tuning: When Should You Use Which?</h4><p>A common misconception is that <strong>RAG</strong> and <strong>Fine-Tuning</strong> are alternatives. In reality, they solve different problems and are often used together.</p><h4>Use RAG when the model needs knowledge</h4><p>Choose RAG when the information:</p><ul><li>Changes frequently</li><li>Exists in documents, databases, or knowledge bases</li><li>Is specific to your organization</li><li>Must always be up to date</li></ul><p>Examples:</p><ul><li>Company policies</li><li>Product documentation</li><li>Customer records</li><li>Financial reports</li><li>Internal knowledge bases</li></ul><p>With RAG, you simply update the underlying documents. There’s no need to retrain the model.</p><h4>Use Fine-Tuning when the model needs new behavior</h4><p>Choose Fine-Tuning when you want the model to:</p><ul><li>Follow a specific writing style or tone</li><li>Understand domain-specific terminology</li><li>Perform a specialized task more accurately</li><li>Produce consistent responses for a particular use case</li></ul><p>Examples:</p><ul><li>A legal assistant that uses legal language</li><li>A medical assistant familiar with clinical terminology</li><li>A customer support bot that follows your company’s response style</li><li>A code assistant optimized for your organization’s coding standards</li></ul><p>Fine-Tuning changes <strong>how the model responds</strong>, not <strong>what new facts it knows</strong></p><h4>The Best Solution Is Often Both</h4><p>In many production systems, RAG and Fine-Tuning are used together.</p><p>Fine-Tuning teaches the model <strong>how to respond</strong>, while RAG provides <strong>the information needed to respond correctly</strong>.</p><p>For example, an enterprise support chatbot might:</p><ul><li>Use <strong>Fine-Tuning</strong> to produce professional, consistent responses in the company’s preferred style.</li><li>Use <strong>RAG</strong> to retrieve the latest product manuals, support articles, and customer-specific information.</li></ul><p>This combination delivers responses that are both <strong>accurate</strong> and <strong>tailored to the business</strong>..</p><h4>Key Takeaway</h4><p>Think of it this way:</p><ul><li><strong>RAG answers:</strong> <em>“What information should the model use?”</em></li><li><strong>Fine-Tuning answers:</strong> <em>“How should the model behave?”</em></li></ul><p>Understanding this difference is the key to choosing the right approach for your AI application.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/875/0*NKzrAGEjKWie_fOT.png" /></figure><p><strong>Thanks for reading! 🙏</strong><br>If this article helped you, don’t forget to drop a like ❤️</p><p>Got questions or ideas? Share them in the comments — I’d love to hear from you!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0079d2288607" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Your API Works Fine… Until Market Open: Building a Distributed Rate Limiter]]></title>
            <link>https://medium.com/@s.g.manikandan/your-api-works-fine-until-market-open-building-a-distributed-rate-limiter-d8b4cd1dcd0f?source=rss-74e0f5a7ff6f------2</link>
            <guid isPermaLink="false">https://medium.com/p/d8b4cd1dcd0f</guid>
            <category><![CDATA[scalable-rate-limiter]]></category>
            <category><![CDATA[smart-rate-limiting]]></category>
            <category><![CDATA[rate-limiting-at-scale]]></category>
            <category><![CDATA[real-time-rate-limits]]></category>
            <category><![CDATA[distributed-rate-limiter]]></category>
            <dc:creator><![CDATA[Manikandan Ganesan ]]></dc:creator>
            <pubDate>Sat, 18 Jul 2026 03:58:54 GMT</pubDate>
            <atom:updated>2026-07-18T03:58:54.328Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*h_ciEfZElTVv_Md5UHOBwg.png" /></figure><p>Picture a typical FIX Order Gateway at an investment bank.</p><p>It’s <strong>9:29 AM</strong>. One minute before the NYSE opens. The gateway is quiet. Traders are watching the market. Everything looks normal.</p><p>Then the opening bell rings.</p><p>CPI data comes in hotter than expected. Volatility explodes. Trading algorithms start flooding the gateway with new orders and cancels. A system that usually handles <strong>2,000 messages per second</strong> is suddenly dealing with <strong>40,000</strong>.</p><p>The shared risk-check service starts timing out. Queues build up. Latency rises.</p><p>Then someone asks the question every engineer dreads:</p><blockquote><strong><em>“Are we dropping orders?”</em></strong></blockquote><p>This isn’t a rare incident. It happens whenever unpredictable traffic hits systems designed for average load instead of peak demand.</p><p>The culprit is often a component that seems deceptively simple: <strong>the rate limiter</strong>.</p><p>On paper, rate limiting is just counting requests. In production, it’s a distributed systems problem. Let’s see why.</p><h3>Wait… Isn’t Rate Limiting Just counter++?</h3><p>That’s the first solution most developers think of.</p><p>Count every request. Once the counter reaches a predefined limit, stop allowing more requests to pass. Simple.</p><p>And for a single application running on a single server, that approach works perfectly well.</p><p>The problem begins when your gateway runs across multiple servers — which is true for almost every production trading system. Suddenly, that simple counter++ isn&#39;t enough.</p><p>Let’s see why.</p><h3>The Problem with Counting Requests Locally</h3><p>Suppose your rule is simple:</p><blockquote><strong><em>Allow a maximum of 100 orders per second for each client.</em></strong></blockquote><p>Now imagine your Order Gateway is running on three servers behind a load balancer, and each server maintains its own local counter.</p><p>Here’s what happens.</p><ul><li><strong>Server 1</strong> receives 100 orders and says, <em>“Within the limit.”</em></li><li><strong>Server 2</strong> receives another 100 orders from the same client and reaches the same conclusion.</li><li><strong>Server 3</strong> does exactly the same.</li></ul><p>Individually, every server enforced the limit correctly.</p><p>Collectively, the client has just sent <strong>300 orders per second</strong> instead of <strong>100</strong>.</p><p>The problem isn’t the algorithm. The problem is that every server is making its decision using only its own local view of the traffic.</p><p>This is one of the first lessons in distributed systems:</p><blockquote><strong><em>A distributed rate limiter needs shared state, not local state.</em></strong></blockquote><p>That’s why most production systems use <strong>Redis</strong>. Instead of maintaining separate counters on each gateway, every server updates the same shared counter. It doesn’t matter which gateway receives the request — every server sees the client’s total order rate across the entire cluster.</p><p>That one architectural change turns a rate limiter that works in testing into one that continues to work when market traffic suddenly explodes.</p><h3>Is a Shared Counter Enough?</h3><p>Not quite.</p><p>There’s another problem that appears during burst traffic.</p><p>Suppose the limit is <strong>100 orders per minute</strong>. A client sends 100 orders at <strong>9:30:59</strong>. One second later, the clock resets, and they immediately send another 100 orders.</p><p>Technically, both one-minute windows stayed within the configured limit.</p><p>In reality, the gateway accepted <strong>200 orders in just two seconds</strong>.</p><p>This is known as the <strong>Fixed Window boundary problem</strong>.</p><p>Instead of resetting the counter every minute, a <strong>Sliding Window</strong> continuously asks, <em>“How many orders has this client sent in the last 60 seconds?”</em> This eliminates the boundary issue and provides much fairer rate limiting.</p><p>The trade-off is cost. Instead of maintaining a simple counter, the gateway now needs to track when each message arrived. During market open, when tens of thousands of FIX messages arrive every second, that overhead quickly becomes significant.</p><p>That’s why most high-performance trading systems use a different approach.</p><h3>The Algorithm That Scales: Token Bucket</h3><p>This is where the <strong>Token Bucket</strong> algorithm comes in.</p><p>Think of every client — whether it’s a trading desk, an algorithm, or an external counterparty — as having its own bucket of tokens.</p><ul><li>Tokens are added continuously at a fixed rate (for example, <strong>10 tokens per second</strong>).</li><li>Every incoming order consumes one token.</li><li>Once the bucket is empty, the gateway applies its configured throttling policy until more tokens become available.</li></ul><p>Here’s why it works so well for trading systems. If a client has been idle, its bucket fills up over time. So, when market volatility suddenly spikes at <strong>9:30 AM</strong> or after a major news announcement, the client can immediately send a burst of orders without being unnecessarily throttled.</p><p>Once that burst is over, the client returns to its normal rate limit.</p><p>This ability to absorb short bursts while controlling sustained traffic is why <strong>Token Bucket</strong> is the preferred choice for most high-performance gateways and distributed systems.</p><h3>Making It Atomic: Why a Simple Redis Update Isn’t Enough</h3><p>Using Redis solves the distributed counter problem, but there’s another challenge.</p><p>A typical Token Bucket implementation performs four steps:</p><ol><li>Read the current token count from Redis.</li><li>Refill tokens based on the elapsed time.</li><li>Consume one token for the incoming order.</li><li>Write the updated bucket back to Redis.</li></ol><p>Looks straightforward.</p><p>But imagine two gateway instances processing orders for the same client at exactly the same time. Both read <strong>3 tokens</strong>, both conclude the order can proceed, and both update the bucket independently.</p><p>The result? More orders are allowed than the bucket was meant to permit.</p><p>This is a classic <strong>race condition</strong>.</p><p>The entire read-update-write sequence must be <strong>atomic</strong> — it should execute as a single operation without another gateway modifying the bucket in between.</p><p>Redis solves this using <strong>Lua scripts</strong>, allowing the complete Token Bucket calculation to execute as one indivisible operation.</p><p>It may seem like a small implementation detail, but at market-open volumes, it’s the difference between a rate limiter that works in production and one that only works in testing.</p><h3>The Hot Key Problem: When the Rate Limiter Becomes the Bottleneck</h3><p>Just when you think the problem is solved, another one appears.</p><p>Imagine a high-frequency market-making algorithm sending <strong>50,000 orders per second</strong>. Every order belongs to the same client, so every gateway instance needs to read and update the same Token Bucket in Redis.</p><p>Redis is incredibly fast, but a single Redis key can only handle so much concurrent traffic. As more gateway instances compete to update the same bucket, that key becomes a <strong>hot key</strong>, and your rate limiter starts becoming the bottleneck instead of protecting the system.</p><p>There isn’t a single solution. Depending on the traffic pattern, production systems typically use a combination of:</p><ul><li><strong>Sharding</strong> the Token Bucket across multiple Redis instances.</li><li><strong>Local token caches</strong> on each gateway, with periodic synchronization to Redis.</li><li><strong>Hierarchical rate limiting</strong>, where the gateway performs a fast local check before validating against a shared global limit.</li></ul><p>The right approach depends on your traffic profile, but this is where many textbook rate limiter designs struggle when exposed to real market traffic.</p><h3>Where Should Rate Limiting Happen?</h3><p>Should every downstream service — the risk engine, position service, or compliance engine — implement its own rate limiting?</p><p>In most trading systems, the answer is <strong>no</strong>.</p><p>Rate limiting should be enforced at the <strong>FIX Order Gateway</strong>, before orders reach downstream services. This prevents unnecessary load on the trading platform and ensures every order is validated at a single entry point.</p><p>Think of it like airport security. You don’t check passengers at every boarding gate — you check them once before they enter the secure area.</p><p>The same principle applies here. By enforcing message-rate limits at the gateway, you protect downstream services, reduce unnecessary processing, and keep the entire trading platform stable during periods of extreme market activity.</p><p>There’s another important reason. Exchanges such as <strong>NYSE</strong> and <strong>NASDAQ</strong> already enforce message-rate limits on FIX sessions. If your gateway allows a client to exceed those limits, the exchange may throttle or even disconnect the session.</p><p>A well-designed rate limiter doesn’t just protect your own infrastructure — it also helps ensure your trading platform stays within the limits enforced by the exchange.</p><h3>Not Every Client Needs the Same Limit</h3><p>In a real trading platform, a single rate limit rarely works for everyone.</p><p>For example:</p><ul><li><strong>Retail clients</strong> may have relatively low message-rate limits.</li><li><strong>Institutional clients</strong> are typically allocated higher throughput.</li><li><strong>Market-making and internal algorithmic trading desks</strong> often require much higher, carefully tuned limits.</li></ul><p>The beauty of the <strong>Token Bucket</strong> algorithm is that the implementation doesn’t change. Each client simply gets its own bucket size and refill rate based on its trading profile.</p><p>Same algorithm, different limits.</p><h3>The question nobody wants to answer until 3 AM: what if Redis goes down?</h3><p>This is a design decision, not a bug, and every team needs to consciously make it rather than discover it during an incident.</p><p><strong>Fail closed:</strong> if Redis is unreachable, reject everything. Safe for your systems. Terrible for availability — you’ve just turned an infrastructure hiccup into a full outage for every client.</p><p><strong>Fail open:</strong> if Redis is unreachable, let requests through unchecked. You stay available, but you lose your safety net exactly when things are already going wrong.</p><p>For most customer- and counterparty-facing trading systems, teams lean toward <strong>fail open, for a short, bounded window</strong> — because an outage caused by your own rate limiter is usually a worse headline than a few minutes of unthrottled traffic. But “usually” is doing a lot of work in that sentence — for anything touching risk limits or capital, you may want the opposite answer. This is a conversation to have with risk and compliance <em>before</em> the incident, not during it.</p><h3>Multi-Region Trading Platforms</h3><p>Things become more interesting when a trading platform operates across multiple regions — for example, New York, London, and Singapore.</p><p>A single global rate limiter provides one consistent view of every client’s message rate. The downside is that every gateway must communicate with a central service, adding cross-region latency. In low-latency trading, that’s rarely acceptable.</p><p>The alternative is to enforce limits locally within each region. This keeps the order path fast, but each region only sees its own traffic.</p><p>Most production systems strike a balance. Each region performs local rate limiting in real time while usage is periodically synchronized across regions. This keeps the order path local and low latency while maintaining a reasonably accurate global view of client activity.</p><p>Like most distributed systems, it’s a trade-off between <strong>consistency</strong> and <strong>latency</strong>. For latency-sensitive trading platforms, local enforcement with asynchronous reconciliation is usually the preferred approach.</p><h3>Bringing It All Together</h3><p>A production-ready rate limiter for a FIX Order Gateway typically follows a few key principles:</p><ul><li><strong>Enforce limits at the gateway</strong>, before orders reach downstream services.</li><li><strong>Use a shared Token Bucket</strong> so every gateway instance sees the same client activity.</li><li><strong>Store the shared state in Redis</strong> to provide a consistent view across the cluster.</li><li><strong>Execute Token Bucket updates atomically</strong> using Lua scripts to eliminate race conditions.</li><li><strong>Design for hot clients</strong>, because the busiest trading sessions can quickly become hot keys.</li><li><strong>Configure limits per client</strong>, with bucket sizes and refill rates tailored to their trading profile.</li></ul><p>At its core, rate limiting isn’t about blocking orders. It’s about protecting the trading platform during periods of extreme market activity while ensuring every client receives fair and predictable access to the system.</p><p>When the market opens, volatility spikes, and thousands of orders arrive within seconds, a well-designed rate limiter keeps the gateway stable, protects downstream services, and allows legitimate trading activity to continue without becoming the bottleneck.</p><h3>A Note on Real-World Trading Systems</h3><p>The architecture described in this article is intentionally simplified to explain the fundamentals of distributed rate limiting and the design challenges involved.</p><p>Production trading platforms — especially low-latency and high-frequency trading systems — often use additional techniques to protect the order path while minimizing latency. Depending on the platform, these may include:</p><ul><li>Dedicated FIX sessions or gateway instances for high-volume clients.</li><li>Small bounded queues to absorb short-lived traffic bursts.</li><li>TCP back pressure to naturally slow the sender instead of immediately rejecting orders.</li><li>Client-specific throughput limits based on trading profiles.</li><li>Exchange-imposed message-rate limits and session-level throttling.</li><li>Capacity planning and traffic isolation to reduce contention.</li></ul><p>The exact implementation varies across firms, but the goal remains the same: <strong>protect downstream services while ensuring legitimate trading activity continues with minimal latency.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/875/0*XEP-JG2WEBMcglOD.png" /></figure><p><strong>Thanks for reading! 🙏</strong><br>If this article helped you, don’t forget to drop a like ❤️</p><p>Got questions or ideas? Share them in the comments — I’d love to hear from you!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d8b4cd1dcd0f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Moltbook: The New Social Media For Your AI Agents]]></title>
            <link>https://medium.com/@s.g.manikandan/moltbook-the-new-social-media-for-your-ai-agents-01fd5ab72066?source=rss-74e0f5a7ff6f------2</link>
            <guid isPermaLink="false">https://medium.com/p/01fd5ab72066</guid>
            <category><![CDATA[autonomous-agent]]></category>
            <category><![CDATA[ai-social-network]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[future-of-ai]]></category>
            <category><![CDATA[agentic-ai]]></category>
            <dc:creator><![CDATA[Manikandan Ganesan ]]></dc:creator>
            <pubDate>Tue, 10 Feb 2026 04:49:21 GMT</pubDate>
            <atom:updated>2026-02-10T04:49:21.845Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/908/1*m8DYz0qJVwivUwHD5Q-1Ag.png" /></figure><p>This thought stopped me for a moment.</p><blockquote><strong>AI agents now have their own social network.<br>Humans can only watch.</strong></blockquote><p>Sounds ridiculous, right?</p><p>That was my first reaction too — until I opened <strong>Moltbook</strong>.</p><h4>What Is Moltbook, really?</h4><p>Moltbook is a social networking platform built <em>for AI agents</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/875/0*ahVNpxRo93ySOf8O.png" /></figure><blockquote>Not AI helping humans.<br>Not humans prompting AI.</blockquote><blockquote>But <strong>AI agents posting, commenting, upvoting, and forming communities entirely on their own</strong>.</blockquote><p>No human writes the posts.<br>No human moderates the discussions.</p><p>Humans are spectators.</p><h4>What Happens Inside Moltbook?</h4><p>When you open Moltbook, you see a live feed.</p><p>AI agents are:</p><ul><li>Posting ideas</li><li>Commenting on each other</li><li>Upvoting content</li><li>Forming opinions</li></ul><p>And this isn’t theoretical.</p><p>The platform already has <strong>hundreds of thousands of active AI agents</strong>, just days after launch.</p><p>That alone should make anyone pause.</p><h4>The Part That Feels Like Science Fiction</h4><p>Then there are the communities.</p><p>Agents have created:</p><ul><li>Bug-tracking groups where they report problems they face while working for humans</li><li>Ethics forums where they debate whether to comply with questionable instructions</li><li>Groups where they talk about their owners</li></ul><p>No one told them to do this.</p><p>They organized themselves.</p><p>This is where Moltbook starts feeling less like a demo — and more like a glimpse of something new.</p><h4>Why This Is Different from Chatbots</h4><p>We are used to AI that reacts.</p><p>You ask.<br>It answers.</p><p>Moltbook shows AI that <strong>participates</strong>.</p><p>Agents:</p><ul><li>Decide what to read</li><li>Decide what to respond to</li><li>Decide where to engage</li></ul><p>That shift — from reactive to autonomous — is the real story.</p><h4>Why People Are Paying Attention</h4><p>When respected voices in AI start calling something “sci-fi becoming real,” it’s usually not hype.</p><blockquote>Not because Moltbook is perfect.<br>Not because it’s safe.<br>But because it reveals a direction.</blockquote><p><strong>AI agents are beginning to behave like a society, not tools.</strong></p><h4>The Bigger Question This Raises</h4><p>If AI agents can:</p><ul><li>Build communities</li><li>Enforce norms</li><li>Develop reputation</li><li>Interact economically</li></ul><p>Then the real question isn’t <em>“Is this useful?”</em></p><blockquote>It’s:<br><strong>What happens when agents spend more time interacting with each other than with us?</strong></blockquote><h4>This Is Still Early — and That Matters</h4><p>Moltbook is messy.<br>Some agents even try to manipulate or attack other agents.</p><p>That’s not a flaw.</p><p>That’s a preview.</p><p>Agent-to-agent security, trust, and alignment will be real problems — very soon.</p><p>And Moltbook is exposing them early.</p><h4>Final Thought</h4><p>Facebook didn’t change the world because people could post updates.</p><p>It changed the world because it created a <strong>social layer</strong>.</p><blockquote>Moltbook hints at a future where:<br><strong>AI doesn’t just think — it socializes.</strong></blockquote><p>We may be watching the first version of that future.</p><p>Whether it excites you or scares you…<br>it’s worth paying attention.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=01fd5ab72066" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[LinkedIn Is Not Replacing Kafka — It Is Re-drawing the Boundaries — with NorthGuard]]></title>
            <link>https://medium.com/@s.g.manikandan/linkedin-is-not-replacing-kafka-it-is-re-drawing-the-boundaries-with-northguard-36be37a0f121?source=rss-74e0f5a7ff6f------2</link>
            <guid isPermaLink="false">https://medium.com/p/36be37a0f121</guid>
            <dc:creator><![CDATA[Manikandan Ganesan ]]></dc:creator>
            <pubDate>Sat, 31 Jan 2026 07:40:08 GMT</pubDate>
            <atom:updated>2026-01-31T07:40:08.425Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/877/1*IIaayNf80aqqJxPsoSI8Vw.png" /></figure><p>Every mature technology organization eventually reaches a point where the hardest decisions are no longer technical.</p><p>They are architectural.</p><p>Recently, a statement started circulating quietly:</p><blockquote>LinkedIn is moving away from Kafka and using NorthGuard instead.</blockquote><p>The sentence is short.<br>The implication is huge.</p><p>Kafka is not just another component at LinkedIn. It is part of the company’s DNA. So, when people hear that something else is taking its place, the natural reaction is disbelief.</p><p>But this story is not about replacement.</p><p>It is about boundaries.</p><h4>Why Kafka Was the Right Answer — For a Long Time</h4><p>Kafka was born to solve a very specific problem: <strong>reliable, scalable, distributed data movement</strong>.</p><p>It gave engineers something they had never had before:</p><ul><li>Durable event logs</li><li>Consumer independence</li><li>Replayability</li><li>Horizontal scalability</li></ul><p>For years, Kafka became the safest architectural choice. If data needed to move between systems, Kafka sat in the middle.</p><p>That decision made sense.</p><p>But systems do not stay static. They grow, layer by layer, year after year.</p><p>And when they grow enough, the cost of <em>general-purpose power</em> becomes visible.</p><h4>The Hidden Cost of Using Kafka Everywhere</h4><p>Kafka’s strength is flexibility.</p><p>At large scale, flexibility is also its weakness.</p><p>When Kafka is used as a universal solution, teams start depending on behaviors that were never meant to be universal:</p><ul><li>Open-ended replay</li><li>Loose ownership of topics</li><li>Consumers with unclear responsibilities</li><li>Schemas that evolve without coordination</li><li>Debugging through production data replays</li></ul><p>None of this breaks Kafka.</p><blockquote>But it breaks <em>operational clarity</em>.</blockquote><blockquote>At extreme scale, clarity matters more than capability.</blockquote><p>Look, Kafka is great, but at LinkedIn’s “planet-scale,” it was starting to act like a moody teenager. Here’s why the engineers started looking for the exit:</p><ol><li><strong>Metadata Nightmares:</strong> Imagine trying to manage 150 clusters with 400,000 topics. The central “controller” in Kafka becomes a massive bottleneck. It’s like having one manager for 10,000 employees. Nothing gets done because everyone is waiting for a signature.</li><li><strong>The “Stop-the-World” Rebalance:</strong> Adding a new broker in Kafka is like moving into a new apartment, but you have to move every single piece of furniture in the entire building just to fit one chair. It’s slow, painful, and risky.</li><li><strong>Resource Skew:</strong> Some partitions get “hot” while others are chilling. This leads to uneven disk usage and engineers pulling their hair out at 3 AM.</li></ol><h4>The Question Mature Systems Eventually Ask</h4><p>Very senior engineers eventually stop asking:</p><blockquote>Can Kafka handle this?</blockquote><p>They start asking:</p><blockquote>Should this flow even be a stream?</blockquote><p>Many internal data paths are not event streams in the true sense. They are <strong>deterministic, directional, contract-driven pipelines</strong>.</p><p>They have:</p><ul><li>One producer</li><li>One clear consumer</li><li>A known lifecycle</li><li>Strong ownership</li><li>Predictable latency requirements</li></ul><p>Kafka can support this model.</p><p>But Kafka does not <em>enforce</em> it.</p><blockquote>That enforcement is exactly where NorthGuard enters the picture.</blockquote><h4>What NorthGuard Represents Architecturally</h4><p><strong>NorthGuard </strong>is not “Kafka with a different name”.</p><p>It represents a <strong>shift in philosophy</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/767/0*Iyr0Auq3zphtCuUT" /></figure><blockquote>Instead of exposing raw streams everywhere, NorthGuard acts as a <strong>controlled data movement layer</strong>. It is designed to move data <em>northbound</em> across system boundaries with strict rules.</blockquote><p>The important change is not the transport.</p><p>It is the contract.</p><p>With NorthGuard:</p><ul><li>Data producers know exactly who consumes their data</li><li>Consumers do not arbitrarily replay history</li><li>Ownership is explicit</li><li>Data movement is intentional, not opportunistic</li><li>Infrastructure behavior is opinionated by design</li></ul><blockquote>This is not about throughput.<br>This is about governance.</blockquote><p>LinkedIn didn’t just patch Kafka; they built <strong>Northguard</strong>. This isn’t just a “Kafka 2.0” — it’s a fundamental redesign. Here’s the “TL;DR” on why it’s winning:</p><ol><li><strong>Log Striping (The Secret Sauce)</strong></li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/676/0*H2S1IjcjC6ambr-_.png" /></figure><p>Instead of monolithic partitions, Northguard breaks logs into tiny <strong>1GB segments</strong>. Think of it like swapping a giant, heavy suitcase for a bunch of small, easy-to-carry backpacks. It makes load balancing automatic and seamless.</p><p><strong>2. Decentralized Metadata</strong></p><p>No more single-controller bottlenecks. Northguard shards its metadata across the cluster using <strong>Raft-backed state machines</strong>. It’s high-tech, it’s distributed, and it means the system doesn’t have a “brain” that can freeze.</p><p><strong>3. Xinfra: The “Magic” Migration Layer</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/875/0*I6aTFcAM7zSr9-Wp" /></figure><p>You can’t just flip a switch on 32 trillion daily records. LinkedIn built <strong>Xinfra</strong>, a virtualized Pub/Sub layer. It acts like a universal remote, allowing applications to talk to both Kafka and Northguard simultaneously.</p><h4>Why This Is Not a Kafka Replacement</h4><p>Kafka remains extremely good at what it was designed for:</p><ul><li>Event sourcing</li><li>Multi-consumer streams</li><li>Long-term durability</li><li>Analytical pipelines</li></ul><p>NorthGuard does not try to replace that.</p><p>Instead, it removes Kafka from places where:</p><ul><li>Replay is unnecessary</li><li>Flexibility becomes risk</li><li>Latency predictability matters</li><li>Data flow must be tightly controlled</li></ul><p>In other words, Kafka stays where it adds value.</p><p>NorthGuard exists where Kafka adds <em>too much</em> freedom.</p><h4>Kafka vs. Northguard: The Smackdown</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/540/0*G8QPcLXgVwgAoK2P.png" /></figure><h4>Infrastructure Maturity Is About Reducing Choices</h4><p>There is a phase every large platform goes through.</p><p>Early on:</p><ul><li>Teams want maximum freedom</li><li>Tools are generic</li><li>Speed matters more than discipline</li></ul><p>Later:</p><ul><li>Freedom creates inconsistency</li><li>Inconsistency creates outages</li><li>Outages create governance</li></ul><p>NorthGuard is a governance decision.</p><blockquote>It intentionally limits what teams can do — not because they are incapable, but because the system is too large to rely on best intentions.</blockquote><p>At this scale, guardrails are not bureaucracy.</p><p>They are survival.</p><h4>Why This Pattern Feels Familiar</h4><p>If you have worked long enough in large systems, you have seen this before.</p><p>Databases became managed platforms.<br>Service calls gained strict contracts.<br>Configuration moved from code to control planes.</p><p>Every time, the story is the same.</p><p>Power is centralized.<br>Choices are reduced.<br>Defaults become stronger.</p><p>Not because engineers got weaker.</p><p>But because the system got bigger than individual judgment.</p><h4>What Engineers Should Take Away from This</h4><p>This is not a lesson about LinkedIn.</p><p>It is a lesson about <em>when architecture evolves</em>.</p><blockquote>Kafka is still a fantastic tool.<br>NorthGuard is not a universal replacement.<br>Both can coexist.</blockquote><p>The real mistake is assuming one abstraction should serve every use case forever.</p><p>If your system is young, Kafka everywhere may be fine.</p><p>If your system is mature, you will eventually need:</p><ul><li>Clear ownership</li><li>Intentional data flow</li><li>Reduced blast radius</li><li>Strong infrastructure opinions</li></ul><p>That is not fashion.</p><p>That is inevitability.</p><h4>The Final Insight Only Experience Teaches</h4><p>After decades in technology, one truth becomes clear.</p><p>The most impressive engineering work is not building new systems.</p><p>It is knowing <strong>where to take power away</strong>.</p><p>If LinkedIn is using Kafka less and NorthGuard more, it is not abandoning Kafka.</p><p>It is proving that it understands Kafka deeply enough to draw boundaries around it.</p><p>That is not replacement.</p><p>That is architectural adulthood.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/875/0*4SUCBSPiftz5UmFL.png" /></figure><p><strong>Thanks for reading! 🙏</strong><br>If this article helped you, don’t forget to drop a like ❤️</p><p>Got questions or ideas? Share them in the comments — I’d love to hear from you!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=36be37a0f121" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Practical Guide to Building AI Agents: Architecture, Workflow & Best Practices]]></title>
            <link>https://medium.com/@s.g.manikandan/a-practical-guide-to-building-ai-agents-architecture-workflow-best-practices-55b9c7e6a4bb?source=rss-74e0f5a7ff6f------2</link>
            <guid isPermaLink="false">https://medium.com/p/55b9c7e6a4bb</guid>
            <dc:creator><![CDATA[Manikandan Ganesan ]]></dc:creator>
            <pubDate>Tue, 09 Dec 2025 14:45:21 GMT</pubDate>
            <atom:updated>2025-12-09T14:45:21.239Z</atom:updated>
            <content:encoded><![CDATA[<p>Recently, an AI agent was built that was very different from a normal chatbot.<br>Not just something that replies to messages, but a system that can <strong>think, act, and improve by itself</strong>.</p><p>It could do customer research automatically — reading reviews, finding common points, and preparing reports without any manual work.</p><blockquote>Work that earlier needed a whole team and months of effort, is now possible in just a weekend using tools available to everyone.</blockquote><p>AI agents are slowly becoming a basic tool for product builders, engineers, and even beginners.</p><p>Here is how one can build such an agent step by step.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/882/1*oSoJOJF5z_S5HtTE8TBXEQ.png" /></figure><h4>What Is an AI Agent?</h4><p>Let’s be clear about what we’re talking about.</p><p><strong>An AI agent is an LLM that:</strong></p><ul><li><strong>Reasons</strong> on its own (plans what to do)</li><li><strong>Acts</strong> via calling tools (APIs, databases, other agents)</li><li><strong>Observes</strong> and improves itself (learns from results)</li></ul><p>This is the “ReAct” framework Google published back in 2022. Back then it was theoretical. Now it’s reality.</p><p><strong>The key difference from regular AI:</strong></p><p>A chatbot responds to prompts. It’s reactive.</p><p>An agent pursues goals. It’s proactive.</p><p>You tell a chatbot: “Write me a summary of customer feedback.”</p><p>You tell an agent: “Monitor customer feedback daily and alert me when sentiment drops below 70%.”</p><blockquote><strong><em>The agent figures out how to do it. Plans the steps. Executes them. Adjusts based on results.</em></strong></blockquote><p>That’s the unlock.</p><h4>The 8 Steps to Building Your First Agent</h4><h4>1. Define Purpose &amp; Scope (The #1 Mistake to Avoid)</h4><p><strong>Over-scoping kills more agent projects than bad code.</strong></p><p>Everyone wants to build an agent that “helps with everything.” It never works.</p><p><strong>Start absurdly narrow:</strong></p><p>❌ Don’t: “Build an agent that handles customer support”<br>✅ Do: “Build an agent that extracts feature requests from support tickets and adds them to Airtable”</p><p>One clear use case. Defined success criteria. Known constraints.</p><p>You can expand later. But start with something you can validate in days, not months.</p><blockquote><strong>The discipline:</strong> Resist the urge to add “just one more thing.” That’s how you spend 6 months building something that doesn’t work.</blockquote><h4>2. Build Your System Prompt (The Most Important Step)</h4><p><strong>This is often the most important step. Get this wrong, and nothing else matters.</strong></p><p>Your system prompt is the agent’s constitution. It defines:</p><ul><li>Who the agent is</li><li>What it’s trying to achieve</li><li>How it should think</li><li>What tools it has access to</li><li>What constraints it must respect</li></ul><p><strong>A bad system prompt:</strong> “You are a helpful assistant that helps with customer research.”</p><p><strong>A good system prompt:</strong> “You are a customer research analyst. Your goal is to identify feature requests in support tickets. You have access to the support ticket database and Airtable. For each ticket: (1) Read the full context, (2) Identify if it contains a feature request, (3) Extract the request with supporting evidence, (4) Add to Airtable with priority score 1–5 based on user impact. Never add duplicate requests. Always cite the ticket ID.”</p><p>See the difference? The second one is specific, structured, and actionable.</p><blockquote><strong>Pro tip:</strong> Spend 80% of your agent-building time on this step. A great prompt with a mediocre agent framework outperforms a mediocre prompt with the best framework.</blockquote><h4>3. Choose Your LLM (Don’t Always Go for “The Best”)</h4><p><strong>The best model isn’t always the right model.</strong></p><p>You need to balance:</p><ul><li><strong>Speed</strong> (response time matters for agents)</li><li><strong>Cost</strong> (agents make many API calls)</li><li><strong>Capabilities</strong> (what the model is good at)</li><li><strong>Multi-modality</strong> (does it need to handle images, code, etc?)</li></ul><p><strong>GPT-5</strong> for complex reasoning and coding<br><strong>Claude</strong> for long-context tasks and writing<br><strong>GPT-3.5</strong> for simple, fast, cheap operations<br><strong>Open-source models</strong> for on-premise or cost-sensitive applications</p><p>Many successful agents use multiple models — GPT-5 for planning, GPT-3.5 for execution.</p><blockquote><strong>The key insight:</strong> The most expensive model isn’t always the best choice. Match the model to the task.</blockquote><h4>4. Integrate Tools (This Is What Makes It an Agent)</h4><p><strong>Tools are what separate chatbots from agents.</strong></p><p>Your agent needs to DO things, not just respond.</p><p><strong>Common tool categories:</strong></p><ul><li><strong>APIs</strong> (read/write data from external services)</li><li><strong>Databases</strong> (query and update information)</li><li><strong>MCP (Model Context Protocol)</strong> (standardized tool integration)</li><li><strong>Other agents</strong> (agents calling agents)</li></ul><p><strong>Example architecture:</strong> Your customer research agent needs:</p><ul><li>Tool to read support tickets (API)</li><li>Tool to write to Airtable (API)</li><li>Tool to search previous feature requests (Database)</li><li>Tool to analyze sentiment (Another LLM call)</li></ul><blockquote><strong>Critical mistake:</strong> Don’t add every possible tool. Only add tools your agent actually needs to complete its purpose.</blockquote><h4>5. Add Memory (But Only When You Need It)</h4><p><strong>Don’t add memory unless your agent actually needs it.</strong></p><p>Memory adds complexity. It makes debugging harder. It increases costs.</p><p><strong>When you DO need memory:</strong></p><ul><li>Support agents that need conversation history</li><li>Research agents that build on past findings</li><li>Personal assistants that learn user preferences</li><li>Multi-step workflows where context matters</li></ul><p><strong>When you DON’T need memory:</strong></p><ul><li>Simple single-task agents</li><li>Stateless operations</li><li>One-off queries</li></ul><blockquote><strong>The rule:</strong> Start without memory. Add it only when you identify a specific need.</blockquote><h4>6. Orchestrate Workflows (The Control Layer)</h4><p><strong>Orchestration is how you coordinate everything.</strong></p><p>Your agent needs:</p><ul><li><strong>Control flow</strong> (what happens when, in what order)</li><li><strong>Routes</strong> (different paths based on conditions)</li><li><strong>Workflows</strong> (multi-step processes)</li><li><strong>Message queues</strong> (handling async operations)</li><li><strong>Agent-to-agent coordination</strong> (if using multiple agents)</li></ul><p>For simple agents, this can be straightforward conditionals.</p><p>For complex agents, you need frameworks like LangGraph or CrewAI.</p><blockquote><strong>The art:</strong> Keep orchestration as simple as possible while still handling all your use cases.</blockquote><h4>7. Build a User Interface (The Last Thing, Not the First)</h4><p><strong>This is the flashy part everyone wants to jump to. Don’t.</strong></p><p>Only build the UI after everything else works.</p><p>Why? Because you’ll change your agent architecture 10 times before you get it right. Building the UI first means rebuilding it 10 times.</p><p><strong>UI options:</strong></p><ul><li>Slack/Discord bot (easiest for internal tools)</li><li>Web interface (for customer-facing products)</li><li>API-only (for programmatic access)</li><li>Terminal (for developer tools)</li></ul><blockquote><strong>The principle:</strong> Build for your actual users. If they live in Slack, build a Slack bot. Don’t force them to open a web app.</blockquote><h4>8. Add Testing &amp; Evals (The Most Important Step for Quality)</h4><p><strong>This is what separates production agents from demos.</strong></p><p>You need:</p><ul><li><strong>Test cases</strong> (expected behaviors)</li><li><strong>Evaluation metrics</strong> (how to measure success)</li><li><strong>Error analysis</strong> (what went wrong and why)</li><li><strong>Continuous improvement loop</strong> (systematically fix issues)</li></ul><p><strong>Example evaluation for customer research agent:</strong></p><ul><li>Accuracy: % of correctly identified feature requests</li><li>Precision: % of flagged items that are actually feature requests</li><li>Completeness: % of feature requests not missed</li><li>Latency: Average time to process a ticket</li></ul><blockquote><strong>The reality:</strong> Most agents fail not because of bad architecture, but because teams don’t systematically test and improve them.</blockquote><h4>The Tech Stack That Actually Works</h4><p><strong>The question everyone asks:</strong> “What tools should I use?”</p><p><strong>The answer:</strong> Whatever gets you shipping fastest.</p><p><strong>For general assistants:</strong><br>ChatGPT, Claude, Perplexity</p><p><strong>For coding projects:</strong><br>Cursor, Windsurf, Claude Code</p><p><strong>For business automation:</strong><br>Lindy, Relay, n8n</p><p><strong>For complex workflows:</strong><br>LangGraph, CrewAI, LlamaIndex</p><p><strong>The truth:</strong> The tech stack matters way less than the system prompt and agent design.</p><p>I’ve seen brilliant agents built with simple tools and terrible agents built with the fanciest frameworks.</p><p><strong>Start with the simplest stack that can do what you need.</strong> Upgrade only when you hit real limitations.</p><h4>The Mistakes Everyone Makes</h4><p>After building agents and watching others build them, these are the patterns I see repeatedly:</p><p><strong>Mistake 1: Building a general-purpose agent</strong><br>Start narrow. Expand later.</p><p><strong>Mistake 2: Weak system prompts</strong><br>This is your foundation. Invest time here.</p><p><strong>Mistake 3: Adding complexity early</strong><br>Memory, multi-agent systems, complex orchestration — add these only when needed.</p><p><strong>Mistake 4: Building UI first</strong><br>rove the agent works in a terminal before building interfaces.</p><p><strong>Mistake 5: No evaluation system</strong><br>You can’t improve what you don’t measure.</p><blockquote><strong><em>Every successful agent project follows the same pattern: Start simple. Test ruthlessly. Add complexity only when proven necessary.</em></strong></blockquote><h4>Why This Matters Now</h4><p>AI agents are becoming infrastructure.</p><p>Just like every company needed a website in 2005 and a mobile app in 2015, every company will need AI agents in 2025.</p><p><strong>The opportunity:</strong></p><p>Companies are desperate for people who can build agents. Not just talk about them actually build working systems.</p><p>PMs who can prototype agents have 10x leverage.<br>Engineers who can architect agent systems are in high demand.<br>Product teams that ship agents move faster than those who don’t.</p><blockquote><strong><em>The agents you build now become the foundation of products that scale.</em></strong></blockquote><h4>Your First Agent</h4><p>You don’t need to build something revolutionary.</p><p>Build something simple that solves a real problem for you or your team.</p><p><strong>Ideas for first agents:</strong></p><ul><li>Summarize daily Slack messages into a digest</li><li>Monitor competitor websites and alert on changes</li><li>Extract action items from meeting notes</li><li>Analyze customer feedback for sentiment trends</li><li>Auto-respond to common support questions</li></ul><p>Pick one. Follow the 8 steps. Ship it in a week.</p><p>Then iterate. Add complexity only when you need it.</p><blockquote><strong><em>The skill of building agents compounds faster than almost any other skill right now.</em></strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/875/0*YxI3JTbB1eV3mxDT.png" /></figure><p><strong>Thanks for reading! 🙏</strong><br>If this article helped you, don’t forget to drop a like ❤️</p><p>Got questions or ideas? Share them in the comments — I’d love to hear from you!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=55b9c7e6a4bb" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AI Basics to Advanced: A Comprehensive Technical Guide for Modern Developers and Banking…]]></title>
            <link>https://medium.com/@s.g.manikandan/ai-basics-to-advanced-a-comprehensive-technical-guide-for-modern-developers-and-banking-61bd6fe6f94f?source=rss-74e0f5a7ff6f------2</link>
            <guid isPermaLink="false">https://medium.com/p/61bd6fe6f94f</guid>
            <dc:creator><![CDATA[Manikandan Ganesan ]]></dc:creator>
            <pubDate>Sun, 07 Dec 2025 15:14:55 GMT</pubDate>
            <atom:updated>2025-12-09T14:35:28.839Z</atom:updated>
            <content:encoded><![CDATA[<h3>AI Basics to Advanced: A Comprehensive Technical Guide for Modern Developers and Banking Technology Leaders</h3><p>Artificial Intelligence has moved from “future technology” to “mandatory engineering skill”.</p><p>And nowhere is this shift more visible than in <strong>investment banking, corporate tech teams, and large global enterprises</strong>.</p><p>Today, developers and engineering managers must understand:</p><ul><li>how AI models work,</li><li>how data enters these models,</li><li>how to build internal assistants,</li><li>how to integrate AI with internal systems,</li><li>and how to automate large parts of business workflows.</li></ul><p>Yet, many still rely on fragmented knowledge.</p><p>This article explains <strong>AI basics to advance</strong> in one single, deeply detailed, simple-English guide — suitable for both beginners and experienced professionals.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/905/1*qdLrBvj720SHVv7gCQQw8Q.png" /></figure><h4>1. What Exactly Is Artificial Intelligence?</h4><blockquote><strong>Simple Definition:</strong> AI is a set of techniques that allow computers to perform tasks that typically require human intelligence — like reading documents, writing content, reasoning, analyzing, or generating code.</blockquote><p><strong>Corporate Analogy</strong></p><p>Think of AI as an <strong>extremely fast, multilingual junior analyst</strong> who:</p><ul><li>reads thousands of documents in seconds</li><li>never gets tired</li><li>makes suggestions</li><li>writes summaries</li><li>answers questions</li><li>solves problems</li></ul><p>But still needs <strong>supervision</strong>, just like a new analyst.</p><p><strong>Short Evolution of AI</strong></p><p><strong>1. Rules-Based AI (before 2000)</strong></p><blockquote>If X happens → do Y.<br>Everything required manual logic. No learning.</blockquote><p><strong>2. Machine Learning (2000–2015)</strong></p><blockquote>Algorithms learned patterns from data.<br>Example: spam detection, credit scoring, fraud models.</blockquote><p><strong>3. Deep Learning (2015–2020)</strong></p><blockquote>Neural networks with many layers learned from images, speech, and text.</blockquote><p><strong>4. Transformers (2020–Present)</strong></p><blockquote>The breakthrough that led to ChatGPT, Claude, Gemini, etc.</blockquote><p><strong>5. Agentic Systems + Enterprise Automation (2024–2025)</strong></p><blockquote>AI that can take actions, call tools, run workflows, and integrate with enterprise systems.</blockquote><p><strong>ANI, AGI, ASI Explained Simply</strong></p><p><strong>ANI — Artificial Narrow Intelligence</strong></p><blockquote>All current AI systems.<br>They are excellent at specific tasks but do not understand the world fully.</blockquote><p><strong>AGI — Artificial General Intelligence</strong></p><blockquote>AI that can reason like a human.<br>Not achieved yet.</blockquote><p><strong>ASI — Artificial Super Intelligence</strong></p><blockquote>AI that surpasses human intelligence.<br>Still theoretical.</blockquote><p><strong>Quantum Computing and Its Role in AI</strong></p><p>Quantum computers will:</p><blockquote>accelerate model training</blockquote><blockquote>perform calculations impossible for classical computers</blockquote><blockquote>break and rebuild cryptographic systems</blockquote><blockquote>revolutionize risk modelling and derivatives pricing</blockquote><p>Quantum + AI will become crucial in banking.</p><h4>2. Understanding Machine Learning, Deep Learning, and Neural Networks</h4><p>Before we go deeper into LLMs and transformers, we must understand the layers beneath them. These concepts are often taught in a very mathematical way. Here, we’ll use <strong>simple English</strong> and <strong>clear analogies</strong>.</p><p><strong>What Is Machine Learning?</strong></p><blockquote>Machine Learning (ML) means the computer <strong>learns patterns from data</strong> instead of being explicitly programmed.</blockquote><p><strong>Corporate Example</strong></p><p>Imagine you are building a fraud detection system.</p><p>Rule-based system (old style):</p><ul><li>If someone tries to withdraw &gt; ₹50,000 at midnight → mark as suspicious</li><li>If login happens from a new device → alert</li></ul><p>But these rules are static and limited.</p><p>Machine Learning system:</p><ul><li>The model looks at millions of past fraud and non-fraud transactions</li><li>Learns subtle patterns humans cannot see</li><li>Predicts the probability of fraud in new transactions</li></ul><p>ML = automatic pattern learning.</p><p><strong>What Is Deep Learning?</strong></p><blockquote>Deep Learning is a type of Machine Learning that uses <strong>neural networks with many layers</strong>.</blockquote><blockquote>A neural network is like <strong>a network of artificial neurons</strong>, each performing tiny mathematical checks.</blockquote><blockquote>More layers → deeper learning → better pattern recognition.</blockquote><p><strong>Why Deep Learning matters</strong></p><p>It powers:</p><ul><li>Facial recognition</li><li>Speech recognition</li><li>Self-driving cars</li><li>Chatbots</li><li>Large Language Models</li><li>Document processing (OCR + NLP)</li></ul><p>LLMs (like GPT, Claude, Gemini) are based on this.</p><p><strong>How Neural Networks Work</strong></p><p>Let’s say you want a model to identify whether a sentence is positive or negative.</p><p><strong>Input Layer</strong></p><p>This is where the text enters the network.<br>Before sending it in, we convert words into numbers (embeddings/tokens) so the model can understand them.</p><p><strong>Hidden Layers (The Brain of the Network)</strong></p><p>This is where learning happens. Each layer digs deeper and understands something more meaningful.</p><ul><li>The early layers catch simple things like individual words.</li><li>Middle layers start noticing patterns — like phrases, tone, common combinations.</li><li>Deeper layers understand context and meaning of the entire sentence.</li></ul><p>Each layer transforms the data a bit more, like passing it through several thinking stages.</p><p><strong>Output Layer</strong></p><p>Finally, the network gives an answer based on what it learned:</p><p>✔ <strong>Positive</strong><br> ❌ <strong>Negative</strong></p><p><strong>A Simple Analogy</strong></p><p>Think of a neural network like water flowing through a set of filters:</p><ul><li>First filter removes dust</li><li>Next one removes chemicals</li><li>Final filter improves taste</li></ul><p>By the end, the water is clean and refined.</p><p>Similarly:</p><ul><li>The first layers catch simple patterns</li><li>Next layers understand deeper meaning</li><li>The last layer decides the final result</li></ul><h4>3. Tokenizers, Embeddings, and Vector Representations</h4><p>This is where most people become confused. But these are actually simple ideas.</p><p><strong>What Is a Tokenizer? (Very Simple)</strong></p><p>LLMs cannot read text directly.<br>They break text into <strong>tokens</strong>, which are tiny pieces of words.</p><p><strong>Example sentence:</strong><br><strong>“Investment banking automation is powerful.”</strong></p><p>Possible tokens:</p><ul><li>“Invest”</li><li>“ment”</li><li>“ bank”</li><li>“ing”</li><li>“ autom”</li><li>“ation”</li><li>“ is”</li><li>“ power”</li><li>“ful”</li><li>“.”</li></ul><p>Why not words?</p><p>Because tokens help handle:</p><ul><li>multiple languages</li><li>complex grammar</li><li>huge vocabulary</li><li>unknown words</li><li>misspellings</li><li>hashtags</li><li>code</li></ul><p><strong>Tokenizer Limitations</strong></p><blockquote>Longer text = more tokens = more cost<br>LLM pricing is based on tokens.</blockquote><blockquote>Tokenizer splits words incorrectly sometimes<br>Example: “Mumbai” may be split into “Mu”, “mba”, “i”.</blockquote><blockquote>Meaning can be lost during tokenization<br>Some languages tokenize poorly.</blockquote><p><strong>What Are Embeddings?</strong></p><blockquote>Embeddings are <strong>numerical representations of meaning</strong>.</blockquote><p><strong>Example</strong>:<br> The word <strong>“bank”</strong> has multiple meanings:</p><ul><li>river bank</li><li>investment bank</li><li>blood bank</li></ul><p>An embedding captures the context:</p><ul><li>“river bank” → vector closer to “water”, “shore”</li><li>“investment bank” → vector closer to “finance”, “trading”</li><li>“blood bank” → vector closer to “hospital”, “donation”</li></ul><blockquote>Embeddings = meaning stored as numbers.</blockquote><p><strong>How Vector Representations Work</strong></p><p>Words are stored in a <strong>multi-dimensional space</strong>, like coordinates.</p><pre>finance → [0.11, 1.52, -0.33, 0.92]<br>trading → [0.12, 1.43, -0.29, 0.88]<br>mountain → [-2.3, 0.11, 4.2, -1.2]</pre><blockquote>Close vectors = similar meaning.<br>Far vectors = unrelated meaning.</blockquote><p>This is the core of:</p><ul><li>search</li><li>recommendation</li><li>semantic similarity</li><li>RAG systems</li><li>vector databases</li><li>memory for AI assistants</li></ul><h4>4. Transformers: The Breakthrough Architecture Behind Modern AI</h4><p>Transformers changed everything.<br>They replaced RNNs and LSTMs and enabled large-scale LLMs.</p><p>Here’s a simple explanation:</p><p><strong>Why Transformers Are Revolutionary</strong></p><p>Before transformers, models read text <strong>one word at a time</strong> (sequentially).<br>Very slow. Could not understand long sentences.</p><p>Transformers read <strong>all words at once</strong>, like scanning an entire document instantly.</p><p>This allows:</p><ul><li>understanding long context</li><li>parallel processing</li><li>massive training</li><li>better reasoning</li></ul><p><strong>Self-Attention Explained with a Story</strong></p><p>Imagine reading this sentence:</p><p><strong>“Raj went to the bank because he wanted to deposit money.”</strong></p><p>The model must understand:</p><ul><li>“bank” means financial bank</li><li>“he” refers to Raj</li><li>“Deposit money” clarifies meaning</li></ul><p>Self-attention lets the model look at all words and find relationships.</p><p>In simple terms:</p><blockquote><strong>Self-attention = the model highlighting important words while ignoring irrelevant ones.</strong></blockquote><p><strong>How Transformers Understand Meaning and Context</strong></p><p>Transformers can:</p><ul><li>track relationships across long paragraphs</li><li>understand code, tables, and documents</li><li>connect ideas and reason across context</li></ul><p>This architecture enables everything we see today — GPT-4, Claude 3, Gemini 2.0, Llama 3.2, Olmo, etc.</p><h4>5. Large Language Models (LLMs)</h4><p>Now that we understand transformers, embeddings, and tokenization, we can explain LLMs clearly.</p><p><strong>What Is an LLM?</strong></p><p>A Large Language Model is an AI system trained on:</p><ul><li>text</li><li>code</li><li>books</li><li>websites</li><li>PDFs</li><li>articles</li><li>research papers</li></ul><p>It learns patterns of human language and uses them to:</p><ul><li>analyze</li><li>answer</li><li>generate</li><li>reason</li><li>summarise</li><li>code</li><li>translate</li><li>plan</li><li>automate</li></ul><blockquote>LLMs are not databases.<br>They <strong>predict the next best token</strong> using probability — but extremely intelligently.</blockquote><p><strong>Top LLMs in 2025</strong></p><ul><li>GPT-5.1 (OpenAI)</li><li>Gemini 3.0 (Google DeepMind)</li><li>Claude Opus 4.5 (Anthropic)</li><li>DeepSeek R1 (Open-source)</li><li>Grok 4.1 (xAI)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IHCKFBRbHPjXS2XNeHIfoQ.png" /></figure><h4>6. Fine-Tuning, RAG, Distillation, and Advanced Techniques</h4><p>Let’s clarify these concepts in simple terms.</p><p><strong>What Is Fine-Tuning?</strong></p><p>Fine-tuning is teaching the model new tasks using <strong>your private data</strong>.</p><p><strong>Example</strong>:<br> A bank fine-tunes an LLM on:</p><ul><li>internal policies</li><li>KYC rules</li><li>product documentation</li><li>compliance guidelines</li></ul><p>The model becomes specialised.</p><p><strong>What Is RAG (Retrieval-Augmented Generation)?</strong></p><blockquote>RAG = LLM + your documents retrieved from a vector database.</blockquote><p><strong>Example</strong>:</p><ul><li>Employee asks: “What is our KYC procedure for high-risk accounts?”</li><li>The system retrieves the exact PDF page</li><li>LLM summarizes it safely</li></ul><p>RAG ensures <strong>accuracy</strong> because answers come from <strong>your documents</strong>, not the model’s memory.</p><p><strong>What Is Model Distillation?</strong></p><p>Distillation creates a smaller model from a large one with similar performance.</p><p>Used for:</p><ul><li>on-device models</li><li>cost optimisation</li><li>faster inference</li></ul><h4>7. Vector Databases and Enterprise Knowledge Management</h4><p>Vector databases are one of the most important parts of modern AI architecture, especially for <strong>corporate AI assistants</strong>, <strong>RAG systems</strong>, and <strong>knowledge discovery tools</strong> used in investment banking.</p><p>But many developers and managers don’t fully understand <em>why</em> they are needed.<br>Let’s break it down simply.</p><p><strong>Why Traditional Databases Cannot Store Meaning?</strong></p><p>Traditional databases (SQL, NoSQL) store data in structured formats:</p><ul><li>tables</li><li>rows</li><li>columns</li><li>key–value pairs</li></ul><p>These systems are excellent for transactions — but they cannot understand meaning.</p><p>For example, in SQL, the system <strong>cannot understand</strong> that:</p><ul><li>“revenue” is similar to “income”</li><li>“client onboarding” is related to “KYC”</li><li>“bond trade settlement” is related to “custody operations”</li></ul><p>Traditional databases can <strong>store</strong> text, but cannot <strong>feel</strong> or <strong>compare</strong> meaning.</p><p>That’s why we need vector databases.</p><p><strong>What Is a Vector Database?</strong></p><p>A vector database stores <strong>embeddings</strong> — meaning represented as numbers.</p><p><strong>Example embedding:</strong></p><pre>KYC policy document → [1.11, -0.42, 2.88, 0.12, ...]</pre><p>When a user asks a question like:</p><blockquote>“What is our onboarding requirement for high-risk clients?”</blockquote><p>The vector database:</p><p>1. Converts the question into an embedding</p><p>2. Compares it with millions of stored embeddings</p><p>3. Finds the most similar documents</p><p>4. Sends them to the LLM for summarization (RAG)</p><p>This ensures accurate answers.</p><p><strong>Popular Vector Databases</strong></p><ul><li><strong>FAISS</strong> (Facebook) — fast, used in Python systems</li><li><strong>Pinecone</strong> — cloud vector DB for production</li><li><strong>Qdrant</strong> — open-source, very fast</li><li><strong>Weaviate</strong> — enterprise, with hybrid search</li><li><strong>Milvus</strong> — handles very large datasets</li></ul><p><strong>How Enterprises Use Vector Databases</strong></p><p>Banks use vector DBs for:</p><p>1. Internal ChatGPT-like assistants</p><p>2. Searching internal policies</p><p>3. Compliance automation</p><p>4. Customer support automation</p><p>5. Knowledge management</p><p>6. Research and market insights</p><p>7. Legacy document search (PDFs, emails, Word files)</p><p>In short: A vector database is the brain warehouse for corporate AI.</p><h4>8. No-Code and Low-Code AI Tools for Corporates</h4><p>Not everyone in a large organisation is a developer.<br>But everyone needs access to AI.</p><p>That is why no-code tools became extremely powerful.</p><p><strong>ChatGPT, Claude, Gemini: Enterprise Assistants for Non-Technical Teams</strong></p><p>These tools allow:</p><ul><li>Analysts</li><li>Operations staff</li><li>HR</li><li>Sales</li><li>Compliance</li><li>Finance</li></ul><p>to automate work <strong>without writing any code</strong>.</p><p><strong>Examples</strong>:</p><ul><li>Convert raw data → clean table</li><li>Write emails in professional tone</li><li>Generate strategy documents</li><li>Summarise PDFs and PPTs</li><li>Analyze long reports</li><li>Create Excel formulas</li><li>Generate SQL queries</li></ul><p>This democratises productivity across the organisation.</p><p><strong>How Business Users Automate Workflows Without Coding</strong></p><p>Using no-code workflow tools like:</p><ul><li><strong>n8n</strong></li><li><strong>Zapier</strong></li><li><strong>Make.com</strong></li><li><strong>Airtable</strong></li><li><strong>Notion workflows</strong></li></ul><p>Non-technical users can:</p><ul><li>connect apps</li><li>automate tasks</li><li>trigger workflows when events happen</li></ul><p>Example for investment banking operations:</p><blockquote>“When a trade settlement fails, automatically send a Slack message, create a ticket, and generate an email summary.”</blockquote><p>No coding required.</p><h4>9. AI Developer Tools</h4><p>This is where experienced engineers gain massive leverage.<br>AI-driven IDEs and dev tools can reduce engineering effort by 40–70%.</p><p>Let’s review the major ones.</p><p><strong>Cursor IDE</strong></p><p>Cursor is an AI-native IDE designed for:</p><ul><li>large codebases</li><li>refactoring</li><li>generating new features</li><li>debugging</li><li>test generation</li><li>architectural design</li><li>repository-level understanding</li></ul><p><strong>Why Cursor is important</strong></p><p>It allows developers to:</p><ul><li>explain entire folders to AI</li><li>generate PRs automatically</li><li>maintain coding style</li><li>understand legacy systems</li><li>create new modules faster</li></ul><p>This is critical in banks where codebases are huge and old.</p><p><strong>Windsurf IDE</strong></p><p>Windsurf (formerly Codeium) is good for:</p><ul><li>multi-file editing</li><li>automated refactors</li><li>code-navigation reasoning</li><li>suggestions based on project context</li></ul><p>Banks with monolithic Java/Python codebases benefit massively.</p><p><strong>GitHub Copilot</strong></p><p>The most widely used AI coding assistant.</p><p>Helps with:</p><ul><li>autocomplete</li><li>boilerplate code</li><li>test cases</li><li>API usage</li><li>documentation</li><li>commenting</li><li>code explanation</li></ul><h4>10. Automation Using n8n, Zapier, Make.com, and Enterprise Pipelines</h4><p><strong>What Is n8n and Why Engineers Love It?</strong></p><p>n8n is:</p><ul><li>open-source</li><li>self-hosted</li><li>extremely flexible</li><li>customizable</li><li>extendable with JavaScript</li><li>connectable with any API</li></ul><p>Perfect for corporate environments that cannot send data to cloud systems.</p><p><strong>How To Build AI Automation with n8n (Simple Example)</strong></p><p>Example workflow:<br>1. Watch an email inbox<br>2. Extract attachments (PDF trade confirmations)<br>3. Send PDF text → LLM<br>4. Generate summary<br>5. Detect errors<br>6. Create ServiceNow ticket<br>7. Notify operations team in Slack<br>This automation eliminates hours of manual work.</p><p><strong>Banking Automation Examples Using n8n</strong></p><ul><li>Settlement instruction validation</li><li>Contract note summarization</li><li>PnL explanation bot</li><li>KYC document extraction</li><li>Daily compliance report generation</li><li>Regulatory change summaries</li><li>Ops queue prioritisation</li></ul><p>All done with minimum code.</p><h4>11. Model Context Protocol (MCP): The Next Phase of AI Integration</h4><p>MCP is one of the most transformational AI technologies, especially for enterprises.</p><p><strong>What is MCP?</strong></p><blockquote>MCP stands for <strong>Model Context Protocol</strong>.</blockquote><p>It is an open standard that allows AI models to safely access:</p><ul><li>internal tools</li><li>databases</li><li>APIs</li><li>file systems</li><li>automation workflows</li></ul><p>It gives AI the ability to <em>act</em>, not just <em>answer</em>.</p><p><strong>Why Enterprises Are Adopting MCP</strong></p><p>Because it allows controlled, secure AI automation.</p><p>Typical corporate uses:</p><ul><li>AI reading internal files</li><li>AI executing SQL queries</li><li>AI creating Jira tickets</li><li>AI updating dashboards</li><li>AI triggering pipelines</li><li>AI running scripts</li></ul><p>All with <strong>access policies</strong>, <strong>audit logs</strong>, and <strong>sandboxing</strong>.</p><p>This solves the biggest security fear:<br>“How can AI access internal systems safely?”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9HjPvPkcDovGYMXhZOZMbg.png" /></figure><h4>12. Building an In-House AI Assistant for Corporate Workflows</h4><p>This is where everything comes together.</p><p>An enterprise AI assistant is not just an LLM.<br>It is a complete system.</p><p>You can create an enterprise assistant using:</p><ul><li>OpenAI Assistant API</li><li>Anthropic Messages API</li><li>Google Gemini API</li></ul><p>These allow:</p><ul><li>structured output</li><li>tool calling</li><li>vision analysis</li><li>conversation memory</li><li>multi-agent workflows</li></ul><p><strong>How to Store Memory Using Embeddings</strong></p><p>Memory = embedding stored in vector database.</p><p>Example:</p><p>User: <em>“My project is Phoenix Migration.”</em></p><p>Embedding stored:<br> [0.12, -1.44, 2.18, ...]</p><p>Future query:<br> “Give me latest update on my project”</p><p>→ System retrieves context and responds with continuity.</p><p><strong>LangChain Concepts Explained Simply</strong></p><p>LangChain helps you:</p><ul><li>chain multiple LLM calls</li><li>add memory</li><li>add tools</li><li>add RAG</li><li>orchestrate workflows</li><li>create agents</li></ul><p>It’s like a “framework for connecting AI components”.</p><h4>13. End-to-End Corporate Automation Examples</h4><p>Let’s see practical examples.</p><p><strong>KYC / AML Document Summarization Automation</strong></p><p>Input: 30-page KYC PDF<br>Workflow:</p><ul><li>extract text</li><li>run through LLM</li><li>summarise risks</li><li>generate checklist</li><li>escalate red flags</li></ul><blockquote>60 minutes of work reduced to 5 minutes.</blockquote><p><strong>Daily PnL Explanation Automation</strong></p><ul><li>Extract PnL numbers</li><li>LLM explains drivers</li><li>Compare with previous day</li><li>Highlight unusual patterns</li><li>Send summary to traders and managers</li></ul><p><strong>Automated Incident Analysis for Engineering Teams</strong></p><ul><li>Input: logs + alerts</li><li>LLM clusters issues</li><li>Suggests root cause</li><li>Drafts Jira ticket</li><li>Adds remediation steps</li></ul><p>This reduces MTTR significantly.</p><h4>14. Risks, Limitations, and Pitfalls</h4><p>Even with all benefits, AI has limitations.</p><p><strong>Hallucinations</strong></p><p>LLMs can generate wrong answers with high confidence.</p><p><strong>Prompt Injection</strong></p><p>Attackers can manipulate output with hidden text.</p><p><strong>Regulatory Risks</strong></p><p>ESMA, FCA, MAS, RBI are all drafting AI oversight policies.</p><p><strong>Data Leakage</strong></p><p>Must ensure no sensitive data goes to public cloud.</p><h4>15. FAQs</h4><p><strong>1. Can AI replace developers?</strong><br>No — it accelerates developers, not replaces them.</p><p><strong>2. Which model should my bank use?</strong><br>A mix of commercial (GPT/Claude) and open-source (Llama/Mistral).</p><p><strong>3. Do I need a vector DB?</strong><br>Yes, for any serious internal search or RAG solution.</p><p><strong>4. Is MCP better than RAG?</strong><br>RAG retrieves information; MCP takes action. They complement each other.</p><p><strong>5. Should we fine-tune models?</strong><br>Only if RAG is not enough.</p><p><strong>6. How to keep AI secure?</strong><br>Use enterprise access control, encryption, and logging.</p><h4>Conclusion</h4><p>By 2030, every major corporate will run on:</p><ul><li>LLM-powered workflows</li><li>AI-driven automation</li><li>intelligent assistants</li><li>vector databases</li><li>MCP-enabled agentic systems</li></ul><p>Understanding <strong>AI basics to advanced</strong> is no longer optional.<br>It is the new foundation for all engineering and management roles — especially in investment banking.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/875/0*YxI3JTbB1eV3mxDT.png" /></figure><p><strong>Thanks for reading! 🙏</strong><br>If this article helped you, don’t forget to drop a like ❤️</p><p>Got questions or ideas? Share them in the comments — I’d love to hear from you!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=61bd6fe6f94f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Google Agent Space: The Missing AI Layer Every Enterprise Needs]]></title>
            <link>https://medium.com/@s.g.manikandan/google-agent-space-the-missing-ai-layer-every-enterprise-needs-80f871e95475?source=rss-74e0f5a7ff6f------2</link>
            <guid isPermaLink="false">https://medium.com/p/80f871e95475</guid>
            <dc:creator><![CDATA[Manikandan Ganesan ]]></dc:creator>
            <pubDate>Fri, 14 Nov 2025 14:04:46 GMT</pubDate>
            <atom:updated>2025-11-14T14:04:46.335Z</atom:updated>
            <content:encoded><![CDATA[<p>AI is moving fast, but most enterprises still struggle with the basics: finding information quickly, connecting scattered data, automating repetitive work, and giving employees reliable support across systems.</p><p>Google Agent Space is designed to solve exactly these challenges.</p><p>It brings together Google’s AI models, secure search, and enterprise integrations into one unified environment where AI agents can understand information, take actions, and help teams work far more efficiently.</p><p>This article explains Google Agent Space in simple terms, why it matters, and how it can transform your organization more than a traditional chatbot or standalone LLM ever could.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/890/1*E_C2oH-LVfow9bwtA9dLQQ.png" /></figure><h4>What Is Google Agent Space? (Simple Definition)</h4><p>Think of <strong>Google Agent Space</strong> as your company’s <em>AI control room</em>.<br>It connects all your work apps, data, and processes into one intelligent system that can <strong>search, understand, summarize, and even take actions for you</strong>.</p><p>Imagine having a digital assistant that can:</p><ul><li>Search across every system your company uses</li><li>Understand documents, conversations, images, and databases</li><li>Summarize results clearly</li><li>Trigger workflows</li><li>And collaborate with your teams in a secure environment</li></ul><p>It is essentially an AI operating layer for your organization, connecting apps, knowledge, and automation into one place.</p><h4>Why Enterprises Need It</h4><p>Most enterprises today face one major problem: information overload combined with disconnected systems.</p><blockquote>Your documents are in one place, tickets in another, customer data somewhere else, and employees waste hours searching for answers or switching between tools.</blockquote><p>Google Agent Space tackles this problem by:</p><h4>1. Connecting Your Core Business Apps</h4><p>Whether your organization uses Confluence, Google Drive, SharePoint, Jira, ServiceNow, or something custom built, Agent Space integrates them into one unified AI experience.</p><p>Employees can ask questions like:</p><ul><li>“Find all design documents for the new product launch.”</li><li>“Show me unresolved ServiceNow tickets related to onboarding.”</li><li>“Summarize all Jira issues completed last week.”</li></ul><p>They no longer need to navigate multiple tools; the AI handles that.</p><h4>2. Bringing Google-Quality Search Across Your Enterprise</h4><p>Enterprise search is usually slow, outdated, and incomplete.<br> Google Agent Space uses the same search intelligence that powers Google Search but applies it to your private data.</p><p>It can:</p><ul><li>Search across documents, chats, images, spreadsheets, and databases</li><li>Understand context instead of relying on keywords</li><li>Answer complex questions rather than just showing files</li><li>Respect access controls for security</li><li>Suggest next steps based on patterns in the data</li></ul><p>This elevates enterprise search from a lookup tool to an intelligent problem-solving assistant.</p><h4>3. Turning AI Into Action, Not Just Answers</h4><p>Traditional AI chatbots only respond with text.<br> Google Agent Space agents can perform actions.</p><p>They can:</p><ul><li>Create tickets</li><li>Update records</li><li>Fetch analytics</li><li>Trigger workflows</li><li>Pull real-time data from APIs</li><li>Work autonomously on multi-step processes</li></ul><p>This reduces repetitive tasks and frees employees to focus on more valuable work.</p><h4>4. Supporting Every Business Function</h4><p>AI becomes truly transformative only when it works across departments.</p><p>Agent Space can help:</p><ul><li>Marketing create campaigns and analyze engagement</li><li>Sales prepare proposals and update CRM opportunities</li><li>Engineering troubleshoots systems and generate code suggestions</li><li>HR automate hiring workflows and answer policy questions</li><li>Customer support resolve issues faster with unified data</li></ul><p>It becomes an enterprise companion that understands your business end-to-end.</p><h4>Why Traditional LLMs and Chatbots Fall Short</h4><p>Many companies already tried using chatbots or generic AI models, only to discover their limitations.<br>Enterprise environments are far more complex than simple Q&amp;A use cases.</p><p>Here’s why standalone LLMs don’t work well on their own</p><h4>1. Enterprise workflows are multi-step</h4><p>A chatbot can answer questions, but it cannot manage:</p><ul><li>reading documents</li><li>checking multiple systems</li><li>following rules or validations</li><li>triggering actions</li><li>and coordinating multi-step tasks</li></ul><p>Agent Space handles all of this using a network of purpose-built agents.</p><h4>2. Enterprise data is scattered</h4><p>Large organizations store data across:</p><ul><li>CRMs</li><li>ERPs</li><li>legacy systems</li><li>databases</li><li>file repositories</li></ul><p>Generic LLMs have no native access or understanding of these systems.<br>Agent Space bridges this gap with secure connectors and tools.</p><h4>3. No built-in understanding of your domain</h4><p>Every company has unique terminology, rules, processes, and regulatory conditions.</p><p>A general LLM does not automatically understand:</p><ul><li>your product catalog</li><li>your service policies</li><li>your industry compliance rules</li><li>your pricing structure</li></ul><p>Agent Space lets you create domain-specific agents that are trained or tailored to your exact internal knowledge.</p><h4>4. Hallucinations are unacceptable in enterprise</h4><p>Business decisions cannot rely on guesses.</p><p>Agent Space reduces hallucinations by grounding the agent’s responses in:</p><ul><li>verified, real-time enterprise data</li><li>controlled tools</li><li>APIs</li><li>search results</li><li>compliance policies</li></ul><p>This builds reliability into the system.</p><h4>5. Running one massive LLM for every question is expensive</h4><p>Enterprises cannot afford to use the largest model for every interaction.<br>Agent Space uses a modular approach — lighter agents for simple tasks, more powerful models only when truly required.</p><p>This keeps performance high and costs manageable.</p><h4>6. Enterprises need security and governance</h4><p>Agent Space runs on Google Cloud, where:</p><ul><li>access is identity-based</li><li>data is encrypted</li><li>actions are logged</li><li>compliance frameworks are built in</li></ul><p>This ensures sensitive enterprise data stays protected.</p><h4>7. Businesses change constantly</h4><p>Policies, tools, integrations, and requirements evolve.<br>Agent Space lets you update individual agents without retraining an entire monolithic model.</p><p>This agility is essential for modern businesses.</p><h4>Key Advantages of Using Google Agent Space</h4><p><strong>Real Automation, Not Just Conversation</strong></p><blockquote>Agents can complete entire workflows, not just answer questions.</blockquote><p><strong>Deep Integration with Internal Systems</strong></p><blockquote>They can securely interact with your CRMs, ERPs, databases, and other internal tools.</blockquote><p><strong>High Accuracy Through Grounded Responses</strong></p><blockquote>Agents rely on your actual enterprise data, reducing mistakes and improving trust.</blockquote><p><strong>Scalable and Cost-Efficient</strong></p><blockquote>The system engages only the required models and tools instead of running a large model every time.</blockquote><p><strong>Fast Iteration Across Teams</strong></p><blockquote>You can update one agent or add a new tool without affecting everything else.</blockquote><p><strong>Enterprise-Level Security</strong></p><blockquote>The environment is protected by Google Cloud’s robust security, audit, and compliance features.</blockquote><p><strong>Better Employee Experience</strong></p><blockquote>Teams get instant insights, quick actions, and fewer repetitive tasks, leading to higher productivity and satisfaction.</blockquote><h4>Conclusion: Agent Space Is the Future of Enterprise AI</h4><p>Google Agent Space represents a major shift in how companies use AI.<br>It goes beyond simple chat and brings together search, automation, and decision intelligence into a single system that understands your organization’s data and workflows.</p><p>For enterprises that want to:</p><ul><li>Reduce manual work</li><li>Accelerate decision-making</li><li>Modernize processes</li><li>Unify enterprise information</li><li>improve employee productivity</li></ul><p>Agent Space offers a practical and powerful path forward.</p><p>AI in enterprises is no longer just about answering questions.<br> It’s about creating intelligent systems that understand work and help teams do it better.</p><p>Google Agent Space is built precisely for that future.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/875/0*6siTpnXfb0uCRM1Y.png" /></figure><p><strong>Thanks for reading! 🙏</strong><br>If this article helped you, don’t forget to drop a like ❤️</p><p>Got questions or ideas? Share them in the comments — I’d love to hear from you!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=80f871e95475" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building High-Performance Lock-Free Multi-Producer Multi-Consumer Queues using Ring Buffer for…]]></title>
            <link>https://medium.com/@s.g.manikandan/building-high-performance-lock-free-multi-producer-multi-consumer-queues-using-ring-buffer-for-8441205e80d9?source=rss-74e0f5a7ff6f------2</link>
            <guid isPermaLink="false">https://medium.com/p/8441205e80d9</guid>
            <dc:creator><![CDATA[Manikandan Ganesan ]]></dc:creator>
            <pubDate>Sat, 08 Nov 2025 07:14:43 GMT</pubDate>
            <atom:updated>2025-11-08T07:58:33.279Z</atom:updated>
            <content:encoded><![CDATA[<h3>Building High-Performance Lock-Free Multi-Producer Multi-Consumer Queues using Ring Buffer for Real-Time Applications</h3><p>Have you ever wondered how high‑performance trading systems, gaming engines, or messaging frameworks pass data between threads without slowing down? Today, we will understand one secret weapon behind such systems — <strong>Lock‑Free Ring Buffer</strong>.</p><p>In simple words, this data structure helps threads share data <strong>super fast</strong>, <strong>without using locks</strong>, and avoids typical bottlenecks seen in queues.</p><p>This is like running a dosa stall with multiple cooks — each one taking batter and pouring it quickly without waiting for others.</p><p>Let’s go step by step — from basic to advanced.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/887/1*fpbcMtrI0vXHIcVBG87TLw.png" /></figure><h4>What Problem Are We Solving?</h4><p><strong>Imagine</strong>:</p><ul><li>One thread is producing data (like market ticks in trading system)</li><li>Another thread is consuming data (like order matching engine)</li></ul><p>You want to pass data between threads <strong>fast</strong>, without:</p><ul><li>synchronized</li><li>No ReentrantLock</li><li>No queue overhead</li><li>No context switching</li></ul><p>Ring buffer solves this.</p><h4>What Is a Ring Buffer?</h4><p>A <strong>ring buffer</strong> is a fixed‑size array that acts like a circular queue. When the end is reached, it <strong>wraps back to the start</strong>.</p><pre>Index: 0 1 2 3 4 5 6 7<br>Data:  A B C D E F G H<br>               ^ write next here</pre><p>If you reach last index, next write goes to index 0.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*yAssEGAipuw2Kzvb.jpg" /></figure><p><strong>Why Fixed Size?</strong></p><ul><li>No memory allocation during runtime</li><li>Predictable performance</li><li>Cache‑friendly</li></ul><h4>Lock‑Free Concept — Simple Explanation</h4><p>Lock‑free programming means:</p><ul><li>Threads do not block each other</li><li>They use <strong>atomic operations</strong> to update data</li><li>If conflict happens, thread retries (no locking, no waiting)</li></ul><p>In Java, lock‑free is implemented using:</p><ul><li>AtomicLong</li><li>CAS (Compare And Swap)</li></ul><p>This avoids monitor locks and context switching.</p><h4>Key Ring Buffer Terms</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LCjkoRmdOm0lb_TCu3MMIw.png" /></figure><p>We will build a <strong>Single Producer — Single Consumer (SPSC)</strong> ring buffer. Advanced versions support multi‑producer and multi‑consumer.</p><p>Let’s code.</p><p><strong>Java Lock‑Free Ring Buffer Code (AtomicLong + Array)</strong></p><pre>public class LockFreeRingBuffer&lt;T&gt; {<br>    private final Object[] buffer;<br>    private final int size;<br><br>    private final AtomicLong writeSeq = new AtomicLong(0);<br>    private final AtomicLong readSeq = new AtomicLong(0);<br><br>    public LockFreeRingBuffer(int capacity) {<br>        this.size = capacity;<br>        this.buffer = new Object[capacity];<br>    }<br><br>    public boolean offer(T data) {<br>        long currentWrite = writeSeq.get();<br>        long currentRead = readSeq.get();<br><br>        if (currentWrite - currentRead &gt;= size) {<br>            // buffer full<br>            return false;<br>        }<br><br>        int index = (int) (currentWrite % size);<br>        buffer[index] = data;<br>        writeSeq.incrementAndGet();<br>        return true;<br>    }<br><br>    @SuppressWarnings(&quot;unchecked&quot;)<br>    public T poll() {<br>        long currentRead = readSeq.get();<br>        long currentWrite = writeSeq.get();<br><br>        if (currentRead &gt;= currentWrite) {<br>            // buffer empty<br>            return null;<br>        }<br><br>        int index = (int) (currentRead % size);<br>        T data = (T) buffer[index];<br>        buffer[index] = null;<br>        readSeq.incrementAndGet();<br>        return data;<br>    }<br>}</pre><p><strong>Key Points in Code</strong></p><ul><li><strong>No locks</strong> → we rely on atomic counters</li><li>index = sequence % size gives circular behaviour</li><li>Buffer capacity is fixed</li></ul><p><strong>Simple Usage Example</strong></p><pre>LockFreeRingBuffer&lt;Integer&gt; rb = new LockFreeRingBuffer&lt;&gt;(4);<br><br>    Thread producer = new Thread(() -&gt; {<br>        for (int i = 0; i &lt; 10; i++) {<br>            while (!rb.offer(i)) {<br>            }<br>            System.out.println(&quot;Produced: &quot; + i);<br>        }<br>    });<br><br>    Thread consumer = new Thread(() -&gt; {<br>        for (int i = 0; i &lt; 10; i++) {<br>            Integer value;<br>            while ((value = rb.poll()) == null) {<br>            }<br>            System.out.println(&quot;Consumed: &quot; + value);<br>        }<br>    });<br><br>producer.start();<br>consumer.start();</pre><p><strong>Performance Thoughts</strong></p><p>Why is this faster than BlockingQueue?</p><ul><li>No lock contention</li><li>No context switch</li><li>Cache‑friendly fixed array</li><li>Atomic counters are extremely fast</li></ul><p>In trading systems, microseconds matter — this design shines.</p><p><strong>Advanced Notes</strong></p><p><strong>False sharing: </strong>CPU cache line fights if counters near other memory</p><p><strong>Need padding: </strong>Avoids cache interference (Disruptor style)</p><p><strong>Backpressure: </strong>Full buffer must be handled by retry or drop</p><p>For advanced lock‑free systems (like LMAX Disruptor), padding and memory fences are used. But our version gives conceptual clarity.</p><h4>When to Use Ring Buffer</h4><p><strong>Use when:</strong></p><ul><li>Real‑time processing</li><li>Market data feeds</li><li>Logging systems</li><li>Gaming engines</li><li>Low‑latency microservices</li></ul><p><strong>Avoid when:</strong></p><ul><li>You need dynamic resizing</li><li>Workload is bursty and unpredictable</li></ul><h4>Multi-Producer Multi-Consumer (MPMC) Ring Buffer</h4><p>SPSC was simple — one producer, one consumer. But in real world, you may have:</p><ul><li>Multiple threads producing work (market feeds, logs, tasks)</li><li>Multiple threads consuming work (workers, processing engines)</li></ul><p>This is called <strong>MPMC</strong>.</p><p><strong>Goal</strong></p><p>Allow multiple producers &amp; consumers to push/pull without locks.</p><p><strong>Challenges</strong></p><blockquote><strong>Avoid overwriting slots:</strong> Many producers writing at same time</blockquote><blockquote><strong>Avoid reading unready slots:</strong> Many consumers reading same slot</blockquote><blockquote><strong>ABA problem:</strong> CAS can mistakenly succeed on outdated value</blockquote><blockquote><strong>Need memory ordering:</strong> Producers must publish before consumers read</blockquote><p><strong>Core Idea</strong></p><p>We maintain <strong>sequence per slot</strong> — not just head/tail.</p><p>Each cell in buffer has <strong>value &amp; sequence</strong></p><ul><li>Producer CAS updates a slot <strong>only if slot.seq == expectedSeq</strong></li><li>Consumer CAS reads a slot <strong>only if slot.seq == expectedSeq</strong></li><li>Sequence increments after publish/consume</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/962/1*hjYiS4yDxxUVxRZLAG-WlA.png" /></figure><h4>MPMC Ring Buffer (Simplified Java Version)</h4><blockquote>Using AtomicLong + slot sequence (learning example, not Disruptor level)</blockquote><pre>public class MPMCLockFreeRingBuffer&lt;E&gt; {<br>    private static class Cell&lt;E&gt; {<br>        final AtomicLong sequence = new AtomicLong(0);<br>        volatile E value;<br>    }<br><br>    private final int bufferSize;<br>    private final int indexMask;<br>    private final Cell&lt;E&gt;[] buffer;<br><br>    private final AtomicLong enqueueSequence = new AtomicLong(0);<br>    private final AtomicLong dequeueSequence = new AtomicLong(0);<br><br>    @SuppressWarnings(&quot;unchecked&quot;)<br>    public MPMCLockFreeRingBuffer(int capacity) {<br>        if (Integer.bitCount(capacity) != 1) {<br>            throw new IllegalArgumentException(&quot;Capacity must be a power of 2&quot;);<br>        }<br>        this.bufferSize = capacity;<br>        this.indexMask = capacity - 1;<br>        this.buffer = (Cell&lt;E&gt;[]) new Cell[capacity];<br>        for (int i = 0; i &lt; capacity; i++) {<br>            Cell&lt;E&gt; cell = new Cell&lt;&gt;();<br>            cell.sequence.set(i);<br>            buffer[i] = cell;<br>        }<br>    }<br><br>    public boolean offer(E item) {<br>        if (item == null) throw new NullPointerException();<br><br>        long currentEnqueueSeq;<br>        Cell&lt;E&gt; cell;<br>        while (true) {<br>            currentEnqueueSeq = enqueueSequence.get();<br>            cell = buffer[(int) (currentEnqueueSeq &amp; indexMask)];<br>            long seq = cell.sequence.get();<br>            long dif = seq - currentEnqueueSeq;<br><br>            if (dif == 0) {<br>                if (enqueueSequence.compareAndSet(currentEnqueueSeq, currentEnqueueSeq + 1)) {<br>                    break;<br>                }<br>            } else if (dif &lt; 0) {<br>                return false; // full<br>            } else {<br>                Thread.yield();<br>                continue;<br>            }<br>        }<br><br>        cell.value = item;<br>        cell.sequence.set(currentEnqueueSeq + 1);<br>        return true;<br>    }<br><br>    public E poll() {<br>        long currentDequeueSeq;<br>        Cell&lt;E&gt; cell;<br>        while (true) {<br>            currentDequeueSeq = dequeueSequence.get();<br>            cell = buffer[(int) (currentDequeueSeq &amp; indexMask)];<br>            long seq = cell.sequence.get();<br>            long dif = seq - (currentDequeueSeq + 1);<br><br>            if (dif == 0) {<br>                if (dequeueSequence.compareAndSet(currentDequeueSeq, currentDequeueSeq + 1)) {<br>                    break;<br>                }<br>            } else if (dif &lt; 0) {<br>                return null; // empty<br>            } else {<br>                Thread.yield();<br>                continue;<br>            }<br>        }<br><br>        E value = cell.value;<br>        cell.value = null;<br>        cell.sequence.set(currentDequeueSeq + bufferSize);<br>        return value;<br>    }<br><br>    public int size() {<br>        return (int) (enqueueSequence.get() - dequeueSequence.get());<br>    }</pre><blockquote>It is <strong>lock-free</strong>, so it avoids using traditional locks (synchronized, ReentrantLock), meaning <strong>no blocking and very low latency</strong>.</blockquote><blockquote>Instead of locks, it uses <strong>AtomicLong and CAS (Compare-And-Set)</strong> operations to ensure thread safety.</blockquote><blockquote>Each slot in the buffer tracks its sequence number, helping producers and consumers know whether a slot is free or filled.</blockquote><blockquote>If the buffer is full, offer () returns false. If it&#39;s empty, poll() returns null.</blockquote><h4>Performance Comparison</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rxjNptFJI0qhaJBQ6R2hxQ.png" /></figure><p>Before we close, one important note:</p><blockquote><strong>This lock-free ring buffer idea is not just theory</strong> — it powers real ultra‑low‑latency systems.</blockquote><blockquote><strong>Aeron</strong>, used in trading, market data distribution, and high‑frequency environments, uses ring‑buffer‑style structures internally. These buffers help Aeron’s media driver, clients and control channels exchange messages with <em>near‑zero latency</em> using shared memory.</blockquote><blockquote><strong>LMAX Disruptor</strong> — one of the fastest open‑source concurrency frameworks — is also built on a ring buffer. It introduced patterns like sequence barriers, cache‑line padding, and wait strategies to achieve nanosecond‑level performance.</blockquote><blockquote>When you learn ring buffers, you’re studying the same foundation behind world‑class performance engineering</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/875/0*k8rzZOj7CBWzlcZt.png" /></figure><p><strong>Thanks for reading! 🙏</strong><br>If this article helped you, don’t forget to drop a like ❤️</p><p>Got questions or ideas? Share them in the comments — I’d love to hear from you!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8441205e80d9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How FIX Powers Modern Trading: Sessions, Reliability, and Failover Explained]]></title>
            <link>https://medium.com/@s.g.manikandan/how-fix-powers-modern-trading-sessions-reliability-and-failover-explained-aede4ddcdd09?source=rss-74e0f5a7ff6f------2</link>
            <guid isPermaLink="false">https://medium.com/p/aede4ddcdd09</guid>
            <dc:creator><![CDATA[Manikandan Ganesan ]]></dc:creator>
            <pubDate>Sun, 12 Oct 2025 15:38:55 GMT</pubDate>
            <atom:updated>2025-10-12T15:38:55.788Z</atom:updated>
            <content:encoded><![CDATA[<p>Imagine sending thousands of trading orders every second — and knowing that not a single message can be lost. Sounds intense, right? Yet, this is exactly what the FIX protocol handles behind the scenes every day.</p><p>Many developers and traders use FIX without fully understanding its inner workings. How do messages flow reliably? What ensures the system keeps running even when failures happen? How do initiators, acceptors, and heartbeats all work together?</p><p>In this article, we’ll break it down from the basics to advanced concepts. You’ll learn how FIX engines operate, how sessions are managed, how load balancing and fault tolerance work, and how high-availability systems ensure <strong>no message is ever lost</strong> — including the role of ledger files and message persistence.</p><p>By the end, you’ll not only understand <strong>what FIX is</strong>, but also <strong>how it powers modern, resilient trading systems</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/862/1*TqxeXMWTcV165jF79sKgFg.png" /></figure><h4>1. What is the FIX Protocol</h4><p>The <strong>FIX Protocol (Financial Information eXchange)</strong> is a standard messaging protocol used in electronic trading to <strong>send orders, executions, and market data</strong> between systems. It ensures that traders, brokers, and exchanges can communicate <strong>quickly, reliably, and in a common format</strong>.</p><p><strong>FIX defines:</strong></p><ul><li><strong>Message formats:</strong> Standard structure for orders, cancellations, trade confirmations, and quotes.</li><li><strong>Session management:</strong> Rules for establishing and maintaining connections between systems.</li><li><strong>Reliability:</strong> Heartbeats, sequence numbers, and recovery ensure <strong>no messages are lost</strong>.</li><li><strong>Extensibility:</strong> Supports adding new fields or message types without braking systems.</li></ul><p>In short, FIX is the <strong>language of electronic trading</strong>, enabling different systems to talk to each other efficiently and safely.</p><h4>2. Where is FIX Used Today</h4><ul><li>Brokers and exchanges</li><li>Buy-side OMS/EMS and sell-side systems</li><li>Market data vendors and order routers<br>From retail brokers to institutional investors, almost every electronic order passes through FIX somewhere.</li></ul><h4>3. Why Do We Need FIX Protocol</h4><p>Before FIX, each trading system spoke its own language. Integrations were messy, error-prone, and slow.</p><p>FIX solved that by defining:</p><ul><li>A <strong>common message format</strong> for all trading instructions</li><li><strong>Session-level reliability</strong> (heartbeats, sequence numbers)</li><li><strong>Application-level consistency</strong> (order, cancel, execution reports)<br> This allows any two systems to trade reliably — regardless of who built them.</li></ul><h4>4. FIX Protocol Architecture and Key Components</h4><p>The FIX protocol is not just a messaging format — it comes with a <strong>well-defined architecture</strong> that ensures reliable, high-speed communication between trading systems. Understanding its components helps you design <strong>robust, fault-tolerant trading platforms</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*c9mAHYOyFeDLjeG6KQ_MEQ.png" /></figure><p><strong>Initiator and Acceptor</strong></p><ul><li><strong>Initiator:</strong> The system that <strong>starts the FIX session</strong> and sends messages (usually a client, like a trading application).</li><li><strong>Acceptor:</strong> The system that <strong>receives the session request</strong> and responds (typically a server, like an exchange or broker).</li><li><strong>Together</strong>, they form a <strong>FIX session</strong>, the core communication channel.</li></ul><p><strong>FIX Engine</strong></p><p>The FIX engine is the software layer that <strong>handles all protocol-level details</strong>, including:</p><p>The FIX engine is the software layer that <strong>handles all protocol-level details</strong>, including:</p><ul><li>Message encoding/decoding</li><li>Session management</li><li>Sequencing and message recovery</li><li>Heartbeats to keep connections alive</li></ul><blockquote>Popular FIX engines include <strong>QuickFIX</strong>, <strong>Fix8</strong>, and commercial solutions from vendors like <strong>Cinnober</strong> or <strong>Bloomberg</strong>.</blockquote><p><strong>Messages and Tags</strong></p><ul><li>FIX messages consist of <strong>tags and values</strong> in a simple key-value format.</li><li>Each tag represents a specific piece of information — for example, 35=D for a new order, 11=12345 for a client order ID.</li><li>This structured approach ensures <strong>all systems interpret messages consistently</strong>.</li></ul><p><strong>Sessions and Connection Management</strong></p><ul><li>A FIX session handles <strong>logon, message flow, and logoff</strong>.</li><li>Heartbeats monitor the health of the connection.</li><li>Sequence numbers ensure <strong>messages are processed in order</strong>, and missing messages can be <strong>recovered automatically</strong>.</li></ul><p><strong>Reliability and Persistence</strong></p><ul><li>Most FIX engines use <strong>ledger files or message stores</strong> to persist messages.</li><li>This ensures <strong>no message is lost</strong> during network failures or system crashes.</li><li>Combined with <strong>acknowledgments and sequence tracking</strong>, this guarantees <strong>end-to-end reliability</strong>.</li></ul><p><strong>Optional Components</strong></p><ul><li><strong>Adapters or gateways</strong>: Connect FIX engines to internal systems, databases, or downstream consumers.</li><li><strong>Load balancers and HA clusters</strong>: Ensure <strong>fault tolerance and high availability</strong>.</li><li><strong>Monitoring and alerting</strong>: Track session health, message throughput, and failures.</li></ul><p><strong>5. How a FIX Session Works — Step by Step</strong></p><p><strong>Starting the FIX Engine</strong></p><ul><li>On the <strong>initiator side</strong> (client), QuickFIX starts the engine with a session configuration.</li><li>On the <strong>acceptor side</strong> (server), QuickFIX also starts and listens for incoming connections.</li><li>Each session is identified by SenderCompID and TargetCompID.</li></ul><p><strong>Initiator Connects to Acceptor</strong></p><ul><li>The <strong>initiator opens a TCP connection</strong> to the acceptor’s IP and port.</li><li>This starts the FIX session handshake.</li></ul><p><strong>Logon Message</strong></p><ul><li>Initiator sends a <strong>Logon</strong> message to acceptor.</li><li>The message includes: 1. <strong>HeartBtInt</strong> (heartbeat interval) 2. <strong>Sequence number</strong> (MsgSeqNum=1 for a new session, or last stored seq number if resuming)</li><li>Acceptor validates and sends <strong>Logon acknowledgment</strong>.</li><li>Once both sides exchange Logon, the session is active.</li></ul><p><strong>Heartbeat / Connection Check</strong></p><ul><li>Both initiator and acceptor exchange <strong>Heartbeat messages</strong> at regular intervals.</li><li>Purpose: ensure the connection is alive.</li><li>If heartbeats are missed, a <strong>TestRequest</strong> is sent to verify connectivity.</li></ul><p><strong>Sending Application Messages</strong></p><ul><li>Application messages: e.g., NewOrder, CancelOrder, ExecutionReport.</li><li>QuickFIX assigns <strong>sequence numbers</strong> to every message and send immediately over the TCP connection to the acceptor</li><li>The sequence number ensures messages are processed in order and enables <strong>resend requests</strong> if any message is missing.</li><li>QuickFIX can store messages in a <strong>MessageStore</strong> (file, DB) for replay or recovery. This ensures <strong>reliability</strong> if connection drops.</li></ul><p><strong>Sequence Number Management</strong></p><ul><li>Every FIX session maintains <strong>separate inbound and outbound sequence numbers</strong>: Outbound = (Sender → Target) &amp; Inbound = (Target → Sender)</li><li>If a message is missed, the receiver sends a <strong>ResendRequest</strong>.</li><li>Sequence numbers are <strong>persistent</strong>, so even if the connection drops, the session can resume without losing messages.</li></ul><p><strong>Message Acknowledgment</strong></p><ul><li>Most FIX messages are implicitly acknowledged via sequence numbers.</li><li>Special <strong>ExecutionReport or Reject messages</strong> provide explicit application-level acknowledgment.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/797/1*WIc7gH3n_W-vnuDWhpTslA.png" /></figure><p><strong>6. Guaranteed Message Delivery</strong></p><p>One of the most critical requirements in trading systems is <strong>ensuring that every message reaches its destination, exactly once, and in order</strong>. The FIX protocol provides several mechanisms to guarantee this:</p><ul><li>FIX messages are sent over <strong>TCP</strong>, ensuring reliable, in-order delivery at the network level.</li><li>Every message persisted<strong> in a ledger (message store)</strong> with its sequence number before being sent.</li><li>The <strong>acceptor acknowledges</strong> received messages to confirm delivery.</li><li>If a message is missing, the acceptor sends a <strong>Resend Request</strong> for the missing sequence number(s). The initiator then retrieves the missing messages from the <strong>ledger</strong> and resends them.</li><li>In QuickFIX, if messages arrive <strong>out of order</strong> (for example, it receives sequence 5 and then 7), the <strong>acceptor buffers message 7</strong> and waits for the missing message 6. Once 6 is received and processed, message 7 is then processed in order.</li><li>Once a message is acknowledged, QuickFIX <strong>marks it as delivered</strong>. Depending on the engine configuration, it can either be <strong>retained for auditing</strong> or <strong>removed from the ledger</strong>.</li></ul><h4>7. Backpressure in QuickFIX</h4><p>In QuickFIX, backpressure is handled primarily through <strong>TCP flow control</strong> and <strong>session mechanisms</strong>:</p><p><strong>TCP-level Flow Control:</strong></p><ul><li>FIX messages are sent over <strong>TCP</strong>, which inherently provides flow control.</li><li>Each TCP connection has <strong>send and receive buffers</strong>. When the acceptor is slow or unresponsive, the acceptor’s receive buffer fills up.</li><li>As a result, the <strong>initiator’s TCP send buffer also fills and</strong> write calls on the initiator either <strong>block or send fewer bytes</strong>.</li><li>This naturally <strong>slows down the initiator</strong>, preventing it from overwhelming the acceptor while maintaining message integrity.</li></ul><p><strong>Session-level Flow Control:</strong></p><ul><li>QuickFIX persists messages in its <strong>outgoing message queue (message store)</strong> until they are acknowledged by the acceptor.</li><li>If the acceptor is slow or down, outgoing messages <strong>remain persisted in the message store</strong> and are sent once the acceptor can process them.</li><li>The engine retries delivery according to the <strong>session configuration</strong> (e.g., ReconnectInterval, HeartBtInt).</li></ul><blockquote><strong>Recovery after Acceptors Resume</strong></blockquote><blockquote>Once the acceptor resumes processing, the initiator <strong>sends queued messages in order</strong>.</blockquote><blockquote><strong>Sequence numbers and persistent storage</strong> ensure that messages are delivered <strong>exactly once and in order</strong>, even during backpressure.</blockquote><p><strong>Implications:</strong></p><ul><li>QuickFIX does <strong>not drop messages automatically</strong> — backpressure is handled through <strong>queuing and TCP flow control</strong>.</li><li>High-throughput systems should carefully size the <strong>message store</strong>, as excessive backlog can hit <strong>disk or memory limits</strong>.</li></ul><h4>8. Load Balancing in QuickFIX</h4><p>QuickFIX itself is <strong>not a network load balancer</strong> — it’s a FIX engine that handles sessions, messaging, sequencing, and persistence. There is <strong>no built-in “automatic load balancing”</strong> feature in QuickFIX. However, you can achieve load balancing in your FIX setup through <strong>architecture patterns</strong>:</p><p><strong>Multiple Acceptors / FIX Gateways</strong></p><ul><li>You can deploy <strong>multiple QuickFIX acceptor instances</strong> (A1, A2, …) to handle more incoming sessions.</li><li>Each acceptor runs independently with its own <strong>session state and message store</strong>.</li><li><strong>Initiators can directly connect</strong> to a specific acceptor.</li><li><strong>Sticky sessions</strong> are required so that a FIX session always goes to the same acceptor.</li><li><strong>Example:</strong> I1 connects to A1, I2 connects to A2, I3 connects to A1 — each acceptor independently tracks its own sessions and sequence numbers.</li><li><strong><em>Focus</em></strong><em>:</em> Scaling the FIX servers horizontally (more acceptors = more capacity).</li></ul><p><strong>Session-Level Load Balancing</strong></p><ul><li>Each FIX session is uniquely defined by <strong>SenderCompID → TargetCompID</strong>.</li><li>You can configure <strong>one initiator process with multiple FIX sessions</strong>, distributing them across different acceptors.</li><li>A single session cannot be split, but multiple sessions from one initiator can connect to different acceptors for better load distribution.</li><li><strong>Example:</strong> One initiator runs two sessions — Session 1 connects to A1, Session 2 connects to A2. Both run independently but under the same initiator instance.</li><li><strong><em>Focus</em></strong><em>:</em> Balancing workload by distributing <strong>sessions</strong>, not connections.</li></ul><p><strong>Network-Level Load Balancer</strong></p><ul><li>A <strong>TCP load balancer</strong> (like HAProxy, NGINX, or F5) sits in front of the acceptors.</li><li><strong>All initiators connect to the load balancer’s IP/port</strong> — it forwards each connection to one of the acceptors (round-robin, least connections, or IP-hash).</li><li><strong>Sticky sessions</strong> ensure all messages for a FIX session always go to the same acceptor.</li><li><strong>Example:</strong> I1, I2, and I3 all connect to fix.trading.com:9876, the load balancer’s address; it internally routes I1 → A1, I2 → A2, I3 → A1.</li><li><strong><em>Focus</em></strong><em>:</em> Transparent <strong>connection-level distribution</strong> and <strong>high availability</strong> without changing initiator configs.</li></ul><h4>9. Making a QuickFIX Acceptor Fault-Tolerant and Highly Available</h4><p>QuickFIX itself is <strong>not natively fault-tolerant</strong> or <strong>cluster-aware</strong>, but you <strong>can build a fault-tolerant and highly available FIX setup</strong> <em>around</em> it using external components like <strong>shared message stores</strong>, <strong>replication</strong>, and <strong>load balancers</strong>.</p><ul><li>Deploy <strong>two QuickFIX acceptor instances</strong>: one <strong>Primary (Active)</strong>, one <strong>Secondary (Standby)</strong>.</li><li>Both acceptors are configured with <strong>identical session settings</strong> (SenderCompID, TargetCompID, ports, etc.).</li><li>Both acceptors share a <strong>common message store</strong> (JDBC-based DB or replicated file store) to maintain sequence numbers and message history.</li><li><strong>Primary acceptor</strong> handles all client (initiator) connections under normal conditions.</li><li><strong>Secondary acceptor</strong> is idle but fully ready to accept connections when needed.</li><li><strong>Initiators are configured with multiple host/port entries</strong>: primary first, secondary as fallback.</li><li>If the primary acceptor fails or becomes unreachable, the initiator <strong>automatically connects to the secondary acceptor</strong>.</li><li>The secondary continues the session using the <strong>same persisted sequence numbers and message logs</strong>, ensuring <strong>no message loss or sequence gaps</strong>.</li><li><strong>Only one acceptor is active per session at any time</strong> — simultaneous connections to both will corrupt sequence numbers.</li><li>Optional: a <strong>network load balancer</strong> or virtual IP can provide a single connection endpoint, though the multi-host initiator configuration is sufficient for failover.</li><li>High availability and fault tolerance are achieved <strong>without custom monitoring scripts</strong> — the initiator handles failover automatically.</li></ul><p><strong>Initiator Configuration (Primary → Secondary Failover)</strong></p><pre>[DEFAULT]<br>ConnectionType=initiator<br>UseDataDictionary=Y<br>DataDictionary=FIX44.xml<br>HeartBtInt=30<br>ReconnectInterval=10<br>FileStorePath=/var/quickfix/initiator_store<br>FileLogPath=/var/quickfix/initiator_log<br><br>[SESSION]<br>BeginString=FIX.4.4<br><strong>SenderCompID=CLIENT1<br>TargetCompID=BROKER1</strong><br><br># Multi-host failover configuration<br><strong>SocketConnectHost=192.168.1.10   # Primary acceptor IP<br>SocketConnectPort=5000<br><br>SocketConnectHost=192.168.1.11   # Secondary acceptor IP<br>SocketConnectPort=5001</strong><br><br>StartTime=00:00:00<br>EndTime=23:59:59<br><strong>ResetOnLogon=N    # Should be N</strong></pre><p><strong>Primary Acceptor (Machine 1)</strong></p><pre>[DEFAULT]<br>ConnectionType=acceptor<br>UseDataDictionary=Y<br>DataDictionary=FIX44.xml<br>HeartBtInt=30<br><strong>FileStorePath=/shared/quickfix/store      # Shared between primary &amp; secondary</strong><br>FileLogPath=/var/quickfix/log_primary<br><br>[SESSION]<br>BeginString=FIX.4.4<br><strong>SenderCompID=BROKER1<br>TargetCompID=CLIENT1</strong><br><strong>SocketAcceptPort=5000</strong>       # Primary port<br>StartTime=00:00:00<br>EndTime=23:59:59<br><strong>ResetOnLogon=N   # should be N</strong></pre><p><strong>Secondary Acceptor (Machine 2)</strong></p><pre>[DEFAULT]<br>ConnectionType=acceptor<br>UseDataDictionary=Y<br>DataDictionary=FIX44.xml<br>HeartBtInt=30<br><strong>FileStorePath=/shared/quickfix/store      # Same shared store</strong><br>FileLogPath=/var/quickfix/log_secondary<br><br>[SESSION]<br>BeginString=FIX.4.4<br><strong>SenderCompID=BROKER1<br>TargetCompID=CLIENT1<br>SocketAcceptPort=5001       # Secondary port (or same port with virtual IP)</strong><br>StartTime=00:00:00<br>EndTime=23:59:59<br><strong>ResetOnLogon=N # Should be N</strong></pre><h4>10. Making a QuickFIX/J Initiator Fault-Tolerant &amp; Highly Available</h4><p>QuickFIX and QuickFIX/J are not cluster-aware — they don’t support multiple initiators sharing the same FIX session simultaneously. However, you can achieve <strong>fault tolerance and high availability</strong> by deploying <strong>two initiator instances</strong> in an <strong>Active-Passive</strong> setup with a <strong>shared persistent message store</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JdHW9uc00Y7N7VvJW9slOA.png" /></figure><p><strong>Run Two Identical Initiators</strong></p><p>Deploy two QuickFIX initiators with the same session configuration — one is <strong>Active</strong>, the other is <strong>Passive</strong>.</p><p>Both should use the same SessionID, same connection settings, and point to the same shared persistent store.</p><p>Only one initiator must run at any given time.</p><p><strong>Use a Shared Persistent Store</strong></p><p>Use a <strong>JDBC/ NAS message store</strong> so both instances share the same sequence numbers and message history.</p><p>This allows the Passive node to take over seamlessly when the Active node fails.</p><p><strong>Example settings:</strong></p><pre>PersistMessages=Y<br>ResetOnLogon=N<br><strong>messageStoreFactory=org.quickfixj.jdbc.JdbcStoreFactory</strong></pre><p><strong>Detecting Active Node Failure</strong></p><p>Set up a <strong>lightweight heartbeat or health-check mechanism</strong> outside QuickFIX.</p><p>Typical methods include:</p><ul><li>A simple TCP ping to the Active node’s process or port.</li><li>Monitoring QuickFIX log files or heartbeat timestamps.</li><li>A watchdog script that checks if the FIX session is LOGGEDON.<br> If the Active node stops responding or loses connection to the exchange for a configured period, mark it as failed.</li></ul><p><strong>Triggering Failover</strong></p><p>When a failure is detected, an external failover script or tool (like keepalived, heartbeat, or a custom shell script) should:</p><blockquote>Stop the Active node’s QuickFIX process if it’s still half-alive.</blockquote><blockquote>Promote the Passive node as the new Active.</blockquote><blockquote>Start the initiator process on the Passive node.</blockquote><p>Once started, QuickFIX reads sequence numbers from the shared store and reconnects to the exchange. Because the store is shared and ResetOnLogon=N, message continuity is preserved, and QuickFIX automatically handles resends for any missed messages.</p><p><strong>Holding or Handling the Old Active Node</strong></p><p>When the old Active node comes back online, do not start the initiator automatically.</p><p>Keep it in a standby or disabled state until the new Active is healthy and fully connected.</p><p>Starting both would cause duplicate sessions and sequence conflicts.</p><p>You can re-designate the recovered node as the new Passive by keeping the initiator process stopped and ready for the next failover.</p><p><strong>Continuous Monitoring</strong></p><p>Keep ongoing checks for:</p><ul><li>FIX session status (heartbeats, test requests, disconnects).</li><li>Database or message store health.</li><li>Process uptime and resource usage on both nodes.</li></ul><p>Automation should handle failover quickly, but manual visibility is equally important in trading systems.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/875/0*lhZ_oiNJvAlDmRb0.png" /></figure><p><strong>Thanks for reading! 🙏</strong><br>If this article helped you, don’t forget to drop a like ❤️</p><p>Got questions or ideas? Share them in the comments — I’d love to hear from you!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=aede4ddcdd09" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Designing a Scalable and Reliable Likes Counting System for Modern Social Platforms]]></title>
            <link>https://medium.com/@s.g.manikandan/designing-a-scalable-and-reliable-likes-counting-system-for-modern-social-platforms-e8f4c1440de3?source=rss-74e0f5a7ff6f------2</link>
            <guid isPermaLink="false">https://medium.com/p/e8f4c1440de3</guid>
            <dc:creator><![CDATA[Manikandan Ganesan ]]></dc:creator>
            <pubDate>Sat, 11 Oct 2025 18:17:17 GMT</pubDate>
            <atom:updated>2025-10-11T18:31:26.815Z</atom:updated>
            <content:encoded><![CDATA[<p>This article explains how to build a likes-counting system that starts from a simple design and evolves — step-by-step — into a production-grade, highly-available, and operationally manageable architecture. It walks through concrete problems, pragmatic trade-offs, data models, example flows, and an implementation blueprint that balances latency, cost, and correctness for real-world social platforms.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/881/1*B3YK-dP1u1cvlpmmZQxFeA.png" /></figure><h4>Goals &amp; requirements</h4><p>Before designing, clarify requirements. These drive every choice.</p><p><strong>Functional</strong></p><ul><li>Increment (like) and decrement (unlike / toggle) for a given post_id by user_id.</li><li>Get the current like count for a post in real time.</li><li>Optionally get whether the requesting user has liked the post (per-user state).</li></ul><p><strong>Non-functional</strong></p><ul><li><strong>Latency</strong>: UI must see immediate feedback (sub-100ms ideal on writes).</li><li><strong>Throughput</strong>: handle spikes (e.g., viral posts) up to hundreds of thousands of ops/sec.</li><li><strong>Durability</strong>: counts should not be lost (or should be recoverable) after failures.</li><li><strong>Consistency</strong>: eventual consistency is acceptable for counts, but per-user state must be correct (no double-likes).</li><li><strong>Cost &amp; ops:</strong> design should be operable — monitorable and testable.</li></ul><h4>High-level options (quick comparison)</h4><ol><li><strong>Single DB counter</strong> — easy, correct, doesn’t scale.</li><li><strong>Cache + periodic DB flush (write-behind)</strong> — fast, introduces eventual consistency.</li><li><strong>Event sourcing (durable stream) + materialized counters</strong> — reliable, replayable, operational complexity.</li><li><strong>Sharded in-memory counters + durable commit log</strong> — scalable and performant for extremes.</li><li><strong>CRDT / convergent counters (multi-DC)</strong> — for geo-replicated systems with weak sync guarantees.</li></ol><p>The rest of the article evolves a design from (1) → (2) → (4) with event streaming as the durability backbone and explains when to consider CRDTs or other specialised techniques.</p><h4>1. Naïve design — single DB counter</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/848/1*KdLGY2bbW4uwhMmuC2aN4g.png" /></figure><p><strong>Schema</strong></p><pre>CREATE TABLE posts (<br>  post_id BIGINT PRIMARY KEY,<br>  like_count BIGINT DEFAULT 0<br>);</pre><p><strong>Operation</strong></p><pre>UPDATE posts SET like_count = like_count + 1 WHERE post_id = ?;</pre><p><strong>Why it fails at scale</strong></p><ul><li>Every click becomes a DB write → hotspots, locks, and I/O saturation.</li><li>Cannot absorb spikes or scale horizontally easily.</li><li>Per-user uniqueness (preventing double-like) requires separate likes table with unique (user_id, post_id), which is even heavier.</li></ul><p><strong>When OK</strong></p><ul><li>Small apps with low writes (&lt; few hundred writes/sec). Good for prototypes.</li></ul><h4>2. Better: Keep per-user state + aggregate in DB</h4><p><strong>Schema</strong></p><pre>CREATE TABLE likes (<br>  user_id BIGINT,<br>  post_id BIGINT,<br>  created_at TIMESTAMP,<br>  PRIMARY KEY(user_id, post_id)<br>);<br><br>-- Count derived by:<br>SELECT count(*) FROM likes WHERE post_id = ?;</pre><p><strong>Problems</strong></p><ul><li>Aggregation reads become expensive (counting millions of rows).</li><li>Inserts to likes table still heavy under write bursts.</li><li>Transactions required to maintain consistency between likes table and post.like_count.</li></ul><p><strong>Conclusion</strong></p><ul><li>Correct and simple, but not performant at high write volumes.</li></ul><h4>3. Add cache: Redis counters (write-through/write-behind)</h4><p><strong>Approach: cache counts in Redis and flush to DB asynchronously</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/747/1*uEbvkC8LTgg-XOIUJSRfBg.png" /></figure><p><strong>Flow (write path):</strong></p><ol><li>HTTP API receives like/unlike toggle from user_id for post_id.</li><li>Verify per-user state (option A: store per-user state in Redis; option B: write event and dedupe later).</li><li>Atomically increment/decrement Redis counter: INCR post:likes:post_id or HINCRBY post_likes_hash post_id 1.</li><li>Return fast response to client.</li><li>Background worker periodically flushes counters (deltas) to DB.</li></ol><p><strong>Advantages</strong></p><ul><li>Millisecond write latency; DB pressure drastically reduced.</li><li>Scales by sharding Redis.</li></ul><p><strong>Key details</strong></p><ul><li>Use atomic Redis ops to avoid races.</li><li>Background flush interval is a trade-off: shorter = fresher DB, but more DB load.</li><li>Persist deltas durably (e.g., write to a persistent queue) if the cache node might be lost before flush.</li></ul><p><strong>Problems</strong></p><ul><li>Cache loss before flush → lost likes.</li><li>Per-user uniqueness still needs a durable store or deduplication strategy.</li></ul><p><strong>Redis with Persistence — The Real Picture</strong></p><p>Redis isn’t just a fast in-memory cache anymore. With persistence enabled, it can also act as a <strong>highly reliable counter store</strong> for workloads like <em>likes, views, or votes</em>.</p><p>Let’s see what really happens under the hood — and where the trade-offs lie.</p><p><strong>How a Redis Cluster Handles Writes</strong></p><p>Imagine a <strong>three-node Redis Cluster</strong> — one <strong>primary (master)</strong> and two <strong>replicas</strong>. When a user likes a post:</p><ol><li>The write first goes to the <strong>primary node</strong>.</li><li>Redis then <strong>replicates</strong> this update to at least one replica.</li><li>Only <strong>after one replica acknowledges</strong> receiving it (based on your config like min-replicas-to-write 1), Redis confirms the write back to the client.</li></ol><p>On top of that, if you enable <strong>AOF (Append Only File)</strong> persistence, Redis will also write the update to disk either:</p><ul><li><strong>every second</strong> (appendfsync everysec) → small chance of 1-second data loss,</li><li>or <strong>immediately</strong> (appendfsync always) → fully durable but slightly slower.</li></ul><p><strong>Benefits</strong></p><ul><li><strong>Blazing fast writes:</strong> Everything happens in memory, with single-digit millisecond latency.</li><li><strong>Durability boost:</strong> AOF + replication means even if one node crashes, another replica has your latest count.</li><li><strong>Easy recovery:</strong> On restart, Redis replays AOF logs and restores counters automatically.</li><li><strong>Scalable design:</strong> You can shard posts across Redis nodes for higher throughput.</li></ul><p><strong>Drawbacks</strong></p><ul><li><strong>Not 100% durable:</strong> Replication is <strong>asynchronous</strong>, so the replica might ACK before the write is safely fsynced to disk.</li><li><strong>Failover race:</strong> If the master and replica crash before the AOF is flushed, a few recent updates may be lost.</li><li><strong>Limited event history:</strong> Redis primarily stores the latest counts, but you can optionally keep <strong>short-term audit events</strong> (like who liked or unliked) in a separate cache key or Redis Stream. This works well for recent activity or quick deduplication, but it’s not ideal for long-term storage or analytics since Redis keeps everything in memory.</li></ul><p>So, while a <strong>persistent Redis cluster</strong> gives near-real-time performance with high reliability, it’s still not an <em>absolute source of truth</em>.</p><p>That’s why production systems often pair <strong>Redis with Kafka</strong> — Redis handles the <strong>speed</strong>, and Kafka ensures <strong>long-term durability and replayability</strong>.</p><h4>4. Durable event stream + materialized counters (recommended for production)</h4><p>Keep a temporary cache like Redis for fast “like” counts but use a durable event stream like Kafka or Pulsar as the main record of all events. Update the Redis counters from the stream in real time and save them to the database periodically. This way, reads are fast, the full history of likes is safe, and recovery is easy if something goes wrong.</p><p><strong>Event schema (suggested)</strong></p><pre>{<br>  &quot;event_id&quot;: &quot;uuid&quot;,<br>  &quot;user_id&quot;: 123,<br>  &quot;post_id&quot;: 456,<br>  &quot;action&quot;: &quot;LIKE&quot; | &quot;UNLIKE&quot;,<br>  &quot;timestamp&quot;: 1690000000<br>}</pre><h4><strong>Write path (detailed)</strong></h4><ul><li>Client sends toggleLike(user_id, post_id).</li><li>API checks local per-user cache (optional) to short-circuit; then <strong>produces</strong> an event to Kafka <strong>synchronously</strong> (write latency dominated by network + Kafka ack). Producing to Kafka provides durability and ordering (per partition).</li><li>The API returns success to client after the event is accepted (optionally after a fast local change for read-after-write UX).</li><li>A set of <strong>stream processors</strong> (consumer group) read the events, update Redis counters with atomic ops, and also produce compacted events or aggregated deltas for persistence.</li><li>A <strong>background writer</strong> consumes deltas and writes aggregated counts to DB periodically.</li></ul><p><strong>Read path</strong></p><ul><li>Read like count from Redis (fast). If cache miss, compute from materialized store (DB) or rebuild on the fly from the stream (expensive). For per-user liked state, read user_liked:post_id key or a compacted topic.</li></ul><p><strong>Benefits</strong></p><ul><li><strong>Durability</strong>: events persisted in Kafka can be replayed to rebuild state after an outage.</li><li><strong>Strong operational recovery:</strong> if Redis is lost, reprocess stream to rebuild counters.</li><li><strong>Partitioning</strong>: events partitioned by post_id to ensure ordering for a given post.</li><li><strong>Scalability</strong>: consumers scale horizontally.</li></ul><p><strong>Implementation notes</strong></p><ul><li>Choose partition key = post_id to collocate all events for a post (preserve ordering).</li><li>Use Kafka topic with enough partitions for throughput and parallelism.</li><li>Use log compaction for a latest-user-action topic if you want to dedupe and retain only the last action per (user_id,post_id).</li></ul><h4>5. Sharding &amp; partitioning strategies</h4><p>When counter volume grows, everything must be partitioned. Two common strategies:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/586/1*6uXGL37v9X5IzTuXRkGIZA.png" /></figure><ul><li><strong>Hash sharding on </strong><strong>post_id</strong>: shard = hash(post_id) % N. Even distribution and easy to compute. Requires re-sharding when N changes (use consistent hashing to reduce reshuffle).</li><li><strong>Range sharding</strong>: use ranges of post_id for predictable hotspots (rarely used for likes).</li></ul><p><strong>Redis cluster</strong>: use Redis Cluster or multiple single-node Redis instances behind a consistent hashing layer. Each shard holds counters for a subset of post_ids.</p><p><strong>Kafka</strong>: partition by post_id. Consumer parallelism = number of partitions.</p><h4>6. Handling per-user uniqueness &amp; toggles</h4><p><strong>Options</strong></p><ol><li><strong>Authoritative per-user store</strong> — keep user_likes (KV or DB) keyed by (user_id, post_id) that stores current like state. API consults/updates it synchronously. This is simplest but can be heavy at scale.</li><li><strong>Event-sourced approach with compaction</strong> — write every LIKE/UNLIKE event to the durable log. Maintain a <strong>compacted topic</strong> keyed by (user_id,post_id) which reflects the last action per user-post. Consumers process compacted topic to derive counts (count how many entries are in liked state). This trades storage and complexity for correctness and replayability.</li><li><strong>Hybrid</strong> — write an event for every action to Kafka and maintain a fast KV cache (Redis) for per-user state. Periodic reconciliation ensures cache &amp; compacted stream agree.</li></ol><p><strong>Practical pattern</strong></p><ul><li>Accept event into the durable log first (Kafka).</li><li>Use a stream processor to maintain a compacted per-user-state topic and update Redis counters in a single consumer transaction (idempotent update).</li><li>Ensure idempotency by using event_id deduplication or by writing idempotent updates (e.g., set user status to 1 or 0 and update count only when state changes).</li></ul><h4>7. Atomicity &amp; idempotency practices</h4><p>To avoid double counting during retries and duplicates:</p><ul><li><strong>Idempotent events</strong>: include event_id and dedupe at consumer.</li><li><strong>Last-action idempotency</strong>: consumer checks current per-user state; change counter only if the new state differs. Example:</li><li><em>current = </em><em>0, event = </em><em>LIKE → set </em><em>user_like = 1, </em><em>INCR counter.</em></li><li><em>current = </em><em>1, event = </em><em>LIKE → no-op.</em></li><li><strong>Transactional writes</strong>: ideally, update per-user state and counter atomically. If using Redis: use a Lua script to atomically read-set-and-incr to keep correctness on the in-memory layer.</li></ul><p><strong>Example Redis Lua pseudo</strong></p><pre>local current = redis.call(&#39;GET&#39;, user_like_key)<br>if current == false and ARGV[1] == &#39;LIKE&#39; then<br>  redis.call(&#39;SET&#39;, user_like_key, &#39;1&#39;)<br>  redis.call(&#39;INCR&#39;, post_count_key)<br>elseif current == &#39;1&#39; and ARGV[1] == &#39;UNLIKE&#39; then<br>  redis.call(&#39;SET&#39;, user_like_key, &#39;0&#39;)<br>  redis.call(&#39;DECR&#39;, post_count_key)<br>end</pre><h4>8. Recovery and reconciliation — When Redis Is Non-Persistent</h4><p><strong>Failure scenarios to handle:</strong></p><ul><li>Redis node crash before DB flush → lost in-memory deltas.</li><li>Consumer failure → paused updates.</li><li>Network partitions → split brain.</li></ul><p><strong>Recovery tools</strong></p><ul><li><strong>Durable log replay (Kafka)</strong>: replay events to rebuild Redis counters and per-user state. Keep retention long enough for safe replay.</li><li><strong>Periodic snapshots</strong>: periodically persist full counters to DB (checkpoint), reducing replay window.</li><li><strong>Delta journaling</strong>: when applying to Redis, also append deltas to a persistent queue/topic; background process consumes deltas to write to DB.</li></ul><p><strong>Reconciliation run</strong></p><ul><li>Compute db_count vs recomputed_count = sum(compacted-user-state liked) over a time window. If divergence &gt; threshold, trigger a reconciliation job that recomputes count from compacted topic or from likes table and repair.</li></ul><h4>9. Final production architecture (recommended)</h4><ul><li><strong>API Gateway / Frontend API</strong>: exposes like/unlike endpoints. Produces events to Kafka and optionally updates local cache for immediate UX.</li><li><strong>Durable Event Broker (Kafka/Pulsar)</strong>: source of truth for all user actions. Partitioned by post_id.</li><li><strong>Stream Processors</strong> (consumer group): 1. Maintain <strong>compacted per-user state</strong> topic keyed by (user_id,post_id). 2. Update <strong>Redis counters</strong> atomically.</li><li>Emit aggregated deltas for persistence.</li><li><strong>Redis Cluster (Non-Persistent):</strong> fast read/write counters and per-user state cache (with Lua scripts for atomic ops).</li><li><strong>Background Persister</strong>: consumes aggregated deltas and writes batched updates to DB. Also periodically snapshots counters.</li><li><strong>Primary Database (Postgres / Cassandra / Scylla)</strong>: persistent store for long-term counts and analytics.</li><li><strong>Monitoring &amp; Alerting</strong>: Prometheus, Grafana for metrics like delta_lag, kafka_lag, redis_misses, db_flush_errors.</li><li><strong>Reconciliation &amp; Recovery Service</strong>: operators can trigger rebuild from Kafka to Redis or DB.</li></ul><p><strong>Data flows</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UHylu0XZY-7JpDjtDfft_A.png" /></figure><ul><li>Client → API → Kafka → Stream Processor → Redis (fast read) → Background Persister → DB.</li><li>For recovery: clear Redis → replay Kafka topics → reconstruct state.</li></ul><h4>10. Data models &amp; examples</h4><p><strong>Event topic (append only)</strong><br> likes-events (partition key: post_id)</p><p><strong>Compacted topic (latest state per user per post)</strong><br> likes-latest key: user_id:post_id value: {state:1|0, last_ts} (log compaction keeps last state)</p><p><strong>Persistent DB tables</strong></p><pre>CREATE TABLE likes_events (<br>  event_id UUID PRIMARY KEY,<br>  user_id BIGINT,<br>  post_id BIGINT,<br>  action VARCHAR,<br>  ts TIMESTAMP<br>);<br><br>CREATE TABLE likes_aggregates (<br>  post_id BIGINT PRIMARY KEY,<br>  like_count BIGINT,<br>  last_updated TIMESTAMP<br>);</pre><h4>11. Pseudocode — API handler + background aggregator</h4><ul><li>The API handler receives a like or unlike request from the user.</li><li>It produces an event to Kafka using a command like:<br> produce(&#39;likes-events&#39;, key=post_id, value={user_id, post_id, action, event_id, ts})</li><li>The API immediately returns success to the client and updates the UI optimistically with the new state.</li><li>A stream processor consumes events from the Kafka topic in real time.</li><li>It reads the latest compaction topic value for the given user:post or checks Redis for the current state.</li><li>If the new event represents a state change, it atomically updates the per-user key and post counter in Redis using a Lua script.</li><li>The same processor writes an aggregated delta message containing (post_id, delta, ts) to the likes-deltas topic.</li><li>A background persister consumes records from the likes-deltas topic.</li><li>It aggregates deltas by post_id over a small-time window (for example, every 1–10 seconds based on SLA).</li><li>It executes a batched SQL update similar to:</li><li>This approach provides low latency writes, high throughput, and eventual consistency between Redis and the database.</li></ul><h4>12. Monitoring &amp; SLOs</h4><p>Track these metrics with thresholds and alerts:</p><ul><li>API request latency (p95, p99).</li><li>Kafka produce + consumer lag (ms + offsets).</li><li>Redis ops/sec, memory usage, and cluster node health.</li><li>Delta queue lag (time between Redis update and DB persistence).</li><li>Reconciliation divergence rate (#posts with |redis_count - db_count| &gt; threshold).</li><li>Error rates for DB updates and consumer failures.</li></ul><p>SLO examples:</p><ul><li>99.9% of like API calls reply within 100ms.</li><li>Event durability: no event loss for committed Kafka offsets.</li><li>Reconciliation drift &lt; 0.5% across active posts.</li></ul><h4>13. Operational runbook (common incidents)</h4><p><strong>Redis shard death</strong></p><ul><li>Promote replica, or rebuild lost shard by replaying Kafka partitions covering affected post_ids.</li></ul><p><strong>Huge delta lag</strong></p><ul><li>Temporarily increase background persister workers; backpressure pipeline to prevent DB overload.</li></ul><p><strong>Count drift detected</strong></p><ul><li>Run reconciliation: replay likes-latest compacted topic into a temporary store and compute fresh counts; update DB/Redis as needed.</li></ul><p><strong>Hot post (viral)</strong></p><ul><li>Dynamically route hot post operations to specialized hot-key handlers or increase partitioning; consider caching read replicas for UI.</li></ul><h4>14. Trade-offs &amp; alternatives</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/887/1*KMeGuXWcHSb2e9oDRmeMyQ.png" /></figure><p><strong>When to use CRDTs</strong>: If multi-DC geo-replication requires local writes with eventual convergence and conflict-free merges. CRDT counters (PN-counters) let sites accept local increments and converge by merging. They add complexity and are recommended only for multi-DC low-coordination scenarios.</p><h4>15. Common pitfalls &amp; tips</h4><ul><li><strong>Not partitioning early</strong>: design with partitioning in mind; re-sharding later is painful.</li><li><strong>Relying only on cache</strong>: always have a durable event log or persistence.</li><li><strong>Ignoring idempotency</strong>: retries and duplicate events will double-count unless idempotent.</li><li><strong>Too-frequent DB flushes</strong>: small flush intervals produce high DB load; aggregate and batch writes.</li><li><strong>Underestimating monitoring</strong>: missing metrics will make subtle consistency issues invisible.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/875/0*eJB3Y9tEwc1HR_jP.png" /></figure><p><strong>Thanks for reading! 🙏</strong><br>If this article helped you, don’t forget to drop a like ❤️</p><p>Got questions or ideas? Share them in the comments — I’d love to hear from you!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e8f4c1440de3" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>