<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Piyush Dubey on Medium]]></title>
        <description><![CDATA[Stories by Piyush Dubey on Medium]]></description>
        <link>https://medium.com/@piyushdubey?source=rss-7d2edfe33219------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*6s6nr3PjiXdElryT.</url>
            <title>Stories by Piyush Dubey on Medium</title>
            <link>https://medium.com/@piyushdubey?source=rss-7d2edfe33219------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 04:51:26 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@piyushdubey/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Choosing the Right AI Agent Framework in 2026: A Practical Guide for Builders]]></title>
            <link>https://medium.com/@piyushdubey/choosing-the-right-ai-agent-framework-in-2026-a-practical-guide-for-builders-3865877dbd0e?source=rss-7d2edfe33219------2</link>
            <guid isPermaLink="false">https://medium.com/p/3865877dbd0e</guid>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[framework]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[llm-agent]]></category>
            <dc:creator><![CDATA[Piyush Dubey]]></dc:creator>
            <pubDate>Wed, 24 Jun 2026 06:27:23 GMT</pubDate>
            <atom:updated>2026-06-24T06:40:19.105Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sqJNt2EzmnteE__BEnJ1Eg.png" /></figure><p><em>LangChain, LangGraph, CrewAI, PydanticAI, OpenAI Agents SDK, Mastra, Vercel AI SDK, n8n, and Vellum — which one should you actually use?</em></p><p>AI agents are no longer just experimental demos stitched together in weekend hackathons. They are starting to power customer support workflows, internal engineering tools, research assistants, sales automations, coding copilots, and enterprise-grade decision systems. But here’s the uncomfortable truth: building an AI agent is easy. Building one that is reliable, debuggable, cost-controlled, secure, and production-ready is much harder. That is where agent frameworks come in.</p><p>The problem is that the agent framework ecosystem has exploded. Every tool promises faster development, better orchestration, easier memory, smoother deployment, or enterprise-grade security. If you are a developer, startup founder, or engineering leader trying to choose the right one, the decision can feel overwhelming. Should you use LangChain because it has the biggest ecosystem? LangGraph because it gives you durable graph execution? CrewAI because it makes multi-agent collaboration feel natural? PydanticAI because you care about type safety? Mastra because your team lives in TypeScript? Or should you skip frameworks altogether and use the OpenAI Agents SDK or Vercel AI SDK?</p><p>Let’s break it down in a practical, human way. This post is inspired by Speakeasy’s detailed comparison of AI agent frameworks, but written as a more conversational guide for builders who need to make a real decision.</p><h4><strong>First: What Is an AI Agent?</strong></h4><p>At its simplest, an AI agent is a loop. An LLM receives an instruction, decides what to do next, takes an action, observes the result, and repeats until it reaches an answer or completes a task. That action might be:</p><ul><li>Calling an API</li><li>Searching the web</li><li>Querying a database</li><li>Writing code</li><li>Reading a document</li><li>Asking a user for clarification</li><li>Delegating work to another agent</li></ul><p>A normal LLM call gives you one response. An agent can take multiple steps. That difference sounds small, but it changes everything. Once an AI system can use tools, remember context, plan steps, and act with some autonomy, you need to think about orchestration, safety, memory, cost, debugging, and deployment. That is where frameworks become useful. <em>Sometimes…</em></p><h4><strong>Do You Actually Need an Agent Framework?</strong></h4><p>Before comparing frameworks, ask the most important question:</p><blockquote><strong>Do you need one at all?</strong></blockquote><p>If your use case is simple for example, <em>“call two tools, summarize the result, and return an answer”</em>, you probably do not need a full framework. A lightweight SDK like the OpenAI Agents SDK or Vercel AI SDK may be enough. If your workflow is extremely custom, latency-sensitive, or requires unusual memory and orchestration patterns, you may be better off building your own thin framework. Frameworks are most useful in the middle:</p><ul><li>You need human-in-the-loop approvals</li><li>You have multi-step workflows</li><li>You need multiple agents working together</li><li>You want tracing and observability</li><li>You need durable execution</li><li>You want consistent tool definitions across a large codebase</li><li>You are building something that must survive production failures</li></ul><p>In other words, frameworks shine when your project is too complex for a simple tool loop but not so custom that a framework gets in your way.</p><p>Let’s walk through the major options.</p><h3>LangChain: The Ecosystem Giant</h3><p>LangChain is still one of the most recognizable names in AI development. Its biggest strength is breadth. If you need an integration, there is a good chance LangChain already has it. It supports a huge number of models, tools, retrievers, vector databases, and agent patterns. For rapid prototyping, this is powerful. You can move quickly and connect many pieces without starting from scratch. But LangChain’s biggest strength can also become its weakness. As applications grow, its abstractions can become hard to debug. When something breaks, you may find yourself digging through framework internals instead of your own business logic. Earlier versions also had a reputation for churn, though the newer v1.0 release has improved stability.</p><p><strong>Use LangChain if:</strong></p><ul><li>You want maximum integration coverage</li><li>You are prototyping quickly</li><li>Your workflow is mostly linear tool-calling</li><li>You want access to a mature ecosystem</li></ul><p><strong>Avoid LangChain if:</strong></p><ul><li>You need deep control over execution</li><li>You want strong type safety</li><li>You are building complex cyclic workflows</li><li>Debuggability is your top priority</li></ul><blockquote>LangChain is great when ecosystem coverage matters more than architectural purity.</blockquote><h3>LangGraph: For Serious Stateful Workflows</h3><p>LangGraph is the lower-level orchestration framework behind many modern LangChain agent patterns. If LangChain is the friendly abstraction layer, LangGraph is the engine room. It lets you model agents as graphs with nodes, edges, state, cycles, checkpoints, and resumable execution. That makes it especially strong for workflows that need durability. This matters in production. If your agent takes 20 steps and your server crashes at step 17, you do not want to start over. LangGraph can checkpoint execution and resume from prior states. It is also strong for human-in-the-loop workflows. You can pause execution, let a person inspect or modify state, and then continue. The tradeoff is complexity. LangGraph is explicit, powerful, and verbose. You will write more code, and your team will need to understand graph-based orchestration.</p><p><strong>Use LangGraph if:</strong></p><ul><li>You are building stateful production agents</li><li>You need checkpointing and crash recovery</li><li>Your workflows involve cycles or branching</li><li>You need human approval steps</li><li>You are a Python team that values control</li></ul><p><strong>Avoid LangGraph if:</strong></p><ul><li>You want the fastest prototype</li><li>Your team prefers TypeScript</li><li>You are deploying primarily to serverless environments</li><li>You want simple role-based agent abstractions</li></ul><blockquote>LangGraph is one of the best choices for production-grade Python agents, especially when reliability matters.</blockquote><h3>CrewAI: The Most Natural Multi-Agent Mental Model</h3><p>CrewAI is popular because it maps well to how humans think about teams. You create agents with roles, goals, and backstories. Then you assign tasks and let the “crew” collaborate. This is intuitive and fun to build with. For example, you might create:</p><ul><li>A researcher agent</li><li>A writer agent</li><li>A reviewer agent</li><li>A manager agent</li></ul><p>Each has a clear responsibility, and together they complete a workflow.</p><p>This makes CrewAI one of the fastest ways to prototype multi-agent systems. It is especially appealing for business workflows, content generation pipelines, analysis teams, and role-based collaboration. But the abstraction can become a black box. Debugging why one agent delegated incorrectly or why a loop became expensive can be painful. Multi-agent systems also multiply token usage quickly.</p><p><strong>Use CrewAI if:</strong></p><ul><li>You want role-based multi-agent collaboration</li><li>You need to prototype quickly</li><li>Your workflow resembles a team structure</li><li>You value simplicity over low-level control</li></ul><p><strong>Avoid CrewAI if:</strong></p><ul><li>You need fine-grained debugging</li><li>You have strict cost controls</li><li>Your workflows require complex branching</li><li>You need predictable execution at scale</li></ul><blockquote>CrewAI is fantastic for demos and early multi-agent prototypes, but production teams should be careful with observability and cost limits.</blockquote><h3>PydanticAI: Type Safety for Practical Python Teams</h3><p>PydanticAI feels like the agent framework Python backend developers were waiting for. It comes from the Pydantic ecosystem, so its core strength is validation. You define structured outputs using Pydantic models, and the framework validates LLM responses against those schemas. That is a big deal. LLMs are unpredictable. They return malformed JSON, missing fields, unexpected values, and strange edge cases. PydanticAI helps catch these problems before they hit downstream systems. It is also lightweight and developer-friendly. If you like FastAPI and clean Python code, PydanticAI will probably feel natural. The downside is ecosystem maturity. Compared with LangChain, it has fewer integrations and fewer community examples.</p><p><strong>Use PydanticAI if:</strong></p><ul><li>You care about type safety</li><li>You want structured outputs</li><li>You are building Python services</li><li>You prefer simple, explicit code</li><li>You want built-in usage limits for cost control</li></ul><p><strong>Avoid PydanticAI if:</strong></p><ul><li>You need a massive integration ecosystem</li><li>You need first-class multi-agent abstractions</li><li>You want managed hosting</li><li>You rely heavily on visual workflow tooling</li></ul><blockquote>PydanticAI is one of the best choices for backend teams that want reliable, maintainable AI code.</blockquote><h3>OpenAI Agents SDK: Minimal, Clean, and Practical</h3><p>The OpenAI Agents SDK is not trying to be everything. That is its strength. It gives you a simple way to define agents, tools, handoffs, guardrails, and sessions. You can build a working agent quickly without learning a large framework. It also integrates naturally with OpenAI’s hosted tools like web search, file search, computer use, and code execution. Tracing is available through the OpenAI dashboard, which makes debugging easier from day one. The tradeoff is lock-in. If you rely heavily on OpenAI-hosted tools, moving away later may be harder.</p><p><strong>Use OpenAI Agents SDK if:</strong></p><ul><li>You are already using OpenAI models</li><li>You want a minimal abstraction</li><li>You need simple handoffs between agents</li><li>You want hosted tools with little setup</li><li>You do not need graph-based orchestration</li></ul><p><strong>Avoid it if:</strong></p><ul><li>You need provider independence</li><li>You require complex workflow graphs</li><li>You need parallel multi-agent execution</li><li>You want full control over hosted tool infrastructure</li></ul><blockquote>For many teams, this is the best starting point. Do not over-engineer if a simple SDK solves the problem.</blockquote><h3><strong>Mastra: The TypeScript-Native Powerhouse</strong></h3><p>Mastra is one of the most exciting options for TypeScript teams. It gives you agents, tools, workflows, memory, and deployment primitives in a style that feels natural to modern JavaScript and TypeScript developers. It also integrates with the Vercel AI SDK ecosystem, which gives it access to many providers and models. Where Mastra stands out is its combination of agent loops and graph-based workflows. You can build sequential, branching, and parallel workflows in TypeScript. It also has a strong memory story, including automatic context compression. That makes it especially compelling for teams building AI products in Next.js, serverless environments, or modern web stacks. The concerns are around maturity, platform coupling, and enterprise compliance. Some auth features are still evolving, and teams with strict SOC 2 requirements may need to evaluate carefully.</p><p><strong>Use Mastra if:</strong></p><ul><li>Your team uses TypeScript</li><li>You need graph-like workflows</li><li>You want serverless-friendly deployment</li><li>You care about memory management</li><li>You are building modern AI web products</li></ul><p><strong>Avoid Mastra if:</strong></p><ul><li>You need mature enterprise compliance today</li><li>You prefer Python</li><li>You want time-travel debugging like LangGraph</li><li>You need CrewAI-style role abstractions</li></ul><blockquote>If I were building a serious TypeScript AI product today, Mastra would be high on my shortlist.</blockquote><h3>Vercel AI SDK: Best for AI-Powered Web Apps</h3><p>The Vercel AI SDK is not a full agent framework in the same way LangGraph or CrewAI is. It is better understood as the best toolkit for building AI experiences into web applications. If you are building with Next.js or React, the SDK is excellent. It handles streaming, chat state, completions, tool calls, and provider abstraction beautifully. For simple agents embedded in web apps, it may be all you need. The limits appear when workflows become long-running, deeply stateful, or highly orchestrated. Serverless timeouts and manual memory management can become constraints.</p><p><strong>Use Vercel AI SDK if:</strong></p><ul><li>You are building AI features in React or Next.js</li><li>You need streaming chat UI</li><li>Your workflows are relatively simple</li><li>You want easy deployment on Vercel</li><li>You prefer TypeScript</li></ul><p><strong>Avoid it if:</strong></p><ul><li>You need long-running agents</li><li>You need complex graph orchestration</li><li>You require native checkpointing</li><li>You want advanced memory built in</li></ul><blockquote>For AI user interfaces, Vercel AI SDK is hard to beat. For deep agent orchestration, pair it with something else.</blockquote><h3>n8n: Visual Automation With AI Agents</h3><p>n8n is different from most tools on this list. It is a visual workflow automation platform with AI agent capabilities. Its biggest advantage is accessibility. Non-developers can build workflows by connecting nodes on a canvas. It has hundreds of integrations and is great for business automation. Want to connect Gmail, Slack, HubSpot, Notion, Postgres, and an AI model? n8n makes that easy. But visual workflows have limits. As branching logic gets complex, the canvas can become difficult to maintain. Persistent memory also requires careful setup with external storage.</p><p><strong>Use n8n if:</strong></p><ul><li>Non-technical users need to build workflows</li><li>You are connecting many SaaS tools</li><li>Your workflows are business-process oriented</li><li>You want visual automation plus AI</li></ul><p><strong>Avoid it if:</strong></p><ul><li>You need complex dynamic orchestration</li><li>You want code-first maintainability</li><li>You need advanced agent memory</li><li>You require token-level cost visibility</li></ul><blockquote>n8n is excellent for AI-powered automation, not necessarily for deeply engineered agent systems.</blockquote><h3>Vellum: Visual Workflows With Production Discipline</h3><p>Vellum is built for teams that want both a visual builder and a real code backend. That combination matters. Many visual tools are great for demos but painful for engineers. Vellum tries to bridge that gap with workflow versioning, deployment environments, evaluations, prompt management, and SDK support. Its strongest feature is production workflow management. You can test prompts, compare models, review outputs, deploy versions, and manage workflows more rigorously than in many other platforms. The downside is lock-in and pricing. Vellum is more of a managed platform than an open framework.</p><p><strong>Use Vellum if:</strong></p><ul><li>You need visual workflows and code sync</li><li>You care about evaluations</li><li>You want production deployment workflows</li><li>You have non-technical and technical users collaborating</li><li>You need enterprise controls</li></ul><p><strong>Avoid it if:</strong></p><ul><li>You want open-source flexibility</li><li>You need low platform lock-in</li><li>You require high concurrency on lower tiers</li><li>You want native checkpoint/time-travel debugging</li></ul><blockquote>Vellum is a strong option for companies operationalizing AI workflows across teams.</blockquote><h3>Which One Should You Choose?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-NAujAsbUfxMsAiqdXpVvw.png" /></figure><h3>Final Thoughts</h3><p>The best AI agent framework is not the one with the most GitHub stars, the most integrations, or the flashiest demo. It is the one that matches your team’s language, your deployment environment, your debugging needs, your security requirements, and your workflow complexity.</p><p>Agent frameworks are not magic. They are tradeoff machines. Some optimize for speed. Some optimize for control. Some optimize for visual collaboration. Some optimize for type safety. Some optimize for enterprise governance. Choose based on the pain you actually have, not the architecture diagram you hope to draw. Because in production, the winning framework is not the one that helps you build the fastest demo. It is the one your team can still understand, debug, secure, and afford six months later.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3865877dbd0e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How AI Agents Search Memory: A Deep Dive into Hybrid Search & Re-ranking]]></title>
            <link>https://medium.com/@piyushdubey/how-ai-agents-search-memory-a-deep-dive-into-hybrid-search-re-ranking-5faa5b8da5e7?source=rss-7d2edfe33219------2</link>
            <guid isPermaLink="false">https://medium.com/p/5faa5b8da5e7</guid>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Piyush Dubey]]></dc:creator>
            <pubDate>Wed, 18 Feb 2026 13:07:25 GMT</pubDate>
            <atom:updated>2026-02-18T13:07:25.722Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uAzvx3h1bKtwaG8Lij0vuQ.png" /></figure><h3>The Search for Intelligence</h3><p>In the world of AI agents, memory is useless if it’s buried. For AI agents, “Searching Memory” is the process of retrieving the most relevant “pages” to answer a user’s query without bloating the context window.</p><h3>1. The Two Pillars: Keyword vs. Semantic Search</h3><p>It begins by distinguishing between two fundamental ways agents find information:</p><ul><li><strong>Keyword Search (Literal Matches):</strong> This is the “Ctrl+F” of memory. It relies on exact word matches. Tools like <strong>BM25</strong> (Best Matching 25) go a step further by ranking results based on how often a term appears and how unique it is. While simple, it is surprisingly robust for finding specific error codes, function names, or unique identifiers.</li><li><strong>Semantic Search (Meaningful Matches):</strong> This searches by <em>intent</em> rather than literal characters. Using <strong>Embeddings</strong> — long lists of numbers representing the “essence” of a text — an agent can understand that “How do I speed up my app?” is conceptually similar to “tips for improving application performance,” even if they share zero words.</li></ul><h3>2. The Hybrid Approach: Why Both are Better</h3><p>A major takeaway is that semantic search isn’t a silver bullet. It often fails with specific technical strings like error_connection_refused.</p><p>The solution used by high-end systems (like <strong>OpenClaw</strong>) is <strong>Hybrid Search</strong>. They run keyword and semantic searches in parallel and then “fuse” the results. Two methods are highlighted:</p><ul><li><strong>Weighted Score Fusion:</strong> Assigning a percentage of importance to each (e.g., 70% weight to semantic, 30% to keyword).</li><li><strong>Reciprocal Rank Fusion (RRF):</strong> Ranking results based on their position in both lists rather than raw scores. If a result is #2 in both, it bubbles to the top.</li></ul><h3>3. Re-ranking: The Final Filter</h3><p>Even with a hybrid search, math-based ranking can be “dumb.” <strong>Re-ranking</strong> introduces a second step where a smaller, refined set of results is passed to a more powerful LLM (or a specialized model like Cohere) to judge nuance. It checks if the search result <em>actually</em> answers the user’s specific query. While this adds latency and cost, it significantly boosts accuracy.</p><h3>4. Real-World Case Study: OpenClaw’s Architecture</h3><p><strong>OpenClaw</strong>, an open-source project that serves as a practical blueprint for these concepts:</p><ul><li><strong>Storage:</strong> It uses <strong>SQLite</strong> with extensions like sqlite-vec for vector search and FTS5 for keyword search.</li><li><strong>Chunking:</strong> It doesn’t store whole files. Instead, it breaks markdown files into <strong>400-token chunks</strong> with an 80-token overlap, ensuring no context is lost at the edges of a “page”.</li></ul><p><strong>The Two-Step Tool Pattern:</strong></p><ul><li><strong>Memory Search:</strong> The agent first searches for snippets and gets a preview with “relevance scores.”</li><li><strong>Memory Get:</strong> Only after finding the right snippet does the agent use a “get” tool to read the full context of that specific file. This keeps the agent’s context window lean.</li></ul><h3>5. Staying in Sync</h3><p>Finally, How memory stays “fresh.” OpenClaw uses a <strong>File Watcher</strong> that detects changes in markdown files. To save money and time, it uses an <strong>Embedding Cache</strong>. If you change one sentence in a large file, it only re-embeds the specific chunk that changed, skipping the ones that already exist in the cache.</p><h3>Conclusion</h3><p>Effective AI memory isn’t just about having a big database; it’s about the sophisticated dance between keyword matching, semantic understanding, and intelligent re-ranking. By combining these methods, agents can feel “human” in their ability to recall exactly what was discussed three weeks ago or find a needle in a haystack of documentation.</p><p><strong>Watch the full video here by </strong>Damian Galarza<strong> :</strong> <a href="https://youtu.be/SpReZZk_13w?si=SYDvqVzIM5lSZh1F">How AI Agents Search Their Memory</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5faa5b8da5e7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Ditch the Subscription: How to Run a Local AI Agent Server for Free]]></title>
            <link>https://medium.com/@piyushdubey/ditch-the-subscription-how-to-run-a-local-ai-agent-server-for-free-c0ef217bd564?source=rss-7d2edfe33219------2</link>
            <guid isPermaLink="false">https://medium.com/p/c0ef217bd564</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[ai-agent]]></category>
            <dc:creator><![CDATA[Piyush Dubey]]></dc:creator>
            <pubDate>Thu, 29 Jan 2026 13:54:15 GMT</pubDate>
            <atom:updated>2026-01-29T13:54:15.307Z</atom:updated>
            <content:encoded><![CDATA[<blockquote><strong><em>TL;DR:</em></strong><em> Stop paying $20/month for AI. Learn how to use </em><strong><em>LM Studio</em></strong><em> to host powerful open-source models (like </em><em>gpt-oss-20b) on your own hardware. This guide covers turning your machine into a private AI server and integrating it into </em><strong><em>VS Code Copilot</em></strong><em> and </em><strong><em>OpenCode</em></strong><em> for a professional, free, and private developer workflow.</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2QBcPmtTLj4u3rKJ9oaX6g.png" /></figure><p>Have you ever wondered what it’s like to have your own AI model that is always there for you — ready to answer questions or act as a personal assistant — without paying a single dollar?</p><p><strong>Yes, you can actually do that.</strong> And no, you don’t need a room full of expensive server hardware.</p><p>There are dozens of powerful “open-weight” models available today that you can download and run on your own computer. Most people are hesitant to try because they assume the setup is too technical or requires coding skills.</p><p>In this guide, I’ll show you how to get up and running for free, integrating these models with open-source agents so you can build your own digital assistant, regardless of whether you are a developer or a casual user.</p><h3>Part 1: Get Your Favorite Model with LM Studio</h3><p>The easiest way to start is with <strong>LM Studio</strong>. It is a clean, user-friendly tool that lets you discover, download, and run local AI models with a single click.</p><h4>1. Download &amp; Install</h4><p>First, head over to <a href="https://lmstudio.ai/">lmstudio.ai</a> and install the version compatible with your operating system (Windows, Mac, or Linux).</p><h4>2. Choose Your Brain: gpt-oss-20b</h4><p>For our setup, we’ll use the <strong>gpt-oss-20b</strong> model. This is an excellent “middle-ground” model:</p><ul><li><strong>Reasoning:</strong> It’s designed to think through complex problems.</li><li><strong>Tool Calling:</strong> It is specifically trained to interact with other apps (perfect for an AI Agent).</li><li><strong>Efficiency:</strong> It works smoothly on most modern consumer laptops without needing a $5,000 GPU.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WzWWlyRolxBV0riX8YYmjg.png" /></figure><h4>3. Start Chatting</h4><p>Once the model is downloaded, click the <strong>Chat</strong> icon on the sidebar. Select your model from the top dropdown, and you’re ready to go. You can ask it questions just like you would with ChatGPT — but this time, the data never leaves your machine.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-A8mScK6YN9BRUYD60e9Ow.png" /></figure><h3>Part 2: Moving Beyond Chat: Building Your Own AI Server</h3><p>If you just want to talk to an AI, the chat interface works perfectly. But if you’re a developer or want to build an <strong>Agentic AI</strong> — an AI that can actually <em>do</em> things like write code, search the web, or manage files — you need to turn your local model into a <strong>Server</strong>.</p><p>Fortunately, LM Studio comes with a built-in Local Inference Server that acts exactly like OpenAI’s API, but runs entirely on your hardware.</p><h4>How to Start the Server</h4><ul><li><strong>Navigate to the Developer Tab:</strong> On the left-hand sidebar, click the <strong>&lt; &gt; (Developer)</strong> icon.</li><li><strong>Configure Your Settings:</strong> In the side panel, you will see a few critical settings:<br> — <strong>Server Port:</strong> By default, this is set to 1234. You can leave it as is.<br> — <strong>Serve on Local Network:</strong> Toggle this <strong>ON</strong>. This allows other devices or applications on your WiFi to “talk” to your AI model.</li><li><strong>Load the Model:</strong> Select your model (like gpt-oss-20b) from the dropdown at the top and click <strong>Start Server</strong>.</li></ul><p>Once the light turns green, your computer is officially hosting its own private AI API.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ByTwBedgzJv7denPd5FZnw.png" /></figure><p>To make sure everything is running correctly, you can use a simple curl command in your terminal. This mimics how a real AI Agent would send a request to your model.</p><p>Open your <strong>Terminal</strong> app, copy the command provided in the LM Studio “Example Request” section (or use the one below), and hit enter:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TIhBQOEi5u6l1Xz2CdwuwA.png" /></figure><pre>curl http://localhost:1234/v1/chat/completions \<br>  -H &quot;Content-Type: application/json&quot; \<br>  -d &#39;{<br>    &quot;model&quot;: &quot;openai/gpt-oss-20b&quot;,<br>    &quot;messages&quot;: [<br>        {<br>            &quot;role&quot;: &quot;system&quot;,<br>            &quot;content&quot;: &quot;Always answer in rhymes. Today is Thursday&quot;<br>        },<br>        {<br>            &quot;role&quot;: &quot;user&quot;,<br>            &quot;content&quot;: &quot;What day is it today?&quot;<br>        }<br>    ],<br>    &quot;temperature&quot;: 0.7,<br>    &quot;max_tokens&quot;: -1,<br>    &quot;stream&quot;: false<br>}&#39;</pre><p>If you see a JSON response with a “message” back from the AI, <strong>congratulations</strong> — you’ve just built a local AI backend!</p><pre>{<br>  &quot;id&quot;: &quot;chatcmpl-b5ftejfj5nl64ttvrrdlc2&quot;,<br>  &quot;object&quot;: &quot;chat.completion&quot;,<br>  &quot;created&quot;: 1769276982,<br>  &quot;model&quot;: &quot;openai/gpt-oss-20b&quot;,<br>  &quot;choices&quot;: [<br>    {<br>      &quot;index&quot;: 0,<br>      &quot;message&quot;: {<br>        &quot;role&quot;: &quot;assistant&quot;,<br>        &quot;content&quot;: &quot;The calendar says with a cheerful cheer,  \nToday’s the day we call Thursday!&quot;,<br>        &quot;reasoning&quot;: &quot;Need to rhyme. Today Thursday.&quot;,<br>        &quot;tool_calls&quot;: []<br>      },<br>      &quot;logprobs&quot;: null,<br>      &quot;finish_reason&quot;: &quot;stop&quot;<br>    }<br>  ],<br>  &quot;usage&quot;: {<br>    &quot;prompt_tokens&quot;: 89,<br>    &quot;completion_tokens&quot;: 34,<br>    &quot;total_tokens&quot;: 123<br>  },<br>  &quot;stats&quot;: {},<br>  &quot;system_fingerprint&quot;: &quot;openai/gpt-oss-20b&quot;<br>}</pre><h3>Optimizing for Your Hardware</h3><p>Your local AI setup is now live and breathing! You can consume your private API using any <strong>OpenAI-compatible SDK</strong> or standard <strong>REST API</strong> calls.</p><p>But before you go full “Tony Stark,” you need to ensure your “suit” (your hardware) can handle the “power” (the model).</p><h3>Choosing the Right Model Size</h3><p>LM Studio is quite helpful here; it provides a color-coded indicator when you search for models. If a model is too large for your system’s RAM, the app will warn you before you even hit download.</p><ul><li><strong>Integrated Graphics (Standard Laptops):</strong> Look for <strong>“Quantized”</strong> models (labeled as Q4_K_M or similar). These are compressed versions that retain most of the intelligence while using significantly less memory.</li><li><strong>The Gamer Advantage:</strong> If you’re a gamer and own an <strong>NVIDIA GPU</strong>, you’re in luck! You can leverage <strong>CUDA</strong> to offload the heavy lifting to your graphics card. This makes the AI’s response time (tokens per second) feel lightning-fast — almost indistinguishable from GPT-4.</li></ul><h3>Why This Setup is a Game Changer</h3><p>Since the server follows the OpenAI format, you don’t have to rewrite your code if you decide to switch models. You can test a small 7B model today and, if you upgrade your hardware tomorrow, swap it for a massive 70B model just by changing a single setting in LM Studio.</p><blockquote><strong><em>Pro Tip:</em></strong><em> If you have an NVIDIA card, make sure to enable “GPU Offload” in the LM Studio settings to see a massive jump in performance.</em></blockquote><h3>Part 3 : Integrating Your Local AI with VS Code Copilot</h3><p>If you’re a developer, the ultimate goal is to have your AI assistant living directly inside your IDE. While <strong>GitHub Copilot</strong> is a powerhouse, you can now swap its “brain” for your local LM Studio server using the <strong>“Bring Your Own Model” (BYOM)</strong> feature.</p><h3>Prerequisites: The Insiders Build</h3><p>At the time of writing, this deep integration requires the <strong>VS Code Insiders</strong> build. This version lets you test the latest AI features before they hit the stable release.</p><ul><li><strong>Download:</strong> <a href="https://code.visualstudio.com/insiders">VS Code Insiders</a></li></ul><h3>Step-by-Step Integration</h3><ul><li><strong>Open Copilot Chat:</strong> Launch VS Code Insiders and open the Copilot Chat window (Ctrl+Alt+I or Cmd+Shift+I).</li><li><strong>Manage Models:</strong> Click the model name at the top of the chat window and select <strong>“Manage Models…”</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5rx-46eKwzEOCiBnwKX-bA.png" /></figure><ul><li><strong>Add Your Local Provider:</strong> In the Language Models editor, click <strong>“Add Model”</strong> and choose the <strong>“OpenAI Compatible”</strong> option.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PwxON9XBEbXvJlhLQPO2Ig.png" /></figure><ul><li><strong>Configure the Connection:</strong> You will be prompted to enter a name and endpoint. Once you do, VS Code will open a configuration file. Replace the contents with this block:</li></ul><pre>{<br>  &quot;name&quot;: &quot;CustomOAI&quot;,<br>  &quot;vendor&quot;: &quot;customoai&quot;,<br>  &quot;models&quot;: [<br>   {<br>    &quot;name&quot;: &quot;gpt-oss-20b&quot;,<br>    &quot;url&quot;: &quot;http://127.0.0.1:1234&quot;,<br>    &quot;toolCalling&quot;: true,<br>    &quot;vision&quot;: false,<br>    &quot;thinking&quot;: true,<br>    &quot;maxInputTokens&quot;: 131072,<br>    &quot;maxOutputTokens&quot;: 8145,<br>    &quot;id&quot;: &quot;openai/gpt-oss-20b&quot;<br>   }<br>  ],<br>  &quot;apiKey&quot;: &quot;${input:chat.lm.secret.2cec8e79}&quot;<br> }</pre><p><strong>Key Mapping Tip:</strong></p><ul><li><strong>URL:</strong> This must match the port you set in LM Studio (default is 1234).</li><li><strong>ID:</strong> Ensure this matches the specific model ID shown in the LM Studio server logs.</li><li><strong>Tool Calling:</strong> Set this to true if your local model (like gpt-oss-20b) supports it; this allows the model to act as a true agent!</li></ul><h3>Activating the Model</h3><p>Once you save the config, look for your new model in the <strong>Language Models</strong> tab. Click the <strong>Eye Icon</strong> to make it visible in your chat window.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dA8YByaqk2Ju6G1UGZ9PcQ.png" /></figure><p>Now, in your next chat session, you can simply select <strong>gpt-oss-20b</strong> from the dropdown. You are now “vibe coding” with a model that is running 100% locally, costing you exactly $0.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aPGhV3xJLfpstsxLHHD5-A.png" /></figure><h3>💡 Troubleshooting Tip: Context Length Errors</h3><p>If you try to run a prompt and see an error like:</p><blockquote>“The number of tokens to keep from the initial prompt is greater than the context length.”</blockquote><p><strong>Don’t panic!</strong> This just means your current conversation has exceeded the memory limit you set when you loaded the model.</p><p><strong>How to fix it:</strong></p><ol><li>Go back to <strong>LM Studio</strong>.</li><li><strong>Eject</strong> the current model.</li><li>Look at the <strong>Configuration</strong> panel on the right-hand side before reloading.</li><li>Find the <strong>Context Length</strong> (or n_ctx) setting.</li><li>Increase this value (e.g., change 2048 to <strong>8192</strong> or higher, depending on your system’s RAM).</li><li><strong>Reload</strong> the model.</li></ol><p>By increasing the context length, you give the AI a “bigger desk” to work on, allowing it to process larger code files and longer conversation histories without crashing.</p><h3>Part 4: Terminal Mastery with OpenCode</h3><p>While VS Code is great for a visual experience, many developers prefer the efficiency of the terminal. <a href="https://opencode.ai/"><strong>OpenCode</strong></a> (similar to ClaudeCode or Gemini CLI) is a terminal-based AI agent that can plan, edit, and refactor your code directly through a TUI (Terminal User Interface).</p><p>The best part? It’s completely open-source and allows you to “bring your own brain” by plugging in any OpenAI-compatible local model.</p><h4>1. Install OpenCode</h4><p>If you haven’t installed it yet, you can do so quickly via your favorite package manager:</p><pre># Using npm<br>npm install -g opencode-ai<br><br># Or using the official install script<br>curl -fsSL https://opencode.ai/install | bash</pre><h4>2. Configure Your Local Provider</h4><p>To tell OpenCode to use your LM Studio server instead of a paid cloud provider, you need to edit its configuration file. Open your terminal and create or edit the config file:</p><pre>vi ~/.config/opencode/opencode.jsonc</pre><p>Paste the following JSON block into the file. This tells OpenCode to use the @ai-sdk/openai-compatible driver to talk to your local machine:</p><pre>{<br>  &quot;$schema&quot;: &quot;https://opencode.ai/config.json&quot;,<br>  &quot;permission&quot;: {<br>    &quot;*&quot;: &quot;ask&quot;,<br>    &quot;bash&quot;: &quot;allow&quot;,<br>    &quot;codesearch&quot;: &quot;allow&quot;,<br>    &quot;read&quot;: &quot;allow&quot;<br>  },<br>  &quot;provider&quot;: {<br>    &quot;LMStudio&quot;: {<br>      &quot;npm&quot;: &quot;@ai-sdk/openai-compatible&quot;,<br>      &quot;name&quot;: &quot;LM Studio (local)&quot;,<br>      &quot;options&quot;: {<br>        &quot;baseURL&quot;: &quot;http://127.0.0.1:1234/v1/&quot;<br>      },<br>      &quot;models&quot;: {<br>        &quot;gpt-oss-20b&quot;: {<br>          &quot;name&quot;: &quot;gpt-oss-20b&quot;<br>        }<br>      }<br>    }<br>  }<br>}</pre><h4>3. Connect and Start Coding</h4><p>Once the config is saved:</p><ul><li>Open your terminal in a project folder and run opencode.</li><li>Type the /connect command.</li><li>Select <strong>LMStudio</strong> from the list.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aA15V0QaGqqXbj2QqpT_Pw.png" /></figure><ul><li>Use the /models command to select <strong>gpt-oss-20b</strong>.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aA15V0QaGqqXbj2QqpT_Pw.png" /></figure><blockquote><strong>Pro Tip:</strong> When it asks for an API key, just type in a random string (like local-dev). LM Studio doesn&#39;t require a real key, but OpenCode expects one to be present.</blockquote><p>That’s it! You now have a high-performance coding agent running in your terminal, using your local hardware as its engine. You can ask it to “Refactor this function” or “Add unit tests for the auth module,” and it will go to work — <strong>completely for free.</strong></p><h3>Why Local AI Wins:</h3><p>We’ve gone from a simple chat interface to a full-blown AI server and integrated it into both a modern IDE and a terminal agent.</p><ul><li><strong>Zero Subscription Fees:</strong> No more $20/month bills.</li><li><strong>Privacy:</strong> Your conversations stay on your hard drive.</li><li><strong>Offline Access:</strong> It works even when your internet doesn’t.</li></ul><blockquote><strong><em>Note:</em></strong><em> If you have a computer with limited RAM, look for the “quantized” versions of models in LM Studio (usually labeled as Q4 or Q5) to ensure they run fast.</em></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c0ef217bd564" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Beyond the Resume: Why Everyone Needs a Portfolio (And How to Build One Without Touching CSS with…]]></title>
            <link>https://medium.com/@piyushdubey/beyond-the-resume-why-everyone-needs-a-portfolio-and-how-to-build-one-without-touching-css-with-c1fc13d80195?source=rss-7d2edfe33219------2</link>
            <guid isPermaLink="false">https://medium.com/p/c1fc13d80195</guid>
            <category><![CDATA[career-advice]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[cloudflare]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[backend]]></category>
            <dc:creator><![CDATA[Piyush Dubey]]></dc:creator>
            <pubDate>Mon, 29 Dec 2025 15:38:47 GMT</pubDate>
            <atom:updated>2025-12-29T15:38:47.524Z</atom:updated>
            <content:encoded><![CDATA[<h3>Beyond the Resume: Why Everyone Needs a Portfolio (And How to Build One Without Touching CSS with Zero Cost)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NcE4UduJgLc0JiUBUZx5rw.png" /></figure><p>Have you ever wondered what the best way is to showcase your unique skills outside of a standard two-page resume? When you’re applying for a job, standing out from the crowd is the goal — and a personal portfolio might be your best secret weapon. <em>While I’m writing this from a backend developer’s perspective, these steps work for anyone looking to build a digital presence for free.</em></p><h3>The “Backend Developer” Dilemma</h3><p>I’ll be honest: I’m a backend developer. I’m not exactly a wizard when it comes to UI/UX design 😛. Like many of you, I find myself more comfortable with terminal and databases than I do with padding and margins.</p><p>However, I still wanted a web app to showcase my skills and give recruiters a deeper look into my achievements while I’m on the hunt for my next gig.</p><h3>Why Your Resume Isn’t Enough</h3><p>Yes, you <em>could</em> cram everything into your resume, but that’s not really what a resume is for. A resume is a snapshot; it isn’t meant to house complete information about your:</p><ul><li><strong>Pet projects</strong> that show off your personality.</li><li><strong>Niche skills</strong> that might not be required for the specific job you’re applying for today but show your range.</li><li><strong>Deep dives</strong> into how you solved specific technical hurdles.</li></ul><p>That’s where a portfolio comes in handy. It gives you the flexibility to tell your story in a way that a PDF simply can’t.</p><h3>The “Clever Route”</h3><p>I know what you’re thinking. Creating a web app feels like it’s against your “backend” nature. If you aren’t familiar with frameworks like React or Angular — or if you simply hate wrestling with HTML/CSS — the barrier to entry feels high.</p><p>But here is the truth: to create a web app today, you don’t need to be a CSS master.</p><p>I decided to take a clever route: <strong>I delegated the UI/UX to AI.</strong> By using AI to handle the design and boilerplate, I was able to focus entirely on the content and the logic. It allowed me to build a professional-looking space without the headache of frontend frustration.</p><h3>Choosing Your Tools: Do You Need a Full-Stack Masterpiece?</h3><p>The next question was: <em>What do I actually need to build?</em> Do I need a full-fledged web app loading dynamic content with heavy frameworks?</p><p><strong>Of course not.</strong> What I needed was a clean, aesthetic HTML page. While I knew AI could help, I wondered if I needed a massive, state-of-the-art model just for a professional landing page.</p><p>Fortunately, I stumbled upon <strong>GLM-4</strong> (by Z.ai). After looking into its background, I found it was specifically optimized for coding capabilities and web projects. It’s a powerhouse for building web apps from scratch. And while you <em>can</em> self-host it if you have a spare H100 GPU lying around (which, let’s face it, most of us don’t!), their web interface works perfectly for this.</p><h3>The AI Success Story</h3><p>I started with a very specific, simple prompt to give the AI a clear vision:</p><blockquote>“Create a simple portfolio HTML web page for me with a dark theme inspired by a ‘hacker developer’ aesthetic. Include buttons like About Me, Blogs, LinkedIn, and GitHub with a magnetic hover effect.”</blockquote><p>The result? It worked like a charm. The model delivered exactly what I was looking for: a complete HTML file with embedded CSS.</p><p>It wasn’t 100% perfect on the first try, but that’s the beauty of the “clever route” — I could refine the design with just a few more prompts. In the end, I had a professional-looking site where all I had to do was hardcode my actual content.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*W9dW5fxa6AGo2LbRXZbcXw.png" /><figcaption>Portfolio page</figcaption></figure><h3>From Local File to Global URL: The “Zero-Cost” Infrastructure</h3><p>So, I built a killer portfolio with a single HTML file that perfectly showcases my skills. Now came the real challenge: <strong>How do I get it online without breaking the bank or becoming a full-time server administrator?</strong></p><p>I wanted to avoid the old-school FTP route. I wanted something modern: Git-driven, globally cached, and fully automated.</p><h4>Why I Skipped Other Options</h4><p>I looked at a few different paths before settling on a solution:</p><ul><li><strong>Oracle Cloud (OCI):</strong> They offer a great “Always Free” tier with 2 CPUs and 1GB of RAM. While it’s powerful enough to host a single page, I didn’t want the operational overhead of managing a Linux instance or dealing with cumbersome DNS setups just for a portfolio.</li><li><strong>Managed Services (WordPress/Squarespace):</strong> These are great for some, but they aren’t free if you want a custom experience, and they feel a bit “bloated” for a simple dev portfolio.</li><li><strong>GitHub Pages:</strong> A solid contender, but I’ve always found it a bit limiting in terms of deep customization and headers.</li></ul><p><strong>The Winner: Cloudflare Pages</strong></p><p>I ultimately went with <strong>Cloudflare Pages</strong>. It’s incredibly developer-friendly and gives you a level of performance that’s hard to beat for free. Here’s why it won me over:</p><ol><li><strong>Git Integration:</strong> I just push my code to a GitHub repo, and Cloudflare automatically detects the change and redeploys the site in seconds.</li><li><strong>Global CDN:</strong> Your site is cached on Cloudflare’s edge network (200+ cities), meaning it loads instantly no matter where the recruiter is located.</li><li><strong>Unlimited Bandwidth:</strong> Unlike other free tiers that cap your traffic, Cloudflare Pages offers unlimited bandwidth on the free plan.</li><li><strong>Free SSL &amp; Custom Domains:</strong> Getting that “lock” icon and connecting a professional .dev domain is seamless and costs nothing extra.</li></ol><p>All I had to do was connect my repository, and within two minutes, my AI-generated portfolio was live and accessible to the world.</p><h3>The Technical Setup: A 5-Minute Deployment Guide</h3><p>Now that we have the vision and the tools, let’s get it live. Here is the exact workflow I used to go from a local file to a globally distributed website.</p><h4>Step 1: Prepare Your Repository (The Foundation)</h4><p>Even for a single HTML file, version control is non-negotiable. It allows you to track changes and enables the automated deployment pipeline.</p><ol><li><strong>Create a GitHub Repository:</strong> Give it a clear name, like my-portfolio.</li><li><strong>Add Your File:</strong> Ensure your main file is named index.html and place it in the root directory.</li></ol><h4>Step 2: Deploy to Cloudflare Pages</h4><p>This is where the magic happens. Cloudflare handles the hosting, SSL, and CDN distribution automatically.</p><ol><li><strong>Cloudflare Dashboard:</strong> Log in, navigate to <strong>Workers &amp; Pages</strong>, and click <strong>Create application</strong>.</li><li><strong>Connect to Git:</strong> Select <strong>Pages</strong>, then <strong>Connect to Git</strong>. Authorize your GitHub account and select your my-portfolio repository.</li><li><strong>Configure Deployment:</strong></li></ol><ul><li><strong>Project Name:</strong> This becomes your subdomain (e.g., my-dev-portfolio.pages.dev).</li><li><strong>Production Branch:</strong> Set this to main.</li><li><strong>Framework Preset:</strong> Select <strong>None</strong> (since we are using plain HTML).</li></ul><p><strong>Deploy!</strong> Click <strong>Save and Deploy</strong>. Within seconds, your site will be live.</p><h3>Advanced Strategy: The _headers File</h3><p>As a backend developer, I couldn’t just leave the defaults. I wanted to ensure the site was secure and optimized. By creating a file named _headers in your root directory, you can control how Cloudflare handles your traffic.</p><p>Here is the configuration I used:</p><pre>/*<br>  # Security Headers - Essential for modern web<br>  X-Frame-Options: DENY<br>  X-Content-Type-Options: nosniff<br>  Referrer-Policy: strict-origin-when-cross-origin<br>  Strict-Transport-Security: max-age=31536000; includeSubDomains; preload<br><br>  # Caching Strategy - Browser vs. Edge<br>  # Tell browsers to revalidate frequently<br>  Cache-Control: public, max-age=60, must-revalidate<br><br>  # Tell Cloudflare&#39;s Edge to cache for much longer (24 hours)<br>  Cloudflare-CDN-Cache-Control: max-age=86400</pre><h4>Why the Cache-Control split?</h4><p>This setup offers the best of both worlds:</p><ul><li><strong>Browsers</strong> revalidate quickly, so if you fix a typo, your visitors see it almost immediately.</li><li><strong>Cloudflare’s Edge</strong> keeps the file for 24 hours, ensuring the site loads instantly from the nearest global node.</li></ul><p>The best part? Cloudflare Pages automatically purges the CDN cache every time you push a new deployment, so your “long” cache never serves outdated content after an update.</p><h3>Tracking Recruiter Visits (Smart Analytics)</h3><p>Building the portfolio is only half the battle. As a developer, you want data: <em>Is anyone actually looking at it? Which job application is getting the most traction?</em></p><p>Forget complex setups like Google Analytics. For a simple portfolio, you want something lightweight and privacy focused.</p><h4>1. Cloudflare Web Analytics (The Easiest Way)</h4><p>Cloudflare offers a privacy-friendly, cookie-less analytics tool that is completely free.</p><ul><li><strong>How to enable it:</strong> Go to your Cloudflare Pages project <strong>Settings &gt; Analytics</strong> and toggle <strong>“Enable Web Analytics.”</strong></li><li><strong>The Magic:</strong> Cloudflare injects the script automatically. You don’t have to touch your code.</li><li><strong>The Benefit:</strong> You can use the “Referrers” section to see if traffic is coming from LinkedIn, GitHub, or direct clicks.</li></ul><h4>2. UTM Parameters: Your Secret Weapon</h4><p>This is how you turn a simple link into a high-signal tracking tool. When you apply for a job, don’t just send your base URL. Instead, append <strong>UTM (Urchin Tracking Module)</strong> parameters to the end of your link.</p><p>By using unique links for different companies, you can see exactly which recruiter clicked your site:</p><ul><li><strong>For your LinkedIn Profile:</strong> yourname.pages.dev?utm_source=linkedin&amp;utm_medium=profile</li><li><strong>For an Apple Application:</strong> yourname.pages.dev?utm_source=apple&amp;utm_medium=job_app</li><li><strong>For a Google Application:</strong> yourname.pages.dev?utm_source=google&amp;utm_medium=job_app</li></ul><h3>Wrapping Up</h3><p>Don’t let the fear of a &lt;div&gt; tag or a CSS flexbox stop you. Use the &quot;clever route&quot;—delegate the design, automate the deployment, and spend your time building the backend logic you actually love.</p><p><strong>Now it’s your turn: What’s stopping you from launching your portfolio today? If you build one using this method, drop a link in the comments — I’d love to see what you’ve created!</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c1fc13d80195" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From Zero to Distributed: How I Built a Replicated Key-Value Store in One Weekend - 1]]></title>
            <link>https://medium.com/@piyushdubey/from-zero-to-distributed-how-i-built-a-replicated-key-value-store-in-one-weekend-1-572ac534ce72?source=rss-7d2edfe33219------2</link>
            <guid isPermaLink="false">https://medium.com/p/572ac534ce72</guid>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[computer-science]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[database]]></category>
            <category><![CDATA[data-replication]]></category>
            <dc:creator><![CDATA[Piyush Dubey]]></dc:creator>
            <pubDate>Tue, 25 Nov 2025 09:27:41 GMT</pubDate>
            <atom:updated>2025-11-25T09:27:41.445Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2QBcPmtTLj4u3rKJ9oaX6g.png" /></figure><p>Many of us read papers on distributed systems like Kafka, Redis, and others. We often wonder what these systems <em>really</em> look like in practice. What happens when you dive deep and experience the challenges of building and running a distributed system?</p><p>Well, one weekend, lost in my thoughts, I decided to take a different approach. Instead of just reading about these systems, I thought, “Why not create one from scratch and experience firsthand issues like leader election, replication, and fault tolerance?”</p><p>The idea was set. But there was one problem: I didn’t want to invest too much time setting up EC2 instances or an AWS account. I needed something closer to me, something on my local system. So, what should I do?</p><p><strong>Docker.</strong> Yes, Docker is my friend here.</p><h3>Why Docker?</h3><p>At this point, Docker became the key. With it, I could create containers that mimic actual EC2 instances, providing me with the perfect local environment for running my distributed system experiments. Docker would let me spin up these containers quickly without having to worry about cloud configurations and costs. It was lightweight, efficient, and easily configurable.</p><h3>What Now?</h3><p>Now that I had my containers set up, the next question was: <em>What am I going to do with them?</em></p><p>The goal was simple — to understand how a distributed system evolves and to experience the challenges that come with it. I knew I had to start small, so I created a list of things I wanted to achieve with my Docker instances:</p><ul><li><strong>Level 1: The Chatty Cluster (Discovery)</strong></li><li><strong>Level 2: Who’s in Charge? (Leader Election)</strong></li><li><strong>Level 3: “Not My Job” (Request Replication)</strong></li><li><strong>Level 4: The Quorum (Strong Consistency)</strong></li><li><strong>Level 5: Amnesia (Durability &amp; WAL)</strong></li></ul><h3>The Setup Stack: The “Shared-Nothing” Rig</h3><p>For this distributed system playground, I’ve chosen the following stack:</p><ul><li><strong>Java 17:</strong> For robust concurrency management, using CompletableFuture.</li><li><strong>Javalin:</strong> A lightweight micro-framework (Spring Boot was too heavy for just 3 nodes on one PC).</li><li><strong>Docker Compose:</strong> To simulate the network and set up distinct “servers” (nodes).</li><li><strong>Maven:</strong> To build the “Fat Jar.”</li></ul><h4>1. Install Docker Desktop &amp; Docker Compose</h4><p>Docker containers will serve as your individual &quot;nodes&quot; (or servers). Docker Compose makes it easy to simulate a cluster of these nodes with a single command.</p><ul><li><strong>Install Docker</strong> on your machine (Docker Desktop will include both Docker and Docker Compose).</li></ul><h4>2. Define Your Cluster</h4><p>The next step is to define your cluster configuration. You’ll use a docker-compose.yml file to simulate the network of nodes. Here’s a template for a 3-node cluster setup:</p><pre>services:<br>  node-1:<br>    build: ./node-app<br>    hostname: node-1<br>    environment:<br>      - NODE_ID=1<br>      - PEERS=node-2,node-3<br>    networks:<br>      - playground-net<br>    ports:<br>      - &quot;7001:7000&quot; # Map host port 7001 to container 7000<br><br>  node-2:<br>    build: ./node-app<br>    hostname: node-2<br>    environment:<br>      - NODE_ID=2<br>      - PEERS=node-1,node-3<br>    networks:<br>      - playground-net<br>    ports:<br>      - &quot;7002:7000&quot;<br><br>  node-3:<br>    build: ./node-app<br>    hostname: node-3<br>    environment:<br>      - NODE_ID=3<br>      - PEERS=node-1,node-2<br>    networks:<br>      - playground-net<br>    ports:<br>      - &quot;7003:7000&quot;<br><br>networks:<br>  playground-net:<br>    driver: bridge</pre><p><strong>Explanation:</strong></p><ul><li>Each node-x is a separate container that runs your application.</li><li>The PEERS environment variable defines the list of other nodes that each node can communicate with.</li><li>All nodes are connected via the cluster-net network, allowing communication across containers.</li></ul><h4>3. Create the Dockerfile</h4><p>Java containers can be large. Use a slim image.</p><pre>FROM maven:3.8-openjdk-17 AS builder<br>WORKDIR /app<br>COPY pom.xml .<br>COPY src ./src<br>RUN mvn clean package -DskipTests<br><br># Stage 2: Create the final lightweight image<br>FROM eclipse-temurin:25-jre-alpine<br>WORKDIR /app<br>COPY --from=builder /app/target/*-jar-with-dependencies.jar app.jar<br>EXPOSE 7000<br>ENTRYPOINT [&quot;java&quot;,&quot;-jar&quot;,&quot;app.jar&quot;]</pre><h4>4. File Structure</h4><p>To keep things organized, your project should have the following file structure:</p><pre>/my-distributed-playground<br>  ├── docker-compose.yml<br>  ├── /node-app (your code)<br>      ├── Dockerfile</pre><ul><li>The docker-compose.yml defines the structure of your containers and network.</li><li>The /node-app folder holds the application code for each node in the cluster.</li></ul><h3><strong>Level 1: The Chatty Cluster (Discovery)</strong></h3><p>Your first goal is simply to get the nodes to recognize each other.</p><p><strong>The Architecture:</strong></p><ul><li><strong>Node 1</strong> starts up.</li><li>It reads PEERS=node-2,node-3 from the environment.</li><li>It enters a while(true) loop sending a POST /ping to those hostnames.</li><li><strong>Node 2</strong> receives the ping and logs: <em>“Received ping from Node 1”</em>.</li></ul><p>Create a class in node-app called Node.java with following file structure.</p><pre>/distributed-playground<br>  ├── docker-compose.yml<br>  ├── /node-app<br>      ├── Dockerfile<br>      ├── pom.xml<br>      └── src<br>          └── main<br>              └── java<br>                  └── com<br>                      └── playground<br>                          └── Node.java</pre><p><em>Pom.xml</em></p><pre>&lt;project xmlns=&quot;http://maven.apache.org/POM/4.0.0&quot;<br>    xmlns:xsi=&quot;http://www.w3.org/2001/XMLSchema-instance&quot;<br>    xsi:schemaLocation=&quot;http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd&quot;&gt;<br>    &lt;modelVersion&gt;4.0.0&lt;/modelVersion&gt;<br><br>    &lt;groupId&gt;com.playground&lt;/groupId&gt;<br>    &lt;artifactId&gt;distributed-node&lt;/artifactId&gt;<br>    &lt;version&gt;1.0&lt;/version&gt;<br><br>    &lt;properties&gt;<br>        &lt;maven.compiler.source&gt;17&lt;/maven.compiler.source&gt;<br>        &lt;maven.compiler.target&gt;17&lt;/maven.compiler.target&gt;<br>    &lt;/properties&gt;<br><br>    &lt;dependencies&gt;<br>        &lt;dependency&gt;<br>            &lt;groupId&gt;io.javalin&lt;/groupId&gt;<br>            &lt;artifactId&gt;javalin&lt;/artifactId&gt;<br>            &lt;version&gt;6.1.3&lt;/version&gt;<br>        &lt;/dependency&gt;<br>        &lt;dependency&gt;<br>            &lt;groupId&gt;org.jetbrains.kotlin&lt;/groupId&gt;<br>            &lt;artifactId&gt;kotlin-stdlib&lt;/artifactId&gt;<br>            &lt;version&gt;1.9.22&lt;/version&gt;<br>        &lt;/dependency&gt;<br>        &lt;dependency&gt;<br>            &lt;groupId&gt;org.slf4j&lt;/groupId&gt;<br>            &lt;artifactId&gt;slf4j-simple&lt;/artifactId&gt;<br>            &lt;version&gt;2.0.7&lt;/version&gt;<br>        &lt;/dependency&gt;<br>        &lt;dependency&gt;<br>            &lt;groupId&gt;com.squareup.okhttp3&lt;/groupId&gt;<br>            &lt;artifactId&gt;okhttp&lt;/artifactId&gt;<br>            &lt;version&gt;4.10.0&lt;/version&gt;<br>        &lt;/dependency&gt;<br>    &lt;/dependencies&gt;<br><br>    &lt;build&gt;<br>        &lt;plugins&gt;<br>            &lt;plugin&gt;<br>                &lt;groupId&gt;org.apache.maven.plugins&lt;/groupId&gt;<br>                &lt;artifactId&gt;maven-compiler-plugin&lt;/artifactId&gt;<br>                &lt;version&gt;3.11.0&lt;/version&gt;<br>                &lt;configuration&gt;<br>                    &lt;source&gt;17&lt;/source&gt;<br>                    &lt;target&gt;17&lt;/target&gt;<br>                &lt;/configuration&gt;<br>            &lt;/plugin&gt;<br><br>            &lt;plugin&gt;<br>                &lt;groupId&gt;org.apache.maven.plugins&lt;/groupId&gt;<br>                &lt;artifactId&gt;maven-assembly-plugin&lt;/artifactId&gt;<br>                &lt;version&gt;3.6.0&lt;/version&gt;<br>                &lt;configuration&gt;<br>                    &lt;descriptorRefs&gt;<br>                        &lt;descriptorRef&gt;jar-with-dependencies&lt;/descriptorRef&gt;<br>                    &lt;/descriptorRefs&gt;<br>                    &lt;archive&gt;<br>                        &lt;manifest&gt;<br>                            &lt;mainClass&gt;com.playground.Node&lt;/mainClass&gt;<br>                        &lt;/manifest&gt;<br>                    &lt;/archive&gt;<br>                &lt;/configuration&gt;<br>                &lt;executions&gt;<br>                    &lt;execution&gt;<br>                        &lt;id&gt;make-assembly&lt;/id&gt;<br>                        &lt;phase&gt;package&lt;/phase&gt;<br>                        &lt;goals&gt;<br>                            &lt;goal&gt;single&lt;/goal&gt;<br>                        &lt;/goals&gt;<br>                    &lt;/execution&gt;<br>                &lt;/executions&gt;<br>            &lt;/plugin&gt;<br>        &lt;/plugins&gt;<br>    &lt;/build&gt;<br>&lt;/project&gt;</pre><p><em>Node.java</em></p><p>In /node-app/src/main/java/com/playground/Node.java:</p><p>This code starts a server and a background thread that pings every peer defined in the environment variables.</p><pre>package com.playground;<br><br>import io.javalin.Javalin;<br>import okhttp3.*;<br>import java.io.IOException;<br>import java.util.Arrays;<br>import java.util.List;<br>import java.util.concurrent.TimeUnit;<br><br>public class Node {<br>    // Shared HTTP Client<br>    private static final OkHttpClient client = new OkHttpClient.Builder()<br>            .readTimeout(2, TimeUnit.SECONDS) // Fail fast if node is down<br>            .build();<br><br>    public static void main(String[] args) {<br>        // 1. Identity Configuration<br>        String myId = System.getenv().getOrDefault(&quot;NODE_ID&quot;, &quot;Unknown&quot;);<br>        String peersEnv = System.getenv().getOrDefault(&quot;PEERS&quot;, &quot;&quot;);<br>        List&lt;String&gt; peers = peersEnv.isEmpty() ? List.of() : Arrays.asList(peersEnv.split(&quot;,&quot;));<br>        int port = 7000;<br><br>        System.out.println(&quot;--- STARTING NODE &quot; + myId + &quot; ---&quot;);<br>        System.out.println(&quot;--- PEERS: &quot; + peers + &quot; ---&quot;);<br><br>        // 2. Start Server (Listener)<br>        Javalin app = Javalin.create().start(port);<br><br>        // Endpoint: Other nodes hit this to check if I am alive<br>        app.get(&quot;/ping&quot;, ctx -&gt; {<br>            ctx.result(&quot;Pong from &quot; + myId);<br>        });<br><br>        // 3. Start Background Broadcaster (Sender)<br>        // This thread runs forever, pinging peers every 2 seconds<br>        new Thread(() -&gt; {<br>            while (true) {<br>                try {<br>                    Thread.sleep(2000); <br>                    for (String peer : peers) {<br>                        pingPeer(peer, myId);<br>                    }<br>                } catch (InterruptedException e) {<br>                    e.printStackTrace();<br>                }<br>            }<br>        }).start();<br>    }<br><br>    private static void pingPeer(String peerHostname, String myId) {<br>        String url = &quot;http://&quot; + peerHostname + &quot;:7000/ping&quot;;<br>        Request request = new Request.Builder()./service/https://medium.com/url(url).build();<br><br>        try (Response response = client.newCall(request).execute()) {<br>            if (response.isSuccessful()) {<br>                System.out.println(&quot;[&quot; + myId + &quot;] -&gt; &quot; + peerHostname + &quot;: SUCCESS&quot;);<br>            } else {<br>                System.out.println(&quot;[&quot; + myId + &quot;] -&gt; &quot; + peerHostname + &quot;: FAILED (Status &quot; + response.code() + &quot;)&quot;);<br>            }<br>        } catch (IOException e) {<br>            System.err.println(&quot;[&quot; + myId + &quot;] -&gt; &quot; + peerHostname + &quot;: UNREACHABLE (Network Error)&quot;);<br>        }<br>    }<br>}</pre><h4>How to run</h4><ul><li><strong>Open your terminal</strong> in the distributed-playground folder.</li><li><strong>Build and start the containers</strong> using Docker Compose by running the following command:</li></ul><pre>docker-compose up --build</pre><h4><strong>What You Will See</strong></h4><ul><li>In your terminal, you will see a stream of logs from all 3 nodes.</li><li><strong>Node 1</strong> will print:</li><li>node-1 will print: [1] -&gt; node-2: SUCCESS</li><li>node-2 will print: [2] -&gt; node-3: SUCCESS</li></ul><h4>Testing Failure</h4><p>To simulate a failure, open a <strong>second terminal window</strong> and stop <strong>Node 2</strong> by running:</p><pre>docker stop distributed-playground-node-2-1</pre><p>Now, go back to the first terminal window. You will see that <strong>Node 1</strong> and <strong>Node 3</strong> will immediately start printing:</p><p>[1] -&gt; node-2: UNREACHABLE</p><p>This simulates a node failure, allowing you to observe how the system reacts when a node becomes unreachable.</p><p><strong>The Learning: </strong>Network partitions are real.</p><h3>Phase 2: Who is the Boss? (Leader Election)</h3><p>We will implement the <strong>Bully Algorithm</strong>.</p><ul><li><strong>The Rule:</strong> The active node with the <strong>Highest Node ID</strong> is the Leader.</li><li><strong>The Behavior:</strong></li><li>Node 1 sees Node 2 and 3 are alive. Max(1, 2, 3) = 3. Node 1 thinks: <em>&quot;Node 3 is Boss.&quot;</em></li><li>If Node 3 crashes, Node 1 sees only Node 2. Max(1, 2) = 2. Node 1 thinks: <em>&quot;Node 2 is Boss.&quot;</em></li></ul><h4>The Code Change</h4><p>You need to modify Node.java to track <strong>state</strong>. Instead of just printing &quot;Success,&quot; we will record the timestamp of the last successful ping.</p><p>Replace your entire src/main/java/com/playground/Node.java with this upgraded version:</p><pre>package com.playground;<br><br>import io.javalin.Javalin;<br>import okhttp3.*;<br>import java.io.IOException;<br>import java.util.*;<br>import java.util.concurrent.ConcurrentHashMap;<br>import java.util.concurrent.TimeUnit;<br>import java.util.stream.Collectors;<br><br>public class Node {<br>    // 1. STATE: Keep track of when we last saw each peer<br>    private static final Map&lt;Integer, Long&gt; peerLastSeen = new ConcurrentHashMap&lt;&gt;();<br>    <br>    // Configuration<br>    private static final OkHttpClient client = new OkHttpClient.Builder().readTimeout(1, TimeUnit.SECONDS).build();<br>    private static int MY_ID;<br>    private static List&lt;String&gt; PEER_HOSTNAMES;<br><br>    public static void main(String[] args) {<br>        // Setup ID and Peers<br>        MY_ID = Integer.parseInt(System.getenv().getOrDefault(&quot;NODE_ID&quot;, &quot;1&quot;));<br>        String peersEnv = System.getenv().getOrDefault(&quot;PEERS&quot;, &quot;&quot;);<br>        PEER_HOSTNAMES = peersEnv.isEmpty() ? List.of() : Arrays.asList(peersEnv.split(&quot;,&quot;));<br><br>        System.out.println(&quot;--- NODE &quot; + MY_ID + &quot; STARTED ---&quot;);<br><br>        // Start Server (So others can ping me)<br>        Javalin app = Javalin.create().start(7000);<br>        app.get(&quot;/ping&quot;, ctx -&gt; ctx.result(&quot;Pong from &quot; + MY_ID));<br><br>        // Start Background Tasks<br>        new Thread(Node::runFailureDetector).start();<br>        new Thread(Node::runLeaderElection).start();<br>    }<br><br>    // TASK 1: Ping everyone to see who is alive<br>    private static void runFailureDetector() {<br>        while (true) {<br>            for (String peerHost : PEER_HOSTNAMES) {<br>                // peerHost looks like &quot;node-2&quot;. We assume ID is the last char.<br>                // In real apps, we&#39;d have a better discovery system.<br>                int peerId = Integer.parseInt(peerHost.split(&quot;-&quot;)[1]);<br>                <br>                if (pingPeer(peerHost)) {<br>                    // Update timestamp if ping success<br>                    peerLastSeen.put(peerId, System.currentTimeMillis());<br>                }<br>            }<br>            try { Thread.sleep(1000); } catch (InterruptedException e) {}<br>        }<br>    }<br><br>    // TASK 2: Decide who is the leader based on who is alive<br>    private static void runLeaderElection() {<br>        while (true) {<br>            // 1. Filter out dead nodes (haven&#39;t seen in 5 seconds)<br>            long now = System.currentTimeMillis();<br>            List&lt;Integer&gt; aliveNodes = new ArrayList&lt;&gt;();<br>            aliveNodes.add(MY_ID); // I am always alive to myself<br><br>            peerLastSeen.forEach((id, lastSeen) -&gt; {<br>                if (now - lastSeen &lt; 5000) { // 5 second timeout<br>                    aliveNodes.add(id);<br>                }<br>            });<br><br>            // 2. The Bully Algorithm: Max ID wins<br>            int leaderId = Collections.max(aliveNodes);<br>            <br>            String status = (leaderId == MY_ID) ? &quot;I AM THE LEADER&quot; : &quot;Leader is Node &quot; + leaderId;<br>            System.out.println(&quot;--- [Status] Alive: &quot; + aliveNodes + &quot; | &quot; + status + &quot; ---&quot;);<br><br>            try { Thread.sleep(2000); } catch (InterruptedException e) {}<br>        }<br>    }<br><br>    private static boolean pingPeer(String host) {<br>        Request request = new Request.Builder().url(https://medium.com/&quot;http://&quot;%20+%20host%20+%20&quot;:7000/ping&quot;).build();<br>        try (Response response = client.newCall(request).execute()) {<br>            return response.isSuccessful();<br>        } catch (Exception e) {<br>            return false;<br>        }<br>    }<br>}</pre><h4>How to Run &amp; Test This</h4><ul><li><strong>Rebuild:</strong> Since you changed the Java code, you must rebuild the jar and image.</li></ul><pre>docker-compose up --build</pre><ul><li><strong>Observe the Logs:</strong> Wait about 5–10 seconds. You will see the cluster “stabilize.”</li><li><strong>Node 1 logs:</strong> Alive: [1, 2, 3] | Leader is Node 3</li><li><strong>Node 2 logs:</strong> Alive: [2, 1, 3] | Leader is Node 3</li><li><strong>Node 3 logs:</strong> Alive: [3, 1, 2] | I AM THE LEADER</li><li><strong>Create Chaos (The Test):</strong> Open a new terminal and assassinate the leader.</li></ul><pre>docker stop distributed-playground-node-3-1</pre><ul><li><strong>Watch the Coup:</strong> Look at the logs for Node 1 and Node 2.</li><li>For ~5 seconds, they might still think Node 3 is alive (because of our 5000ms timeout).</li><li>Suddenly, the logs will change:</li><li><strong>Node 2 logs:</strong> Alive: [2, 1] | I AM THE LEADER <em>(Node 2 realizes 3 is gone, and </em><em>Max(1, 2) = 2, so it takes over)</em>.</li><li><strong>Node 1 logs:</strong> Alive: [1, 2] | Leader is Node 2</li></ul><p><strong>The Learning: </strong>You built a <strong>Self-Healing Distributed System</strong>.</p><ul><li><strong>Failure Detection:</strong> The logic that checks now - lastSeen &lt; 5000.</li><li><strong>Consensus:</strong> The logic where independent nodes agree on the same truth (who the leader is) without a human telling them.</li></ul><h3><strong>Level 3: “Not My Job” (Request Replication)</strong></h3><p>We are going to turn nodes into a functional <strong>Key-Value Store</strong>.We will implement two things:</p><ol><li><strong>Request Routing:</strong> If you send data to a Follower, it says “I’m not the boss” and redirects you to the Leader.</li><li><strong>Basic Replication:</strong> The Leader saves the data and immediately tells all Followers to save it too.</li></ol><h4>The Architecture Change</h4><ol><li><strong>Client</strong> sends POST /data to <strong>Node 2</strong> (Follower).</li><li><strong>Node 2</strong> replies: HTTP 307 Redirect -&gt; Node 3 (Leader).</li><li><strong>Client</strong> sends POST /data to <strong>Node 3</strong>.</li><li><strong>Node 3</strong> saves to HashMap.</li><li><strong>Node 3</strong> loops through peers and sends POST /internal/replicate to Node 1 and Node 2.</li></ol><h4>The Code Update (Node.java)</h4><p>Replace your entire Node.java with this code. I have added the <strong>Data Store</strong>, the <strong>Redirect Logic</strong>, and the <strong>Replication Loop</strong>.</p><pre>package com.playground;<br><br>import io.javalin.Javalin;<br>import io.javalin.http.Context;<br>import okhttp3.*;<br>import java.io.IOException;<br>import java.util.*;<br>import java.util.concurrent.ConcurrentHashMap;<br>import java.util.concurrent.TimeUnit;<br><br>public class Node {<br>    // 1. THE DATABASE (In-Memory Key-Value Store)<br>    private static final Map&lt;String, String&gt; kvStore = new ConcurrentHashMap&lt;&gt;();<br><br>    // 2. STATE<br>    private static final Map&lt;Integer, Long&gt; peerLastSeen = new ConcurrentHashMap&lt;&gt;();<br>    private static volatile int currentLeaderId = -1; // Who is the boss?<br>    <br>    // Configuration<br>    private static final OkHttpClient client = new OkHttpClient.Builder().readTimeout(1, TimeUnit.SECONDS).build();<br>    private static int MY_ID;<br>    private static List&lt;String&gt; PEER_HOSTNAMES;<br><br>    public static void main(String[] args) {<br>        MY_ID = Integer.parseInt(System.getenv().getOrDefault(&quot;NODE_ID&quot;, &quot;1&quot;));<br>        String peersEnv = System.getenv().getOrDefault(&quot;PEERS&quot;, &quot;&quot;);<br>        PEER_HOSTNAMES = peersEnv.isEmpty() ? List.of() : Arrays.asList(peersEnv.split(&quot;,&quot;));<br><br>        System.out.println(&quot;--- NODE &quot; + MY_ID + &quot; STARTED ---&quot;);<br><br>        Javalin app = Javalin.create().start(7000);<br><br>        // --- ENDPOINTS ---<br><br>        // 1. Health Check<br>        app.get(&quot;/ping&quot;, ctx -&gt; ctx.result(&quot;Pong from &quot; + MY_ID));<br><br>        // 2. Public Write Endpoint (The Entry Point)<br>        app.post(&quot;/data/{key}/{value}&quot;, ctx -&gt; handleWriteRequest(ctx));<br><br>        // 3. Public Read Endpoint<br>        app.get(&quot;/data/{key}&quot;, ctx -&gt; {<br>            String value = kvStore.get(ctx.pathParam(&quot;key&quot;));<br>            if (value == null) ctx.status(404).result(&quot;Key not found on Node &quot; + MY_ID);<br>            else ctx.result(value);<br>        });<br><br>        // 4. Internal Replication Endpoint (Only for Leader -&gt; Follower)<br>        app.post(&quot;/internal/replicate/{key}/{value}&quot;, ctx -&gt; {<br>            String key = ctx.pathParam(&quot;key&quot;);<br>            String value = ctx.pathParam(&quot;value&quot;);<br>            kvStore.put(key, value);<br>            System.out.println(&quot;--- [Replication] Leader told me to save &quot; + key + &quot;=&quot; + value + &quot; ---&quot;);<br>            ctx.status(200);<br>        });<br><br>        // --- BACKGROUND TASKS ---<br>        new Thread(Node::runFailureDetector).start();<br>        new Thread(Node::runLeaderElection).start();<br>    }<br><br>    // --- LOGIC HANDLERS ---<br><br>    private static void handleWriteRequest(Context ctx) {<br>        String key = ctx.pathParam(&quot;key&quot;);<br>        String value = ctx.pathParam(&quot;value&quot;);<br><br>        // CASE A: I am not the leader -&gt; Redirect<br>        if (currentLeaderId != MY_ID) {<br>            if (currentLeaderId == -1) {<br>                ctx.status(503).result(&quot;No leader available&quot;);<br>                return;<br>            }<br>            // Redirect client to the leader using host-mapped port (host: 7000 + leaderId)<br>            int leaderHostPort = 7000 + currentLeaderId; // host ports: 7001,7002,7003<br>            String leaderUrl = &quot;http://localhost:&quot; + leaderHostPort + &quot;/data/&quot; + key + &quot;/&quot; + value;<br>            ctx.status(307).redirect(leaderUrl);<br>            System.out.println(&quot;--- [Redirect] Redirecting client to Leader (Node &quot; + currentLeaderId + &quot;) at &quot; + leaderUrl + &quot; ---&quot;);<br>            return;<br>        }<br><br>        // CASE B: I am the leader -&gt; Save and Replicate<br>        System.out.println(&quot;--- [Write] I am Leader. Saving &quot; + key + &quot;=&quot; + value + &quot; ---&quot;);<br>        <br>        // 1. Save Locally<br>        kvStore.put(key, value);<br><br>        // 2. Replicate to others (Fire and Forget for now)<br>        for (String peer : PEER_HOSTNAMES) {<br>            replicateToPeer(peer, key, value);<br>        }<br><br>        ctx.result(&quot;Write Successful on Leader &quot; + MY_ID);<br>    }<br><br>    private static void replicateToPeer(String peerHost, String key, String value) {<br>        // Send to /internal/replicate endpoint<br>        String url = &quot;http://&quot; + peerHost + &quot;:7000/internal/replicate/&quot; + key + &quot;/&quot; + value;<br>        Request request = new Request.Builder()./service/https://medium.com/url(url).post(RequestBody.create(new byte[0])).build();<br>        <br>        // Asynchronous call (don&#39;t wait for response to keep it simple)<br>        client.newCall(request).enqueue(new Callback() {<br>            @Override public void onFailure(Call call, IOException e) {<br>                System.out.println(&quot;Failed to replicate to &quot; + peerHost);<br>            }<br>            @Override public void onResponse(Call call, Response response) {<br>                response.close();<br>            }<br>        });<br>    }<br><br>    // --- BACKGROUND THREADS (Same as before) ---<br><br>    private static void runFailureDetector() {<br>        while (true) {<br>            for (String peerHost : PEER_HOSTNAMES) {<br>                int peerId = Integer.parseInt(peerHost.split(&quot;-&quot;)[1]);<br>                if (pingPeer(peerHost)) peerLastSeen.put(peerId, System.currentTimeMillis());<br>            }<br>            try { Thread.sleep(1000); } catch (InterruptedException e) {}<br>        }<br>    }<br><br>    private static void runLeaderElection() {<br>        while (true) {<br>            long now = System.currentTimeMillis();<br>            List&lt;Integer&gt; aliveNodes = new ArrayList&lt;&gt;();<br>            aliveNodes.add(MY_ID);<br>            peerLastSeen.forEach((id, lastSeen) -&gt; {<br>                if (now - lastSeen &lt; 5000) aliveNodes.add(id);<br>            });<br><br>            int newLeader = Collections.max(aliveNodes);<br>            if (newLeader != currentLeaderId) {<br>                System.out.println(&quot;--- [Election] New Leader is Node &quot; + newLeader + &quot; ---&quot;);<br>                currentLeaderId = newLeader;<br>            }<br>            try { Thread.sleep(2000); } catch (InterruptedException e) {}<br>        }<br>    }<br><br>    private static boolean pingPeer(String host) {<br>        Request request = new Request.Builder().url(https://medium.com/&quot;http://&quot;%20+%20host%20+%20&quot;:7000/ping&quot;).build();<br>        try (Response response = client.newCall(request).execute()) { return response.isSuccessful(); } <br>        catch (Exception e) { return false; }<br>    }<br>}</pre><h4>How to Test the “Redirect” &amp; “Replication”</h4><ul><li>Rebuild and Run:</li></ul><pre>docker-compose up --build</pre><p>Wait for the logs to say New Leader is Node 3.</p><p><strong>Test 1: The Redirect (Writing to a Follower)</strong> Open a terminal. We will try to write user=john to <strong>Node 1</strong> (who is NOT the leader).</p><pre># -L means &quot;Follow Redirects&quot;, -X POST forces it to be a POST<br>curl -L -X POST http://localhost:7001/data/user/john</pre><p><strong>What happens internally:</strong></p><ol><li>Request hits Node 1 (Port 7001).</li><li>Node 1 sees it is not leader. Replies 307 Redirect to node-3.</li><li>curl automatically follows to Node 3.</li><li>Node 3 saves it and returns “Write Successful”.</li></ol><p><strong>Test 2: Verification (Did it replicate?)</strong> Now, ask <strong>Node 2</strong> (Follower) if it has the data.</p><pre>curl http://localhost:7002/data/user</pre><ul><li><strong>Output:</strong> john</li><li><strong>Success!</strong> You wrote to Node 1, it bounced to Node 3, which replicated to Node 2.</li></ul><p><strong>The Learning:</strong></p><ul><li><strong>Master-Slave Architecture:</strong> Writes go to one node; Reads can go anywhere.</li><li><strong>Asynchronous Replication:</strong> The risk of the leader saying “Success” before followers actually have the data.</li></ul><h4>What is the flaw here?</h4><p>Currently, this is <strong>Asynchronous Replication</strong>. If Node 3 saves “john”, sends the “Success” message to you, but crashes <em>before</em> sending the data to Node 2… <strong>Data is lost.</strong></p><p>The next level (Professional Grade) is <strong>Synchronous Replication (Quorums)</strong>. The Leader must wait for Node 2 to say “I got it” before telling the user “Success”.</p><h3><strong>Level 4: The Quorum (Strong Consistency)</strong></h3><p>This is the jump from “Toy Project” to “Real Distributed System.”</p><p>Currently, your Leader says “Success” immediately after saving to its own memory. This is dangerous. If the Leader crashes 1ms later, the data is lost forever because the followers never got it.</p><p>We will implement <strong>Synchronous Replication with Quorums</strong>.</p><p>This is the jump from “Toy Project” to “Real Distributed System.”</p><p>Currently, your Leader says “Success” immediately after saving to its own memory. This is dangerous. If the Leader crashes 1ms later, the data is lost forever because the followers never got it.</p><p>We will implement <strong>Synchronous Replication with Quorums</strong>.</p><h3>The New Rules (Strong Consistency)</h3><p><strong>The Formula:</strong> You need N/2+1 nodes to agree.</p><ul><li>For 3 nodes: 3/2+1=2 nodes.</li></ul><p><strong>The Flow:</strong></p><ul><li>Leader saves data.</li><li>Leader sends to Node 1 &amp; Node 2.</li><li>Leader <strong>waits.</strong></li><li>Leader counts the “ACKs” (Acknowledgements).</li><li>If ACKs &gt;= 2, tell User: &quot;Success&quot;.</li><li>If ACKs &lt; 2, tell User: &quot;Write Failed (Cluster Unstable)&quot;.</li></ul><h3>The Code Upgrade</h3><p>We need to use Java’s CompletableFuture to send requests in parallel but wait for the results before responding to the user.</p><p>Replace your <strong>entire</strong> Node.java with this version.</p><pre>package com.playground;<br><br>import io.javalin.Javalin;<br>import io.javalin.http.Context;<br>import okhttp3.*;<br>import java.io.IOException;<br>import java.util.*;<br>import java.util.concurrent.CompletableFuture;<br>import java.util.concurrent.ConcurrentHashMap;<br>import java.util.concurrent.TimeUnit;<br><br>public class Node {<br>    // 1. THE DATABASE (In-Memory Key-Value Store)<br>    private static final Map&lt;String, String&gt; kvStore = new ConcurrentHashMap&lt;&gt;();<br><br>    // 2. STATE<br>    private static final Map&lt;Integer, Long&gt; peerLastSeen = new ConcurrentHashMap&lt;&gt;();<br>    private static volatile int currentLeaderId = -1; // Who is the boss?<br>    <br>    // Configuration<br>    private static final OkHttpClient client = new OkHttpClient.Builder().readTimeout(1, TimeUnit.SECONDS).build();<br>    private static int MY_ID;<br>    private static List&lt;String&gt; PEER_HOSTNAMES;<br><br>    public static void main(String[] args) throws IOException {<br>        MY_ID = Integer.parseInt(System.getenv().getOrDefault(&quot;NODE_ID&quot;, &quot;1&quot;));<br>        String peersEnv = System.getenv().getOrDefault(&quot;PEERS&quot;, &quot;&quot;);<br>        PEER_HOSTNAMES = peersEnv.isEmpty() ? List.of() : Arrays.asList(peersEnv.split(&quot;,&quot;));<br><br>        System.out.println(&quot;--- NODE &quot; + MY_ID + &quot; STARTED ---&quot;);<br><br>        Javalin app = Javalin.create().start(7000);<br><br>        // --- ENDPOINTS ---<br><br>        // 1. Health Check<br>        app.get(&quot;/ping&quot;, ctx -&gt; ctx.result(&quot;Pong from &quot; + MY_ID));<br><br>        // 2. Public Write Endpoint (The Entry Point)<br>        app.post(&quot;/data/{key}/{value}&quot;, ctx -&gt; handleStrongWrite(ctx));<br><br>        // 3. Public Read Endpoint<br>        app.get(&quot;/data/{key}&quot;, ctx -&gt; {<br>            String value = kvStore.get(ctx.pathParam(&quot;key&quot;));<br>            if (value == null) ctx.status(404).result(&quot;Key not found on Node &quot; + MY_ID);<br>            else ctx.result(value);<br>        });<br><br>        // 4. Internal Replication Endpoint (Only for Leader -&gt; Follower)<br>        app.post(&quot;/internal/replicate/{key}/{value}&quot;, ctx -&gt; {<br>            String key = ctx.pathParam(&quot;key&quot;);<br>            String value = ctx.pathParam(&quot;value&quot;);<br><br>            kvStore.put(key, value);<br>            System.out.println(&quot;--- [Replication] Leader told me to save &quot; + key + &quot;=&quot; + value + &quot; ---&quot;);<br>            ctx.status(200);<br>        });<br><br>        // --- BACKGROUND TASKS ---<br>        new Thread(Node::runFailureDetector).start();<br>        new Thread(Node::runLeaderElection).start();<br>    }<br><br>    // --- LOGIC HANDLERS ---<br><br>    // private static void handleWriteRequest(Context ctx) {<br>    //     String key = ctx.pathParam(&quot;key&quot;);<br>    //     String value = ctx.pathParam(&quot;value&quot;);<br><br>    //     // CASE A: I am not the leader -&gt; Redirect<br>    //     if (currentLeaderId != MY_ID) {<br>    //         if (currentLeaderId == -1) {<br>    //             ctx.status(503).result(&quot;No leader available&quot;);<br>    //             return;<br>    //         }<br>    //         // Redirect client to the leader using host-mapped port (host: 7000 + leaderId)<br>    //         int leaderHostPort = 7000 + currentLeaderId; // host ports: 7001,7002,7003<br>    //         String leaderUrl = &quot;http://localhost:&quot; + leaderHostPort + &quot;/data/&quot; + key + &quot;/&quot; + value;<br>    //         ctx.status(307).redirect(leaderUrl);<br>    //         System.out.println(&quot;--- [Redirect] Redirecting client to Leader (Node &quot; + currentLeaderId + &quot;) at &quot; + leaderUrl + &quot; ---&quot;);<br>    //         return;<br>    //     }<br><br>    //     // CASE B: I am the leader -&gt; Save and Replicate<br>    //     System.out.println(&quot;--- [Write] I am Leader. Saving &quot; + key + &quot;=&quot; + value + &quot; ---&quot;);<br>        <br>    //     // 1. Save Locally<br>    //     kvStore.put(key, value);<br><br>    //     // 2. Replicate to others (Fire and Forget for now)<br>    //     for (String peer : PEER_HOSTNAMES) {<br>    //         replicateToPeer(peer, key, value);<br>    //     }<br><br>    //     ctx.result(&quot;Write Successful on Leader &quot; + MY_ID);<br>    // }<br><br>    // --- THE HARD PART: QUORUM LOGIC ---<br>    private static void handleStrongWrite(Context ctx) {<br>        String key = ctx.pathParam(&quot;key&quot;);<br>        String value = ctx.pathParam(&quot;value&quot;);<br><br>        // 1. Redirect if not leader<br>        if (currentLeaderId != MY_ID) {<br>            if (currentLeaderId == -1) { ctx.status(503).result(&quot;No Leader&quot;); return; }<br>            int leaderHostPort = 7000 + currentLeaderId; // host ports: 7001,7002,7003<br>            String leaderUrl = &quot;http://localhost:&quot; + leaderHostPort + &quot;/data/&quot; + key + &quot;/&quot; + value;<br>            ctx.status(307).redirect(leaderUrl);<br>            return;<br>        }<br><br>        // 2. I am Leader: Start the Clock<br>        System.out.println(&quot;--- [Write] Processing Quorum Write for &quot; + key + &quot; ---&quot;);<br><br>        kvStore.put(key, value);<br>        <br>        // Step A: Save locally (1 vote)<br>        kvStore.put(key, value);<br>        <br>        // Step B: Send to all peers in PARALLEL and wait for results<br>        List&lt;CompletableFuture&lt;Boolean&gt;&gt; futures = new ArrayList&lt;&gt;();<br>        <br>        for (String peer : PEER_HOSTNAMES) {<br>            futures.add(CompletableFuture.supplyAsync(() -&gt; sendReplicationSync(peer, key, value)));<br>        }<br><br>        // Step C: Wait for all attempts to finish (or timeout)<br>        try {<br>            CompletableFuture.allOf(futures.toArray(new CompletableFuture[0])).join();<br>        } catch (Exception e) {<br>            // Ignore exceptions, we just want to count successes<br>        }<br><br>        // Step D: Count Votes<br>        long successCount = 1; // Start with 1 (Myself)<br>        for (CompletableFuture&lt;Boolean&gt; f : futures) {<br>            try {<br>                if (f.getNow(false)) successCount++;<br>            } catch (Exception e) {}<br>        }<br><br>        // Step E: Quorum Decision<br>        int quorum = (PEER_HOSTNAMES.size() + 1) / 2 + 1; // (2+1)/2 + 1 = 2<br>        <br>        if (successCount &gt;= quorum) {<br>            System.out.println(&quot;--- [Commit] Quorum reached (&quot; + successCount + &quot;/&quot; + (PEER_HOSTNAMES.size()+1) + &quot;) ---&quot;);<br>            ctx.result(&quot;Write Successful (Replicated to &quot; + successCount + &quot; nodes)&quot;);<br>        } else {<br>            System.out.println(&quot;--- [Fail] Quorum failed (&quot; + successCount + &quot; votes). Data is unsafe! ---&quot;);<br>            ctx.status(500).result(&quot;Write Failed: Cluster unstable (Only &quot; + successCount + &quot; nodes acknowledged)&quot;);<br>            // Note: In a real DB, we would rollback here!<br>        }<br>    }<br><br>    // private static void replicateToPeer(String peerHost, String key, String value) {<br>    //     // Send to /internal/replicate endpoint<br>    //     String url = &quot;http://&quot; + peerHost + &quot;:7000/internal/replicate/&quot; + key + &quot;/&quot; + value;<br>    //     Request request = new Request.Builder()./service/https://medium.com/url(url).post(RequestBody.create(new byte[0])).build();<br>        <br>    //     // Asynchronous call (don&#39;t wait for response to keep it simple)<br>    //     client.newCall(request).enqueue(new Callback() {<br>    //         @Override public void onFailure(Call call, IOException e) {<br>    //             System.out.println(&quot;Failed to replicate to &quot; + peerHost);<br>    //         }<br>    //         @Override public void onResponse(Call call, Response response) {<br>    //             response.close();<br>    //         }<br>    //     });<br>    // }<br><br>    // Synchronous replication helper used by quorum writes (returns true on success)<br>    private static boolean sendReplicationSync(String peerHost, String key, String value) {<br>        String url = &quot;http://&quot; + peerHost + &quot;:7000/internal/replicate/&quot; + key + &quot;/&quot; + value;<br>        Request request = new Request.Builder()./service/https://medium.com/url(url).post(RequestBody.create(new byte[0])).build();<br>        try (Response response = client.newCall(request).execute()) {<br>            return response.isSuccessful();<br>        } catch (Exception e) {<br>            System.out.println(&quot;Failed to replicate (sync) to &quot; + peerHost + &quot;: &quot; + e.getMessage());<br>            return false;<br>        }<br>    }<br><br>    // --- BACKGROUND THREADS (Same as before) ---<br><br>    private static void runFailureDetector() {<br>        while (true) {<br>            for (String peerHost : PEER_HOSTNAMES) {<br>                int peerId = Integer.parseInt(peerHost.split(&quot;-&quot;)[1]);<br>                if (pingPeer(peerHost)) peerLastSeen.put(peerId, System.currentTimeMillis());<br>            }<br>            try { Thread.sleep(1000); } catch (InterruptedException e) {}<br>        }<br>    }<br><br>    private static void runLeaderElection() {<br>        while (true) {<br>            long now = System.currentTimeMillis();<br>            List&lt;Integer&gt; aliveNodes = new ArrayList&lt;&gt;();<br><br>            aliveNodes.add(MY_ID);<br>            peerLastSeen.forEach((id, lastSeen) -&gt; {<br>                if (now - lastSeen &lt; 5000) aliveNodes.add(id);<br>            });<br><br>            int newLeader = Collections.max(aliveNodes);<br>            if (newLeader != currentLeaderId) {<br>                System.out.println(&quot;--- [Election] New Leader is Node &quot; + newLeader + &quot; ---&quot;);<br>                currentLeaderId = newLeader;<br>            }<br>            try { Thread.sleep(2000); } catch (InterruptedException e) {}<br>        }<br>    }<br><br>    private static boolean pingPeer(String host) {<br>        Request request = new Request.Builder().url(https://medium.com/&quot;http://&quot;%20+%20host%20+%20&quot;:7000/ping&quot;).build();<br>        try (Response response = client.newCall(request).execute()) { return response.isSuccessful(); } <br>        catch (Exception e) { return false; }<br>    }<br>}</pre><h4>How to Verify the “Strong Consistency”</h4><p>This tests if your system really behaves like a Database.</p><ol><li><strong>Rebuild:</strong> docker-compose up --build</li><li><strong>Wait for Leader:</strong> Ensure a Leader is elected (check logs).</li></ol><h4>Test 1: The Happy Path (3 Nodes Alive)</h4><p>Run this in your terminal:</p><pre>curl -L -X POST http://localhost:7001/data/money/100</pre><ul><li><strong>Result:</strong> Write Successful (Replicated to 3 nodes)</li><li><strong>Why:</strong> Leader + Node 1 + Node 2 all saved it. 3 &gt;= 2.</li></ul><h4>Test 2: The “Tolerable Failure” (1 Node Dead)</h4><p>Kill one follower (e.g., Node 1):</p><pre>docker stop distributed-playground-node-1-1</pre><p>Try the write again to Node 2 (or Leader):</p><pre>curl -L -X POST http://localhost:7002/data/money/200</pre><ul><li><strong>Result:</strong> Write Successful (Replicated to 2 nodes)</li><li><strong>Why:</strong> Leader is alive. Node 1 is dead. Node 2 is alive.</li><li>Votes = 2. Quorum is 2. <strong>Success.</strong> The system survives the failure!</li></ul><h4>Test 3: The “Cluster Failure” (2 Nodes Dead)</h4><p>Kill another follower (Node 2). Now only the Leader is alive.</p><pre>docker stop distributed-playground-node-2-1</pre><p>Try to write:</p><pre>curl -L -X POST http://localhost:7003/data/money/300</pre><ul><li><strong>Result:</strong> Write Failed: Cluster unstable (Only 1 nodes acknowledged)</li><li><strong>Why:</strong> The Leader is alive, but it cannot find any friends.</li><li>Votes = 1. Quorum = 2. <strong>Refused.</strong></li></ul><h4>Why is Test 3 so important?</h4><p>This effectively prevents <strong>“Split Brain.”</strong> If the Leader was cut off from the network, it might <em>think</em> it’s the leader, but it can’t reach anyone. By refusing the write, it prevents saving data that no one else knows about.</p><p><strong>The Learning:</strong></p><ul><li><strong>Trade-offs:</strong> We sacrificed <em>Speed</em> (latency) for <em>Safety</em> (consistency).</li></ul><h3><strong>Level 5: Amnesia (Durability &amp; WAL)</strong></h3><p>This is the final piece of the puzzle. Currently, your database has <strong>Amnesia</strong>. If you restart a node, it forgets everything.</p><p>We are going to give it <strong>Long-Term Memory</strong> using a <strong>Write-Ahead Log (WAL)</strong>.</p><h4>The Concept: Why “Write-Ahead”?</h4><p>You might ask: <em>“Why not just save the HashMap to a file every time it changes?”</em> <strong>Answer:</strong> Because writing the whole HashMap to disk for every single request is incredibly slow.</p><p><strong>The Solution:</strong></p><ol><li><strong>The Log:</strong> We append a simple line SET key=value to a text file. (Appending to a file is extremely fast).</li><li><strong>The Memory:</strong> We update the HashMap in RAM.</li><li><strong>The Rule:</strong> You <strong>MUST</strong> write to the file (WAL) <em>before</em> you update the RAM. If the power cuts after the file write but before the RAM update, you are safe (you can replay the file on startup).</li></ol><h4>Step 1: Prepare the Hard Drive (Docker Volumes)</h4><p>We need to tell Docker to map a folder on your real PC to the container, so files survive even if the container is deleted.</p><p>Update your docker-compose.yml. Add the volumes section to <strong>every node</strong>:</p><pre>node-1:<br>    # ... keep other settings ...<br>    volumes:<br>      - ./cluster-data/node-1:/app/data # Maps host folder to container folder<br><br>  node-2:<br>    # ... keep other settings ...<br>    volumes:<br>      - ./cluster-data/node-2:/app/data<br><br>  node-3:<br>    # ... keep other settings ...<br>    volumes:<br>      - ./cluster-data/node-3:/app/data</pre><h4>Step 2: The WAL Class (WalLogger.java)</h4><p>Create a new file src/main/java/com/playground/WalLogger.java. This handles the raw file I/O. It has two jobs: <strong>Append</strong> (Write) and <strong>Replay</strong> (Read).</p><pre>package com.playground;<br><br>import java.io.*;<br>import java.util.Map;<br><br>public class WalLogger {<br>    private final File walFile;<br>    private final BufferedWriter writer;<br><br>    public WalLogger(int nodeId) throws IOException {<br>        // Ensure directory exists inside the container<br>        File dir = new File(&quot;/app/data&quot;);<br>        if (!dir.exists()) dir.mkdirs();<br><br>        // The specific log file for this node<br>        this.walFile = new File(dir, &quot;node_&quot; + nodeId + &quot;.wal&quot;);<br>        <br>        // Append mode = true (Don&#39;t overwrite, just add to end)<br>        this.writer = new BufferedWriter(new FileWriter(walFile, true));<br>    }<br><br>    // 1. WRITE: Append command to disk<br>    public synchronized void write(String key, String value) {<br>        try {<br>            writer.write(key + &quot;,&quot; + value);<br>            writer.newLine();<br>            writer.flush(); // Force write to physical disk immediately<br>        } catch (IOException e) {<br>            e.printStackTrace();<br>            throw new RuntimeException(&quot;Disk Write Failed! Database stopped.&quot;);<br>        }<br>    }<br><br>    // 2. RECOVERY: Read file line by line and fill the HashMap<br>    public void replay(Map&lt;String, String&gt; kvStore) {<br>        if (!walFile.exists()) return;<br><br>        System.out.println(&quot;--- [Recovery] Replaying WAL from disk... ---&quot;);<br>        try (BufferedReader reader = new BufferedReader(new FileReader(walFile))) {<br>            String line;<br>            int count = 0;<br>            while ((line = reader.readLine()) != null) {<br>                String[] parts = line.split(&quot;,&quot;);<br>                if (parts.length == 2) {<br>                    kvStore.put(parts[0], parts[1]);<br>                    count++;<br>                }<br>            }<br>            System.out.println(&quot;--- [Recovery] Restored &quot; + count + &quot; entries from disk ---&quot;);<br>        } catch (IOException e) {<br>            e.printStackTrace();<br>        }<br>    }<br>}</pre><h4>Step 3: Integrate WAL into Node.java</h4><p>We need to modify your main Node.java to:</p><ol><li>Initialize the WAL on startup.</li><li>Replay old data into the HashMap.</li><li>Write to WAL <em>before</em> updating the HashMap.</li></ol><p>Update src/main/java/com/playground/Node.java:</p><pre>package com.playground;<br><br>import io.javalin.Javalin;<br>import io.javalin.http.Context;<br>import okhttp3.*;<br>import java.io.IOException;<br>import java.util.*;<br>import java.util.concurrent.CompletableFuture;<br>import java.util.concurrent.ConcurrentHashMap;<br>import java.util.concurrent.TimeUnit;<br><br>public class Node {<br>    // 1. THE DATABASE (In-Memory Key-Value Store)<br>    private static final Map&lt;String, String&gt; kvStore = new ConcurrentHashMap&lt;&gt;();<br><br>    // 2. STATE<br>    private static final Map&lt;Integer, Long&gt; peerLastSeen = new ConcurrentHashMap&lt;&gt;();<br>    private static volatile int currentLeaderId = -1; // Who is the boss?<br>    <br>    // Configuration<br>    private static WalLogger wal;<br>    private static final OkHttpClient client = new OkHttpClient.Builder().readTimeout(1, TimeUnit.SECONDS).build();<br>    private static int MY_ID;<br>    private static List&lt;String&gt; PEER_HOSTNAMES;<br><br>    public static void main(String[] args) throws IOException {<br>        MY_ID = Integer.parseInt(System.getenv().getOrDefault(&quot;NODE_ID&quot;, &quot;1&quot;));<br>        String peersEnv = System.getenv().getOrDefault(&quot;PEERS&quot;, &quot;&quot;);<br>        PEER_HOSTNAMES = peersEnv.isEmpty() ? List.of() : Arrays.asList(peersEnv.split(&quot;,&quot;));<br><br>        System.out.println(&quot;--- NODE &quot; + MY_ID + &quot; STARTED ---&quot;);<br><br>        // 1. INITIALIZE WAL &amp; RECOVER<br>        wal = new WalLogger(MY_ID); <br>        wal.replay(kvStore); // Load old data from disk into RAM<br><br>        System.out.println(&quot;--- NODE &quot; + MY_ID + &quot; STARTED (With WAL Persistence) ---&quot;);<br>        <br>        Javalin app = Javalin.create().start(7000);<br><br>        // --- ENDPOINTS ---<br><br>        // 1. Health Check<br>        app.get(&quot;/ping&quot;, ctx -&gt; ctx.result(&quot;Pong from &quot; + MY_ID));<br><br>        // 2. Public Write Endpoint (The Entry Point)<br>        app.post(&quot;/data/{key}/{value}&quot;, ctx -&gt; handleStrongWrite(ctx));<br><br>        // 3. Public Read Endpoint<br>        app.get(&quot;/data/{key}&quot;, ctx -&gt; {<br>            String value = kvStore.get(ctx.pathParam(&quot;key&quot;));<br>            if (value == null) ctx.status(404).result(&quot;Key not found on Node &quot; + MY_ID);<br>            else ctx.result(value);<br>        });<br><br>        // 4. Internal Replication Endpoint (Only for Leader -&gt; Follower)<br>        app.post(&quot;/internal/replicate/{key}/{value}&quot;, ctx -&gt; {<br>            String key = ctx.pathParam(&quot;key&quot;);<br>            String value = ctx.pathParam(&quot;value&quot;);<br><br>            // WAL STEP: Write to disk first!<br>            wal.write(key, value);<br><br>            kvStore.put(key, value);<br>            System.out.println(&quot;--- [Replication] Leader told me to save &quot; + key + &quot;=&quot; + value + &quot; ---&quot;);<br>            ctx.status(200);<br>        });<br><br>        // --- BACKGROUND TASKS ---<br>        new Thread(Node::runFailureDetector).start();<br>        new Thread(Node::runLeaderElection).start();<br>    }<br><br>    // --- LOGIC HANDLERS ---<br><br>    // private static void handleWriteRequest(Context ctx) {<br>    //     String key = ctx.pathParam(&quot;key&quot;);<br>    //     String value = ctx.pathParam(&quot;value&quot;);<br><br>    //     // CASE A: I am not the leader -&gt; Redirect<br>    //     if (currentLeaderId != MY_ID) {<br>    //         if (currentLeaderId == -1) {<br>    //             ctx.status(503).result(&quot;No leader available&quot;);<br>    //             return;<br>    //         }<br>    //         // Redirect client to the leader using host-mapped port (host: 7000 + leaderId)<br>    //         int leaderHostPort = 7000 + currentLeaderId; // host ports: 7001,7002,7003<br>    //         String leaderUrl = &quot;http://localhost:&quot; + leaderHostPort + &quot;/data/&quot; + key + &quot;/&quot; + value;<br>    //         ctx.status(307).redirect(leaderUrl);<br>    //         System.out.println(&quot;--- [Redirect] Redirecting client to Leader (Node &quot; + currentLeaderId + &quot;) at &quot; + leaderUrl + &quot; ---&quot;);<br>    //         return;<br>    //     }<br><br>    //     // CASE B: I am the leader -&gt; Save and Replicate<br>    //     System.out.println(&quot;--- [Write] I am Leader. Saving &quot; + key + &quot;=&quot; + value + &quot; ---&quot;);<br>        <br>    //     // 1. Save Locally<br>    //     kvStore.put(key, value);<br><br>    //     // 2. Replicate to others (Fire and Forget for now)<br>    //     for (String peer : PEER_HOSTNAMES) {<br>    //         replicateToPeer(peer, key, value);<br>    //     }<br><br>    //     ctx.result(&quot;Write Successful on Leader &quot; + MY_ID);<br>    // }<br><br>    // --- THE HARD PART: QUORUM LOGIC ---<br>    private static void handleStrongWrite(Context ctx) {<br>        String key = ctx.pathParam(&quot;key&quot;);<br>        String value = ctx.pathParam(&quot;value&quot;);<br><br>        // 1. Redirect if not leader<br>        if (currentLeaderId != MY_ID) {<br>            if (currentLeaderId == -1) { ctx.status(503).result(&quot;No Leader&quot;); return; }<br>            int leaderHostPort = 7000 + currentLeaderId; // host ports: 7001,7002,7003<br>            String leaderUrl = &quot;http://localhost:&quot; + leaderHostPort + &quot;/data/&quot; + key + &quot;/&quot; + value;<br>            ctx.status(307).redirect(leaderUrl);<br>            return;<br>        }<br><br>        // 2. I am Leader: Start the Clock<br>        System.out.println(&quot;--- [Write] Processing Quorum Write for &quot; + key + &quot; ---&quot;);<br><br>        // WAL STEP: Write to local disk first!<br>        wal.write(key, value);<br>        kvStore.put(key, value);<br>        <br>        // Step A: Save locally (1 vote)<br>        kvStore.put(key, value);<br>        <br>        // Step B: Send to all peers in PARALLEL and wait for results<br>        List&lt;CompletableFuture&lt;Boolean&gt;&gt; futures = new ArrayList&lt;&gt;();<br>        <br>        for (String peer : PEER_HOSTNAMES) {<br>            futures.add(CompletableFuture.supplyAsync(() -&gt; sendReplicationSync(peer, key, value)));<br>        }<br><br>        // Step C: Wait for all attempts to finish (or timeout)<br>        try {<br>            CompletableFuture.allOf(futures.toArray(new CompletableFuture[0])).join();<br>        } catch (Exception e) {<br>            // Ignore exceptions, we just want to count successes<br>        }<br><br>        // Step D: Count Votes<br>        long successCount = 1; // Start with 1 (Myself)<br>        for (CompletableFuture&lt;Boolean&gt; f : futures) {<br>            try {<br>                if (f.getNow(false)) successCount++;<br>            } catch (Exception e) {}<br>        }<br><br>        // Step E: Quorum Decision<br>        int quorum = (PEER_HOSTNAMES.size() + 1) / 2 + 1; // (2+1)/2 + 1 = 2<br>        <br>        if (successCount &gt;= quorum) {<br>            System.out.println(&quot;--- [Commit] Quorum reached (&quot; + successCount + &quot;/&quot; + (PEER_HOSTNAMES.size()+1) + &quot;) ---&quot;);<br>            ctx.result(&quot;Write Successful (Replicated to &quot; + successCount + &quot; nodes)&quot;);<br>        } else {<br>            System.out.println(&quot;--- [Fail] Quorum failed (&quot; + successCount + &quot; votes). Data is unsafe! ---&quot;);<br>            ctx.status(500).result(&quot;Write Failed: Cluster unstable (Only &quot; + successCount + &quot; nodes acknowledged)&quot;);<br>            // Note: In a real DB, we would rollback here!<br>        }<br>    }<br><br>    // private static void replicateToPeer(String peerHost, String key, String value) {<br>    //     // Send to /internal/replicate endpoint<br>    //     String url = &quot;http://&quot; + peerHost + &quot;:7000/internal/replicate/&quot; + key + &quot;/&quot; + value;<br>    //     Request request = new Request.Builder()./service/https://medium.com/url(url).post(RequestBody.create(new byte[0])).build();<br>        <br>    //     // Asynchronous call (don&#39;t wait for response to keep it simple)<br>    //     client.newCall(request).enqueue(new Callback() {<br>    //         @Override public void onFailure(Call call, IOException e) {<br>    //             System.out.println(&quot;Failed to replicate to &quot; + peerHost);<br>    //         }<br>    //         @Override public void onResponse(Call call, Response response) {<br>    //             response.close();<br>    //         }<br>    //     });<br>    // }<br><br>    // Synchronous replication helper used by quorum writes (returns true on success)<br>    private static boolean sendReplicationSync(String peerHost, String key, String value) {<br>        String url = &quot;http://&quot; + peerHost + &quot;:7000/internal/replicate/&quot; + key + &quot;/&quot; + value;<br>        Request request = new Request.Builder()./service/https://medium.com/url(url).post(RequestBody.create(new byte[0])).build();<br>        try (Response response = client.newCall(request).execute()) {<br>            return response.isSuccessful();<br>        } catch (Exception e) {<br>            System.out.println(&quot;Failed to replicate (sync) to &quot; + peerHost + &quot;: &quot; + e.getMessage());<br>            return false;<br>        }<br>    }<br><br>    // --- BACKGROUND THREADS (Same as before) ---<br><br>    private static void runFailureDetector() {<br>        while (true) {<br>            for (String peerHost : PEER_HOSTNAMES) {<br>                int peerId = Integer.parseInt(peerHost.split(&quot;-&quot;)[1]);<br>                if (pingPeer(peerHost)) peerLastSeen.put(peerId, System.currentTimeMillis());<br>            }<br>            try { Thread.sleep(1000); } catch (InterruptedException e) {}<br>        }<br>    }<br><br>    private static void runLeaderElection() {<br>        while (true) {<br>            long now = System.currentTimeMillis();<br>            List&lt;Integer&gt; aliveNodes = new ArrayList&lt;&gt;();<br><br>            aliveNodes.add(MY_ID);<br>            peerLastSeen.forEach((id, lastSeen) -&gt; {<br>                if (now - lastSeen &lt; 5000) aliveNodes.add(id);<br>            });<br><br>            int newLeader = Collections.max(aliveNodes);<br>            if (newLeader != currentLeaderId) {<br>                System.out.println(&quot;--- [Election] New Leader is Node &quot; + newLeader + &quot; ---&quot;);<br>                currentLeaderId = newLeader;<br>            }<br>            try { Thread.sleep(2000); } catch (InterruptedException e) {}<br>        }<br>    }<br><br>    private static boolean pingPeer(String host) {<br>        Request request = new Request.Builder().url(https://medium.com/&quot;http://&quot;%20+%20host%20+%20&quot;:7000/ping&quot;).build();<br>        try (Response response = client.newCall(request).execute()) { return response.isSuccessful(); } <br>        catch (Exception e) { return false; }<br>    }<br>}</pre><h4>Phase 4: The “Amnesia Test”</h4><ul><li><strong>Start Clean:</strong></li></ul><pre>docker-compose down<br># Delete old data to start fresh<br>rm -rf cluster-data <br>docker-compose up --build</pre><ul><li><strong>Write Important Data:</strong> Wait for a leader, then write data.</li></ul><pre>curl -L -X POST http://localhost:7001/data/balance/1000</pre><ul><li><em>Result:</em> Write Successful.</li><li><strong>Verify Data Exists:</strong></li></ul><pre>curl http://localhost:7001/data/balance</pre><ul><li><em>Result:</em> 1000.</li><li><strong>The Crash (Simulate Power Failure):</strong> Stop <strong>ALL</strong> containers.</li></ul><pre>docker-compose stop</pre><ul><li><strong>The Resurrection:</strong> Start them back up.</li></ul><pre>docker-compose up</pre><p><strong>Watch the logs closely!</strong> You should see lines like: node-1 | --- [Recovery] Replaying WAL from disk... --- node-1 | --- [Recovery] Restored 1 entries from disk ---</p><ul><li><strong>Verify Memory:</strong> Ask Node 1 for the balance again.</li></ul><pre>curl http://localhost:7001/data/balance</pre><p><em>Result:</em> 1000.</p><p><strong>The Learning:</strong></p><ul><li><strong>Persistence:</strong> RAM is volatile; Disk is forever.</li><li><strong>Log Replay:</strong> How databases “boot up” and recover state.</li></ul><h3>Conclusion: What I Built</h3><ul><li>I didn’t just build a Java app; I built a <strong>System</strong>.</li><li>It survives network failures.</li><li>It survives process crashes.</li><li>It survives full restarts.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=572ac534ce72" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>