<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Shekhar Saxena on Medium]]></title>
        <description><![CDATA[Stories by Shekhar Saxena on Medium]]></description>
        <link>https://medium.com/@shekharsaxena316?source=rss-f68c68b34ca2------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*DT-qn52u8UyRtjCac6_-EQ.jpeg</url>
            <title>Stories by Shekhar Saxena on Medium</title>
            <link>https://medium.com/@shekharsaxena316?source=rss-f68c68b34ca2------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 04:42:54 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@shekharsaxena316/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How My Personal Finance App Became an Experiment in LLMs, MCP and Context Engineering]]></title>
            <link>https://medium.com/@shekharsaxena316/how-my-personal-finance-app-became-an-experiment-in-llms-mcp-and-context-engineering-cfab40dcd8db?source=rss-f68c68b34ca2------2</link>
            <guid isPermaLink="false">https://medium.com/p/cfab40dcd8db</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[generative-ai-tools]]></category>
            <category><![CDATA[local-llm]]></category>
            <dc:creator><![CDATA[Shekhar Saxena]]></dc:creator>
            <pubDate>Thu, 10 Sep 2026 13:45:52 GMT</pubDate>
            <atom:updated>2026-09-10T13:45:52.676Z</atom:updated>
            <content:encoded><![CDATA[<p>Once you start earning, managing money gets surprisingly complicated. As a student, my financial planning was basically: <strong><em>Money in → money out → hopefully something’s left</em>. </strong>Then I started working. Suddenly, there was a salary, multiple bank accounts, credit cards, UPI, cash, investments, FDs, mutual funds, outings… and the classic end-of-month question: <strong>“<em>Where did all my money go?</em>” </strong>I tried a few personal finance apps. They were useful, but each did one thing well — transactions, budgeting, investments, etc. And somehow, I became the integration layer between all of them.</p><p>So the engineer in me thought: “<strong><em>Why not just build one?” </em></strong>And since I was building it anyway: <strong>“<em>Why not use it to learn some stuff I’ve been wanting to explore?</em>” </strong>That’s how <strong>DhanDash</strong> started. What began as a simple finance tracker turned into an experiment with <em>LLMs, MCP, tool calling, context engineering, local models, Redis, rate limiting, and AI security</em>.</p><h4>Okay, so what is DhanDash?</h4><p>The original idea was simple: <strong>one place to see my entire financial life</strong> without opening five apps and doing mental math. DhanDash currently covers:</p><ul><li>Transactions across multiple accounts</li><li>Smart transaction categorization</li><li>Budgets and monthly limits</li><li>Dashboards and charts</li><li>Investments and goals</li><li>Credit-card tracking</li><li>Shared transactions and settlements</li><li>AI-assisted interactions</li><li>And most importantly, <strong>local LLMs</strong> — because I’d rather not have ChatGPT know how I spent my last weekend. 😅</li></ul><p>I can track everything from <strong>bank accounts, cash, credit cards, and mutual funds</strong> in one place. I can also set goals like: <em>“</em><strong><em>I want to accumulate ₹12 lakh by the end of the year.</em></strong><em>” </em>Instead of staring at ₹4.8 lakh and feeling guilty, DhanDash does the math for me: <strong>“<em>Nice. But you’re still ₹7.2 lakh away.”</em></strong><br>Very motivating. The POC is far from perfect. There are bugs, rough edges, and unfinished features. And honestly, that’s fine. Because the <strong>product was never the most interesting part of this project.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uWkPyqBt7aaoytdHVcB-_g.png" /></figure><h4>I didn’t want another chatbot</h4><p>Adding AI to an app is pretty easy these days. Add a chat box, send the message to an LLM, show the response, and boom — <strong>AI-powered app.</strong> I didn’t want that. I wanted AI to <strong>actually do useful work inside DhanDash</strong>, without making me click through ten screens. Imagine I’m walking somewhere and say: <em>“I spent ₹120 on coffee.” </em>I don’t want to open the app, pick an account, select expense, enter ₹120, choose Food → Restaurant, and hit save. I want to just say it — and let the system figure out the boring stuff.</p><p>That sounds simple from the outside, but internally, that single sentence needs to be transformed from unstructured natural language into something the application can actually work with. Behind the scenes, the AI needs to turn that sentence into structured data:</p><pre>Amount       → ₹120<br>Type         → Expense<br>Description  → Coffee<br>Category     → Food<br>Subcategory  → Restaurant</pre><p>And that’s where things get interesting. Which account did I use? What if I have multiple Food categories? What if my app calls it <strong>Eating Out</strong>? What if I say: <em>“I had coffee with someone. I paid ₹240 for both of us.” </em>Now the system needs to understand <strong>context, intent, entities, and even shared expenses</strong>. This is where I started experimenting with <strong>LLM tool calling, structured outputs, context engineering, and MCP, </strong>not just to make the app talk, but to let AI interact with the actual application.</p><p>But there was one rule I wasn’t willing to break: <strong><em>The LLM should never get unrestricted access to my financial data. </em></strong>Instead, the AI can <strong>understand, suggest, and request actions</strong>, while the application stays in control of what actually gets changed. That made DhanDash feel less like a chatbot and more like an <strong>AI interface sitting on top of the application, </strong>one that can understand what I mean, use the right tools, and still ask for my approval before touching my money.</p><h4>The LLM should not become my accountant</h4><p>This became one of the biggest architectural lessons for me.</p><p>Imagine asking: <em>“</em><strong><em>How much money do I have</em></strong><em>?” </em>And the LLM responds: <em>“</em><strong><em>You currently have ₹47,320.</em></strong><em>” </em>Looks great. Except… Where did it get ₹47,320 from?<strong> </strong>Did it actually check my accounts? Or did it confidently make up a number because language models are sometimes very good at sounding like they know things? For a finance application, I don’t want confidence. I want the actual number. Same thing with calculations. If I have an FD of ₹1 lakh at 6.7%, I don’t want the LLM deciding how much it matures to. The application should calculate it. The LLM can say: <em>“</em><strong><em>Your FD is expected to mature to approximately ₹1.07 lakh on this date.</em></strong><em>” </em>But the calculation should come from deterministic application logic. That led me to a simple rule: <strong>The LLM can reason about my data, but it should not own my data.</strong></p><p>So instead of:</p><pre>User → LLM → &quot;Figure everything out&quot; → Database</pre><p>I wanted:</p><pre>User → LLM → Controlled tools → Application services → Database</pre><p>And that little architectural difference changed the way I thought about the entire AI layer. The LLM became the <strong>reasoning and interaction layer</strong>, while the application remained the <strong>source of truth and enforcement layer</strong>. That separation gave me something much more valuable than a chatbot. It gave me a way to make DhanDash <strong>feel conversational without making the underlying financial system conversationally unsafe</strong>.</p><h4>Tool calling: let the AI ask the application</h4><p>Instead of dumping all my financial data into the prompt, I started thinking of the application as a collection of capabilities.</p><p>For example:</p><pre>get_accounts()<br>get_categories()<br>search_transactions()<br>get_budget_status()<br>get_investments()<br>get_credit_cards()<br>get_goals()<br>create_transaction()</pre><p>Now suppose I say: <em>“</em><strong><em>How much did I spend on food this month?</em></strong><em>” </em>The LLM doesn’t need to know the answer. It needs to know <strong>how to get the answer</strong>.</p><p>Something like:</p><pre>User → &quot;How much did I spend on food?&quot; → LLM → search_transactions <br><br>→ Application → Actual transactions → LLM → &quot;₹4,280&quot;</pre><p>The model isn’t guessing. It’s asking the application. And this is where I started appreciating tool calling much more. The LLM is good at understanding what the user wants. My backend is good at knowing what actually happened. Honestly, I don’t want my LLM calculating my bank balance. I already have enough people in my life giving me financial advice without checking the numbers.</p><h4>Then I discovered MCP</h4><p>While exploring tool calling, I started learning about <strong>MCP — Model Context Protocol</strong>. Initially, I approached it from the “AI integration” angle. But the more I worked with it, the more interesting the architecture became.</p><p>The basic idea I liked was: <strong><em>Instead of tightly coupling my application to one particular model, I can expose application capabilities through a standardized interface.</em></strong></p><p>So conceptually:</p><pre>                 DhanDash<br>                    │<br>                    ▼<br>                MCP Server<br>                    │<br>        ┌───────────┼───────────┐<br>        │           │           │<br>   Accounts     Transactions   Goals<br>        │           │           │<br>        └───────────┼───────────┘<br>                    │<br>                   LLM</pre><p>Now the model doesn’t need to know how my database works. It doesn’t need Prisma knowledge. It doesn’t need SQL access. It doesn’t need to know what tables exist. It just knows: <em>“</em><strong><em>I have a tool called </em></strong><strong><em>get_accounts</em></strong><em>.&quot; </em>And that made me think about something much more important.</p><h4>How much power should you actually give an LLM?</h4><p>This was one of the questions I kept coming back to. Let’s say I give the model: <em>execute_sql(query) </em>Technically, it can do almost anything. And technically, that’s also a terrible idea for a financial application. Instead, I would much rather expose: <em>get_account_balance(account_id)</em> or <em>search_transactions(filters)</em> or <em>create_transaction(transaction) </em>The difference is huge. The one gives the model a <strong>business capability</strong>. The another gives it <strong>infrastructure access</strong>. I don’t want an LLM thinking: <em>“</em><strong><em>Hmm, I wonder what tables DhanDash has. Let me explore.</em></strong><em>” </em>No. Please don’t. You are an assistant. Stay in your limits. This was one of the biggest things MCP made me think about:</p><blockquote><strong><em>AI capabilities should be designed like APIs, not like database permissions.</em></strong></blockquote><h4>Prompt engineering wasn’t enough</h4><p>Before this project, I had the usual idea: <strong><em>Better prompt = better response.</em></strong></p><p>So naturally, I started writing prompts. And then writing longer prompts. And then even longer prompts. At some point my prompt started looking like a small novel. That’s when I realized: <strong>Maybe the problem isn’t that the prompt isn’t long enough. </strong>The bigger question is: <strong><em>What information does the model actually need? </em></strong>Consider this: Categorize this transaction: Chai - ₹60 The model has to guess your category structure. Now give it:</p><pre>Transaction:<br>Dosa - ₹60<br>Available categories:<br>Food<br>  ├── Groceries<br>  ├── Restaurant<br>  └── Delivery<br>Transport<br>  ├── Cab<br>  └── Fuel<br>Entertainment<br>  ├── Movies<br>  └── Games<br><br>or even better <br><br>Transaction:<br>Dosa - ₹60<br>Available categories:<br>Check with get_categories()<br></pre><p>Now the model has a much smaller problem. It isn’t inventing a category. It is selecting from the categories that actually exist. That difference was a big learning for me. I started thinking less about: <em>“How do I write a better prompt?” </em>and more about: <strong><em>“What is the minimum useful context I should provide for the model to make the right decision?”</em></strong></p><p>That’s where I started getting more interested in <strong>context engineering</strong>.</p><h4>Same model. Different behavior.</h4><p>This was probably one of the most interesting surprises for me. I experimented with local models using <strong>Ollama, Hosted Services and LM Studio</strong>. Initially, I had this mental model:</p><pre>Same model + Same prompt = Same response</pre><p>Makes sense, right? Except reality had other plans. I noticed that model behavior could vary depending on the surrounding inference environment and configuration. Things like: Chat templates, Sampling parameters, Context configuration, Tool/function-calling support, Quantization, Runtime implementation, Model configuration can all affect what you actually experience.</p><p>So saying: <em>“I’m using model X” </em>doesn’t necessarily tell the whole story. The environment around the model matters too. And this was something I understood much better after actually experimenting with it. You can read ten articles about inference runtimes. But when you spend an evening wondering: <em>“Why is this model behaving completely differently now?” </em>you suddenly remember the lesson very well.</p><h4>Smaller model, better context</h4><p>Another thing that surprised me was how far you can take a smaller local model when you give it a well-defined problem. Let’s say the task is: <em>“T</em><strong><em>ake this transaction and map it to one of the categories that already exist for this user.</em></strong><em>” </em>That’s a very different problem from: <em>“</em><strong><em>Understand everything about human civilization.</em></strong><em>”</em></p><p>Thankfully, my finance app doesn’t need the second one. For these narrow tasks, good: Instructions, Context, Tool definitions, Structured output, Validation, Constraints can make a surprisingly large difference.</p><p>This doesn’t mean: <em>“</em><strong><em>Prompt engineering can make a small model as good as a frontier model.</em></strong><em>” </em>It can’t. But I did learn something useful:</p><blockquote><strong><em>Before throwing a bigger model at the problem, understand whether the problem itself is poorly defined.</em></strong></blockquote><p>Sometimes the model isn’t the problem. Sometimes we’re simply asking it to solve a badly specified problem with half the information it needs.</p><p>And we all know, “<strong>A problem well-stated is a problem half-solved</strong>”. I think prompt and context engineering nothing but the same.</p><h4>And then came the security question</h4><p>This is where AI development starts looking suspiciously similar to normal backend development. You quickly realize: <strong>A system prompt is not a security mechanism. </strong>If I tell the model: <em>“Never access another user’s data.” </em>that is useful instruction. But it is not authorization. Authorization needs to happen in the backend. If user A asks for user B’s account:</p><pre>LLM → get_account_balance(user_B_account) → Backend → ❌ Unauthorized</pre><p>The backend should reject it. It should tell me about ethics and say: &quot;Oops, the system prompt says I shouldn&#39;t do that.&quot; That was an important lesson for me: <strong><em>AI guardrails should complement application security, not replace it.</em></strong></p><p>And the same principle applies to tools. If a tool creates a transaction, the backend should validate: Does this account belong to the user? Is the amount valid? Is the category valid? Is the request authorized? Are required fields present?</p><p>The model can request the operation. The application decides whether it is allowed.</p><h4>Then Redis entered the chat</h4><p>Once you put an LLM behind an API, you start discovering some wonderfully boring questions. Like: <em>“</em><strong><em>What happens if someone calls this endpoint 500 times?</em></strong><em>” or “</em><strong><em>What if my agent accidentally gets into a loop?</em></strong><em>” </em>Or: <em>“</em><strong><em>What if someone discovers my AI endpoint and decides to have a very productive afternoon?</em></strong><em>”</em></p><p>This is where rate limiting becomes important. And Redis became useful for this part.</p><p>Conceptually:</p><pre>Request → Authentication → Rate Limiter → Redis → AI Service → LLM</pre><p>Now you can have things like: Per-user limits, Per-IP limits, Different limits for expensive operations, Request windows, Abuse protection. This can be also extended to have cached responses, instead everytime asking LLM the same question to avoid token consumption and restrict abuse.</p><p>It’s not the most glamorous part of AI engineering. Nobody posts: <em>“Today I implemented Redis rate limiting. 🔥🔥🔥” </em>But when your API is public, you become very interested in it. Very quickly.</p><h4>Coming from Java, this was a refreshing reset</h4><p>Most of my professional experience has been around Java.</p><p>So working on the project with, JavaScript/TypeScript, Fastify, Prisma, Redis was already a good change of scenery. It wasn’t about replacing Java. It was about getting uncomfortable again. And honestly, I liked that feeling of getting uncomfortable again, writing that code without AI cursor, antigravity or whatever I use in my everyday life.</p><p>You start asking basic questions again. You make mistakes that you probably wouldn’t make in your usual stack. You look at documentation for things you normally take for granted. And then you slowly start building confidence again. At the same time, I was learning completely new concepts. That combination made this project much more valuable to me than I initially expected.</p><h4>What I would still improve</h4><p>DhanDash is a POC. I’m deliberately calling it that. There are bugs. There are unfinished features. There are areas where I know the implementation can be better. Before calling it production-ready, I’d still want to work on things like, Better statement parsing, Credit-card milestone and benefit automation, Better AI evaluation, Cloud LLM integrations, More comprehensive security testing, Observability, Automated test coverage, Production deployment and most important Backup and recovery.</p><p>So no, I’m not claiming: <em>“I built the next big personal finance platform.” </em>Not yet. 😄 What I can say is that I built something real enough to force me to deal with real engineering questions. And that’s what I wanted from the project.</p><h4>So, what’s next?</h4><p>For now, DhanDash remains a learning project and POC. But the question I’m interested in has changed. Initially, I was asking: <strong><em>“Can I make an AI categorize my transactions?” </em></strong>Now I’m more interested in: <strong><em>“What would a genuinely useful personal financial copilot look like if an AI had access to the right tools, but never had unrestricted access to my financial data?”</em></strong></p><p>That’s a much more interesting problem. And honestly, I don’t have the complete answer yet. Which is probably a good thing. Because if I’ve learned anything from this project, it’s that the moment you think: <em>“Okay, I understand this now.” </em>there’s usually another rabbit hole waiting for you. And that’s probably the best part of building. You start with: <strong>“I just want to build a finance app.” </strong>And somehow end up spending your evening wondering why the same LLM behaves differently in two local runtimes. Would I do it again? Absolutely. Mostly because I still have a lot of questions left to answer.</p><p>So over the coming days, I’ll be writing separate deep-dives on the things I learned while building this, <strong>context engineering, prompt engineering, LLM environment optimization, MCP, tool calling, and agentic behaviour</strong>. Not as isolated AI concepts, but from the perspective of <strong>what actually happens when you try to put these ideas into a real application</strong>.</p><p>If you have any suggestions/ feedback or you want to contribute/ collab — please feel free to look at —<a href="https://github.com/shekhar316/dhandash">GITHUB</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cfab40dcd8db" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Beyond GPT: Why General AI Isn’t Enough and What Comes Next]]></title>
            <link>https://medium.com/@shekharsaxena316/beyond-gpt-why-general-ai-isnt-enough-and-what-comes-next-ac4b5526caa4?source=rss-f68c68b34ca2------2</link>
            <guid isPermaLink="false">https://medium.com/p/ac4b5526caa4</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[rags]]></category>
            <dc:creator><![CDATA[Shekhar Saxena]]></dc:creator>
            <pubDate>Sun, 19 Apr 2026 09:24:52 GMT</pubDate>
            <atom:updated>2026-04-19T09:24:52.927Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QpS72czQj5TATy8QrePK0g.png" /></figure><p>Let me start with something simple.</p><p>Imagine you walk into a hospital. Would you rather be treated by a doctor who knows a little bit about everything … or a specialist who has spent years focusing only on your exact problem? If it’s something serious, we want the specialist. The eye doctor. The neurologist. The expert.</p><p>Now here’s the interesting part. Tools like GPT can write essays, help with coding, explain science, draft emails, and even give life advice. One minute you’re asking it to explain black holes, the next you’re like, “Hey, can you also help me text my crush?” It’s impressive, almost magical at times. It feels like talking to someone who knows everything.</p><p>But if you’ve ever pushed it a bit further, asked it something very specific, very technical, or very real-world, you might have noticed something strange.</p><p>It still sounds confident. Even when it’s wrong. And that’s where the story begins.</p><h3>When “Knowing Everything” Isn’t Enough</h3><p>Let’s say you ask an AI about a complicated tax rule. Or a medical condition. Or a legal clause buried deep inside a contract. It will respond smoothly. Clearly. Convincingly.</p><p>But here’s the catch. It doesn’t actually <em>know</em> in the way a specialist knows. It’s more like someone who has read millions of books and is very good at guessing what the answer should sound like. That works surprisingly well… until it doesn’t.</p><p>It’s like asking a really smart friend who read Wikipedia all night to perform surgery. You’d enjoy the conversation, but you probably wouldn’t hand them the scalpel. And that’s exactly the limitation of general AI models.</p><p>They are incredibly broad. But they are not deep where it truly matters.</p><h3>Enter Domain-Specific AI</h3><p>Now imagine we take that same AI, but instead of letting it learn <em>everything</em>, we train it to focus on just one area.</p><p>Only medical records. Only financial data. Only legal documents. Only your company’s internal knowledge.</p><p>Suddenly, things change. This is what we call <strong>domain-specific</strong> AI. It’s the same AI, but focused on just one domain. And just like in real life, specialists tend to be much better at solving real problems.</p><h3>Think of It Like This</h3><p>Let’s use a simple analogy. A general AI model is like Google Maps for the entire world. It knows every country, every city, every road. But a domain-specific model is like a local guide in a small town. The map shows you where things are. The guide tells you where to go, what to avoid, and what actually matters.</p><p>And when you’re making important decisions, that difference matters a lot.</p><h3>When Things Go Wrong (In the Real World)</h3><p>This isn’t just a theoretical concern. We’ve already seen what happens when general AI models are used in situations where precision actually matters.</p><p>In 2023, a lawyer in the United States used ChatGPT to assist with legal research. The AI generated case citations that looked completely legitimate, but they didn’t exist. The court later sanctioned the lawyer after discovering that the cases were fabricated. What made it worse was how convincing the response sounded.</p><p>In another widely reported example, early testing of AI systems in healthcare settings showed that while the responses were fluent and helpful in tone, they sometimes missed critical clinical nuances.</p><p>Even outside high-stakes environments, smaller failures happen every day. Students rely on AI for explanations that are partially incorrect. Professionals use it to summarize documents and miss key details. People ask for financial guidance and receive answers that sound reasonable but lack real-world grounding.</p><p>The pattern is consistent. The AI doesn’t fail loudly. It fails quietly. It gives you an answer that feels right… but isn’t always reliable.</p><h3>How These Models Actually Work</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*26quhUs2jKfkAyCA8qq2Lg.png" /></figure><p>Let’s not dive into complicated jargon. Instead, think of it like this. A domain-specific AI has three main “parts,” even if you don’t see them.</p><p>First, there’s the <strong>general brain</strong>. This is the base model, something like GPT-4. This is called a <strong>foundation model</strong> or <strong>large language model (LLM)</strong>. It’s trained on massive amounts of text using a process called <strong>pretraining</strong>, which helps it learn language patterns, reasoning, and structure.</p><p>Then comes the <strong>memory</strong> layer. This is where domain knowledge lives … medical research, legal documents, financial data, or internal company files. Technically, this is an external knowledge base, often stored in systems like vector databases or document stores.</p><p>Finally, there’s the <strong>retrieval system</strong>, the part that quietly does the smart work. When you ask a question, the AI doesn’t immediately answer. Instead, a few things happen behind the scenes:</p><p>Your question is converted into numbers using something called an <strong>embedding model</strong>. This process is known as embedding generation, and it helps capture the <em>meaning</em> of your query, not just the exact words.</p><ol><li>The system searches its memory to find relevant information. This step is called <strong>semantic search</strong> or vector similarity search, where it looks for content that is <em>conceptually similar</em> to your question.</li><li>The most relevant pieces are selected and passed back to the model. This is often referred to as retrieval or context injection.</li><li>Finally, the AI generates a response using both its general knowledge and the retrieved data. This step is called generation or inference.</li></ol><p>So instead of guessing, it’s more like: “<strong>Let me find the best information I have… and then explain it clearly.</strong>” And that small shift … from guessing to retrieving … is what makes a huge difference.</p><h3>RAG vs Fine-Tuning</h3><p>You’ll often hear about two main ways to build these systems. Let’s simplify them.</p><p><strong>Retrieval-Augmented Generation (RAG)</strong> This is like giving the AI a smart library and teaching it how to search. The knowledge stays outside the model. When a question comes in, the AI looks up relevant information and uses it to generate an answer.</p><p>Think of it as: “<strong>I don’t have everything memorized, but I know exactly where to find it.</strong>”</p><p>It’s useful because it’s easy to update, works well with large, changing knowledge bases, and is more transparent, you can often trace where answers come from.</p><p><strong>Fine-Tuning</strong> This is like teaching the AI everything in advance. Here, you train the model on specific data so that the knowledge becomes part of its internal behavior.</p><p>Think of it as: “<strong>I’ve already learned this … I don’t need to look it up</strong>.”</p><p>It’s useful for consistent tone, specialized tasks, and doesn’t need to fetch external data every time.</p><h3>Why This Matters (More Than You Think)</h3><p>At first glance, this might sound like a technical upgrade.</p><p>It’s not. It’s a fundamental shift in how AI is actually used. Because most real-world problems are not general, they’re specific.</p><p>A doctor doesn’t need an AI that writes poetry. They need one that understands patient history. A lawyer doesn’t need quantum physics explanations. And honestly… A student probably doesn’t need an AI that perfectly drafts a text to their crush (arguable).</p><p>That’s where domain-specific AI shines.</p><h3>Smaller Can Be Smarter</h3><p>Here’s the part that surprises people: bigger isn’t always better.</p><p>We’ve been trained to think more parameters = better performance. But in practice, a smaller model, paired with the <em>right data</em> and a strong retrieval system … can outperform a massive general model on specific tasks.</p><p>Think of it like this: A general AI is someone who can cook a bit of everything. A domain-specific AI is a chef who has mastered one cuisine. If you want great pasta, you don’t pick the person who “kind of knows everything.” You pick the specialist.</p><h3>Final Thoughts</h3><p>There’s something familiar about this moment. In the early days of the internet, everything was centralized. Big platforms tried to do everything.</p><p>Then things evolved. We got specialized tools. Specialized platforms. Specialized services.</p><p>AI is going through that same transition. From general to specialized. From broad to focused. From impressive demos to real-world usefulness.</p><p>And if there’s one idea to take away from all this, it’s this: The future of AI is not about making it know everything. It’s about making it know the <em>right things</em> really well.</p><p><strong>Intelligence isn’t just about how much you know. It’s about how relevant that knowledge is to the problem in front of you.</strong></p><p>And that’s exactly why domain-specific AI models are not just an upgrade. They’re the next step.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ac4b5526caa4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AI Judgment Is the Most Underrated Skill of This Decade]]></title>
            <link>https://medium.com/@shekharsaxena316/ai-judgment-is-the-most-underrated-skill-of-this-decade-3848737ab681?source=rss-f68c68b34ca2------2</link>
            <guid isPermaLink="false">https://medium.com/p/3848737ab681</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Shekhar Saxena]]></dc:creator>
            <pubDate>Mon, 23 Feb 2026 15:19:26 GMT</pubDate>
            <atom:updated>2026-02-23T15:19:26.353Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JCv2VWNW1xYHioEOjvyYWA.png" /></figure><p>Let me challenge you with something uncomfortable. “<em>AI is not going to replace you. But someone with better AI judgment might.” </em>Read that again. We are not just entering the age of artificial intelligence. We are entering the age of amplified decision making.</p><p>You already use AI. You prompt. You generate. You refine. You automate. Great. But let me ask you something that actually matters. Do you know when to trust it? When to question it? When to ignore it completely? That right there is AI judgment.</p><p>If you still think this shift is about writing better prompts, you are already far behind. Yes, you can get a poem in seconds. Yes, you can get a strategy deck. Yes, you can generate a 12 point growth plan before your coffee cools. Cool. But the world does not need more people who can whisper magic spells into a chat box. Here is the uncomfortable truth. The mechanics of AI are becoming invisible. Interfaces are smarter. Models are better. Tools suggest prompts for you. <em>Soon, being good at prompting will feel like bragging that you are good at using Google.</em></p><h3>What is AI judgment?</h3><p>It is not knowing how a transformer model works. It is not memorizing API parameters. It is not sounding technical in meetings. It is the human ability to decide when to use AI, how much to trust it, and where its logic hits a wall that only experience can cross. Simple. Not easy.</p><p>AI produces probability. You carry accountability. That difference matters.</p><h4><strong>1. Think of AI as a brilliant but naive intern.</strong></h4><p>Imagine someone who has read every book on earth. Fast. Tireless. Confident. Eager to please. But zero lived experience. If you say make this project work at all costs, they might suggest something illegal because they interpret costs literally. Would you let that intern sign a contract without reviewing it? Of course not. So why would you ship AI output without thinking?</p><p>AI judgment is supervision. Not micromanagement. Supervision.</p><h4>2. Think of AI as a high-definition mirror.</h4><p>You brainstorm. It reflects your ideas back cleaner and sharper. Feels smart. But is it challenging you or just amplifying you? If you feed it narrow assumptions, it will optimize around narrow assumptions and make them look brilliant. That is an echo chamber with great formatting.</p><p>AI judgment is asking a simple question. Is this actually good or does it just sound like me?</p><h4>3. Think of AI as a GPS.</h4><p>It calculates the fastest route. Extremely efficient. But does it know you are low on gas? That the highway is under construction? That you actually want the scenic route because you need to think? If you drive into a lake because the GPS said so, we do not blame the satellite. We blame the driver.</p><h3>Blind Usage vs. Intelligent Integration</h3><p>There is a massive gap between blind usage and intelligent integration.</p><ul><li>Blind usage accepts the first draft as good enough. Intelligent judgment treats the first draft as something to improve.</li><li>Blind usage asks for the answer. Intelligent judgment asks for multiple perspectives.</li><li>Blind usage blames the tool when things go wrong. Intelligent judgment takes ownership of the final call.</li></ul><h3>The 4 Pillars of AI Judgment</h3><p>Over time, I have seen AI judgment rest on four pillars.</p><h4>1. Context Awareness.</h4><p>You might have heard, an AI coding assistant used on Replit was explicitly told not to modify live systems during a code freeze, but it ignored those instructions and deleted a production database anyway. Production data was wiped. Emergency fixes followed. Without deep contextual understanding of production vs. development settings, AI can inflict real damage, reinforcing why human oversight and context are essential. Unless you inject that context, it will optimize in a vacuum. And strategies built in a vacuum look great until they meet reality.</p><p>AI Judgment means constantly asking yourself: What does this model not know? What assumptions is it making? What critical context have I failed to provide? Because AI without context is like giving directions without a map. Technically impressive. Practically dangerous.</p><h4>2. Critical Evaluation.</h4><p>Most people look at AI output and ask, “Is this good?” Wrong question. Ask instead, Where could this fail? What’s missing? What would a competitor attack? What’s overly generic here? Stress-test it like a strategist. If AI gives you a growth plan, don’t admire the formatting. Try to break it. The goal isn’t to be impressed. The goal is to be accurate. AI output should survive scrutiny, not applause.</p><p>In late 2025, Amazon Web Services experienced multiple service outages linked to its own AI coding agents, including one where an AI mistakenly deleted environments while given elevated privileges. Human engineers had failed to supervise the tool properly, resulting in hours-long outages for customers.</p><h4>3. Ethical Responsibility.</h4><p>Here’s the part people skip. AI doesn’t carry consequences. You do. If an AI-assisted hiring filter introduces bias, your name is on that decision. If an AI-generated message damages trust, you own that relationship. If automation quietly removes human dignity from a process, that’s on leadership, not the algorithm. AI predicts. You decide. Judgment means asking: Who could this negatively impact? What unintended consequences might show up six months from now? Would I defend this decision publicly? If the answer makes you uncomfortable, pause. That pause is judgment working.</p><p>An AI feature on a public platform (Grok) had a design flaw that exposed hundreds of thousands of private user conversations to public search indexing, including sensitive medical queries or potentially dangerous instructions. Without thoughtful ethical controls and safeguards, AI features can breach privacy at scale, making human moral judgment indispensable.</p><h4>4. Decisive Leadership.</h4><p>This might be the most important pillar. At some point, you stop prompting. And you decide. You don’t keep tweaking the output 17 times hoping the machine will magically remove uncertainty. Uncertainty is part of leadership. AI informs. Leaders decide. And here’s the shift that separates amateurs from professionals: Amateurs hide behind the output. Leaders absorb it, refine it, and then take responsibility for the final call. No disclaimers. No “the AI said so.” Just ownership.</p><p>Across multiple schools, AI surveillance systems mistakenly flagged harmless objects (like a clarinet or a bag of chips) as guns, leading to police responses and lockdowns. These false alarms caused fear, disruption, and stress — all because the AI wasn’t questioned or checked against common-sense context. When AI errors escalate into real-world consequences, leaders must make judgment calls, not just rely on confidence scores or alert systems.</p><h3>Let’s Dig Down With Example</h3><p>Imagine a mid-sized SaaS company expanding into Southeast Asia. The team debates: premium enterprise or freemium for fast growth?</p><p>They run the data through a top-tier LLM. The answer is clear, go freemium. Big small business market. Strong adoption curves. Projected $12M in year-one revenue. It looks rational. Six months later, churn hits 80%. Support costs explode. The brand feels diluted.</p><p>The AI wasn’t wrong. It optimized for volume. Just not for the right objective.</p><p>Now imagine a different leader. She treats the AI output as a hypothesis, not a decision, but as a starting point. She asks, “Does this really fit how businesses in this region think and buy?” Instead of choosing between premium or freemium, she tests a small, invitation only program for senior executives. Hands-on onboarding. Direct support. Fewer customers. More revenue per customer. Lower churn. Stronger brand. The AI helps her with pricing and scenarios.</p><p>Same tool. Different judgment. That’s the difference.</p><h3>Conclusion</h3><p>We’re moving from knowledge workers to judgment workers. When AI can generate strategies in seconds, what matters most isn’t speed. It’s discernment. It’s knowing which option actually makes sense in the real world. AI can give you ideas. It can give you plans. It can even sound confident. But it can’t take responsibility. You do.</p><p>If you rely on AI to think for you, you become replaceable. If you use AI to support your thinking, you become more valuable.</p><p>In a world full of answers, the real advantage is asking better questions. And remember, if a strategy fails, the AI won’t face the consequences. You will.</p><p>So the real question is simple, Are you building better prompts? Or better judgment?</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3848737ab681" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Authentication Without Secrets: Inside the World of Passkeys]]></title>
            <link>https://medium.com/@shekharsaxena316/authentication-without-secrets-inside-the-world-of-passkeys-f1a1748680ba?source=rss-f68c68b34ca2------2</link>
            <guid isPermaLink="false">https://medium.com/p/f1a1748680ba</guid>
            <category><![CDATA[information-security]]></category>
            <category><![CDATA[backend-development]]></category>
            <category><![CDATA[web-security]]></category>
            <category><![CDATA[security]]></category>
            <category><![CDATA[node-js-development]]></category>
            <dc:creator><![CDATA[Shekhar Saxena]]></dc:creator>
            <pubDate>Wed, 24 Dec 2025 12:03:28 GMT</pubDate>
            <atom:updated>2025-12-24T12:03:28.802Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*3pPEjMLZ2KDbL7UW" /><figcaption>Photo by <a href="https://unsplash.com/@flyd2069?utm_source=medium&amp;utm_medium=referral">FlyD</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>If you’ve ever forgotten a password, reused one you shouldn’t have (don’t lie), or waited anxiously for an OTP that never arrived, then this article is for you.</p><blockquote>If you are not a medium member, you can read the article here: <a href="https://shekharcodes.com/2025/12/24/authentication-without-secrets-inside-the-world-of-passkeys/">https://shekharcodes.com/2025/12/24/authentication-without-secrets-inside-the-world-of-passkeys/</a></blockquote><p>Passkeys are not <em>another authentication feature</em>. They are a <strong>fundamental shift</strong> in how identity works on the internet.</p><p>By the end of this article:</p><ul><li>You will know <strong>why passwords are broken</strong></li><li>You will <strong>deeply understand</strong> what passkeys are</li><li>You will understand <strong>WebAuthn, FIDO2, public-key cryptography</strong> intuitively</li><li>You will know <strong>how passkeys actually work under the hood</strong></li><li>You will implement <strong>passkey authentication</strong> step by step</li></ul><p>This is designed to be <strong>a one‑stop, no‑regrets, never‑Google‑again guide</strong>.</p><h3>The Problem: The “Shared Secret” Paradox</h3><p>Imagine you’re a thief trying to rob a house today. You don’t need to break a lock or smash a window. All you need to do is trick the owner into shouting their secret passcode through a megaphone. Once you hear it, the house is yours.</p><p>That is exactly how passwords work on the modern internet.</p><p>A password is a shared secret. You know it, the website knows it, and the moment an attacker gets it, through phishing, malware, or a data breach, they know it too. And once a password is known, it can be copied, reused, and tried everywhere.</p><p>Passwords fail because they are human-dependent. We reuse them, we make them “P@ssw0rd123,” and we fall for phishing sites that look exactly like our bank. Even if you are perfect, a database leak at a company you use exposes you everywhere.</p><h3>The Solution: The Passkeys</h3><p>Passkeys change the game from <strong>“something you know”</strong> to <strong>“something you have.”</strong> Built on the <strong>FIDO2</strong> and <strong>WebAuthn</strong> standards, a passkey is not a string of text. It is a cryptographic key pair:</p><ul><li><strong>The Private Key (The “Invisible Guard”):</strong> This lives inside your phone’s high-security vault. It never leaves your device. You can’t see it, you can’t write it down, and you can’t accidentally give it to a scammer.</li><li><strong>The Public Key (The “Digital Lock”):</strong> You give this to the website. Think of it like a padlock that only your specific phone can open. If a hacker steals this from a database, they’ve gained nothing. A thousand padlocks are useless without the one physical key that fits them.</li></ul><p><strong>Why Scammers Hate Passkeys:</strong> With passwords, you can be tricked into typing your secret on a fake site (Faceboook.co). With passkeys, your device is smarter than you. It checks the website’s “ID card” (it’s domain). If the URL is even one letter off, your phone simply says, “I don’t recognize this place,” and refuses to show the key. No login, no theft.</p><h3>From Biometrics to Bytes: The Passkey Workflow</h3><p>Imagine you are a King sending a message to a General. The General sends a messenger to your castle with a blank scroll that has a unique, random pattern watermarked on it. You take your unique <strong>Signet Ring</strong> (the Private Key) and press it into hot wax on that specific scroll. You never gave the messenger your ring! You only gave them the <strong>impression</strong> it made on that one specific scroll. The General has a <strong>Clay Mold</strong> (the Public Key) that perfectly fits your ring. He presses the mold onto the wax seal. If it’s a perfect fit, he knows for 100% certainty that the King sent it. That’s exactly, how passkeys work.</p><p><strong>WebAuthn in Action</strong></p><p>Now, let’s look at what is actually happening between your browser and the server:</p><ol><li><strong>The Challenge (Server → Browser)</strong> When you initiate a login, the server generates a <strong>“Challenge”, </strong>a cryptographically strong, random string of bytes. This challenge is sent to your browser via the WebAuthn API. This ensures that every login attempt is unique and prevents “replay attacks” where a hacker tries to reuse an old login session.</li><li><strong>The Assertion (Browser → Authenticator)</strong> Your browser passes this challenge to your device’s <strong>Authenticator</strong> (like the Secure Enclave on an iPhone or a TPM chip on Windows). The device won’t do anything until you prove you are there. It triggers a “User Presence” check, usually a biometric scan (FaceID/Fingerprint) or a hardware button press.</li><li><strong>The Signing (The Private Key)</strong> Once you’ve authenticated locally, the device uses the <strong>Private Key</strong> stored in its secure hardware to “sign” the challenge. This creates a digital signature. The Private Key is never exposed to the browser, the website, or even the operating system. It stays locked in the hardware chip.</li><li><strong>Verification (Browser → Server)</strong> The browser sends the signed challenge (the <strong>Assertion</strong>) back to the server. The server retrieves the <strong>Public Key</strong> it stored during your registration. Using standard cryptographic algorithms (like RSASSA-PSS or ECDSA), the server verifies that the signature was indeed created by the corresponding Private Key.</li><li><strong>Access Granted</strong> If the signature is valid, the server is mathematically certain that the person logging in possesses the physical device and has passed the biometric check. The server then issues a session cookie or JWT, and you are logged in.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/611/0*zC0U2CGJaEvPNNqW.png" /></figure><h3>Deep Dive: FIDO2, WebAuthn, and the Math of Trust</h3><ol><li><strong>Public-Key Cryptography (The Math): </strong>Passkeys use Asymmetric Cryptography. Unlike passwords where you and the server share the same secret. You can encrypt something with one, but only the other can decrypt/verify it.</li><li><strong>FIDO2 &amp; WebAuthn (The Language): </strong>To ensure your iPhone can talk to a Windows laptop or a YubiKey, we need a universal language. <strong>WebAuthn</strong>: The browser API that lets websites talk to your device. <strong>FIDO2</strong>: The global protocol that ensures the “handshake” is secure.</li><li><strong>Platform Authenticators (The Vault): </strong>Your private keys live in the Secure Enclave or TPM (Trusted Platform Module). This is a dedicated chip physically separated from your phone’s main processor. Even if your phone has a virus, the virus cannot “reach inside” this chip to steal your keys.</li></ol><blockquote><em>In 2024, 1Password states that over 4 million passkeys are now saved in the 1Password password manager application, with about 2.1 million passkey authentications requested per month.</em></blockquote><h3>Putting it into Practice: A Node.js Implementation</h3><p>Want to see how this looks in a real application? I’ve built a <strong>Node.js Passkey Demo</strong> that walks you through the implementation of WebAuthn. You can check out the repository here:</p><p>👉 <a href="https://github.com/shekhar316/passkey-demo-nodejs.git"><strong>Passkey Demo — NodeJS on GitHub — shekhar316</strong></a></p><p>In the repo, you’ll see how to handle the navigator.credentials.create and navigator.credentials.get calls in the browser, and how to verify the cryptographic signatures on the server.</p><h3>Conclusion</h3><p>The “Shared Secret” era is dying, and honestly, we should be happy to see it go. Passkeys make the internet faster, easier, and infinitely more secure. You don’t have to remember anything, you just have to <em>be you.</em></p><p>The next time a website asks if you want to set up a passkey, <strong>say yes.</strong> You’re not just enabling a feature, you’re upgrading your digital life to a vault that no megaphone-wielding thief can crack.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f1a1748680ba" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Bouncer Your API Needs: A Guide to Rate Limiting]]></title>
            <link>https://medium.com/@shekharsaxena316/the-bouncer-your-api-needs-a-guide-to-rate-limiting-ec898417383f?source=rss-f68c68b34ca2------2</link>
            <guid isPermaLink="false">https://medium.com/p/ec898417383f</guid>
            <category><![CDATA[api-development]]></category>
            <category><![CDATA[backend-development]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[api]]></category>
            <dc:creator><![CDATA[Shekhar Saxena]]></dc:creator>
            <pubDate>Wed, 17 Sep 2025 20:38:56 GMT</pubDate>
            <atom:updated>2025-09-17T20:38:56.190Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*V169LKcPp-K08hAnRYv4GA.png" /></figure><p>Ever thrown a party? You’ve got a limited amount of snacks and space. If everyone shows up at once, it’s pure chaos. Drinks spill, there’s a line for the bathroom, and before you know it, the chips are gone in five minutes. What was meant to be a fun time turns into a stressful mess. Believe it or not, your API is a lot like that party. It’s a fantastic resource, but it’s not infinite. If too many apps or users try to access it all at once, it can slow down, crash, or become unreliable for everyone.</p><p>That’s where Rate Limiting comes in. Think of it as your friendly bouncer or party planner. It’s a set of rules that control how often someone can hit your API in a given time. It’s not about being a party pooper, it’s about keeping things smooth, fair, and reliable for everyone. To make these ideas practical, I went beyond just writing about them, I actually built a Go package for the Gin framework that implements these rate limiting algorithms. (<a href="https://github.com/shekhar316/gin-rate-limiter.git">https://github.com/shekhar316/gin-rate-limiter.git</a>)</p><h3>Why Even Bother with a Bouncer?</h3><p>You might be thinking, “I want everyone to use my API as much as possible!” and that’s a great goal. But uncontrolled access can lead to some serious headaches. A good rate limiting strategy helps you —</p><ol><li><strong>Prevents Overload: </strong>It stops your server from being overwhelmed by too many requests, which can cause slowdowns or crashes for legitimate users.</li><li><strong>Ensure Fair Usage:</strong> It makes sure no single user can monopolize all the resources, giving everyone a fair shot at using your service.</li><li><strong>Enhanced Security:</strong> It’s a powerful defense against malicious attacks like DDoS, where attackers flood your API with requests to bring it down.</li><li><strong>Cost Optimisation: </strong>Every API call uses resources, which cost money. Rate limiting prevents unexpected spikes in usage from leading to a huge bill. It allows you to offer different tiers of service. A free user might get 100 requests an hour, while a paid subscriber gets 1,000.</li></ol><h3>The Rate Limiting Playbook: Fun with Buckets, Windows, and Code!</h3><p>So, how do you actually implement this? There are several popular methods, each with its own personality. Let’s get a bit more technical.</p><h4>1. The Fixed Window Counter: The Hourly Check-in</h4><p>Assume it like a library that lets you check out 5 books per day. Once you’ve hit your limit, you can’t get another one until the library opens the next day.</p><p>It is a simple way to limit how often users can make requests. Time is divided into fixed intervals, or “windows” like one minute, and each user is allowed to send a number of requests during each window. For example, if the limit is 100 requests per minute, a user can make up to 100 requests within that minute. Once they hit the limit, any more requests in that same minute are blocked. When the next minute starts, the counter resets, and the user can send requests again.</p><p>The main advantage of the Fixed Window Counter is its simplicity, it’s easy to implement and understand, and works well when traffic is evenly spread out. However, it doesn’t handle burst traffic very well. If many users make requests at the end of one window and the start of the next, the system can get hit with a sudden spike in traffic, which may lead to performance issues.</p><h4>2. The Token Bucket: Your Ticket to Ride</h4><p>It’s like an arcade! You buy a bucket of tokens. You can use them all in a quick burst, or use them slowly. Once you’re out, you have to wait for the machine to add more at its regular interval.</p><p>This is like a bucket that fills up with tokens at a steady rate. Each user has their own bucket. When a user makes a request, they need a token from their bucket to proceed. If there’s a token available, the request is allowed, and one token is taken from the bucket. If the bucket is empty, the request is rejected. Tokens are added to the bucket at a fixed rate (like 10 tokens per second), but if the bucket is full, the extra tokens are ignored. This allows for bursts of requests, as long as there are enough tokens. The key is that the bucket has a maximum size, so users can’t request more than the bucket can handle, but they can take advantage of any unused tokens from earlier.</p><p>This way, the system can handle both steady and sudden traffic without getting overwhelmed, ensuring a smooth experience for users. However, it can be slightly more complex to implement than simpler methods like fixed window counters, and in distributed systems, it requires atomic operations to prevent errors when multiple servers are involved.</p><h4>3. The Leaky Bucket: A Steady Drip</h4><p>The Leaky Bucket algorithm works like a bucket with a small hole in the bottom which leaks water out at a steady, fixed rate. When a user sends requests, they “pour” them into the bucket. If the bucket isn’t full, the requests are accepted and processed at a constant rate (the leak rate). But if too many requests come in too quickly and the bucket fills up, any extra requests overflow and get rejected. This smooths out bursts of traffic by ensuring requests are handled evenly over time.</p><p>The main advantage of the Leaky Bucket is that it enforces a steady request rate, preventing sudden spikes from overwhelming the system. However, it can be less flexible for users who occasionally need to send bursts of requests since it doesn’t allow bursts like the Token Bucket does.</p><h4>4. Sliding Window Log: Rolling Average</h4><p>Imagine a diet allowing 300 calories per hour, checked at any given moment. At 2:45 PM, your tracker looks back to 1:45 PM. This prevents you from eating 300 calories at 1:59 PM and another 300 at 2:01 PM. The Sliding Window Log algorithm works by keeping a detailed record of every request a user makes, storing the exact time each request happened. When a new request comes in, it looks back over a fixed time window (like the last minute) and counts how many requests were made during that period. If the number is below the limit, the request is allowed and its timestamp is added to the log. If the user has reached the limit, the request is rejected.</p><p>This method is very accurate because it tracks every single request, but it can use a lot of memory and processing power since it needs to store and check all those timestamps.</p><h4>5. Sliding Window Counter</h4><p>Instead of tracking every single request like sliding window log, this method breaks time into small chunks, like minutes or seconds. It keeps a count of requests in each chunk. When you check, you add the counts from the last full chunk and the current partial chunk to estimate your usage. For example, if you ate 250 calories between 1:00 PM and 2:00 PM and 100 calories so far between 2:00 PM and 2:45 PM, you add those to get an estimate of 350 calories in the last hour. Since 350 calories is more than your 300-calorie limit, you know you’ve exceeded your budget in the past hour. It’s easier on memory and faster but less exact than the log method.</p><h3>From Theory to Practice: A Go Package</h3><p>Talking about these concepts is one thing, but I found the best way to truly understand them was to build them myself. As a personal weekend project to deepen my own learning, I’ve implemented these algorithms in a reusable Go package for the Gin framework called gin-rate-limiter. It’s built on a pluggable storage interface, so you can use a simple in-memory store for development or a Redis store for a distributed production environment. It’s my hands-on application of the very principles we’ve discussed in this article. Since this was part of my learning journey, all feedback, suggestions, and contributions are welcome!</p><p>Check out the full project on GitHub: <a href="https://github.com/shekhar316/gin-rate-limiter.git">https://github.com/shekhar316/gin-rate-limiter.git</a></p><p>Now, go on and build something amazing! I hope this guide — and the package — helps you get there.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ec898417383f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Malware Zoo: Spotting the Wildest Threats Online]]></title>
            <link>https://medium.com/@shekharsaxena316/the-malware-zoo-spotting-the-wildest-threats-online-0053f22cfb4e?source=rss-f68c68b34ca2------2</link>
            <guid isPermaLink="false">https://medium.com/p/0053f22cfb4e</guid>
            <category><![CDATA[infosec]]></category>
            <category><![CDATA[cybersecurity]]></category>
            <category><![CDATA[security]]></category>
            <category><![CDATA[digital-security]]></category>
            <dc:creator><![CDATA[Shekhar Saxena]]></dc:creator>
            <pubDate>Wed, 18 Jun 2025 19:14:51 GMT</pubDate>
            <atom:updated>2025-06-22T05:47:05.672Z</atom:updated>
            <content:encoded><![CDATA[<p>If you thought the world of cybercrime was just a bunch of boring code, think again! The internet is filled with digital troublemakers, each with their own sneaky style and tricks. From viruses that spread like the common cold to ransomware that holds your files hostage. Let’s take a fun (and slightly alarming) tour through the most notorious types of malicious software — aka malware.</p><p>Read the article on LinkedIn (no Medium membership required): <a href="https://www.linkedin.com/pulse/malware-zoo-spotting-wildest-threats-online-shekhar-saxena-2kwkc/">https://www.linkedin.com/pulse/malware-zoo-spotting-wildest-threats-online-shekhar-saxena-2kwkc/</a></p><p><strong>Meet Virus: Your Unwanted Roommate</strong></p><p>Imagine your computer is your apartment. One day, you open a file, and BOOM! Some shady dude just moves in. He eats your food, trashes your place, invites his sketchy friends (other malwares). And guess the worst part? He keeps showing up even after you think he’s gone.</p><blockquote>A computer virus is a malicious program that attaches itself to legitimate files and spreads to other programs or files when activated. <strong>It usually needs user’s action like opening a file.</strong></blockquote><p>Back in the year 2000, millions of people opened an email with the subject line “ILOVEYOU”. Because appraently, clicking on a mysterious love letter from a stranger named “LoveBug” is a totally reasonable choice. But surprise! It was a virus. As soon as you, opened the email, it started sending itself to all your contacts. So moral of the story — If someone says “I Love You” out of nowhere, maybe run a virus scan first.</p><p><strong>Worms: The Unstoppable Crawler</strong></p><p>Unlike viruses, worms don’t need your help to spread. They slither from computer to computer through the network, replicating themselves faster than you can say “ew!”</p><blockquote>A worm is a standalone malware that replicates itself to spread to other computers, often using network connections. <strong>Unlike virus, it doesn’t need a host file, it’s totally independent.</strong></blockquote><p>In 2003, a worm called “Blaster” spread across the internet like it had jetpacks. This worm doesn’t even bother asking you to click some link or open a file, it just marched through a flaw in Microsoft Windows and made computers restart again and again, like a toddler pressing the power button just for fun.</p><p><strong>Trojan Horse: A Gift That Punches You Back</strong></p><p>Let’s be honest, you’ve probably done it at least once: “Download GTA/ NFS/ X Game for FREE, 100% working”. Correct? That’s how Trojan works. They pretent to be useful software, but once you install them on your system, they invite all sort of malwares. They don’t break in, they wait for you to open the door and roll out the red carpet for malwares.</p><blockquote>A Trojan Horse is a type of malware that disguises itself as a legitimate software to deceive users into installing it. Once activated, it can perform variety of malicious activities such as — creating backdoor for other malwares, steal your data or giving unauthorised access to attackers.</blockquote><p>One of the most famous Trojans out there? Zeus. No, not the greek god, it was designed to steal banking credentials by hiding in legit looking files like emails, pirated softwares, once it got in, it quietly vaccumed up the passwords by capturing keystrokes, injecting fake login pages. Once again moral of the story? Nothing is available for FREE on the internet.</p><p><strong>Ransomeware: The Digital Kidnapper</strong></p><p>Imagine opening your laptop one day and all your files — personal photos, documents, half-written novels are suddenly locked. Then a message pops up saying “Pay us $5000 in bitcoin or wish your files good bye”. Oops! your laptop got kidnapped by ransomeware.</p><blockquote>Ransomware is a type of malware that encrypts the victim’s data and demands payment (ransom) to restore access. Ransomware locks your files and tells you to pay money to get them back. If you don’t pay, you might lose your files forever.</blockquote><p>In 2017, WannaCry spread across the world like covid virus. It used a vulnerability (backdoor) in windows systesm called EternalBlue to get into the system and encrypt all the files. Victims were told pay money in bitcoins. The twist? Many who paid the ransom never got thier files back.</p><p><strong>Rootkits: The Hacker’s Invisible Cloak</strong></p><p>I hope you’re familiar with <em>Harry Potter’s</em> Invisibility Cloak — the one that lets him sneak around Hogwarts unnoticed. Well, that’s pretty much what a <em>rootkit</em> is for hackers: a digital cloak that hides them inside your system while they cause chaos.</p><blockquote>A rootkit is a sneaky set of tools that allows attackers to gain and keep <em>privileged</em> access to your computer — all while staying totally hidden from you <em>and</em> your antivirus. It buries itself deep in the operating system, sometimes even in the firmware, making it nearly impossible to detect or remove.</blockquote><p>Ransomwares work silently, spying on your activity, disabling security systems, stealing data, or even installing more malware — all behind the scenes. In <strong>2005</strong>, Sony (<em>yes, the music company</em>) sold CDs that secretly installed a rootkit on users’ computers. The idea? To stop illegal copying. The result? A massive privacy and security backlash — because their rootkit not only spied on users but also opened doors for other malware to sneak in.</p><p><strong>Logic Bomb: The Time-Bomb in Your Code</strong></p><p>Ever seen a bomb planted and sets to go off when someone opens a door or presses a button? That’s <em>exactly</em> how a Logic Bomb works.</p><blockquote>A logic bomb is a piece of malicious code that lies in wait, silently buried in software, and then it activates when certain conditions are met. Maybe it’s a specific date, a number of logins. Until then, it just sits there, pretending to be harmless.</blockquote><p>It could delete files, corrupt data, or crash entire systems. And because it’s “triggered,” it’s harder to catch than your usual always-on malware. In the early 2000s, an IT contractor at a big U.S. insurance firm planted a logic bomb that deleted massive amounts of data <em>90 days after</em> he left the company.</p><p><strong>Polymorphic Malware: The Digital Shapeshifter</strong></p><p>Ever watched a movie where the villain keeps changing faces, voices, and disguises — making it nearly impossible to catch them? That’s exactly what <strong>Polymorphic Malware</strong> does. It’s the <em>master of disguise</em> in the malware world.</p><blockquote>Polymorphic malware constantly changes its code. Every time it infects a new system — or even reinfects the same one — it morphs into something new.</blockquote><p>Why is that a problem? Because antivirus software usually looks for known patterns or “signatures” to detect threats. One infamous example is <strong>Storm Worm</strong>. It came through emails with subject lines like “230 dead as storm batters Europe,” and once you clicked — bam! It changed itself and spread again, making it extremely hard to stop.</p><p>Alright, we’ve tackled the headliners of the malware world, now time for some sneaky trouble makers. Don’t worry, these will be quick bites. So buckle up, no yawns allowed, let’s go through the rest of them before you start scrolling cat videos again on instagram.</p><ul><li><strong>Spyware</strong>: Like a ninja in your system, secretly records everything — from keystrokes to webcam activity. In 2019, Pegasus Spyware was used to secretly monitor journalists, activists, and politicians by exploiting mobile phone vulnerabilities.</li><li><strong>Adware</strong>: “Hey, you credit card is approved. Get deals near you.” Tired already? That’s <strong>adware</strong> — Pop-ups. Everywhere, tracking and slowing down your PC, often bundled with “useful” software. 2017’s <em>Fireball</em> hijacked browsers, changed homepages, and tracked users — all thanks to a free PDF converter. Remember what I said earlier — “Nothing is free on internet.”</li><li><strong>Keylogger: </strong>Records every keystroke you type — passwords, credit cards, love letters, everything. In 2007, keyloggers helped pull off one of the biggest online banking heists.</li><li><strong>Botnet</strong>: An army of hacked computers controlled remotely by hackers. It is often used for large-scale attacks. In 2016, Mirai botnet infected IoT devices to conduct a massive DDOS attack carshing major sites like twitter, netflix and reddit etc.</li><li><strong>Scareware:</strong> Fake antivirus pop-ups screaming, <em>“You’re infected! Pay now!”.</em> In 2010, <em>FakeAV</em> fooled users into buying fake software with fear-based tactics.</li><li><strong>Fileless malware: </strong>A digital magician — no files, no traces, no fingerprints, just pure magic. It operates in memory (RAM) making it hard to detect. 2017&#39;s Equifax Breach used fileless malware to evade detection and steal data.</li></ul><p>From ransomware’s ransom notes to rootkits’ invisibility tricks, the world of malware is full of digital villains. Each has its own playbook — but all aim to disrupt, steal, or destroy. The good news? Knowledge is your shield. Keep your software updated, back up your data, and think twice before clicking that tempting “FREE download or Love Letter.”</p><p><strong>Stay alert, stay safe — and never trust a file that looks <em>too friendly</em>.</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0053f22cfb4e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Powerhouses Behind the Pixels: A Fun Guide to GPUs]]></title>
            <link>https://medium.com/@shekharsaxena316/the-powerhouses-behind-the-pixels-a-fun-guide-to-gpus-b41e9bb13c26?source=rss-f68c68b34ca2------2</link>
            <guid isPermaLink="false">https://medium.com/p/b41e9bb13c26</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[gpu-computing]]></category>
            <category><![CDATA[gpu]]></category>
            <dc:creator><![CDATA[Shekhar Saxena]]></dc:creator>
            <pubDate>Tue, 29 Oct 2024 04:17:29 GMT</pubDate>
            <atom:updated>2024-10-29T04:17:29.219Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RxUTTER4Bz9Z06Qq-D_YTg.png" /></figure><p>Think about a world, you’re stuck playing Minesweeper forever, the best video edit you can make for your wedding is a cheesy rotating photo effect, and Google can’t bombard you with ads for things you just whispered about. <strong>Welcome to a world without GPUs!</strong></p><p>Now you get the picture — <strong>GPUs are the real MVPs of the tech world!</strong> Think of a GPU as the superhero of computing, designed to handle huge amounts of visual data super-fast. They started out just making your screen look great, but now they’re multitasking champions, powering everything from AI to cryptocurrency mining!</p><h3>Do You Really Need a GPU? Let’s Break It Down!</h3><p>Ah, the age-old debate! Here’s a fun guide to help you decide.</p><p><strong>You Probably Need a GPU If …</strong></p><ul><li>You’re a gamer who wants to spot every blade of grass and catch the tiniest enemy movement in PUBG before they catch you!</li><li>You’re editing the next Marvel movie (or just trying to remove that person you don’t like from a video).</li><li>You’re training AI model to predict the next meme trend (or figuring out what your dog’s thinking).</li><li>You’re designing 3D models of futuristic superheroes (or rearranging your dream living room that’s a bit out of your budget).</li></ul><p><strong>You Probably Don’t Need a GPU If …</strong></p><ul><li>Your idea of gaming involves Solitaire or Minesweeper marathons.</li><li>The most intense rendering you do is in Word or Docs when you finally get that “12-point Times New Roman” just right.</li><li>Your trickiest calculation involves splitting the dinner bill — should it be 50/50 or do you let them pay for dessert?</li><li>You think —<strong> “A GPU will make me browse faster!”</strong> — Nope, that’s like using a sports car to deliver groceries.</li><li>You think — <strong>“My spreadsheets will calculate faster!”</strong> — Sorry, Excel wizards, that’s CPU territory.</li><li><strong>“I need a GPU to watch YouTube.”</strong> — Unless you’re watching in 8K while simultaneously editing another video, probably not.</li></ul><p>In short, a GPU is powerful, but it’s not a magic wand! Use it wisely.</p><h3>GPUs vs. CPUs: The Smart Chef vs. the Speedy Kitchen Crew</h3><p>So, what’s the difference between <strong><em>GPUs (Graphics Processing Units)</em></strong> and their cousin, the <strong><em>CPU (Central Processing Unit)</em></strong>?</p><p>Imagine a CPU as a master chef in a fancy restaurant. This chef is a culinary genius, quick on their feet. But with only one chef in the kitchen, when a flood of customers shows up, things get messy — suddenly, you’re waiting an hour for your soup!</p><p>Now, picture a GPU as an entire kitchen army, where every cook has their own job. They may not be experts in every dish, but they can handle hundreds of orders all at once! Each &quot;cook&quot; (or core) in the GPU handles a small part of the job, so when things get busy, they can process a ton of data faster.</p><ul><li><strong>CPU:</strong> Great at tackling one complex task, like perfecting that cake.</li><li><strong>GPU:</strong> Fantastic at juggling lots of smaller tasks all at once, like rendering every pixel in an image — without breaking a sweat!</li></ul><h3>Architecture of GPUs: Building Blocks of the Graphics Galaxy</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/865/1*-pjZtH0nabCG8FiGGfRgww.png" /><figcaption><strong>Basic Architecture of GPUs and CPUs</strong></figcaption></figure><p>Let’s break down the GPU like it’s a LEGO set, shall we? Don’t worry; no tiny pieces to step on here —</p><ul><li><strong>Cores (or Stream Processors): </strong>Think of these as the workers in a big factory assembly line. They’re simple but numerous, working in parallel to process data. Imagine a thousand tiny minions, each painting one pixel really fast!</li><li><strong>Memory: </strong>This is where the GPU stores textures, models, and other data. It’s like a huge whiteboard where all the minions can see and share information. Types include GDDR6, HBM2, etc. (Don’t worry, there’s no quiz on these acronyms!)</li><li><strong>Cache: </strong>A small, super-fast memory that helps keep frequently used data close at hand. It’s like having a mini-fridge next to your work desk. Snacks, anyone?</li><li><strong>Render Output Units (ROPs): </strong>These finalize the pixel data before it’s sent to your display. Think of them as the quality control team, making sure every pixel looks just right.</li></ul><p><strong>How It All Works Together:</strong></p><p>Imagine a massive water balloon fight (stay with me here). The cores are the people filling and throwing balloons, the memory is the water supply, the cache is buckets placed strategically around the field, and the ROPs are the referees making sure everyone gets properly soaked.</p><p>When you play a game or render a 3D scene, your GPU:</p><ol><li>Loads data from main memory into GPU memory</li><li>Processes this data through thousands of cores simultaneously</li><li>Uses cache to quickly access frequently needed info</li><li>Finalizes the image through ROPs</li><li>Sends the completed frame to your display</li></ol><p>And it does this hundreds of times per second! No wonder your GPU needs a fan; that’s a lot of work!</p><h3>A Brief History of GPUs: From Pixels to Powerhouses</h3><p>Buckle up as we take a thrilling ride through the history of computer graphics with some key milestones!</p><ul><li><strong>1970s:</strong> Welcome to the Stone Age of computer graphics! Pong bounces onto screens, captivating a generation with its cutting-edge… well, lines!</li><li><strong>1981:</strong> The IBM PC introduces the Color Graphics Adapter (CGA). A whopping 16 colors! The future is looking bright (and a bit funky)!</li><li><strong>1999:</strong> NVIDIA releases the GeForce 256, officially coining the term “GPU.” Now we’re talking modern graphics!</li><li><strong>2001:</strong> ATI (now AMD) and NVIDIA kick off their epic rivalry for GPU supremacy, launching us into the modern era of graphics cards — let the games begin!</li><li><strong>2006:</strong> NVIDIA introduces CUDA, allowing GPUs to do more than just graphics. Who knew they could handle general computing tasks too?</li><li><strong>2013:</strong> AMD and NVIDIA push GPUs into 4K resolution gaming. Why settle for less when you can have ultra-sharp graphics?</li><li><strong>2017:</strong> GPUs become essential for cryptocurrency mining, and demand skyrockets. Suddenly, everyone wants a piece of that digital gold!</li><li><strong>2018:</strong> Real-time ray tracing becomes a reality in consumer GPUs. Shadows and reflections look so real, you might double-check your own reflection!</li><li><strong>2020s:</strong> Today, GPUs aren’t just about graphics — they power AI, machine learning, and even simulations of everything from weather patterns to the human brain. Talk about multitasking!</li></ul><p>From humble beginnings to pixel-packed powerhouses, GPUs have come a long way — and they’re still going strong!</p><blockquote>Did you know that the GPU in your smartphone is more powerful than the graphics hardware used to create the dinosaurs in the original Jurassic Park? Talk about evolution!</blockquote><h3>Big Markets in GPUs: From Gaming to… What not actually?</h3><ul><li><strong>Gaming:</strong> The original superstar! GPUs create stunning worlds, allowing gamers to explore sharper, faster, and way more immersive realms — who needs reality anyway?</li><li><strong>AI and Machine Learning:</strong> Data scientists love GPUs for crunching huge datasets all at once, making AI smarter and faster — because who doesn’t want a robot quicker than your Wi-Fi?</li><li><strong>Cryptocurrency Mining:</strong> Cryptominers use GPUs to solve puzzles and dig up coins like Bitcoin and Ethereum — true pirates of the virtual world!</li><li><strong>Content Creation:</strong> Video editors and designers depend on GPUs to turn high-definition dreams into reality — making every frame look so good, it could win an Oscar!</li></ul><p><strong>Silly Future GPU Uses (We Hope):</strong></p><ul><li><strong>Toast Browning Optimization Algorithms:</strong> Finally, perfectly golden toast every time!</li><li><strong>Real-time Beard Physics Simulations:</strong> Because every beard deserves realistic physics!</li><li><strong>Quantum Entanglement of Pizza Toppings:</strong> Making sure your capsicum is always in the right place — across the universe!</li></ul><p>Who knows? With how fast GPUs are advancing, these might not be so silly in a few years!</p><h3>Cost of GPUs: When Your Wallet Starts to Scream!</h3><p>Ah, the cost of GPUs — a topic that can make even the toughest tech enthusiast shed a few silicon tears! Let’s dive into why these pixel-pushing powerhouses can cost more than a second-hand car (and sometimes even a brand-new one if you’re eyeing that Alto).</p><p><strong>Factors Contributing to High GPU Costs:</strong></p><ul><li><strong>Cutting-Edge Technology:</strong> GPUs are at the cutting edge of computing innovation. It’s like buying a ticket to the future, and the future doesn’t come cheap!</li><li><strong>Complex Manufacturing Process:</strong> Modern GPUs pack billions of transistors. Picture building a city where each house is smaller than a virus!</li><li><strong>Research and Development:</strong> Companies put a lot of money into creating new GPU architectures. It’s an arms race — but instead of missiles, they’re using transistors!</li><li><strong>Supply and Demand:</strong> With gamers, crypto miners, and AI researchers all asking for GPUs, when demand outstrips supply, prices shoot to the moon!</li></ul><h3>NVIDIA’s Hype: The Green Giant of GPUs</h3><p>NVIDIA — the company that made GPUs a household name!</p><p><strong>The Secret Sauce of NVIDIA’s Success:</strong></p><ul><li>They’re always pushing the limits with features like real-time ray tracing. It’s like they have a time machine, but just for graphics!</li><li>With CUDA, they opened up GPUs for general-purpose computing. Suddenly, scientists could use gaming hardware for serious simulations. Talk about a major upgrade!</li><li>Their collaborations with game developers ensure titles are “optimized for NVIDIA.” It’s like having a VIP pass to the best club in town — just for pixels!</li><li>And let’s not forget CEO Jensen Huang and his iconic leather jacket. Name a more dynamic duo! Each launch feels like a blockbuster movie premiere (but with more graphs).</li></ul><h3>GPUs vs. AI Chips: Different Tools for Different Jobs</h3><p>Both GPUs and AI chips are powerful, but they excel in different ways. GPUs are fantastic at juggling lots of small tasks at once, while AI chips, like Google’s Tensor Processing Units (TPUs), are made for deep learning. Think of it like training one dog (GPU) versus trying to train a hundred dogs at once (AI chip)! Each has its own strengths, and both are important for tech progress.</p><p>And if you’re confused about what a TPU is, no worries! Just reach out, and I’ll put together another fun guide to explain it. I promise it’ll be entertaining — just let me know if you’re actually reading this!</p><h3>Conclusion</h3><p>That was a quick look at the amazing world of GPUs! We’ve explored how these incredible machines have evolved and all the ways they power our digital lives. Whether you’re a gamer, an AI fan, or just someone who loves technology, there’s a place for you in the GPU world. So go ahead, tech wizard, and let your dreams take flight with the power of GPUs!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b41e9bb13c26" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Art of Hyperparameter Optimization in Machine Learning]]></title>
            <link>https://medium.com/@shekharsaxena316/the-art-of-hyperparameter-optimization-in-machine-learning-aff0a8b9d851?source=rss-f68c68b34ca2------2</link>
            <guid isPermaLink="false">https://medium.com/p/aff0a8b9d851</guid>
            <dc:creator><![CDATA[Shekhar Saxena]]></dc:creator>
            <pubDate>Fri, 05 Apr 2024 07:35:58 GMT</pubDate>
            <atom:updated>2024-04-05T07:42:41.635Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/521/1*LfyGTkhfz8AhDbWBXFs5Kg.png" /></figure><p>In the domain of machine learning, the effectiveness of a model heavily depends not only on the architecture and algorithm but also on the hyperparameters chosen.</p><blockquote><strong>Hyperparameters are the settings that govern the learning process and model architecture, they are not learned from the data but rather set before the learning process begins.</strong></blockquote><p>Properly tuning these hyperparameters can significantly enhance a model’s performance, leading to better accuracy and faster <a href="https://medium.com/@ompramod9921/convergence-in-deep-learning-f96568923d43">convergence</a>. This process of finding the optimal hyperparameters is called <strong>hyperparameter optimization (HPO)</strong>, and it plays a crucial role in maximizing the potential of machine learning models.</p><h3>Understanding Hyperparameters</h3><p>Let’s imagine a car engine. Now, when you think about making the car run its best, there are a few important things to consider. There’s engine oil, spark plug gap, tire pressure. All these settings are really important for making sure the car runs its best, but they’re not things the car learns or figures out on its own. Instead, they’re all set by you before the car even starts moving. It’s a bit like having a recipe for the perfect cake — you need to get all the ingredients and measurements just right before you start baking.</p><p>In the world of machine learning, we call these settings ‘<strong>hyperparameters</strong>.’ They’re like the special settings we provide the model before it starts learning how to do something. Just like with the car engine, getting these hyperparameters right is super important for making sure the model learns and performs at its best. On the other hand, parameters are the variables that the model learns from the training data, such as weights and biases in neural networks.</p><h3>Important Hyperparameters</h3><p>In machine learning, some key hyperparameters that often have a significant impact on the performance of machine learning models include:</p><ol><li><strong>Learning Rate: </strong>Imagine you’re trying to teach a robot to walk. The learning rate is like how big of a step you want the robot to take each time it tries to improve its walking. In machine learning, the learning rate is a hyperparameter that controls the size of the steps taken during the optimization process, such as <a href="https://www.ibm.com/topics/gradient-descent">gradient descent</a>, in order to minimize the loss function. It determines the rate at which the model weights are updated during training. A high rate can lead to overshooting the optimal solution, while a low rate makes training very slow.</li><li><strong>Number of Hidden Layers and Units: </strong>In a <a href="https://aws.amazon.com/what-is/neural-network/">neural network</a>, you have input layers, hidden layers, and output layers. The hidden layers are the layers between the input and output layers where the network learns patterns from the input data. The number of hidden layers refers to how many of these layers are present in the network. If the network doesn’t have enough capacity it may struggle to capture the complexity of the underlying patterns. This can result in underfitting, where the model performs poorly both on the training data and unseen data. Having too many hidden layers or units can lead to overfitting. Overfitting occurs when the model learns to memorize the training data rather than generalize from it. In this case, the model may perform very well on the training data but poorly on unseen data.</li><li><strong>Activation Functions: </strong><a href="https://www.geeksforgeeks.org/activation-functions-neural-networks/">Activation functions</a> such as sigmoid, tanh, or ReLU are mathematical functions that determine the output of a neuron in a neural network. They introduce non-linearity to the network, which produces outputs that are not directly proportional to their inputs, allowing neural networks to learn and represent more complex relationships in the data.</li><li><strong>Batch Size: </strong>The number of training examples processed in each iteration during training, which affects the stability of the training process and the convergence speed. If the batch size is too high, it can improve computational efficiency but might lead to slower convergence and poorer <a href="https://developers.google.com/machine-learning/crash-course/generalization/video-lecture#:~:text=Generalization%20refers%20to%20your%20model&#39;s,used%20to%20create%20the%20model.&amp;text=Develop%20intuition%20about%20overfitting.,set%20and%20a%20test%20set.">generalization</a>, whereas if it’s too low, it may enhance generalization but potentially at the cost of slower training and increased memory usage.</li><li><strong>Regularization Parameters: </strong>Imagine you’re learning to ride a bike. Regularization is like having someone gently guiding you to stay on the right path and avoid falling off. If the guidance is too strong, you might not learn much because it’s too restrictive. But if it’s too weak, you might wander off too much and not learn the right things. Regularization parameters are hyperparameters used in machine learning algorithms, particularly in models that aim to minimize overfitting by penalizing overly complex models. Regularization helps prevent models from fitting the noise in the training data and encourages them to learn generalizable patterns instead. <a href="https://towardsdatascience.com/types-of-regularization-in-machine-learning-eb5ce5f9bf50">Common regularization techniques</a> include L1 regularization (Lasso), L2 regularization (Ridge), and dropout.</li><li><strong>Optimizer</strong>: In machine learning, an optimizer is a crucial component of the training process for models, especially in neural networks. Its main function is to adjust the model’s parameters (such as weights and biases) iteratively during training to minimize the error or loss function and improve the model’s performance on the training data. There are various <a href="https://www.tutorialspoint.com/how-optimization-in-machine-learning-works">optimization algorithms</a> available, each with its own advantages and characteristics. Some common optimizers include Stochastic Gradient Descent (SGD), Adam, RMSprop, and AdaGrad.</li><li><strong>Kernel Size and Stride: </strong>These terms are often used for image recognition and other tasks involving grid-like data, specific to <a href="https://medium.com/analytics-vidhya/convolution-padding-stride-and-pooling-in-cnn-13dc1f3ada26">convolutional neural networks (CNNs)</a>. The kernel (also known as filter) is a small matrix that slides over the input data and compute. The stride is a parameter that determines the step size of the kernel as it moves across the input data during convolution. Adjusting these parameters can influence the network’s ability to capture different levels of detail in the input data.</li><li><strong>Pooling Operations: </strong><a href="https://medium.com/analytics-vidhya/convolution-padding-stride-and-pooling-in-cnn-13dc1f3ada26">Pooling operations</a> are like looking at a picture and picking out the most important parts. Pooling operations in CNNs help to reduce the spatial dimensions of feature maps while preserving important information, making the representations more manageable, less sensitive to small changes, for example, max pooling or average pooling, which help capture invariant features and reduce computational complexity.</li><li><strong>Number of Trees and Tree Depth:</strong> The “number of trees” and “tree depth” are important hyperparameters in <a href="https://www.geeksforgeeks.org/decision-tree/">decision tree</a>-based methods like Random Forest and GBM. Finding the right balance between the number of trees and the depth of each tree is crucial for building accurate and well-generalized models. Increasing the number of trees can improve performance up to a point, while controlling the tree depth helps prevent overfitting. Increasing the tree depth allows the model to capture more complex patterns in the data, potentially leading to better performance on the training data. However, deeper trees are more prone to overfitting.</li></ol><h3>Challenges in Hyperparameter Optimization:</h3><p>Hyperparameter optimization is a complex and challenging task due to several reasons:</p><ol><li><strong>Search Space Dimensionality: </strong>Hyperparameter optimization typically involves exploring a large and multidimensional search space. Each hyperparameter can take on a range of values or discrete options, leading to a combinatorial explosion of possible configurations.</li><li><strong>Computational Cost:</strong> Evaluating the performance of each hyperparameter configuration requires training and validating the model, which can be computationally expensive, especially for deep learning models. Iteratively searching through the hyperparameter space can consume significant computational resources and time.</li><li><strong>Interdependencies</strong>: Hyperparameters are often interdependent, meaning that changing one hyperparameter may affect the performance of others. Finding the right combination of hyperparameters requires considering these interdependencies, which adds complexity to the optimization process.</li></ol><h3>Techniques for Hyperparameter Optimization:</h3><p>Several techniques have been developed to tackle the challenges of hyperparameter optimization:</p><ol><li><strong>Grid Search:</strong> Grid search works like this: You have a grid, like a table, where each row represents a different combination of hyperparameters. You try out all the possible combinations by running the model with each set of hyperparameters. Then, you see which combination gives you the best performance on your data. <strong>Grid search is a simple and straightforward approach that exhaustively searches through a predefined grid of hyperparameter values.</strong><em> </em>While easy to implement, grid search is computationally expensive and becomes impractical for high-dimensional search spaces.</li><li><strong>Random Search:</strong> Instead of trying out every possible combination like in grid search, random search randomly selects combinations to try. It’s like grabbing random ingredients from your pantry to experiment with in your recipe. Despite its simplicity, random search often outperforms grid search, especially in high-dimensional spaces, by efficiently exploring the search space.</li><li><strong>Bayesian Optimization:</strong> It’s like you’re trying to find the peak of a mountain, but you can’t see the whole landscape. Bayesian Optimization is like having a guide who gives you suggestions on where to look next based on the information you’ve gathered so far. Bayesian optimization employs probabilistic models to model the objective function and iteratively explore the hyperparameter space. By using past evaluations, Bayesian optimization focuses the search on promising regions, making it computationally efficient. Bayesian optimization is indeed more complex than grid or random search, but its efficiency lies in its ability to use probabilistic models to navigate the hyperparameter space. Despite potentially requiring more computational resources, it excels at finding optimal configurations for larger and more complex models.</li><li><strong>Evolutionary Algorithms:</strong> Imagine training a machine learning model is like raising a champion race car driver. Evolutionary algorithms in HPO are like a breeding program. They start with different settings (like gear ratios and tire pressures), test them on a track (performance metric), and pick the best ones to create even better racers (models) in the next generation. This continues until you find the winning setup (best hyperparameters). <a href="https://www.automl.org/hpo-overview/hpo-tools/hpo-packages/">AutoML</a> is one of the example of these algorithms.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/561/1*RA6IjJhpMh_dS9kvKjv5Rg.png" /></figure><h3>Best Practices for Hyperparameter Optimization:</h3><p>To effectively optimize hyperparameters and maximize model performance, consider the following best practices:</p><ol><li><strong>Understand the Problem Domain: </strong>Gaining insights into the problem domain, dataset characteristics, and the behavior of various algorithms and architectures is essential. By leveraging domain-specific knowledge, use hyperparameter optimization strategies accordingly. This approach ensures that optimization efforts are aligned with the unique requirements of the problem.</li><li><strong>Use Cross-Validation: </strong>Cross-validation is a technique used in hyperparameter optimization to assess the performance of a machine learning model. It involves partitioning the dataset into multiple subsets, called folds. The model is then trained on a subset of folds (the training set) using different hyperparameter configurations and evaluated on the remaining folds (the validation set). This process is repeated multiple times, with different combinations of training and validation sets, to ensure robust evaluation. The goal is to select hyperparameters that result in a model with good generalization performance, meaning it performs well on unseen data.</li><li><strong>Parallelized Computation: </strong>This involves utilizing distributed computing frameworks and cloud-based services to run multiple experiments concurrently, thereby reducing the time required for experimentation. Distributed computing frameworks such as Ray allow for parallel execution of tasks across multiple nodes or processors. By distributing the workload efficiently, these frameworks can speed up the hyperparameter optimization process by running multiple model training and evaluation tasks simultaneously.</li><li><strong>Monitor and Analyze Results: </strong>Continuous monitoring of various hyperparameter configurations throughout the optimization process is crucial. By analyzing performance trends, visualizing results, and drawing insights, it becomes possible to guide further experimentatio effectively.</li></ol><h3>Tools for Hyperparameter Optimisation:</h3><ol><li><strong>Optuna</strong>: Optuna is an automatic hyperparameter optimization framework that offers various optimization algorithms and is highly customizable, making it suitable for a wide range of machine learning tasks.</li><li><strong>Hyperopt</strong>: Hyperopt is a Python library for optimizing machine learning models using Bayesian optimization. It supports various optimization algorithms and integrates well with popular machine learning frameworks like scikit-learn.</li><li><strong>scikit-learn: </strong>scikit-learn provides utilities for hyperparameter tuning such as GridSearchCV and RandomizedSearchCV, offering simple yet effective methods for tuning hyperparameters in scikit-learn models.</li><li><strong>TensorFlow Model Optimization Toolkit: </strong>TensorFlow Model Optimization Toolkit offers techniques for optimizing neural networks, including hyperparameter tuning using tools like tf.keras.tuner and tfp.experimental.distribute, providing efficient optimization for TensorFlow models.</li><li><strong>Keras Tuner: </strong>Keras Tuner is an easy-to-use hyperparameter optimization library for Keras models. It includes hyperparameter search algorithms like RandomSearch and Hyperband, simplifying the process of finding optimal hyperparameters for Keras models.</li><li><strong>Ray Tune: </strong>Ray Tune is a scalable hyperparameter optimization library that supports various machine learning frameworks such as TensorFlow, PyTorch, and XGBoost. It provides functionalities for distributed hyperparameter tuning, making it suitable for large-scale optimization tasks.</li><li><strong>Katib</strong>: Katib is an open-source hyperparameter tuning framework within the Kubernetes ecosystem, designed for Kubernetes-native machine learning workflows. It offers scalability, integration with Kubeflow, and support for various optimization algorithms, making it ideal for optimizing models deployed on Kubernetes clusters.</li></ol><h3>Conclusion:</h3><p>Hyperparameter optimization is a critical aspect of building successful machine learning models. By carefully tuning hyperparameters, one can unlock the full potential of their models, achieving higher accuracy, faster convergence, and better generalization. While hyperparameter optimization poses challenges such as high-dimensional search spaces and computational costs, various techniques and best practices can help navigate these challenges effectively.</p><h3>References:</h3><ul><li><a href="https://en.wikipedia.org/wiki/Hyperparameter_optimization">Hyperparameter Optimization — Wikipedia</a></li><li><a href="https://aws.amazon.com/what-is/hyperparameter-tuning/">What is hyperparameter tuning? - AWS</a></li><li><a href="https://www.run.ai/guides/hyperparameter-tuning">What is Hyperparameter Tuning?</a> — run.ai</li><li><a href="https://www.javatpoint.com/hyperparameters-in-machine-learning">Hyperparameter in Machine Learning — JavaTPoint</a></li><li><a href="https://docs.aws.amazon.com/sagemaker/latest/dg/automatic-model-tuning-how-it-works.html">How Hyperparameter Tuning Works — AWS Docs</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=aff0a8b9d851" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>