<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Anuranjan Yadav on Medium]]></title>
        <description><![CDATA[Stories by Anuranjan Yadav on Medium]]></description>
        <link>https://medium.com/@anuranjan.yadav?source=rss-27d4fba372b9------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*UuRiGnSkrtEyvYEpfDodVQ.jpeg</url>
            <title>Stories by Anuranjan Yadav on Medium</title>
            <link>https://medium.com/@anuranjan.yadav?source=rss-27d4fba372b9------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 06:57:35 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@anuranjan.yadav/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[I Fine-Tuned a 1.5B Model to Solve Grade 6 Math Olympiad Problems]]></title>
            <link>https://medium.com/@anuranjan.yadav/i-fine-tuned-a-1-5b-model-to-solve-grade-6-math-olympiad-problems-43488f4d8040?source=rss-27d4fba372b9------2</link>
            <guid isPermaLink="false">https://medium.com/p/43488f4d8040</guid>
            <category><![CDATA[hugging-face]]></category>
            <category><![CDATA[synthetic-data]]></category>
            <category><![CDATA[artifical-intellegence]]></category>
            <category><![CDATA[edtech]]></category>
            <category><![CDATA[fine-tuning]]></category>
            <dc:creator><![CDATA[Anuranjan Yadav]]></dc:creator>
            <pubDate>Sun, 12 Jul 2026 14:28:18 GMT</pubDate>
            <atom:updated>2026-07-12T14:28:18.593Z</atom:updated>
            <content:encoded><![CDATA[<p><em>From synthetic data to a working demo — a complete, beginner-friendly fine-tuning pipeline using Unsloth, LoRA, and a lot of Grade 6 maths.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PqQutY7a54OLs28RB_H0DQ.png" /></figure><p>My kid is preparing for the SOF International Mathematics Olympiad (IMO) this year, and somewhere between printing practice papers and checking answers, I had a thought: <strong>could I train a small language model to solve these problems step-by-step, the way a tutor would?</strong></p><p>I’m still learning how model training works, so I treated this as a hands-on project: build the dataset, fine-tune a model, and ship a demo. This post walks through every step, with code. Everything is open:</p><p><a href="https://huggingface.co/anuranjany/grade6-math-assistant">- 🤗 <strong>Model</strong>: [anuranjany/grade6-math-assistant](https://huggingface.co/anuranjany/grade6-math-assistant)</a></p><p><a href="https://github.com/anuranjanyadav/maths-model">- 💻 <strong>Code</strong>: [github.com/anuranjanyadav/maths-model](https://github.com/anuranjanyadav/maths-model)</a></p><p><a href="https://colab.research.google.com/github/anuranjanyadav/maths-model/blob/main/demo.ipynb">- 🎮 <strong>Try it in one click</strong>: [open the demo notebook in Colab](https://colab.research.google.com/github/anuranjanyadav/maths-model/blob/main/demo.ipynb) (free T4 GPU — `Runtime → Run all`)</a></p><p><strong>The pipeline at a glance</strong></p><p>Olympiad PDFs ──► Template engine ──► 1,000 verified questions (20 mock papers)</p><p>▼</p><p>GPT-4o structured-output augmentation</p><p>▼</p><p>~1,240-record seed dataset (JSONL, with CoT)</p><p>▼</p><p>LoRA fine-tune on Qwen2.5-Math-1.5B (Unsloth)</p><p>▼</p><p>Gradio demo + HF Hub</p><p>The most important lesson of this project is hiding in that diagram: <strong>most of the work is data work.</strong> The actual fine-tuning was the easy part.</p><p><strong>Step 1: Generate seed data you can actually trust</strong></p><p>LLM-generated training data has a well-known problem: the answers are sometimes wrong. For maths, that’s fatal — you’d be teaching the model to be confidently incorrect.</p><p>So the first data source is a <strong>deterministic template engine</strong> (`data_engine/generate_question_bank.py`): ~65 parameterized question templates across the full SOF Class 6 syllabus — HCF/LCM, fractions, ratios, mensuration, early algebra, logical reasoning, and more. Each template generates the question <em>and computes the answer with code</em>, so the answer key is correct by construction.</p><p>One template, for example, generates “tank is 3/5 full, remove X litres, now it’s 1/4 full” problems — and computes the capacity from the fractions it just picked. Run it once and you get <strong>1,000 unique MCQs organised as 20 SOF-style mock papers</strong> (Level 1 and Level 2), each with an answer key and step-by-step solutions:</p><p>json</p><p>{</p><p>“source_concept”: “SOF IMO Class 6 Level 2 — Mathematical Reasoning — Fractions (application)”,</p><p>“question”: “A tank is 3/5 full of water. When 14 litres are removed, it becomes 1/4 full. What is the capacity of the tank?”,</p><p>“chain_of_thought”: [</p><p>“The water removed equals (3/5–1/4) = 7/20 of the capacity.”,</p><p>“So 7/20 of capacity = 14 L, giving capacity = 14 x 20/7 = 40 L.”</p><p>],</p><p>“answer”: “40 L”</p><p>}</p><p>Those step-by-step solutions matter: they become the <strong>chain-of-thought supervision</strong> the model learns to imitate.</p><p><strong>Step 2: Add linguistic variety with GPT-4o</strong></p><p>Template data has one weakness — the phrasing is uniform. A model trained only on it risks overfitting to template wording instead of learning the maths.</p><p>The second stage (`data_engine/maths_generator.py`) feeds the generated papers to <strong>GPT-4o with Structured Outputs</strong>, asking it to extract the core concepts and write fresh variations — new scenarios, new numbers, same mathematical substance. Pydantic schemas keep the output machine-parseable:</p><p><em>class MathProblemVariation(BaseModel):</em></p><p><em>question: str</em></p><p><em>chain_of_thought: List[str]</em></p><p><em>final_answer: str</em></p><p><em>completion = client.beta.chat.completions.parse(</em></p><p><em>model=”gpt-4o-2024–08–06&quot;,</em></p><p><em>messages=[…],</em></p><p><em>response_format=DataEngineResponse,</em></p><p><em>)</em></p><p>Because the source PDFs include worked solutions, GPT-4o stays accurate — I hand-verified a sample and every record checked out. After merging and de-duplicating all sources, the final seed dataset is <strong>~1,240 records</strong>, each with a question, a reasoning chain, and an answer.</p><p><strong>Step 3: Fine-tune with Unsloth + LoRA</strong></p><p>For the model itself I picked <strong>Qwen2.5-Math-1.5B</strong> — small enough to train on a single GPU, and already maths-flavoured. Rather than full fine-tuning (updating all 1.5B weights), I used <strong>LoRA</strong>: small trainable adapter matrices injected into the attention and MLP layers, with the base model frozen. [Unsloth](https://github.com/unslothai/unsloth) makes this fast and memory-friendly with 4-bit quantization.</p><p>The config that matters (`training/train_math_model.py`):</p><p><em>model = FastLanguageModel.get_peft_model(</em></p><p><em>model,</em></p><p><em>r=16, lora_alpha=16, lora_dropout=0,</em></p><p><em>target_modules=[“q_proj”, “k_proj”, “v_proj”, “o_proj”,</em></p><p><em>“gate_proj”, “up_proj”, “down_proj”],</em></p><p><em>)</em></p><p>Each training example is formatted with a fixed prompt template — the same one used at inference time:</p><p>Below is a math problem. Solve it step-by-step.</p><p>### Problem:</p><p>{question}</p><p>### Reasoning:</p><p>{chain_of_thought}</p><p>### Final Answer:</p><p>{answer}</p><p>Training hyperparameters: batch size 2 with gradient accumulation 4 (effective batch 8), learning rate 2e-4, linear schedule with 5 warmup steps, `adamw_8bit` optimizer, 60 optimizer steps. On a 1.5B model with 4-bit LoRA, this finishes in minutes, not hours.</p><p><strong>Step 4: Serve it with Gradio</strong></p><p>The inference app (`inference/app.py`) loads the LoRA adapter with Unsloth in 4-bit and exposes a simple textbox UI. One decision worth highlighting:</p><p><em>outputs = model.generate(**inputs, max_new_tokens=512, do_sample=False)</em></p><p>`do_sample=False` forces <strong>greedy decoding</strong>. For creative writing you want sampling; for arithmetic you really don’t — sampling makes the model occasionally “drift” mid-calculation (9 × 7 becomes 64 somewhere down the chain). Deterministic decoding noticeably improved answer consistency.</p><p>`demo.launch(share=True)` gives you a public link instantly — great for showing a friend, but those links expire after a few days. And here’s something I learned the hard way: <strong>hosting a Gradio Space on Hugging Face now requires a PRO subscription</strong> (only static Spaces are free). The zero-cost permanent alternative: a [Colab notebook](https://colab.research.google.com/github/anuranjanyadav/maths-model/blob/main/demo.ipynb) in the repo that loads the model on Colab’s free T4 and launches the same Gradio UI — readers get a <em>faster</em> demo than a CPU Space would have given them, and it never expires.</p><p><strong>Using the model</strong></p><p>The adapter is on the Hugging Face Hub. Here’s the minimal recipe with `transformers` + `peft`:</p><p><em>from transformers import AutoModelForCausalLM, AutoTokenizer</em></p><p><em>from peft import PeftModel</em></p><p><em>base = “Qwen/Qwen2.5-Math-1.5B”</em></p><p><em>tok = AutoTokenizer.from_pretrained(base)</em></p><p><em>model = AutoModelForCausalLM.from_pretrained(base, torch_dtype=”auto”, device_map=”auto”)</em></p><p><em>model = PeftModel.from_pretrained(model, “anuranjany/grade6-math-assistant”)</em></p><p><em>prompt = “””Below is a math problem. Solve it step-by-step.</em></p><p>### Problem:</p><p>The ratio of marbles with Rohan and Soham is 4 : 7. If Soham gives 12 marbles</p><p>to Rohan, they have an equal number. How many marbles do they have in total?</p><p>### Reasoning:</p><p>“””</p><p><em>inputs = tok(prompt, return_tensors=”pt”).to(model.device)</em></p><p><em>out = model.generate(**inputs, max_new_tokens=512, do_sample=False)</em></p><p><em>print(tok.decode(out[0][inputs[“input_ids”].shape[1]:], skip_special_tokens=True))</em></p><p>Two practical tips:</p><p>1. <strong>Keep the prompt template exact.</strong> The model was trained on this format; deviating from it degrades output quality.</p><p>2. <strong>One objective per prompt.</strong> The model struggles with multi-part questions — split “find the area AND the perimeter” into two prompts.</p><p><strong>What I learned</strong></p><p>- <strong>Data quality beats data quantity.</strong> A thousand template questions with code-verified answers taught the model more reliably than any amount of unverified LLM output could.</p><p>- <strong>CoT supervision is the point.</strong> Training on bare question→answer pairs teaches guessing; training on question→reasoning→answer teaches <em>method</em>.</p><p>- <strong>Small models are underrated.</strong> A 1.5B model with a focused dataset handles a narrow domain surprisingly well — and it runs on very modest hardware.</p><p>- <strong>Greedy decoding for maths.</strong> The cheapest accuracy win in the whole project.</p><p><strong>What’s next</strong></p><p>An honest gap in the current version: I haven’t built a proper held-out evaluation set yet, so “it works well” is anecdotal. Next steps: split an eval set by concept (not randomly), score exact-match accuracy on final answers, and try the 7B base for comparison.</p><p>If you’re a parent, a teacher, or just someone who wants a gentle end-to-end fine-tuning project — this whole pipeline is reproducible from the [repo](https://github.com/anuranjanyadav/maths-model). Questions and PRs welcome.</p><p>— -</p><p><em>*If you found this useful, a 👏 on Medium or a ⭐ on GitHub goes a long way.*</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=43488f4d8040" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[2.718: How This One Number Impacts What Your AI Says Next]]></title>
            <link>https://medium.com/@anuranjan.yadav/2-718-how-this-one-number-impacts-what-your-ai-says-next-2df3a23949a0?source=rss-27d4fba372b9------2</link>
            <guid isPermaLink="false">https://medium.com/p/2df3a23949a0</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[generative-ai-solution]]></category>
            <category><![CDATA[softmax-function]]></category>
            <category><![CDATA[eulers-number]]></category>
            <dc:creator><![CDATA[Anuranjan Yadav]]></dc:creator>
            <pubDate>Thu, 19 Mar 2026 12:19:38 GMT</pubDate>
            <atom:updated>2026-03-19T12:23:37.110Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mze_7XhNuMKorCONt9317A.png" /></figure><p>Have you ever wondered how an AI model, like ChatGPT, always seems to find the perfect next word in a sentence? Or how it knows to be more “creative” when you ask it to tell a story?</p><p>The secret often lies in a specific, magical number from mathematics:</p><p><strong><em>e ≈ 2.718</em></strong></p><p>often called <strong><em>Euler’s number</em></strong>. This isn’t just a number you might learn in an advanced math class; it’s the heartbeat behind a lot of modern Artificial Intelligence, especially Large Language Models (LLMs). Let’s dive into what <strong><em>e</em></strong> is and how it powers the coolest features of these AI models, explaining it in a way that anyone, even in secondary school, can understand!</p><p><strong>What exactly is e ?</strong></p><p>So, what is this <strong><em>e</em></strong> number all about? You might be familiar with numbers like <strong>π (Pi)</strong>, which is roughly <strong>3.14</strong>. Just like <strong>π </strong>describes something special about circles, <strong><em>e</em></strong> (which stands for <strong><em>Euler’s Number</em></strong>) describes something special about <strong>growth and probability</strong>.</p><p>It’s often defined in calculus as the unique base for an exponential function (e^x) whose rate of growth is equal to its value.</p><p><strong>How is value of e calculated: The Story of the Magic Dollar</strong></p><p>To understand why <strong><em>e</em></strong> is <strong>2.718</strong> , imagine you have <strong>$1</strong> in a special bank account that gives you <strong>100% interest per year</strong>. The formula for compound interest is:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/372/1*ZzDAvWP0dl2s3Jup4l_B6A.png" /></figure><p>(Where n is the number of times you calculate the interest in a year). Let’s watch what happens to your dollar as we calculate interest more often:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ErRt9tx7F9Yi09huLFQvyA.png" /></figure><p>Notice how the number doesn’t keep growing to infinity? Even if you calculate interest every millisecond, you can’t beat the “speed limit” of nature. That limit is <strong><em>e</em></strong>. In LLMs, we use this same “natural growth” math to help the AI grow</p><p>For LLMs, though, we care less about interest and more about probability. <strong><em>e</em></strong> is special because functions built around it (like <strong>e^x</strong> ) have very useful and “smooth” properties when we use calculus to train the model. But more importantly, it lets us turn the raw scores a model gives for a word into real percentages. Let’s see how!</p><p><strong>e as the Gatekeeper: Choosing the Next Word</strong></p><p>When an LLM is about to generate the next word in a sentence, it doesn’t just know the answer. Instead, its internal math calculates a “score” for every single word in its massive vocabulary. These raw scores are called <strong>Logits</strong>.</p><p>Let’s imagine our LLM is writing the sentence “The cat likes to eat…”. Its vocabulary is: Mice, Yarn, Carrots, The. The model might generate these raw <strong>logit</strong> scores:</p><ul><li>Mice: 4</li><li>Yarn: 2</li><li>Carrots: 1</li><li>The: -1</li></ul><p>How do we turn these into an actual probability? A probability needs to be between 0 and 1, and all options must add up to 1 . This is where <strong><em>e</em></strong> steps in via the <strong>Softmax</strong> Function.</p><p><strong>The Gateway of Probability: The Softmax Function</strong></p><p>To turn scores ‘Zi’ into probabilities that add up to 100%, LLMs use e :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/296/1*NnxTu30YZXuc5qo0XuCj6w.png" /></figure><p>Why <strong><em>e</em></strong> ? Using <strong><em>e</em></strong> as the base ensures the function is monotonically increasing (higher scores = higher probabilities) and differentiable, allowing the model to “smoothly” learn during training.</p><p>The <strong>“Winner Takes All” Effect</strong>: <strong><em>e</em></strong> is an exponential base, so it magnifies small differences in raw scores, helping the model commit to the most likely word rather than being “wishy-washy.”</p><p><strong>The Role of </strong>e<strong> and Softmax: Step-by-Step</strong></p><p>Using our “cat” example where <strong>e = 2.718</strong> :</p><p><strong>Step 1: Exponentiation (The Power of </strong>e<strong> )</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/374/1*RIFEUv8xOhEA_IrxdnbOWg.png" /></figure><p><strong>Step 2: Calculate the sum of these values Total Sum</strong></p><p><strong>54.60+7.39+2.72+0.37= 65.08</strong></p><p><strong>Step 3: Calculate the probability (Value / Total Sum)</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/606/1*uY7oVFfZheoQ5eViB-5SQg.png" /></figure><p>This makes ‘Mice’ the strongest choice by far. Exponentially increasing the value helps in making a clear decision.</p><p><strong>Controlling Creativity: Temperature Scaling</strong></p><p>When you adjust “Temperature” ( T ), you modify the exponent of e. The formula becomes:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/62/1*rKw0qCM-m3ituwC7T4l5RA.png" /></figure><p><strong>Scenario 1: Low Temperature ( </strong>T=0.7<strong> ) — Predictable</strong></p><p>We divide each raw score by 0.7 . This makes the scores more spread out or “sharpened.”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/598/1*CD-phlbBGc0InWLD5UfJrg.png" /></figure><p>New Probability</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/524/1*UCF49bFyf7oiEPFiAyUlEA.png" /></figure><p>The model is now much more certain.</p><p><strong>Scenario 2: High Temperature ( </strong>T=1.3<strong> ) — Creative</strong></p><p>We divide each raw score by 1.3. This makes the scores less spread out or “flattened.”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/642/1*3R3lT2Tm_edFmFvaaG4C7w.png" /></figure><p>New Probability</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/486/1*hkKlYlwSTFf_7DEbm8s9JQ.png" /></figure><p>This aligns with LLM being more creative. Higher the temperature, higher the chance of LLM being more creative.</p><p><strong>Conclusion: The Unsung Hero of AI</strong></p><p>The next time you’re using an LLM, take a moment to think about this simple number, <strong><em>e</em></strong>. It’s not just a mathematical curiosity. From turning raw internal math into a structured list of word probabilities, to controlling whether the model is a predictable helper or a creative storyteller, <strong><em>Euler’s number</em></strong> (<strong>2.718</strong>) is the unsung hero that makes Artificial Intelligence feel truly “intelligent.”</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2df3a23949a0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Username Suggestion: Building a Semantic Username Suggestion Engine]]></title>
            <link>https://medium.com/@anuranjan.yadav/username-suggestion-building-a-semantic-username-suggestion-engine-28ce534deb3c?source=rss-27d4fba372b9------2</link>
            <guid isPermaLink="false">https://medium.com/p/28ce534deb3c</guid>
            <category><![CDATA[vector-embeddings]]></category>
            <category><![CDATA[word2vec]]></category>
            <category><![CDATA[bloom-filter]]></category>
            <category><![CDATA[faiss]]></category>
            <category><![CDATA[semantic-search]]></category>
            <dc:creator><![CDATA[Anuranjan Yadav]]></dc:creator>
            <pubDate>Tue, 10 Feb 2026 07:12:58 GMT</pubDate>
            <atom:updated>2026-02-10T07:12:58.232Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hwhPXepaSMjkdCWR6yhg9A.jpeg" /></figure><p>A follow-up to [<a href="https://medium.com/@anuranjan.yadav/bloom-filters-fast-userid-existence-check-using-bloom-filters-5c6f82ecbb0e">“Bloom Filters: Fast UserID Existence Check”</a>]</p><p>In my [<a href="https://medium.com/@anuranjan.yadav/bloom-filters-fast-userid-existence-check-using-bloom-filters-5c6f82ecbb0e">previous article</a>], we explored how to use Bloom Filters to perform lightning-fast checks for username availability. We solved the problem of defensive validation: ensuring a user doesn’t pick a name that’s already taken.</p><p>But what happens when the user’s desired name is taken?</p><p>Simply saying “Username Taken” is a bad user experience. A great application should offer alternatives. And not just random numbers (cool_coder_123), but meaningful alternatives that capture the essence of what the user wanted.</p><p>If I want “CodeWizard”, but it’s taken, the system should suggest “DevGuru”, “ScriptSorcerer”, or “SyntaxMage”.</p><p>To achieve this, we need Semantic Search.</p><h3>The Problem with Traditional Search</h3><p>Traditional database searches rely on lexical matching (exact keywords). If you search for “dog”, you find documents containing the string “dog”. You won’t find “puppy”, “canine”, or “hound” unless you explicitly program those synonyms.</p><p>For a username generator, this is limiting. We want to understand the intent and context behind the user’s input, not just the characters.</p><h3>The Solution: Vector Embeddings</h3><p>The core magic behind semantic search is the concept of Vector Embeddings.</p><p>As explained in “<a href="https://medium.com/@manansuri/a-dummys-guide-to-word2vec-456444f3c673">A Dummy’s Guide to Word2Vec</a>”, computers don’t understand words; they understand numbers. Vectorization is the process of turning words into lists of numbers (vectors) in a multi-dimensional space.</p><p>The key property of these vectors is that semantically similar words end up close together in this mathematical space.</p><ul><li>`King` — `Man` + `Woman` ≈ `Queen`</li><li>`Paris` is to `France` as `Tokyo` is to `Japan`</li></ul><p>By converting our dictionary of cool usernames into vectors, we can mathematically calculate which words are “nearest” to the user’s input.</p><h3>Powering Up: Transformers and Attention</h3><p>While Word2Vec was a revolution, modern AI has moved further. We now use Transformers.</p><p>In the landmark article “<a href="https://medium.com/@joshuaanang783/the-math-and-logic-behind-chatgpt-this-paper-is-all-you-need-57b82a0527f9">The Math and Logic behind ChatGPT: This Paper is All You Need</a>”, the architecture of the Transformer is explained. The secret sauce is the Attention Mechanism, which allows the model to weigh the importance of different words in a sentence relative to each other.</p><p>For our username generator, we use Sentence Transformers (specifically all-MiniLM-L6-v2), which are pre-trained models capable of capturing deep semantic meaning from short text snippets.</p><h3>Implementation: The “Smart” Suggestion Pipeline</h3><p>Here is how we implemented the semantic suggestion engine using Python, sentence-transformers, and FAISS.</p><h3>1. The Vector Database (FAISS)</h3><p>Searching through millions of vectors one by one is slow. To make it fast, we use FAISS (Facebook AI Similarity Search).</p><p><em>import faiss<br>from sentence_transformers import SentenceTransformer</em></p><p><em># 1. Load the Model<br>model = SentenceTransformer(‘all-MiniLM-L6-v2’)</em></p><p><em># 2. Encode our dictionary of “cool words”<br>words = [“ninja”, “swift”, “hyper”, “cyber”, “guru”, …]<br>embeddings = model.encode(words)</em></p><p><em># 3. Build the Index (L2 Distance)<br>dimension = embeddings.shape[1]<br>index = faiss.IndexFlatL2(dimension)<br>index.add(embeddings)</em></p><h3>2. The Search Query</h3><p>When a user types “fast”, we don’t look for the string “f-a-s-t”. We encode the concept of “fast” into a vector and ask FAISS for the nearest neighbors.</p><p><em>def semantic_search(query):<br># Turn input into vector<br>query_vector = model.encode([query])</em></p><p><em># Find 5 nearest neighbors<br>D, I = index.search(query_vector, 5)</em></p><p><em>return [words[i] for i in I[0]]</em></p><p>If the user types “Fast”, the model might return:</p><p>1. Swift (Distance: 0.2)</p><p>2. Rapid (Distance: 0.3)</p><p>3. Turbo (Distance: 0.35)</p><h3>3. Combining with Logic</h3><p>Once we have these semantically similar bases, we apply combinatorial logic (prefixes, suffixes, leetspeak) to generate the final usernames:</p><ul><li>`TheSwift`</li><li>`RapidOfficial`</li><li>`Turbo_HQ`</li></ul><h3>The Full Picture: Bloom Filters + Semantic Search</h3><p>By combining the two concepts from these blog posts, we created a robust username system:</p><p>1. Input: User types CloudWalker.</p><p>2. Bloom Filter Check: Is CloudWalker taken? -&gt; Yes.</p><p>3. Semantic Search: What means the same thing as “Cloud” and “Walker”? -&gt; Returns Sky, Mist, Ranger, Traveler.</p><p>4. Generation: Create variations like SkyRanger, MistTraveler.</p><p>5. Bloom Filter Re-check: Are these new suggestions taken?</p><p>6. Output: Display unique, verified, and relevant suggestions to the user.</p><h3>Conclusion</h3><p>Building a “smart” application isn’t just about using the latest LLM APIs. By understanding the building blocks — Bloom Filters for probabilistic efficiency and Vector Embeddings for semantic understanding — we can build systems that are both incredibly fast and surprisingly intelligent.</p><p>— -</p><p>References:</p><p>1. Anuranjan Yadav, “Bloom Filters: Fast UserID Existence Check using Bloom Filters”</p><p>2. Manan Suri, “A Dummy’s Guide to Word2Vec”</p><p>3. Joshua Anang, “The Math and Logic behind ChatGPT: This Paper is All You Need”</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=28ce534deb3c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Bloom filters: Fast UserID existence check using bloom filters]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/@anuranjan.yadav/bloom-filters-fast-userid-existence-check-using-bloom-filters-5c6f82ecbb0e?source=rss-27d4fba372b9------2"><img src="https://cdn-images-1.medium.com/max/856/1*btM_pH-rKJW9-5_FRuWA7A.png" width="856"></a></p><p class="medium-feed-snippet">Modern systems routinely handle millions or even billions of user IDs.
A seemingly simple question can become surprisingly expensive:</p><p class="medium-feed-link"><a href="https://medium.com/@anuranjan.yadav/bloom-filters-fast-userid-existence-check-using-bloom-filters-5c6f82ecbb0e?source=rss-27d4fba372b9------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://medium.com/@anuranjan.yadav/bloom-filters-fast-userid-existence-check-using-bloom-filters-5c6f82ecbb0e?source=rss-27d4fba372b9------2</link>
            <guid isPermaLink="false">https://medium.com/p/5c6f82ecbb0e</guid>
            <category><![CDATA[user-login]]></category>
            <category><![CDATA[bloom-filter]]></category>
            <dc:creator><![CDATA[Anuranjan Yadav]]></dc:creator>
            <pubDate>Fri, 09 Jan 2026 10:50:59 GMT</pubDate>
            <atom:updated>2026-01-09T10:50:59.704Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[What’s the Middle, Anyway? ]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-snippet">Remember that moment in primary school when the teacher asked, &#x201C;What&#x2019;s the middle of the data?&#x201D; and you were taught to add everything up&#x2026;</p><p class="medium-feed-link"><a href="https://medium.com/@anuranjan.yadav/whats-the-middle-anyway-4f5d1423d9c6?source=rss-27d4fba372b9------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://medium.com/@anuranjan.yadav/whats-the-middle-anyway-4f5d1423d9c6?source=rss-27d4fba372b9------2</link>
            <guid isPermaLink="false">https://medium.com/p/4f5d1423d9c6</guid>
            <dc:creator><![CDATA[Anuranjan Yadav]]></dc:creator>
            <pubDate>Sun, 17 Aug 2025 07:55:13 GMT</pubDate>
            <atom:updated>2025-08-17T07:55:13.273Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Daylight Savings Time — Should US ditch DST]]></title>
            <link>https://medium.com/@anuranjan.yadav/daylight-savings-time-should-us-ditch-dst-fd031b1bcf00?source=rss-27d4fba372b9------2</link>
            <guid isPermaLink="false">https://medium.com/p/fd031b1bcf00</guid>
            <category><![CDATA[daylight-saving-time]]></category>
            <category><![CDATA[business-intelligence]]></category>
            <category><![CDATA[time-series-analysis]]></category>
            <category><![CDATA[data-analytics]]></category>
            <dc:creator><![CDATA[Anuranjan Yadav]]></dc:creator>
            <pubDate>Wed, 22 Jan 2025 09:22:04 GMT</pubDate>
            <atom:updated>2025-01-22T09:22:04.984Z</atom:updated>
            <content:encoded><![CDATA[<h3>Daylight Savings Time — Should US ditch DST</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*R6C9oc78WjgHKgHL3ppMgg.png" /></figure><h3><strong>Should the US Ditch Daylight Saving Time?</strong></h3><p>Daylight Saving Time (DST), that annual ritual of shifting our clocks forward and back, has been a fixture in the US for over a century. But is it a relic of the past, or does it still serve a valuable purpose? Let’s delve into the history, benefits, drawbacks, and potential consequences of eliminating DST in the United States.</p><p><strong>A Brief History of DST</strong></p><p>The concept of adjusting clocks seasonally to maximize daylight hours originated in Germany during World War I as a fuel-saving measure. The US adopted DST in 1918, primarily to conserve energy by shifting daylight hours towards the evening, thus reducing the need for artificial lighting.</p><p><strong><em>Benefits of DST</em></strong></p><ul><li><strong>Energy Conservation:</strong> Proponents argue that DST reduces evening energy consumption by shifting daylight hours towards the evening, leading to decreased reliance on artificial lighting.</li><li><strong>Increased Recreational Time:</strong> Longer evenings during summer months provide more opportunities for outdoor activities and recreation.</li><li><strong>Reduced Traffic Accidents:</strong> Some studies suggest a potential decrease in traffic accidents during daylight hours.</li></ul><p><strong><em>Disadvantages of DST</em></strong></p><ul><li><strong>Health Impacts:</strong> The abrupt shift in sleep-wake cycles can disrupt circadian rhythms, leading to sleep disturbances, increased fatigue, and potential health issues.</li><li><strong>Increased Accidents:</strong> While some studies suggest a decrease in evening accidents, others point to an increase in morning accidents due to drowsiness.</li><li><strong>Economic Impacts:</strong> The spring forward transition can lead to decreased productivity and increased workplace accidents.</li><li><strong>Agricultural Disruptions:</strong> Farmers and agricultural workers rely on consistent schedules, and DST shifts can disrupt their routines.</li></ul><h3><strong>DST in different countries</strong></h3><p><strong>United States:</strong></p><ul><li><strong>Start:</strong> Second Sunday in March</li><li><strong>End:</strong> First Sunday in November</li></ul><p><strong>European Union:</strong></p><ul><li><strong>Start:</strong> Last Sunday in March</li><li><strong>End:</strong> Last Sunday in October</li></ul><p><strong>Canada (most parts):</strong></p><ul><li><strong>Start:</strong> Second Sunday in March</li><li><strong>End:</strong> First Sunday in November</li></ul><p><strong>Australia (parts):</strong></p><ul><li><strong>Start:</strong> First Sunday in October</li><li><strong>End:</strong> First Sunday in April</li></ul><p><strong>Mexico:</strong></p><ul><li><strong>Start:</strong> First Sunday in April</li><li><strong>End:</strong> Last Sunday in October</li></ul><p><strong><em>Important Notes:</em></strong></p><ul><li><strong>Variations:</strong> The specific dates and times can vary slightly within countries and even within regions of a country.</li><li><strong>Exceptions:</strong> Many countries do not observe DST at all. Some countries may observe it only in certain regions.</li><li><strong>Changes:</strong> DST rules can change from year to year, so it’s always best to check the most up-to-date information for a specific country.</li></ul><p>For details on each country do refer to : <a href="https://en.wikipedia.org/wiki/Daylight_saving_time_by_country">https://en.wikipedia.org/wiki/Daylight_saving_time_by_country</a></p><p>Fun fact: <strong>India also had DST for brief periods during World War II, 1962 Sino-India war, 1965 Pakistan war, 1971 Pakistan War</strong></p><h3><strong>Software Impact of DST</strong></h3><p>Modern software systems, from operating systems to scheduling applications, must account for DST. This involves complex logic to handle time zone transitions, often leading to:</p><ul><li><strong>Bugs and Errors:</strong> Incorrectly implemented DST rules can cause unexpected behavior in software, leading to incorrect timestamps, scheduling conflicts, and data inconsistencies.</li><li><strong>Maintenance Overhead:</strong> Maintaining and updating software to accommodate DST changes requires significant development and testing effort.</li><li><strong>Security Risks:</strong> Incorrectly handled DST transitions can create vulnerabilities that malicious actors could exploit.</li><li><strong>Scheduling and Calendaring:</strong></li></ul><p><strong>Recurring Events:</strong> Recurring events scheduled to occur during the DST transition hour can be missed or double-booked if the software doesn’t correctly handle the time shift. For example, a daily meeting scheduled for 2:00 AM might not occur at all during the spring forward transition, or it might occur twice during the fall back transition.</p><p><strong>Travel and Logistics:</strong> Flight and train schedules, hotel reservations, and other travel-related bookings can be disrupted if the software doesn’t account for DST changes in different time zones. This can lead to missed connections, incorrect arrival times, and frustrated travelers.</p><ul><li><strong>Financial Systems:</strong></li></ul><p><strong>Stock Market Trading:</strong> High-frequency trading systems rely on precise timestamps. DST transitions can introduce inaccuracies in trade execution times, potentially leading to financial losses.</p><p><strong>Billing and Invoicing:</strong> Software that generates invoices based on time-based events (e.g., hourly billing) can produce incorrect amounts if it doesn’t account for the DST hour shift.</p><ul><li><strong>Network and Security Systems:</strong></li></ul><p><strong>Security Logs and Audits:</strong> Security logs often rely on timestamps for analysis. DST transitions can create confusion in log analysis, making it difficult to pinpoint the exact time of security events.</p><p><strong>Network Monitoring:</strong> Network monitoring tools that rely on time-based alerts can be triggered incorrectly during DST transitions, leading to false alarms or missed critical events.</p><ul><li><strong>Industrial Control Systems:</strong></li></ul><p><strong>Manufacturing Processes:</strong> Industrial control systems that rely on precise timing for production processes can be disrupted by DST transitions. This can lead to production delays, equipment damage, and safety hazards.</p><ul><li><strong>Scientific Research:</strong></li></ul><p><strong>Data Analysis:</strong> Scientific experiments often involve precise time-stamped data. DST transitions can introduce errors in data analysis, leading to inaccurate conclusions.</p><h3><strong>DST and Data</strong></h3><p>Daylight Saving Time (DST) can cause a variety of issues and bugs in analytics systems, primarily due to the abrupt shift in time zones. Here are some major ones:</p><ul><li><strong>Data Inconsistency and Gaps:</strong></li></ul><p><strong>Missing Data:</strong> The “missing hour” during the spring forward transition can create gaps in time-series data, leading to inaccurate analysis of trends and patterns.</p><p><strong>Duplicate Data:</strong> During the fall back transition, where the clock “repeats” an hour, some systems may record duplicate data points, skewing averages and other metrics.</p><ul><li><strong>Incorrect Time Calculations:</strong></li></ul><p><strong>Durations and Intervals:</strong> Calculations involving time differences (e.g., session duration, time between events) can be inaccurate if the system doesn’t correctly handle the DST transition.</p><p><strong>Time-Based Segmentation:</strong> Segmenting data by time of day (e.g., hourly, daily) can produce misleading results if the system doesn’t account for the DST hour shift.</p><ul><li><strong>Reporting and Visualization Errors:</strong></li></ul><p><strong>Misaligned Charts and Graphs:</strong> Visualizations that rely on time-based axes can be distorted by DST transitions, making it difficult to interpret trends and patterns.</p><p><strong>Inaccurate Reporting:</strong> Reports that rely on time-based metrics (e.g., website traffic by hour, daily active users) can generate incorrect results.</p><ul><li><strong>Cross-Timezone Analysis:</strong></li></ul><p><strong>Challenges in Comparing Data:</strong> Analyzing data across time zones becomes more complex during DST, as the relationship between time zones changes.</p><ul><li><strong>Alerting and Monitoring Issues:</strong></li></ul><p><strong>False Alarms:</strong> Time-based alerts (e.g., for unusual activity) can be triggered incorrectly during DST transitions.</p><p><strong>Missed Alerts:</strong> Critical events might be missed if the monitoring system doesn’t account for the DST hour shift.</p><p><strong>Mitigating These Issues:</strong></p><ul><li><strong>Robust Time Zone Handling:</strong> Implement robust time zone libraries and functions that accurately handle DST transitions.</li><li><strong>Data Validation and Cleaning:</strong> Thoroughly validate and clean time-related data, especially around DST transitions.</li><li><strong>Testing and QA:</strong> Rigorously test analytics systems for DST-related issues before and after the transitions.</li><li><strong>Documentation:</strong> Clearly document how the system handles DST to aid in troubleshooting and maintenance.</li><li><strong>Consider DST-Aware Libraries:</strong> Utilize libraries specifically designed to handle time zone and DST complexities.</li></ul><p>By carefully addressing these issues, organizations can ensure the accuracy and reliability of their analytics systems throughout the year, even during the complexities of Daylight Saving Time.</p><h3><strong>What if We Remove DST?</strong></h3><p>Eliminating DST would mean setting clocks to standard time year-round. This would have several potential impacts:</p><ul><li><strong>More Consistent Sleep-Wake Cycles:</strong> A consistent time schedule would improve sleep quality and overall well-being.</li><li><strong>Reduced Risk of Accidents:</strong> Consistent daylight hours could lead to a more predictable traffic pattern and potentially reduce accidents.</li><li><strong>Simplified Software:</strong> Eliminating DST would significantly simplify software development and maintenance by removing the need for complex DST handling logic.</li></ul><p><strong>Software Impact of Removing DST</strong></p><p>Removing DST would have a significant positive impact on software:</p><ul><li><strong>Reduced Complexity:</strong> Software systems would no longer need to handle the complexities of DST transitions, leading to simpler, more robust, and easier-to-maintain code.</li><li><strong>Improved Reliability:</strong> Eliminating DST would reduce the risk of bugs and errors related to time zone transitions.</li><li><strong>Increased Security:</strong> Removing the need for DST handling would reduce potential security vulnerabilities.</li></ul><h3><strong>Conclusion</strong></h3><p>The debate over Daylight Saving Time is complex and multifaceted. While DST has its proponents and detractors, the potential benefits of eliminating it, particularly in terms of improved public health, increased safety, and simplified software development, warrant serious consideration.</p><p><strong><em>Disclaimer:</em></strong><em> This blog post provides general information and should not be considered legal or professional advice.This information is for general guidance only. Please refer to official government sources for the most accurate and up-to-date information on DST in any particular country.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fd031b1bcf00" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Time — Interesting times]]></title>
            <link>https://medium.com/@anuranjan.yadav/time-interesting-times-fa47779cf3cc?source=rss-27d4fba372b9------2</link>
            <guid isPermaLink="false">https://medium.com/p/fa47779cf3cc</guid>
            <category><![CDATA[daylight-saving-time]]></category>
            <category><![CDATA[timezone]]></category>
            <category><![CDATA[time]]></category>
            <category><![CDATA[timezone-difference]]></category>
            <category><![CDATA[utc]]></category>
            <dc:creator><![CDATA[Anuranjan Yadav]]></dc:creator>
            <pubDate>Sun, 07 Apr 2024 12:44:13 GMT</pubDate>
            <atom:updated>2024-04-07T12:48:51.371Z</atom:updated>
            <content:encoded><![CDATA[<h3>Time — Interesting times</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Lt5rwS3K7obIiKo0q5qdCA.jpeg" /></figure><p>Time is a very interesting concept. In today’s times, with AI all around, time has become even more relevant. Any prediction is based on what the software learns from the past and this is where time plays an important role. If you deal with a lot of time based data, you would know what all challenges it leads to. You think time and time zones are easy — you are in for a surprise. If you think the libraries or utility functions take care of all time related things — you are in for a big surprise. If you think you are working on this super cool product which has been running for years or you work with a high performing team and intelligent people around so time based issues are not for you — you are in for a huge surprise on how less people have understanding of time. Everyone assumes that it must already have been taken care of.</p><p>Some of the problems related to time which have been hugely debated online:</p><ol><li><strong>Y2K problem</strong> — half a trillion problem or rather the problem which created opportunities for so many Software Engineers around the world. (no one knew what will happen on 1st January 2000 — will all software stop working) (<a href="https://en.wikipedia.org/wiki/Year_2000_problem">https://en.wikipedia.org/wiki/Year_2000_problem</a>)</li><li><strong>Microsoft Exchange going down on 1st Jan, 2022 </strong>— Y2K2022 bug (<a href="https://www.theverge.com/2022/1/2/22863950/microsoft-exchange-y2k22-bug">https://www.theverge.com/2022/1/2/22863950/microsoft-exchange-y2k22-bug</a>)</li><li><strong>All petrol pumps in New Zealand stopped working on 29th Feb,2024</strong> (software did not take care of leap year — before you laugh off — do check if the code you have worked on has taken care of all time bound things) (<a href="https://www.wired.com/story/leap-year-glitch-gas-pumps-broken-new-zealand/%23:~:text=But%2520that%2520unique%2520calendar%2520date,gas%2520pumps&#39;%2520payment%2520processing%2520software">https://www.wired.com/story/leap-year-glitch-gas-pumps-broken-new-zealand/#:~:text=But%20that%20unique%20calendar%20date,gas%20pumps&#39;%20payment%20processing%20software</a>.)</li><li><strong>Azure service disruption on 29th February, 2012</strong> (<a href="https://azure.microsoft.com/en-us/blog/summary-of-windows-azure-service-disruption-on-feb-29th-2012/%23:~:text=The%2520leap%2520day%2520bug%2520immediately,resulted%2520in%2520new%2520VM%2520creation">https://azure.microsoft.com/en-us/blog/summary-of-windows-azure-service-disruption-on-feb-29th-2012/#:~:text=The%20leap%20day%20bug%20immediately,resulted%20in%20new%20VM%20creation</a>.)</li><li><strong>ChapGPT answered that 29th Feb, 2024 is not a valid date</strong> as 2024 is not a leap year — try that it still makes the same mistake.</li></ol><p>The list can be endless. Just search online or go on reddit or twitter or any social media platforms to check on how many things failed on leap year this year — it is super interesting. It makes you wonder — how come no one can fix this time thing once and for all. But hey — look at it this way — it leads to some interesting problems to take care of.</p><p>Let’s delve into some interesting insights on how time becomes an interesting concept</p><p><strong>UTC</strong></p><ul><li>UTC offset ranges from -12 to + 14. So that is 27 hours of offset(don’t forget the zero offset). All this because some islander folks decided they wanted to be on one side of the date line. Date line is a jagged one and not straight.</li><li>UTC -12 has the same time as UTC+12 , just a day behind.</li><li>UTC offset can map to multiple timezones UTC+5 has 10 timezones.</li><li>UTC offsets need not be every hour, IST is UTC+5:30 , Nepal uses 45 minute offset.</li><li>UTC offset can change for a country depending upon DST — adds to the confusion</li><li>Saving future time in UTC is definitely adding complexity as no one can predict what happens</li></ul><p><strong>Daylight Savings:</strong></p><ul><li>Every country can decide when DST starts and what is the DST offset they use. It can be debated that there is no scientific reason for this — just practical reasons. DST can be changed by governments anytime they choose.</li><li>DST does not start on the same day every year. It can be decided by the government.</li><li>Arizona state in US does not follow DST (imagine the confusion it creates)</li><li>Converting from one timezone to another may not be simple — What if the date and time are 1:30 am on Nov 1st, 2020, right when US DST ends and the clock moves backwards? 1:30am occurs twice that morning, how do you know which instance was intended?</li></ul><p><strong>Timezones</strong>:</p><ul><li>244 timezone being used in 195 countries.</li><li>Same timezone can have multiple names — Eastern Standard Time, Tiempo del Este, and Heure Normale de l’Est are all different names for the exact same time zone</li><li>Most libraries support 1 timezone only. So the default it uses is the one installed on the local machine but as called out there can be multiple timezones in which machine could be — this is what adds complexity.</li><li>Indiana in US has an interesting case — some cities follow EST while some follow CST.</li><li>North Pole and South Pole have no timezones, each country follows their own time there.</li></ul><p>Time related problems are some of the best to deal with. With AI becoming more and more vital — timeseries data and comparisons will take centerstage.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fa47779cf3cc" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>