<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Venumadhavjidugu on Medium]]></title>
        <description><![CDATA[Stories by Venumadhavjidugu on Medium]]></description>
        <link>https://medium.com/@venumadhavjidugu?source=rss-92f6feb0c064------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*jp44ABXhff5FCUpB</url>
            <title>Stories by Venumadhavjidugu on Medium</title>
            <link>https://medium.com/@venumadhavjidugu?source=rss-92f6feb0c064------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 12:08:40 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@venumadhavjidugu/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Recurrent Neural Networks (RNNs) Explained]]></title>
            <link>https://medium.com/@venumadhavjidugu/recurrent-neural-networks-rnns-explained-35347625da40?source=rss-92f6feb0c064------2</link>
            <guid isPermaLink="false">https://medium.com/p/35347625da40</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[rnn-architecture]]></category>
            <category><![CDATA[lstm]]></category>
            <category><![CDATA[recurrent-neural-network]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Venumadhavjidugu]]></dc:creator>
            <pubDate>Fri, 20 Nov 2026 08:02:28 GMT</pubDate>
            <atom:updated>2026-11-20T08:02:28.931Z</atom:updated>
            <content:encoded><![CDATA[<h3>1. Introduction to Neural Networks</h3><p>Neural networks are machine learning models loosely inspired by the brain. They are built from layers of simple units called neurons, and each neuron takes some numbers in, multiplies them by learned weights, adds a bias, and passes the result through an activation function such as ReLU or tanh.</p><p>A typical network has an input layer, one or more hidden layers and an output layer. During training, the network compares its predictions with the correct answers, measures the error with a loss function, and adjusts its weights through backpropagation and gradient descent. The simplest design, the feedforward network, treats every input as independent, which works well for images of fixed size or rows in a table but poorly for data that arrives as a sequence.</p><h3>2. What is an RNN?</h3><p>A recurrent neural network (RNN) is a neural network designed for sequential data. Its defining feature is a loop: the output of the hidden layer at one time step is fed back in as an extra input at the next step. That feedback gives the network a form of memory, so what it computes now depends on both the current input and a summary of everything it has seen before.</p><h3>3. Why Do We Need RNNs?</h3><p>Much of the world’s data is ordered, and the order carries meaning. Consider “dog bites man” versus “man bites dog”: the same words, very different sentences. Speech, text, stock prices, sensor readings and music all share this property.</p><p>Feedforward networks struggle here for three reasons:</p><ul><li>They expect inputs of a fixed size, while sentences and recordings vary in length.</li><li>They have no memory, so each input is processed in isolation.</li><li>They use separate weights for each position, so a pattern learned at the start of a sequence is not recognised at the end.</li></ul><p>RNNs address all three by processing one element at a time, carrying a hidden state forward, and reusing the same weights at every step.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/751/0*2S-6V_RBnVU9WBH7.png" /><figcaption>RNN Architecture</figcaption></figure><h3>4. How Does an RNN Work?</h3><p>An RNN reads a sequence one step at a time. Take the sentence “I love learning” as an example:</p><ol><li>At step 1, the network receives the vector for “I” along with an initial hidden state, usually all zeros, and produces a new hidden state.</li><li>At step 2, it receives “love” together with the hidden state from step 1, so the new state reflects both words.</li><li>At step 3, it receives “learning” and the previous state, producing a state that summarises the whole sentence so far.</li></ol><p>At any step the network can also produce an output from its current hidden state. If you “unroll” the loop across time, an RNN looks like a chain of identical copies of the same network, each passing a message to the next. Crucially, every copy shares the same weights.</p><h3>5. RNN Architecture</h3><p>A basic RNN has four parts:</p><ul><li><strong>Input layer:</strong> receives the element of the sequence at the current time step, such as a word embedding or a sensor reading, as a vector.</li><li><strong>Hidden state:</strong> a vector that acts as the network’s memory. It is recomputed at every step from the current input and the previous hidden state.</li><li><strong>Recurrent connection:</strong> the loop that carries the hidden state from one step to the next. It is what separates an RNN from a feedforward network.</li><li><strong>Output layer:</strong> turns the hidden state into a prediction, for example a probability distribution over the next word or a class label. It can run at every step or only at the last one.</li></ul><h3>6. Mathematical Representation of RNN</h3><p>At time step t, a simple RNN updates its hidden state and produces an output with two equations:</p><pre>h_t = \tanh(W_{xh} x_t + W_{hh} h_{t-1} + b_h)</pre><pre>y_t = W_{hy} h_t + b_y</pre><p>Here x_t is the input at step t, h_t is the hidden state (with h_0 usually set to zero), and y_t is the output. W_xh, W_hh and W_hy are weight matrices, and b_h and b_y are biases. The same weights are reused at every time step, which keeps the parameter count independent of sequence length. For classification, y_t is typically passed through a softmax. Training uses backpropagation through time (BPTT), which unrolls the network across the sequence and propagates the error backwards through every step.</p><h3>7. Types of RNN Architectures</h3><p>RNNs can be wired in different ways depending on how many inputs and outputs the task has.</p><ul><li><strong>One-to-One:</strong> a single input gives a single output. This is the standard feedforward setup, such as image classification, with no sequence involved.</li><li><strong>One-to-Many:</strong> a single input gives a sequence of outputs. Examples include image captioning and generating music from a seed note.</li><li><strong>Many-to-One:</strong> a sequence of inputs gives a single output. Sentiment analysis and spam detection work this way.</li><li><strong>Many-to-Many:</strong> a sequence of inputs gives a sequence of outputs. Machine translation, speech recognition and video labelling are typical examples.</li></ul><p>Many-to-many comes in two forms. When input and output have equal length and align step by step, as in part-of-speech tagging, the network emits an output at every step. When lengths differ, as in translation, an encoder reads the whole input first and a decoder then generates the output.</p><h3>8. Applications of RNNs</h3><p>RNNs and their variants have been used wherever order matters:</p><ul><li><strong>Natural language processing:</strong> language modelling, text generation, sentiment analysis and named-entity recognition.</li><li><strong>Machine translation:</strong> encoder-decoder models that read a sentence in one language and write it in another.</li><li><strong>Speech recognition:</strong> converting audio signals into text, as in voice assistants.</li><li><strong>Time-series forecasting:</strong> predicting stock prices, energy demand, weather and sensor readings.</li><li><strong>Music and text generation:</strong> producing the next note or character one step at a time.</li><li><strong>Video and handwriting analysis:</strong> labelling activities across frames and recognising cursive writing.</li></ul><p>Transformers now dominate many language tasks, but RNN-style models remain popular for small devices, streaming data and low-latency settings where processing one step at a time is an advantage.</p><h3>9. Limitations of Traditional RNNs</h3><p>Plain RNNs are hard to train on long sequences, largely because of how gradients behave during backpropagation through time.</p><h3>Vanishing Gradient Problem</h3><p>During BPTT, the gradient is multiplied by the recurrent weights and the activation derivative once for every step it travels back. When these factors are smaller than 1, the gradient shrinks exponentially, so early time steps receive almost no learning signal and the network fails to learn from distant inputs.</p><h3>Exploding Gradient Problem</h3><p>When those factors are larger than 1, the gradient grows exponentially instead. Weight updates become huge, training turns unstable and the loss can jump or become NaN. A common remedy is gradient clipping, which caps the gradient’s size before each update.</p><h3>Long-term dependency problem</h3><p>The practical result of vanishing gradients is that a simple RNN struggles to connect information across long gaps. In “I grew up in France, and after many years abroad I still speak fluent …”, the word “France” is needed to predict the answer, but it may be dozens of steps away. A basic RNN tends to remember only the most recent words.</p><h3>10. Advanced RNN Architectures</h3><p>Two gated architectures were introduced to fix these problems. Gates are small learned layers that decide what information to keep, update or discard.</p><h3>LSTM (Long Short-Term Memory)</h3><p>Proposed in 1997 by Hochreiter and Schmidhuber, the LSTM adds a separate cell state that runs through the sequence with only small, controlled changes, giving gradients a more direct path backwards. Three gates manage it:</p><ul><li><strong>Forget gate:</strong> decides which parts of the old cell state to erase.</li><li><strong>Input gate:</strong> decides which new information to write into the cell state.</li><li><strong>Output gate:</strong> decides how much of the cell state to expose as the hidden state.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/627/0*-P5UEucxZfrftpsm" /><figcaption>LSTM Architecture</figcaption></figure><h3>GRU (Gated Recurrent Unit)</h3><p>Introduced in 2014, the GRU simplifies the LSTM by merging the cell and hidden states and using just two gates: an <strong>update gate</strong>, which balances old memory against new information, and a <strong>reset gate</strong>, which controls how much of the past to ignore when computing new content. It has fewer parameters, trains faster, and often matches the LSTM’s accuracy, though LSTMs can still do better on some tasks with very long sequences.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*beYWLXAkXh97PXp0.png" /><figcaption>GRU Architecture</figcaption></figure><h3>12. Conclusion</h3><p>RNNs extended neural networks to sequential data by adding a hidden state that carries context from one step to the next, using the same weights at every step. That idea powered progress in translation, speech recognition and forecasting. Simple RNNs, however, suffer from vanishing and exploding gradients and cannot reliably learn long-range dependencies, which led to LSTMs and GRUs and their gated memory. Even as Transformers have taken over many large-scale language tasks, understanding RNNs remains valuable: they introduced the core ideas of sequence modelling, and they are still a practical choice for streaming and resource-constrained applications.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=35347625da40" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building an AI-Powered Pothole Detection System: From Dataset Analysis to Deployment]]></title>
            <link>https://medium.com/@venumadhavjidugu/building-an-ai-powered-pothole-detection-system-from-dataset-analysis-to-deployment-15baec189b0b?source=rss-92f6feb0c064------2</link>
            <guid isPermaLink="false">https://medium.com/p/15baec189b0b</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[computer-vision-project]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Venumadhavjidugu]]></dc:creator>
            <pubDate>Mon, 09 Nov 2026 11:38:49 GMT</pubDate>
            <atom:updated>2026-11-09T11:38:49.968Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/886/1*pLYHexBd0ybZ3WAmM0Lhsw.jpeg" /></figure><p>Machine learning projects are not just about training a model and achieving good accuracy. The real learning happens when we understand the data, investigate problems, experiment with different configurations, evaluate performance, and finally deploy the solution.</p><p>Recently, I worked on an <strong>AI-Powered Road Damage Detection System using YOLO11n</strong>, focusing on detecting potholes in road images and videos.</p><p>In this article, I want to share my complete development journey, including the technical decisions, challenges, experiments, and lessons learned.</p><h3>1. Understanding the Problem</h3><p>Potholes are a major road safety concern. Identifying them manually across large road networks can be time-consuming.</p><p>My objective was to develop a computer vision system that could:</p><ul><li>Detect potholes in road images.</li><li>Identify their locations using bounding boxes.</li><li>Predict confidence scores.</li><li>Process road videos.</li></ul><p>I selected object detection rather than image classification because I wanted to identify not only whether potholes existed but also where they were located.</p><h3>2. Dataset Collection and Exploration</h3><p>I used a pothole detection dataset from Roboflow Universe.</p><p>Before training the model, I explored the dataset to understand its structure and quality.</p><p>The dataset contained three major splits:</p><ul><li>Training dataset</li><li>Validation dataset</li><li>Testing dataset</li></ul><p>I performed Exploratory Data Analysis (EDA) to investigate:</p><ul><li>Image distribution across dataset splits.</li><li>Class distribution.</li><li>Bounding-box dimensions.</li><li>Annotation quality.</li><li>Object size distribution.</li></ul><p>One important lesson I learned was that understanding the dataset is just as important as selecting the model.</p><h3>3. Data Cleaning and Annotation Validation</h3><p>During dataset analysis, I identified 66 invalid annotations caused by bounding boxes exceeding image boundaries.</p><p>This highlighted an important aspect of computer vision: poor annotation quality can affect model learning.</p><p>I investigated the annotation boundaries and worked on correcting invalid coordinates.</p><p>This step helped me understand why data validation should happen before model training.</p><h3>4. Model Selection and Training</h3><p>For object detection, I selected YOLO11n.</p><p>I chose the Nano variant because I wanted a lightweight model that could be trained and deployed using my available computing resources.</p><p>I started with pretrained weights, applying transfer learning to adapt the model to pothole detection.</p><h3>Experiment 1: YOLO11n at 640 resolution</h3><p>The first experiment used:</p><ul><li>Model: YOLO11n</li><li>Image size: 640</li><li>Epochs: 100</li><li>Batch size: 16</li><li>GPU: NVIDIA Tesla T4</li></ul><h3>Experiment 2: YOLO11n at 800 resolution</h3><p>For the second experiment, I increased the image resolution to 800 and reduced the batch size to 8.</p><p>The motivation was to provide more visual detail for detecting relatively small potholes.</p><p>The second experiment achieved the following validation results:</p><p>MetricResultPrecision89.5%Recall83.9%mAP@5091.0%mAP@50–9557.9%</p><p>These results helped me understand how object detection models are evaluated beyond simple accuracy.</p><p><em>Note: These are validation results, not independent final test results.</em></p><h3>5. Understanding Model Evaluation</h3><p>One of the most valuable parts of this project was learning how to evaluate object detection models.</p><p>I explored several important metrics:</p><p><strong>Precision:</strong> How many predicted potholes were actually correct?</p><p><strong>Recall:</strong> How many actual potholes did the model successfully detect?</p><p><strong>Intersection over Union (IoU):</strong> How accurately does the predicted bounding box overlap with the ground-truth box?</p><p><strong>mAP@50:</strong> Measures average precision at an IoU threshold of 0.50.</p><p><strong>mAP@50–95:</strong> Evaluates average precision across multiple IoU thresholds, from 0.50 to 0.95.</p><p>I also learned why a model with good precision may still miss several actual potholes.</p><p>Understanding these metrics helped me look beyond a single performance number.</p><h3>6. Confidence Threshold Optimization</h3><p>Another interesting experiment involved testing different confidence thresholds.</p><p>I evaluated thresholds ranging from 0.10 to 0.50 using custom prediction-to-ground-truth matching.</p><p>The results were:</p><p>ConfidencePrecisionRecallF1-score0.1069.59%91.55%79.07%0.2078.14%88.45%82.98%0.2581.17%87.11%84.04%0.3082.85%85.67%84.24%0.4087.12%82.99%85.01%0.5091.02%79.38%84.80%</p><p>The highest F1-score in this experiment was obtained at a confidence threshold of 0.40.</p><p>This helped me understand the trade-off between precision and recall.</p><p>I also explored Non-Maximum Suppression (NMS) to understand how overlapping duplicate predictions are removed.</p><h3>7. Error Analysis</h3><p>I learned that evaluation metrics alone cannot explain everything about model performance.</p><p>I visually inspected predictions and compared them against ground-truth annotations.</p><p>I investigated:</p><ul><li>False positives.</li><li>False negatives.</li><li>Duplicate bounding boxes.</li><li>Bounding-box localization.</li><li>Confidence scores.</li></ul><p>I also calculated IoU for individual predictions to understand how closely predicted boxes matched actual annotations.</p><p>This was one of the most valuable learning experiences because it helped connect mathematical evaluation metrics with actual model predictions.</p><h3>8. Building the Streamlit Application</h3><p>After training and evaluating the model, I integrated the best checkpoint into a Streamlit application.</p><p>The application supports:</p><ul><li>Image uploads.</li><li>Pothole detection.</li><li>Bounding-box visualization.</li><li>Confidence score display.</li><li>Detection count.</li><li>Annotated image downloads.</li><li>Video processing functionality.</li></ul><p>I used Python, Streamlit, Ultralytics YOLO, and OpenCV.</p><p>This transformed the trained model into an interactive application that users can test.</p><h3>9. Key Learnings</h3><p>This project helped me understand that developing a machine learning solution involves much more than training a model.</p><p>Some of my biggest takeaways were:</p><ol><li>Why dataset quality matters.</li><li>How to perform EDA on object detection datasets.</li><li>How bounding-box annotations work.</li><li>How transfer learning helps model training.</li><li>How precision, recall, IoU, and mAP measure performance.</li><li>Why confidence threshold optimization matters.</li><li>How to perform error analysis.</li><li>How to deploy a trained computer vision model using Streamlit.</li></ol><p>Most importantly, I learned to question model predictions rather than blindly trust evaluation metrics.</p><h3>10. What’s Next?</h3><p>There is still room for improvement.</p><p>My next steps include:</p><ul><li>Testing on more diverse road conditions.</li><li>Improving detection of small potholes.</li><li>Investigating false positives and missed detections.</li><li>Evaluating performance on an independent test set.</li><li>Exploring model optimization for faster inference.</li><li>Improving the video-processing experience.</li></ul><h3>Conclusion</h3><p>Building this project gave me practical exposure to the complete computer vision workflow, from understanding the problem and exploring data to training, evaluation, error analysis, and deployment.</p><p>I believe the most important part of a machine learning project is not just achieving a good metric but understanding why the model performs the way it does.</p><p>This project is another step in my journey of becoming a better Machine Learning Engineer.</p><p><strong>Thanks to my mentor and trainer for their guidance and support throughout this learning journey.</strong></p><p>#MachineLearning #ComputerVision #YOLO11 #ObjectDetection #DeepLearning #Python #Streamlit</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=15baec189b0b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Word Embeddings Explained: From Bag-of-Words to Word2Vec, GloVe and FastText]]></title>
            <link>https://medium.com/@venumadhavjidugu/word-embeddings-explained-from-bag-of-words-to-word2vec-glove-and-fasttext-a6ea47a0c2de?source=rss-92f6feb0c064------2</link>
            <guid isPermaLink="false">https://medium.com/p/a6ea47a0c2de</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[transformers]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Venumadhavjidugu]]></dc:creator>
            <pubDate>Wed, 04 Nov 2026 09:50:09 GMT</pubDate>
            <atom:updated>2026-11-04T09:50:09.116Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/678/1*gbuHbMbraz6_oix55VbtZg.jpeg" /></figure><p>Machine learning models only understand numbers, so every NLP system must first turn words into vectors, and the way you do that decides how much language the model can ever “understand”. This post walks the path from the simplest answer (Bag of Words), through its smarter cousin (TF-IDF), to the idea that changed NLP in 2013 (Word2Vec) and the two methods built on it, GloVe and FastText. You will see what each method does, a small worked example, where it fails, and why each next method was needed.</p><h3>Why words need numbers</h3><p>A neural network or a linear classifier cannot read the string “cat”. It needs a fixed-length list of numbers. Text is awkward for this in three ways:</p><ul><li><strong>Variable length.</strong> One review has 5 words, another has 500, but a model expects inputs of one fixed size.</li><li><strong>No built-in order or distance.</strong> The letters in “cat” tell a computer nothing about whether “cat” is closer to “dog” or to “car”.</li><li><strong>Huge vocabulary.</strong> English has hundreds of thousands of word forms, and most of them are rare.</li></ul><p>A <strong>word embedding</strong> (or more broadly, a text representation) is the answer to one question: <em>what list of numbers should stand for this word or document so that a model can learn from it?</em> Everything below is a different answer to that question.</p><p>The approaches fall into two families:</p><ol><li><strong>Sparse, count-based representations:</strong> Bag of Words and TF-IDF. Long vectors, mostly zeros, each position tied to one specific word.</li><li><strong>Dense, learned representations:</strong> Word2Vec, GloVe and FastText. Short vectors (typically 100 to 300 numbers), all non-zero, where <em>meaning</em> is spread across the positions.</li></ol><h3>Bag of Words: count the words, ignore the order</h3><p>Bag of Words (BoW) represents a document as a vector of word counts, treating the text like a bag of loose words with no order. It is the oldest and simplest text representation, and it is still a strong baseline for tasks like spam filtering and topic classification.</p><p><strong>How it works</strong></p><ol><li>Collect every unique word across all your documents. This is the <strong>vocabulary</strong>.</li><li>Give each word a fixed position (index) in a vector.</li><li>For each document, write down how many times each vocabulary word appears at its position.</li></ol><p><strong>Worked example.</strong> Take three tiny documents:</p><ul><li>D1: “the cat sat on the mat”</li><li>D2: “the dog sat on the log”</li><li>D3: “the cat chased the dog”</li></ul><p>The vocabulary has 8 words: cat, chased, dog, log, mat, on, sat, the. Each document becomes 8 counts, in that order:</p><ul><li>D1 becomes [1, 0, 0, 0, 1, 1, 1, 2]</li><li>D2 becomes [0, 0, 1, 1, 0, 1, 1, 2]</li><li>D3 becomes [1, 1, 1, 0, 0, 0, 0, 2]</li></ul><p>Now every document, whatever its length, is a vector of the same size, so any classifier can use it. You can compare two documents with cosine similarity: D1 and D2 share “the”, “on” and “sat”, so they come out similar.</p><p><strong>Common variants</strong></p><ul><li><strong>Binary BoW:</strong> store 1 or 0 for present or absent instead of the count.</li><li><strong>N-grams:</strong> also count pairs (bigrams) or triples of consecutive words, such as “not good” as one feature. This recovers a little local word order, at the cost of a much bigger vocabulary.</li><li><strong>One-hot encoding:</strong> the same idea applied to a single word, where the vector has a 1 at that word’s position and 0 everywhere else.</li></ul><p><strong>Where it breaks</strong></p><ul><li><strong>Word order is lost.</strong> “The dog chased the cat” and “the cat chased the dog” get identical vectors, though they mean opposite things.</li><li><strong>Common words dominate.</strong> “The” has the biggest counts above, yet tells you nothing about what a document is about.</li><li><strong>Vectors are huge and mostly zero.</strong> A real vocabulary has 50,000 to 500,000 words, so each document vector has that many positions and almost all are 0.</li><li><strong>Unknown words are invisible.</strong> A word that was not in the vocabulary when you built it has no position.</li></ul><h3>TF-IDF: count the words, but reward the rare ones</h3><p>TF-IDF fixes BoW’s biggest flaw by down-weighting words that appear in almost every document and boosting words that are distinctive to a few. The name stands for <strong>Term Frequency, Inverse Document Frequency</strong>.</p><p><strong>1. Term frequency (TF)</strong> asks how important a word is <em>inside this document</em>: the word’s count divided by the document’s length.</p><p>tf(t, d) = count of t in d / total words in d</p><p><strong>2. Inverse document frequency (IDF)</strong> asks how rare a word is <em>across the whole collection</em>. If N is the number of documents and df(t) is how many of them contain the word t:</p><p>idf(t) = log( N / df(t) )</p><p>A word in every document gets idf = log(1) = 0, so it is switched off. A word in only one document out of many gets a large idf.</p><p><strong>The final weight is the product of the two:</strong></p><p>tf-idf(t, d) = tf(t, d) × idf(t)</p><p><strong>Worked example.</strong> Using the same three documents (N = 3, natural log), here is how the words of D1 (“the cat sat on the mat”, 6 words) are weighted:</p><ul><li><strong>the</strong> appears in 3 of 3 documents, so idf = 0.000 and tf-idf = 0.000</li><li><strong>cat</strong> appears in 2 of 3, so idf = 0.405 and tf-idf = 0.167 × 0.405 = 0.068</li><li><strong>sat</strong> and <strong>on</strong> are the same as “cat”: tf-idf = 0.068</li><li><strong>mat</strong> appears in 1 of 3, so idf = 1.099 and tf-idf = 0.167 × 1.099 = 0.183</li></ul><p>In BoW, “the” had the largest value (2). In TF-IDF it drops to exactly 0, and “mat”, which only D1 contains, becomes the most important word in D1. That matches intuition: “mat” tells you what makes D1 different.</p><p><strong>A practical note.</strong> Libraries do not use the plain formula. scikit-learn, for example, uses a smoothed idf of ln((1+N)/(1+df)) + 1 and then normalizes each vector to unit length, so its numbers will differ from the ones above even though the idea is the same.</p><p><strong>What TF-IDF is good for.</strong> It is a strong, cheap baseline for search ranking, document similarity, keyword extraction and text classification. For many small and medium problems, TF-IDF with logistic regression is hard to beat.</p><p><strong>What it still does not fix.</strong> TF-IDF only changes the <em>weights</em>. The vectors are still huge, still mostly zeros, still ignore word order, and still treat every word as an unrelated symbol. That last point is the real problem.</p><h3>The wall that BoW and TF-IDF hit</h3><p>BoW and TF-IDF treat every word as an isolated symbol, so they have no idea that “great” and “excellent” mean nearly the same thing. Four problems follow from that.</p><p><strong>1. No meaning, only identity.</strong> In one-hot form, every pair of different words is exactly equally far apart. The vector for “cat” has a 1 in one slot and the vector for “dog” has a 1 in another, so their dot product is 0. “Cat” is as unrelated to “dog” as it is to “carburetor”.</p><p><strong>2. Synonyms look different, opposites can look similar.</strong> Compare two pairs of short reviews with BoW and cosine similarity:</p><ul><li>“the movie was great” vs “the film was excellent” (same meaning): they share only “the” and “was”, so the similarity is 0.50.</li><li>“the movie was great” vs “the movie was terrible” (opposite meaning): they share “the”, “movie” and “was”, so the similarity is 0.75.</li></ul><p>The opposite-meaning pair scores higher, because only the surface words are compared. A model built on these vectors has to relearn from scratch that “great” and “excellent” go together, and it can only do that for words it saw in its training data.</p><p><strong>3. The curse of dimensionality.</strong> One position per vocabulary word means 50,000 to 500,000 dimensions. Each document uses only a few dozen of them, so the data is extremely sparse. Models need more training data, more memory and more compute, and they overfit easily.</p><p><strong>4. Little or no word order and context.</strong> “Bank” in “river bank” and “bank loan” gets the same single column. N-grams help a bit with order, but they make the sparsity problem much worse.</p><p><strong>What we actually want</strong> is a representation where:</p><ul><li>words with similar meanings get similar vectors,</li><li>the vectors are short (a few hundred numbers) and dense,</li><li>the vectors are <em>learned from raw text</em>, without anyone labeling meaning by hand.</li></ul><p>The key to getting there is an old idea from linguistics, the <strong>distributional hypothesis</strong>: words that appear in similar contexts tend to have similar meanings. The linguist J. R. Firth summed it up in 1957 by saying you know a word by the company it keeps. “Coffee” and “tea” both appear near “drink”, “hot”, “cup” and “morning”, so they should end up with similar vectors. Word2Vec turns this idea into an algorithm.</p><h3>Word2Vec: learn each word’s vector by predicting its neighbours</h3><p>Word2Vec, published by Tomas Mikolov and colleagues at Google in 2013, learns a short dense vector for every word by training a tiny neural network to predict words from their context. The prediction task is only a vehicle. What you keep at the end are the network’s learned weights, and those weights are the word vectors.</p><p><strong>The setup in four steps</strong></p><ol><li>Read a very large amount of plain text (Wikipedia, news, books). No labels are needed, which makes this <em>self-supervised</em>: the text itself supplies the answers.</li><li>Slide a small <strong>window</strong> along each sentence, for example 5 words wide. Within each window there is one <em>centre</em> word and its surrounding <em>context</em> words.</li><li>Train a shallow network to predict one from the other (two ways of doing this are in the next section).</li><li>Throw away the prediction layer and keep the input weight matrix. Row <em>i</em> of that matrix is the embedding of word <em>i</em>.</li></ol><p><strong>Why that works.</strong> The network has to give “coffee” and “tea” similar internal representations, because both need to predict the same neighbours: “drink”, “hot”, “cup”, “morning”. Words used in similar contexts are pushed to similar vectors.</p><p><strong>What an embedding looks like.</strong> If the vocabulary has V words and you choose an embedding size d (commonly 100 to 300), the weight matrix is V × d. Looking up a word is just selecting a row. A made-up example with d = 4:</p><ul><li>king = [0.52, 0.81, -0.30, 0.12]</li><li>queen = [0.49, 0.78, -0.27, 0.58]</li><li>apple = [-0.40, 0.05, 0.66, -0.21]</li></ul><p>These numbers are invented to show the shape. In a trained model, “king” and “queen” would sit close together and “apple” far from both. Individual dimensions have no human-readable meaning; the meaning lives in the direction and distance between whole vectors.</p><p><strong>Measuring similarity.</strong> Closeness between two word vectors is usually measured with cosine similarity, which is 1 for the same direction, 0 for unrelated and negative for opposite:</p><p>cos(a, b) = (a · b) / (|a| × |b|)</p><p><strong>The famous side effect: vector arithmetic.</strong> Mikolov’s team found that relationships between words show up as consistent directions in the space. The most quoted example is that the vector for “king” minus “man” plus “woman” lands very near “queen”. Other examples follow the same pattern: country to capital (France to Paris, Japan to Tokyo), and verb tense (walk to walked, swim to swam).</p><p>king - man + woman ≈ queen</p><p>This worked far better than anyone expected from a model that was never told anything about gender or geography. It was learned purely from co-occurrence patterns, and it is the reason Word2Vec caught the field’s attention.</p><h3>CBOW vs Skip-gram: two ways to train the same idea</h3><p>Word2Vec comes in two architectures that differ only in the direction of the prediction: CBOW predicts the centre word from its context, and Skip-gram predicts the context words from the centre word. Both learn useful embeddings.</p><p>Take the sentence “the quick brown fox jumps over the lazy dog” with a window of 2 on each side, and let the centre word be “fox”. The context is “quick”, “brown”, “jumps”, “over”.</p><p><strong>CBOW (Continuous Bag of Words)</strong></p><ul><li>Input: the context words (quick, brown, jumps, over). Target: the centre word (fox).</li><li>One training example per window. The context vectors are averaged into one vector.</li><li>Faster, and good for frequent words.</li></ul><p><strong>Skip-gram</strong></p><ul><li>Input: the centre word (fox). Target: each context word in turn.</li><li>Four training examples from this window: fox to quick, fox to brown, fox to jumps, fox to over.</li><li>Slower, but better for rare words and small datasets.</li></ul><p><strong>How CBOW trains, step by step</strong></p><ol><li>Look up the embedding of each context word (a row of the input matrix).</li><li>Average those vectors into a single hidden vector of size d.</li><li>Multiply by a second matrix (the output matrix, d × V) to get a score for every word in the vocabulary.</li><li>Turn the scores into probabilities with softmax.</li><li>Compare with the true centre word, compute the error, and nudge both matrices with gradient descent so “fox” becomes more likely next time.</li></ol><p><strong>How Skip-gram trains.</strong> The same steps, with the roles swapped: the input is the single centre word’s embedding, and the network is asked to give high probability to each of its actual context words. Because every (centre, context) pair is its own training example, rare words get many more updates than in CBOW, which is why Skip-gram usually does better on rare words.</p><p><strong>Which one should you use?</strong> If you have a large corpus and care about speed, CBOW is a fine default. If your vocabulary has many rare words, or your dataset is smaller, Skip-gram is generally the safer choice.</p><p><strong>Hyperparameters that matter</strong></p><ul><li><strong>Embedding size (d):</strong> 100 to 300 is typical.</li><li><strong>Window size:</strong> small windows (2 to 5) capture syntactic similarity, such as which words can replace each other in a sentence. Larger windows capture more topical similarity.</li><li><strong>Minimum count:</strong> ignore words below some frequency, such as 5, because they do not appear often enough to learn a good vector.</li><li><strong>Epochs:</strong> how many passes over the corpus.</li></ul><h3>Making it fast: negative sampling and subsampling</h3><p>The naive version of Word2Vec is too slow to use, and two tricks from the original papers are what made training on billions of words practical. The problem is the softmax step: to compute the probability of the right word, the network must score <em>every</em> word in the vocabulary, and with a vocabulary of a million words that is a million calculations for every single training example.</p><p><strong>Negative sampling</strong> replaces that giant multi-class problem with a few small yes/no questions.</p><ol><li>For a real pair such as (fox, jumps), the label is <strong>1</strong>: these two words really appeared together.</li><li>Pick k random words that did <em>not</em> appear in this context, such as (fox, pizza) and (fox, tax). Their label is <strong>0</strong>.</li><li>Train a simple binary classifier to say yes to the real pair and no to the fake ones.</li></ol><p>Now each update touches only 1 + k output vectors instead of the whole vocabulary. The Word2Vec authors suggest k of 5 to 20 for small datasets and 2 to 5 for large ones. Maximizing this objective pulls real neighbours together and pushes random words apart. The negatives are not picked uniformly: they are drawn from the word-frequency distribution raised to the power 3/4, which samples rare words a bit more often than raw frequency would.</p><p><strong>Subsampling of frequent words</strong> tackles a different waste. Words like “the”, “a” and “of” appear in huge numbers of windows but carry little information about their neighbours. Word2Vec randomly drops each occurrence of a word with a probability that rises with its frequency. This speeds up training and also improves the vectors of rarer, more meaningful words.</p><p><strong>Hierarchical softmax</strong> is the other speed-up from the original work. It arranges the vocabulary as a binary tree so a word’s probability costs about log(V) steps instead of V. In practice, negative sampling is the more popular choice.</p><h3>GloVe: learn from global co-occurrence counts</h3><p>GloVe (Global Vectors), published by Jeffrey Pennington, Richard Socher and Christopher Manning at Stanford in 2014, learns word vectors by factorizing a table of how often words appear together across the <em>whole</em> corpus. Word2Vec learns from one small window at a time. GloVe first counts everything, then fits vectors to those counts.</p><p><strong>The idea in three steps</strong></p><ol><li>Build a <strong>co-occurrence matrix</strong> X of size V × V, where X[i][j] is how many times word j appeared within the window of word i, over the entire corpus.</li><li>Look at <em>ratios</em> of co-occurrence probabilities, not raw counts. The authors’ example: “ice” co-occurs with “solid” far more than “steam” does, “steam” co-occurs with “gas” far more than “ice” does, and both co-occur about equally with “water”. Meaning shows up in these ratios.</li><li>Train vectors so that the dot product of two word vectors approximates the logarithm of how often the words co-occur.</li></ol><p><strong>The objective</strong> is a weighted least-squares fit. In words: for every pair of words, make (vector of word i · vector of word j + biases) as close as possible to log(X_ij), and weight each pair by a function f(X_ij):</p><p>J = sum over i, j of f(X_ij) × ( w_i · w_j + b_i + b_j - log X_ij )²</p><p>The weighting function f caps the influence of very frequent pairs such as (the, of) and ignores pairs that never co-occur, since log 0 is undefined.</p><p><strong>GloVe vs Word2Vec</strong></p><ul><li><strong>Learns from:</strong> Word2Vec uses local windows, one at a time. GloVe uses a global co-occurrence matrix.</li><li><strong>Training style:</strong> Word2Vec is predictive (a small neural network). GloVe is a count-based fit (weighted least squares).</li><li><strong>Memory:</strong> Word2Vec streams over text. GloVe needs the V × V matrix built first.</li><li><strong>Results:</strong> comparable quality, with each sometimes ahead on different benchmarks.</li></ul><p>In practice the two give similar quality. Stanford publishes pre-trained GloVe vectors (trained on Wikipedia and Gigaword, on Common Crawl, and on Twitter) in sizes such as 50, 100, 200 and 300 dimensions, so many people simply download them rather than train their own.</p><p><strong>Its limits are the same as Word2Vec’s.</strong> GloVe gives one fixed vector per word, so “bank” has a single meaning, and a word that was not in the training vocabulary has no vector at all.</p><h3>FastText: use pieces of words, so no word is unknown</h3><p>FastText, from Piotr Bojanowski, Edouard Grave, Armand Joulin and Tomas Mikolov at Facebook AI Research (2016), extends Skip-gram by representing each word as a bag of its character n-grams, which lets it build a vector for words it has never seen. Word2Vec and GloVe treat “run”, “running” and “runner” as three unrelated symbols. FastText sees that they share letters.</p><p><strong>How it works</strong></p><ol><li>Wrap the word in boundary markers, for example “&lt;where&gt;”.</li><li>Break it into character n-grams, typically of length 3 to 6. For n = 3, “&lt;where&gt;” gives: “&lt;wh”, “whe”, “her”, “ere”, “re&gt;”. The whole word “&lt;where&gt;” is also kept as its own token.</li><li>Give each n-gram its own vector, learned during training.</li><li>The vector for the word is the <strong>sum (or average) of its n-gram vectors</strong>, plus the whole-word vector.</li><li>Train with the Skip-gram objective and negative sampling, exactly as in Word2Vec, but using these composed word vectors.</li></ol><p><strong>What this buys you</strong></p><ul><li><strong>Out-of-vocabulary words get vectors.</strong> A new word, a rare word, or a misspelling such as “recieve” shares many n-grams with “receive”, so FastText builds a sensible vector for it instead of failing.</li><li><strong>Better for morphologically rich languages.</strong> In Finnish, Turkish, German, Hindi and similar languages, a single root can appear in dozens of inflected forms. Sharing n-grams lets rare forms borrow strength from common ones.</li><li><strong>Better on rare words.</strong> A rare word has few training examples, but its n-grams appear in many other words, so its vector is still well trained.</li></ul><p><strong>Example.</strong> Suppose “unhappiness” never appeared in your training text. Its n-grams (“&lt;un”, “unh”, “nha”, “hap”, “app”, “ppi”, “pin”, “ine”, “nes”, “ess”, “ss&gt;” and so on) overlap with “happy”, “happiness”, “unhappy” and “kindness”, all of which were seen. FastText adds up those n-gram vectors and gets a vector that lands near “unhappy” and “sadness”.</p><p><strong>Trade-offs</strong></p><ul><li><strong>Bigger model and slower training</strong>, since there are many n-gram vectors to store and update. The paper hashes n-grams into a fixed number of buckets to keep memory bounded.</li><li><strong>Still static and context-free.</strong> “Bank” is still one vector.</li><li><strong>Spelling similarity can mislead</strong> for pairs that look alike but differ in meaning.</li></ul><p>FastText also ships a very fast <strong>text classifier</strong> that averages word and n-gram embeddings and feeds them to a linear layer. It trains in minutes on millions of sentences and is a strong baseline, so the name is used for both the embedding method and this classifier. Facebook released pre-trained FastText vectors for around 157 languages, which makes it a popular choice for non-English work.</p><h3>Try it yourself in Python</h3><p>Short snippets reproduce everything above: scikit-learn for BoW and TF-IDF, and gensim for Word2Vec, GloVe and FastText. Install with pip install scikit-learn gensim.</p><p><strong>Bag of Words and TF-IDF</strong> [GIST]</p><pre>from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer</pre><pre>docs = [<br>    &quot;the cat sat on the mat&quot;,<br>    &quot;the dog sat on the log&quot;,<br>    &quot;the cat chased the dog&quot;,<br>]</pre><pre>bow = CountVectorizer()<br>X = bow.fit_transform(docs)<br>print(bow.get_feature_names_out())<br>print(X.toarray())</pre><pre>tfidf = TfidfVectorizer()<br>T = tfidf.fit_transform(docs)<br>print(T.toarray().round(3))   # &quot;the&quot; is down-weighted</pre><p>Values differ slightly from the hand-worked numbers because scikit-learn smooths the idf and normalizes each row.</p><p><strong>Word2Vec</strong> [GIST]</p><pre>from gensim.models import Word2Vec<br>import gensim.downloader as api</pre><pre>sentences = [doc.split() for doc in docs]</pre><pre>model = Word2Vec(<br>    sentences,<br>    vector_size=100,   # embedding size d<br>    window=5,<br>    min_count=1,       # only OK for a toy corpus<br>    sg=1,              # 1 = Skip-gram, 0 = CBOW<br>    negative=5,<br>    epochs=50,<br>)<br>print(model.wv[&quot;cat&quot;][:5])</pre><pre># Real effect: Google&#39;s vectors trained on ~100 billion words (about 1.6 GB)<br>wv = api.load(&quot;word2vec-google-news-300&quot;)<br>print(wv.most_similar(positive=[&quot;king&quot;, &quot;woman&quot;], negative=[&quot;man&quot;], topn=1))</pre><p>On three sentences the toy model’s similarities are noise. The pre-trained vectors show the real behaviour, with “queen” at or near the top.</p><p><strong>GloVe</strong> [GIST]</p><pre>import gensim.downloader as api</pre><pre>glove = api.load(&quot;glove-wiki-gigaword-100&quot;)   # about 130 MB<br>print(glove.most_similar(&quot;coffee&quot;, topn=3))<br>print(glove.similarity(&quot;cat&quot;, &quot;dog&quot;))</pre><p><strong>FastText</strong> [GIST]</p><pre>from gensim.models import FastText</pre><pre>ft = FastText(sentences, vector_size=100, window=5, min_count=1,<br>              sg=1, min_n=3, max_n=6, epochs=50)</pre><pre>print(&quot;cats&quot; in ft.wv.key_to_index)   # False: never seen as a whole word<br>vec = ft.wv[&quot;cats&quot;]                   # still works, built from n-grams<br>print(ft.wv.similarity(&quot;cat&quot;, &quot;cats&quot;))</pre><p>A plain Word2Vec model raises a KeyError on “cats” here, while FastText returns a vector. For real use, load Facebook’s pre-trained vectors with gensim’s load_facebook_vectors and the .bin file, since only that file keeps the n-gram vectors.</p><h3>Side by side, and what came after</h3><p>Word2Vec exists because count-based vectors cannot tell you that two words mean similar things, while learned dense vectors can. GloVe and FastText keep that idea and fix different weaknesses. Here is how all five methods compare:</p><ul><li><strong>Bag of Words:</strong> sparse counts, one column per vocabulary word. Represents a document. No similarity of meaning, no word order, unknown words are ignored. Fully interpretable. Good for quick baselines and small data.</li><li><strong>TF-IDF:</strong> sparse weights, same size as BoW. Represents a document. Rewards rare, informative words, but still has no similarity of meaning. Good for search, keyword extraction and baselines.</li><li><strong>Word2Vec:</strong> dense vectors of 100 to 300 numbers, learned from local context windows. Represents a word. Captures similarity of meaning. Unknown words have no vector. Good for semantic similarity and as input to neural networks.</li><li><strong>GloVe:</strong> dense vectors of 50 to 300 numbers, learned from a global co-occurrence matrix. Represents a word. Similar quality to Word2Vec, and easy to download pre-trained. Unknown words have no vector.</li><li><strong>FastText:</strong> dense vectors built from character n-grams. Captures similarity of meaning, handles word forms like run and running, and builds vectors for unseen words and typos. Good for rare words and morphologically rich languages.</li></ul><p><strong>So why Word2Vec?</strong> In short:</p><ul><li><strong>Meaning becomes geometry.</strong> Similar words sit near each other, so a model that learns something about “good” automatically transfers it to “great”.</li><li><strong>Small and dense.</strong> 300 numbers instead of hundreds of thousands, which is easier on memory, faster to compute with, and less prone to overfitting.</li><li><strong>Learned from free data.</strong> No manual labels are needed, just raw text, and the vectors can be reused across many different tasks (an early form of <em>transfer learning</em> for NLP).</li><li><strong>Relationships are captured.</strong> Analogies such as king, man, woman and queen fall out of the geometry without being programmed.</li></ul><p><strong>A fair word of caution.</strong> BoW and TF-IDF are not obsolete. For small datasets, quick baselines, search and keyword extraction they are fast, transparent and often hard to beat. Word2Vec also gives you vectors for <em>words</em>, so to represent a whole document you usually need to average its word vectors, which loses order again.</p><p><strong>Word2Vec’s own limits, and its successors</strong></p><ul><li><strong>One vector per word.</strong> “Bank” has a single vector whether it means a river bank or a money bank. This is called the polysemy problem.</li><li><strong>Unknown words.</strong> A word never seen in training has no vector.</li><li><strong>Static.</strong> The vector does not change with the sentence it appears in.</li></ul><p>The field moved on to fix these, step by step. GloVe and FastText, covered above, are the first two steps:</p><ol><li><strong>GloVe (2014)</strong> replaced sliding-window prediction with a fit to global co-occurrence counts. Quality is similar to Word2Vec.</li><li><strong>FastText (2016)</strong> builds vectors from character n-grams, which fixes unknown words and rich word forms, but each word still has a single vector.</li><li><strong>ELMo (2018)</strong> and <strong>BERT (2018)</strong> produce <em>contextual</em> embeddings, where the vector for “bank” depends on the whole sentence. This solves the polysemy problem, and the Transformer models that followed, including today’s large language models, build on this idea.</li></ol><p><strong>Takeaway.</strong> Bag of Words counts words, TF-IDF counts them smartly, and Word2Vec finally gave words <em>meaning</em> by learning from context. GloVe and FastText refined that idea with global statistics and sub-word pieces. Every modern embedding, from sentence encoders to LLMs, carries forward the same insight: you know a word by the company it keeps.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a6ea47a0c2de" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How Does Gmail Know Which Emails Are Spam?]]></title>
            <link>https://medium.com/@venumadhavjidugu/how-does-gmail-know-which-emails-are-spam-788fb494d09d?source=rss-92f6feb0c064------2</link>
            <guid isPermaLink="false">https://medium.com/p/788fb494d09d</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[classification-algorithms]]></category>
            <category><![CDATA[naive-bayes]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[spam-detection]]></category>
            <dc:creator><![CDATA[Venumadhavjidugu]]></dc:creator>
            <pubDate>Thu, 20 Aug 2026 10:26:18 GMT</pubDate>
            <atom:updated>2026-08-20T10:26:18.459Z</atom:updated>
            <content:encoded><![CDATA[<p>Every day your inbox fills up with messages — some useful, some ads, some obvious spam. Gmail sorts most of this automatically, before you even open the app. How?</p><p>It’s not just scanning for words like “free” or “winner.” It’s <strong>machine learning</strong> — a system that learns patterns from huge amounts of past data and uses them to judge new emails. Here’s the idea, broken into simple steps.</p><h3>1. Spam Detection Is a Classification Problem</h3><p>At its core, the task is simple: look at an email and decide between two options.</p><pre>New Email → [Model] → Spam or Not Spam</pre><p>This is called <strong>binary classification</strong> — there are only two possible answers.</p><h3>2. Turning an Email Into Something a Machine Can Read</h3><p>A computer doesn’t “read” an email like a human does. It first needs the text converted into useful signals, a process called <strong>feature extraction</strong>. Common signals include:</p><ul><li><strong>Words</strong> — “free,” “winner,” “urgent,” “click here”</li><li><strong>Structure</strong> — too many links, odd formatting, excessive capital letters</li><li><strong>Sender info</strong> — where the message came from and how it was delivered</li><li><strong>Behavior</strong> — how people typically react to similar messages</li></ul><p>No single signal decides the outcome — the model weighs many of them together.</p><h3>3. Learning From Labeled Examples</h3><p>The model is trained on thousands (or millions) of emails, each already tagged as <strong>Spam</strong> or <strong>Not Spam</strong>. It searches for patterns — for instance, noticing that words like <em>“FREE,” “PRIZE,”</em> and <em>“CLICK HERE”</em> tend to appear together far more often in spam. It’s not memorizing exact emails; it’s learning patterns that generalize to new, unseen ones.</p><h3>4. A Classic Algorithm: Naive Bayes</h3><p>One simple but powerful method for text classification is <strong>Naive Bayes</strong>, based on probability. It asks:</p><blockquote><em>Given the words in this email, how likely is it to be spam?</em></blockquote><p>It compares the probability of spam vs. not-spam given the words present, and picks whichever is higher.</p><p><strong>Why “naive”?</strong> The algorithm assumes each word contributes independently to the prediction — ignoring how words relate to each other in real sentences. That’s an oversimplification, but it still works surprisingly well in practice.</p><h3>5. Turning Words Into Numbers</h3><p>Machine learning models need numbers, not raw text. A simple technique called <strong>Bag of Words</strong> does this: it checks whether each word from a known vocabulary appears in the email, turning a sentence into a list of 1s and 0s.</p><h3>6. Handling Words the Model Has Never Seen</h3><p>New emails often contain words the model never encountered during training. A related issue: if a word never appeared in spam examples, its estimated probability could be exactly zero — which breaks the math (multiplying by zero wipes out everything). The fix is called <strong>Laplace smoothing</strong>, a small adjustment that keeps probabilities from ever hitting zero.</p><h3>7. A Minimal Python Example</h3><pre>from sklearn.feature_extraction.text import CountVectorizer<br>from sklearn.naive_bayes import MultinomialNB</pre><pre>emails = [<br>    &quot;Win a free prize now&quot;,<br>    &quot;Free lottery reward&quot;,<br>    &quot;Meeting at 10 AM&quot;,<br>    &quot;Project discussion tomorrow&quot;,<br>]<br>labels = [&quot;spam&quot;, &quot;spam&quot;, &quot;not_spam&quot;, &quot;not_spam&quot;]</pre><pre>vectorizer = CountVectorizer()<br>X = vectorizer.fit_transform(emails)</pre><pre>model = MultinomialNB()<br>model.fit(X, labels)</pre><pre>new_email = [&quot;Congratulations you won a free prize&quot;]<br>prediction = model.predict(vectorizer.transform(new_email))<br>print(prediction)</pre><p>Pipeline: <strong>Raw Email → Clean Text → Numbers → Model → Prediction</strong></p><h3>8. Why Real Spam Filters Are More Complex</h3><p>Spammers adapt. If “FREE” gets flagged, they switch to “F.R.E.E,” “Fr33,” or hide the word inside an image. This creates an ongoing back-and-forth:</p><pre>Filters improve → Spammers change tactics → New patterns appear → Filters adapt again</pre><h3>9. Two Kinds of Mistakes</h3><ul><li><strong>False Positive</strong> — a real email wrongly marked as spam (e.g., an important message buried in your Spam folder)</li><li><strong>False Negative</strong> — an actual spam email that slips into your inbox</li></ul><p>Both matter, and reducing one often increases the other.</p><h3>10. Why “Accuracy” Alone Is Misleading</h3><p>Imagine 10,000 emails, where only 100 are actually spam. A lazy model that labels <em>everything</em> as “Not Spam” would be <strong>99% accurate</strong> — yet it would catch zero spam. That’s why we also track:</p><ul><li><strong>Precision</strong> — of the emails flagged as spam, how many really were spam?</li><li><strong>Recall</strong> — of all the real spam, how much did the model actually catch?</li></ul><p>There’s usually a trade-off between the two.</p><h3>11. Why Machine Learning Beats Hand-Written Rules</h3><p>Early spam filters used rigid rules like <em>“if the email contains ‘free,’ mark it spam.”</em> But that fails instantly on something like “The free meeting room is available at 3 PM.” Piling on more and more manual rules becomes unmanageable. Machine learning instead lets the system <strong>learn the rules itself</strong> from examples.</p><h3>12. The Big-Picture Workflow</h3><p>Spam detection is a great mini-tour of the entire ML process:</p><ol><li><strong>Define the problem</strong> — spam vs. not spam</li><li><strong>Collect data</strong> — historical emails</li><li><strong>Label the data</strong> — mark each as spam/not spam</li><li><strong>Extract features</strong> — words, links, structure, sender info</li><li><strong>Convert to numbers</strong> — text → numerical form</li><li><strong>Train a model</strong> — learn patterns from labeled data</li><li><strong>Test it</strong> — check performance on unseen emails</li><li><strong>Evaluate</strong> — precision, recall, F1-score</li><li><strong>Improve</strong> — better data, features, and models over time</li></ol><h3>Conclusion</h3><p>Gmail doesn’t just look for the word “FREE.” It combines many signals, converts them into numbers, and uses a trained model to estimate the likelihood that a message is spam — all in milliseconds. The same basic idea (classification: learn from labeled examples, then predict on new data) powers many other systems too, like credit approval, disease prediction, and fraud detection.</p><p>The core lesson of machine learning: <strong>it’s not about memorizing past examples — it’s about finding patterns that generalize to new, unseen situations.</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=788fb494d09d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Descriptive Statistics]]></title>
            <link>https://medium.com/@venumadhavjidugu/descriptive-statistics-b7072f49de1c?source=rss-92f6feb0c064------2</link>
            <guid isPermaLink="false">https://medium.com/p/b7072f49de1c</guid>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[descriptive-statistics]]></category>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[statistics]]></category>
            <dc:creator><![CDATA[Venumadhavjidugu]]></dc:creator>
            <pubDate>Tue, 19 May 2026 14:41:49 GMT</pubDate>
            <atom:updated>2026-05-19T14:41:49.777Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*EpN1QKNwdCI-2wwX.jpeg" /></figure><blockquote>Statistics helps us understand data and make better decisions. Businesses, apps, hospitals, banks, and even social media platforms use statistics every day.</blockquote><p>Imagine you own a small tea shop. Every day, customers spend different amounts of money. Some spend ₹10, some ₹50, and a few spend ₹500 for large orders. Looking at all these numbers directly can be confusing. Statistics helps us summarize and understand the data easily.</p><p>Descriptive statistics mainly answers three questions:</p><ol><li>What is the center of the data?(Central Tendency)</li><li>How spread out is the data?(Dispersion)</li><li>What shape does the data follow?(Shape)</li></ol><h3>1. Measures of Central Tendency</h3><h4>Introduction</h4><blockquote>Measures of central tendency help us find the “center” or “typical value” of data.</blockquote><p>Suppose a teacher asks:<br> “What is the average mark of the class?”</p><p>The teacher is trying to understand the general performance of students using one representative value.</p><p>But sometimes a single value can be misleading.</p><p>For example:</p><ul><li>9 employees earn ₹30,000</li><li>1 manager earns ₹5,00,000</li></ul><p>The average salary becomes very high, even though most employees earn much less.</p><p>This is why we use different methods like Mean, Median, and Mode.</p><h3>Mean</h3><h4>What Mean Is:</h4><blockquote>Mean is what most people call the “average.”</blockquote><p>We calculate it by adding all values and dividing by the number of values. It gives a single value representing the entire dataset.</p><p>Example:</p><p>A shop earns:<br> ₹1000, ₹1200, ₹1300, ₹1500, ₹1000</p><p>The mean income is: (1000 + 1200 + 1300 + 1500 + 1000) / 5 = ₹1200</p><p>So the shop usually earns around ₹1200 daily.</p><h4>Why It Is Called the Balance Point:</h4><p>Mean is called the balance point because data balances around it.</p><p>Imagine weights placed on a ruler. The mean acts like the center where the ruler balances perfectly.</p><p>Values below the mean and above the mean balance each other.</p><h4>Effect of Extreme Values</h4><p>Mean gets heavily affected by very large or very small values.</p><p>Example: Marks:<br> 70, 75, 80, 85, 500</p><p>The value 500 changes the average drastically even though it is unusual.</p><p>This makes the mean misleading sometimes.</p><h4>When Mean Is Useful</h4><p>Mean is useful when:</p><ul><li>Data is fairly consistent</li><li>There are no extreme outliers</li><li>We want overall performance</li></ul><p>Examples:</p><ul><li>Average temperature</li><li>Average sales</li><li>Average marks</li></ul><h4>When Mean Gives Wrong Understanding :</h4><p>Mean gives wrong understanding when data contains outliers.</p><p>Examples:</p><ul><li>Salary distribution</li><li>House prices</li><li>Income data</li></ul><p>A few rich people can increase the average income even though most people earn less.</p><h3>Median</h3><h4>What Median Means :</h4><blockquote>Median is the middle value after arranging data in order.</blockquote><p>Example: 2, 4, 5, 8, 10</p><p>Median = 5</p><p>It divides data into two equal halves.</p><h4>Why It Is Not Affected by Outliers :</h4><p>Median only depends on position, not extreme values.</p><p>Example:</p><p>10, 12, 15, 18, 500</p><p>Median = 15</p><p>Even though 500 is huge, the median stays stable.</p><h4>Why Businesses Prefer Median :</h4><p>Businesses often prefer median because it represents the “typical” value better.</p><h4>Salary Example</h4><p>If one CEO earns crores while employees earn normal salaries, the mean becomes misleading.</p><p>Median salary shows what a typical employee earns.</p><h4>House Price Example</h4><p>A few luxury villas can increase average house prices.</p><p>Median house price gives a more realistic market picture.</p><h3>Mode</h3><h4>What Mode Represents</h4><blockquote>Mode is the value that appears most frequently.</blockquote><p>Example:</p><p>2, 3, 3, 5, 7</p><p>Mode = 3</p><h3>Why It Is Useful for Categorical Data</h3><p>Mode works well even for non-numerical data.</p><p>Example:</p><p>Most popular:</p><ul><li>Mobile brand</li><li>Ice cream flavor</li><li>Clothing size</li></ul><p>A clothing store may find that “Medium” size sells the most. That size becomes the mode.</p><p>This helps businesses manage inventory properly.</p><h3>2. Measures of Dispersion</h3><h4>Introduction</h4><p>Finding the center alone is not enough.</p><p>Two classes may have the same average marks but very different student performance.</p><p>Example:</p><p>Class A:<br> 48, 49, 50, 51, 52</p><p>Class B:<br> 10, 20, 50, 80, 90</p><p>Both may have similar means, but Class B is much more spread out.</p><blockquote>Measures of dispersion tell us how scattered data is.</blockquote><h3>Range</h3><h4>Meaning</h4><blockquote>Range is the difference between maximum and minimum values.</blockquote><p>Example:</p><p>Marks:<br> 40, 50, 60, 90</p><p>Range = 90 − 40 = 50</p><h4>Advantages</h4><ul><li>Very simple</li><li>Quick understanding of spread</li></ul><h4>Limitations</h4><ul><li>Uses only two values</li><li>Highly affected by outliers</li></ul><p>Example:</p><p>10, 12, 13, 15, 500</p><p>Range becomes huge because of one unusual value.</p><h3>Variance</h3><h4>What Variance Measures</h4><blockquote>Variance measures how far data points are from the mean on average.</blockquote><p>If values are close to the mean, variance is small.</p><p>If values are scattered far away, variance is large.</p><h4>Why Differences Are Squared :</h4><p>Some values are above the mean and some are below it.</p><p>If we add differences directly, positives and negatives cancel out.</p><p>Squaring makes all differences positive.</p><p>It also gives more importance to large deviations.</p><h4>Meaning of Large Variance</h4><p>Large variance means:</p><ul><li>Data is inconsistent</li><li>Performance varies a lot</li><li>Risk or uncertainty is high</li></ul><p>Example:<br> A business with highly changing monthly profits has large variance.</p><h3>Standard Deviation</h3><h4>Why It Is Preferred Over Variance</h4><p>Variance uses squared units, which are hard to interpret.</p><p>Standard deviation converts it back to normal units.</p><p>This makes it easier to understand.</p><h4>Data Consistency</h4><p>Small standard deviation:</p><ul><li>Data is consistent</li><li>Values stay near the average</li></ul><p>Large standard deviation:</p><ul><li>Data changes a lot</li><li>Results are unpredictable</li></ul><h4>Relationship with Uncertainty and Risk</h4><p>Finance uses standard deviation to measure investment risk.</p><p>Low standard deviation:</p><ul><li>Stable returns</li><li>Lower risk</li></ul><p>High standard deviation:</p><ul><li>Unstable returns</li><li>Higher risk</li></ul><p>Companies also use it for:</p><ul><li>Demand prediction</li><li>Sales forecasting</li><li>Quality control</li></ul><h4><strong>The 68–95–99.7 Rule</strong> (for normal distributions):</h4><ul><li>68% of data falls within 1 standard deviation of the mean</li><li>95% falls within 2 standard deviations</li><li>99.7% falls within 3 standard deviations</li></ul><p>This is incredibly useful for understanding probability and risk!</p><h3>Quartiles and Interquartile Range (IQR)</h3><h4>Q1, Q2, Q3</h4><blockquote>Quartiles divide data into four equal parts.</blockquote><ul><li>Q1 → 25% of data</li><li>Q2 → Median</li><li>Q3 → 75% of data</li></ul><h4>Dividing Data into Four Parts</h4><p>Example:</p><p>10, 20, 30, 40, 50, 60, 70, 80</p><ul><li>Q1 = 25</li><li>Q2 = 45</li><li>Q3 = 65</li></ul><p>This helps understand how data is distributed.</p><h3>Interquartile Range (IQR)</h3><blockquote>IQR = Q3 − Q1</blockquote><p>It measures the spread of the middle 50% of data.</p><h4>Detecting Outliers Using IQR</h4><p>Outliers are unusually large or small values.</p><p>Using IQR:</p><ul><li>Lower limit = Q1 − 1.5 × IQR</li><li>Upper limit = Q3 + 1.5 × IQR</li></ul><p>Values outside these limits are considered outliers.</p><h3>Business Examples</h3><h4>E-Commerce</h4><p>An online store checks unusual spending behavior to detect fraud.</p><h4>Banking</h4><p>Banks identify suspicious transactions using outlier detection.</p><h4>Manufacturing</h4><p>Factories detect defective products with unusual measurements.</p><h3>Part 3: Measures of Shape</h3><h4>Why Data Is Not Always Symmetrical</h4><p>Not all data looks like a bell curve. Some data is skewed (lopsided), some has heavy tails, and some is perfectly balanced. Understanding the shape helps us:</p><ul><li>Choose the right measure of center</li><li>Understand what’s typical</li><li>Spot unusual patterns</li></ul><h3>Skewness: When Data Leans to One Side</h3><blockquote>Skewness measures whether data is symmetrical or lopsided.</blockquote><p><strong>Left-skewed (Negatively Skewed) Data:</strong> The tail points to the left. Most data is on the right side.</p><p><strong>Visual example</strong>: Exam scores in a difficult class Mean &lt; Median &lt; Mode</p><ul><li>The small left tail pulls the mean down</li><li>Median is more representative than mean</li></ul><p><strong>Business example</strong>: If most customers are satisfied (high ratings) but a few had terrible experiences (very low ratings), satisfaction scores are left-skewed.</p><p><strong>Right-skewed (Positively Skewed) Data:</strong> The tail points to the right. Most data is on the left side.</p><p><strong>Real example</strong>: Income distribution. Most people earn $30K-$100K, but a few CEOs and entrepreneurs earn millions. The distribution is right-skewed.</p><p><strong>Characteristics:</strong></p><ul><li>Mode &lt; Median &lt; Mean</li><li>The long right tail pulls the mean up</li><li>Median is more representative than mean</li></ul><p><strong>Business example</strong>: This is why the median salary story earlier was so important! Right-skewed income distribution is common everywhere.</p><p><strong>Relationship between mean and median:</strong></p><ul><li><strong>Symmetric data</strong>: Mean ≈ Median</li><li><strong>Left-skewed</strong>: Mean &lt; Median</li><li><strong>Right-skewed</strong>: Mean &gt; Median</li></ul><p>This is a quick trick to spot skewness without drawing graphs!</p><h3>Kurtosis: Understanding the Tails</h3><blockquote>Kurtosis measures how “thick” or “heavy” the tails of a distribution are, and how peaked the center is.</blockquote><p><strong>Meaning of tails:</strong> “Tails” are the extreme ends of your data — the unusual, rare values.</p><ul><li><strong>Heavy tails</strong>: Extreme values appear more often than expected (outliers are common)</li><li><strong>Light tails</strong>: Extreme values are rare (outliers are very unusual)</li></ul><p><strong>Three types of distributions:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*8BqzTIDI-2eeLvMV.png" /></figure><p><strong>1. Normal Distribution (Mesokurtic)</strong></p><ul><li>Moderate tails</li><li>Fits the bell curve we expect</li><li>Not too many outliers</li><li><strong>Example</strong>: Human height, test scores in large classes</li></ul><p><strong>2. Flat/Heavy-tailed Distribution (Leptokurtic)</strong></p><ul><li>More extreme values than normal</li><li>Tails are thicker</li><li>More outliers</li><li><strong>Example</strong>: Stock market returns (sudden crashes happen more often than models predict), customer purchase amounts, fraud amounts</li></ul><p><strong>3. Peaked/Light-tailed Distribution (Platykurtic)</strong></p><ul><li>Fewer extreme values than normal</li><li>Flatter distribution</li><li>Very few outliers</li><li><strong>Example</strong>: Temperature in a stable climate, quality control in excellent manufacturing</li></ul><p><strong>Importance in finance:</strong> This is critical! Most financial models assume normal distribution. But real markets have heavy tails.</p><p><strong>Real story</strong>: In 2008, the financial crisis happened because banks assumed stock returns follow a normal distribution. They didn’t expect the extreme movements that occurred. With heavy-tailed distribution understanding, they could have better predicted and prepared.</p><p><strong>Importance in fraud detection:</strong> Fraud often shows heavy tails. Fraudsters make unusual, extreme transactions. By identifying heavy-tailed distributions, fraud detection systems spot suspicious patterns.</p><p><strong>Business example</strong>:</p><ul><li>Normal transactions: $20, $50, $100, $120, $95, $150… (mostly between $50-$150)</li><li>Fraud pattern: $50, $45, $800, $30, $5000, $65… (includes extreme values)</li></ul><p>The second shows heavy tails — a red flag for fraud!</p><h3>Practical Summary: Choosing Your Tools</h3><h4>When to use what:</h4><p><strong>For center:</strong></p><ul><li><strong>Mean</strong>: General data with no extreme outliers</li><li><strong>Median</strong>: Skewed data or when outliers exist (salaries, house prices)</li><li><strong>Mode</strong>: Categorical data or “most popular” items</li></ul><p><strong>For spread:</strong></p><ul><li><strong>Range</strong>: Quick, rough estimate</li><li><strong>Variance</strong>: Mathematical calculations (further analysis)</li><li><strong>Standard Deviation</strong>: Most practical, intuitive measure</li><li><strong>IQR</strong>: Robust measure that handles outliers well; great for detecting anomalies</li></ul><p><strong>For shape:</strong></p><ul><li><strong>Skewness</strong>: Understand if data is balanced or lopsided</li><li><strong>Kurtosis</strong>: Expect outliers or not</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b7072f49de1c" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>