<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by SannySingh on Medium]]></title>
        <description><![CDATA[Stories by SannySingh on Medium]]></description>
        <link>https://medium.com/@sannysingh1?source=rss-dd796636e530------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*EyCKMgfQ6huekPqmjWB2fw.png</url>
            <title>Stories by SannySingh on Medium</title>
            <link>https://medium.com/@sannysingh1?source=rss-dd796636e530------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 01:11:51 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@sannysingh1/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[The Architect’s Guide to Machine Learning: Building Your First Pipeline]]></title>
            <link>https://medium.com/@sannysingh1/the-architects-guide-to-machine-learning-building-your-first-pipeline-2f365d33437f?source=rss-dd796636e530------2</link>
            <guid isPermaLink="false">https://medium.com/p/2f365d33437f</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[normalization]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[data-preprocessing]]></category>
            <dc:creator><![CDATA[SannySingh]]></dc:creator>
            <pubDate>Thu, 07 May 2026 21:01:00 GMT</pubDate>
            <atom:updated>2026-05-07T21:01:00.687Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UAUgBFBpuJqtM9S3JJ-NGw.png" /><figcaption>Machine Learning Pipeline</figcaption></figure><p>Welcome back to our journey into the heart of artificial intelligence! If you’ve been following along, you know that building a Machine Learning (ML) model isn’t just about writing a few lines of code and calling it a day. It’s more like building a skyscraper; if the foundation is shaky, the whole thing comes crashing down.<br>To help us navigate the roadmap ahead, we’re diving straight into the core foundations. Together, we are going to explore:</p><h3><strong>Module 1: The Machine Learning Pipeline and Data Preprocessing</strong></h3><p>The Story of the “<em>Eco-Guardian</em>”<br>Imagine a small village that wants to build a robot called the “Eco-Guardian”. Its job is to predict exactly when the community garden needs water so they don’t waste a single drop. To build this robot, the villagers can’t just give it a bucket and hope for the best. They need a Machine Learning Lifecycle — a structured process that defines how models are developed, deployed, and maintained.</p><p><strong>Step 1: Defining the Mission (<em>Problem Definition</em>)</strong><br>First, the villagers must identify the business problem . They define the success criteria: “The robot must predict soil dryness with 95% accuracy to save water.”</p><p><strong>Step 2: Gathering the Ingredients (<em>Data Collection</em>)</strong><br>Next, they collect datasets from soil sensors, weather stations, and old farming logs. It is important to remember that the quality and variety of this raw data directly affect how well the final model will perform. They need a high quantity of diverse data to capture different weather patterns, like summer heatwaves or unexpected spring rain.</p><p><strong>Step 3: The Scrub-Down (<em>Data Cleaning &amp; Preprocessing</em>)</strong><br>Raw data is almost always messy and unstructured. The villagers might find that some sensors ran out of battery, resulting in missing values, or others gave impossible readings, known as outliers. This step involves standardizing formats and scaling values to ensure the data is well-organized for analysis.</p><blockquote><strong>Do You Know?</strong> Data Preprocessing and Cleaning are often considered the most time-consuming parts of a Machine Learning Engineer’s job, sometimes taking up to <strong>80% of the total project time</strong>. It is a critical foundational step because even the most advanced algorithm cannot overcome the “Garbage In, Garbage Out” rule — properly processed data is the only way to prevent misleading inputs and ensure reliable predictions.</blockquote><p><strong>Step 4: Finding the “Why” (<em>Exploratory Data Analysis</em>)</strong><br>The villagers use statistical and visual tools to explore patterns in the data. By identifying underlying trends and potential challenges, they gain valuable insights that help with decision-making in later stages.</p><p><strong>Step 5: Shaping the Data (<em>Feature Engineering &amp; Selection</em>)</strong><br>This is a transformative process where they create new features or transform existing ones to capture better patterns. They also identify a subset of features that most significantly impact the model’s performance, reducing complexity while maintaining accuracy.</p><p><strong>Step 6: Choosing the Brain (<em>Model Selection</em>)</strong><br>Now they must find an algorithm that aligns with the nature of the data and the complexity of the problem. They evaluate factors like performance and scalability, often experimenting with different models to find the best fit.</p><p><strong>Step 7: The Training Ground (<em>Model Training</em>)</strong><br>The model is exposed to historical data, allowing it to learn the patterns and relationships within the dataset. This is an iterative process where parameters are adjusted to minimize errors and enhance predictive capabilities.</p><p><strong>Step 8: The Final Exam (<em>Evaluation and Tuning</em>)</strong><br>Before the system goes live, it undergoes rigorous testing against validation or test datasets to see how it handles new, unseen data. If the performance isn’t high enough, they tune the hyperparameters to improve accuracy and reliability.</p><p><strong>Step 9: Going Live (<em>Model Deployment</em>)</strong><br>The predictive model is finally integrated with the village’s existing systems. This allows the community to use the predictions for real-world decision-making through dedicated APIs or pipelines.</p><p><strong>Step 10: The Long Watch (<em>Monitoring and Maintenance</em>)</strong><br>The world changes — perhaps the climate shifts or a sensor becomes faulty. Models must be monitored regularly to detect “data drift” or drops in accuracy. When performance dips, the model is updated and retrained to stay reliable.</p><h4><strong>Why Preprocessing is the Real Hero?</strong></h4><p>Data preprocessing involves cleaning, transforming, and organizing raw data so it is ready for modeling. Properly processed data prevents “misleading inputs”, which leads to much more reliable predictions.</p><p><strong>Common Data “Villains” (<em>Anomalies</em>)</strong></p><ul><li><strong>Missing Values:</strong> Gaps caused by system failures or human error during collection.</li><li><strong>Duplicate Records:</strong> Redundant observations that do not add new information.</li><li><strong>Incorrect Data Types:</strong> Having numerical data accidentally stored as text.</li><li><strong>Outliers:</strong> Extremely high or low values that can distort statistical measures.</li><li><strong>Structural Errors:</strong> Having data in different formats or inconsistent measurement scales.</li></ul><blockquote><strong>Do You Know?</strong> A model that works perfectly today might fail tomorrow due to something called <strong>Data Drift</strong>. This happens when real-world patterns change over time, causing the model’s accuracy to drop. This is why continuous monitoring and regular retraining are essential parts of the machine learning lifecycle to keep models reliable in the real world.</blockquote><h4><strong>The Preprocessing Toolkit: Scaling and Normalization</strong></h4><p>Feature engineering is the transformative process of creating, transforming, or selecting the most important features from our raw data to significantly improve how our model performs. By applying these scaling and normalization techniques, you help our model learn complex patterns more effectively, improve the accuracy of its predictions, and help it generalize better to new data while reducing the risk of overfitting. Here is a closer look at the five key tools in this toolkit:</p><h4><strong>1. Absolute Maximum Scaling</strong></h4><p>This method rescales our data by dividing each individual value by the maximum absolute value of that specific feature. This transformation ensures all values fall within a range of -1 to 1.</p><ul><li><strong>Best Use Case:</strong> It works best for clean datasets. However, it is very sensitive to outliers; a single extreme value can drastically increase the maximum and reduce the overall quality of the scaling for the rest of the data.</li></ul><h4><strong>2. Min-Max Scaling</strong></h4><p>One of the most common methods, Min-Max Scaling, rescales features by subtracting the minimum value and dividing by the range (the difference between the maximum and minimum).</p><ul><li><strong>The Result:</strong> It maps all feature values to a neat range between 0 and 1 while preserving the original distribution of the data.</li><li><strong>Implementation:</strong> In Python, we can use the MinMaxScaler() object to calculate these values and transform the dataset.</li></ul><h4><strong>3. Normalization (Vector Normalization)</strong></h4><p>Unlike other methods that look at the size of the numbers, Normalization focuses on the direction of the data points rather than their magnitude. It scales each data sample so that its vector length (the Euclidean norm) becomes exactly 1.</p><ul><li><strong>Best Use Case:</strong> This is incredibly useful for tasks like text classification and clustering where the relationship between features matters more than their raw size.</li><li><strong>Implementation:</strong> We can use the Normalizer() object to ensure each row in our data has a consistent vector length of 1.</li></ul><h4><strong>4. Standardization (Z-score Scaling)</strong></h4><p>Standardization transforms our data so that features have a mean of 0 and a variance of 1. It does this by centering the data around the mean and scaling it based on the standard deviation.</p><ul><li><strong>Best Use Case:</strong> It is highly effective for data that is approximately normally distributed (following a bell curve) and helps many algorithms — especially those that assume Gaussian distributions — perform much better.</li><li><strong>Implementation:</strong> Use <em>StandardScaler()</em> to subtract the mean (µ) and divide by the standard deviation (σ) across your dataset.</li></ul><h4><strong>5. Robust Scaling</strong></h4><p>When our data is “noisy” or contains extreme values (outliers) that would break other scalers, Robust Scaling is the answer. Instead of using the mean and standard deviation, which outliers can easily pull away, it uses the median and the Interquartile Range (IQR).</p><ul><li><strong>The Benefit:</strong> This makes the scaling much less sensitive to extreme outliers and skewed data, ensuring those rare extreme points don’t distort the rest of your features during the learning process.</li><li><strong>Implementation:</strong> Use <em>RobustScaler()</em> to scale data while effectively reducing the negative influence of outliers.</li></ul><blockquote><strong>Do You Know? Robust Scaling</strong> is your best friend when dealing with “noisy” data. Unlike other scaling methods that can be easily distorted by a single extreme value, Robust Scaling uses the <strong>median</strong> and <strong>Interquartile Range (IQR)</strong> to ensure that outliers don’t ruin the scaling quality for the rest of your dataset.</blockquote><p>Expanding our technical vocabulary is key to mastering the pipeline. Here are several more critical terms with more detailed explanations:</p><p><strong>Feature Engineering:</strong> This is the transformative process of creating new features or modifying existing ones from raw data to better capture underlying patterns. By doing this, we help the model learn complex relationships more effectively and improve overall prediction accuracy.</p><p><strong>Data Drift:</strong> This occurs when the statistical properties of the target variable change over time in unforeseen ways, making a previously accurate model less reliable in the real world. It often requires the model to be updated or retrained to maintain its performance levels.</p><p><strong>Hyperparameters:</strong> These are the external configuration settings of a model that cannot be learned directly from the data during training. They must be manually adjusted and “tuned” to find the perfect balance for the model’s predictive accuracy and robustness.</p><p><strong>Imputation:</strong> This is a statistical technique used to handle missing data by filling in the gaps with estimated values like the mean, median, or mode. It ensures the integrity of our dataset so that missing information doesn’t introduce bias or force us to delete valuable records.</p><p><strong>Feature Selection:</strong> This process involves identifying and choosing only the subset of features that significantly impact the model’s performance. It helps reduce the computational complexity of the model while ensuring it remains efficient and accurate.</p><p><strong>Interquartile Range (IQR):</strong> This represents the range between the 25th and 75th percentiles of our data, capturing the middle 50% of the distribution. It is a key component in methods like “Robust Scaling” because it helps measure the spread of data without being influenced by extreme outliers.</p><h4><strong>Pros and Cons of Data Preprocessing</strong><br><strong>Pros</strong></h4><ul><li>Models can learn much more meaningful patterns from well-structured and clean data.</li><li>Predictions become more reliable by preventing misleadning or noisy inputs from reaching the models</li><li>The process helps identify and remove sensitive information, which improves overall data security.</li><li>Organized and standardized data is simpler to process, enhancing the model’s speed and performance.</li><li>Proper scaling and transformation allow the model to generalize better to new, unseen data.</li></ul><p><strong>Cons</strong></p><ul><li>The process is often extremely time-consuming, particularly when dealing with large or complex datasets.</li><li>Aggressive cleaning can lead to the accidental loss of important information if not handled carefully.</li><li>Performing these steps correctly requires significant expertise, effort, and sometimes specialized tools.</li><li>If too much data is removed during the cleaning phase, it can inadvertently contribute to overfitting.</li></ul><h4><strong>Wrapping Up: The Road to Robust Models</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rqbkf3H_dEKFAWVRZV9cVA.png" /></figure><p>Data preprocessing might not always feel as “exciting” as training a complex neural network, but it is the silent engine that drives every successful AI project. By focusing on detecting and correcting errors in raw data, we are ensuring that our analysis is not just effective, but truly reliable for real-world use.<br>While the process requires a significant investment of time, effort, and expertise, the payoff is undeniable. Clean and well-structured data allows our models to learn the deep, meaningful patterns it need to make accurate predictions, rather than getting distracted by noise and errors.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2f365d33437f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Pandas — From Math to Management]]></title>
            <link>https://medium.com/@sannysingh1/pandas-from-math-to-management-f26e93e01656?source=rss-dd796636e530------2</link>
            <guid isPermaLink="false">https://medium.com/p/f26e93e01656</guid>
            <category><![CDATA[pandas]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[SannySingh]]></dc:creator>
            <pubDate>Tue, 14 Apr 2026 21:31:01 GMT</pubDate>
            <atom:updated>2026-04-14T21:31:01.464Z</atom:updated>
            <content:encoded><![CDATA[<p>Welcome back! In our last log, we built our “Mathematical Engine” using NumPy. We learned how to process numbers at lightning speed. But in the real world of Machine Learning, data almost never comes to us as neat, mathematical arrays. It comes as messy spreadsheets, giants CSV files, and complex SQL databases.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*n84zlb-FPdYGVBuqSIZkbw.png" /></figure><p>To bridge the gap between raw data and our ML models, Lets dive into <strong>Pandas, </strong>the library that makes data analysis, cleaning, and manipulation feel like magic.</p><h3>What is Pandas?</h3><p>Think of Pandas as Excel o steroids, but powered by Python. While Excel might freeze or crash if we try to load a million rows, Pandas handles massive datasets with ease. It provides highly optimized functions for analyzing, cleaning, exploring, and manipulating data, making it the absolute industry standard for data science.</p><p>To get started, install it via terminal (using pip install pandascommand) and import it using its standard alias:</p><pre>import pandas as pd</pre><blockquote><strong>Do You Know?</strong> The name “Pandas” doesn’t actually come from the cute bear! It’s a portmanteau of “Panel Data”, an econometrics term for data sets that include observations over multiple time periods for the same individuals. It was created by developer <strong>Wer Mckinney</strong> in 2008 because he was frustrated by the lack of good data analysis tools in Python.</blockquote><h3>1. Series &amp; DataFrames: The Core Structures</h3><p>Pandas handles data `using` two main structures. If we understand these, we understand Pandas.</p><h4><strong>The Series (1-D)</strong></h4><p>A Series is like a single column in a table. It’s a one-dimensional array holding data of any type. Think of a Series as a single “feature” in our dataset — like a whole column dedicated just to “Age” or “Price.” By default, Pandas assigns index numbers (0, 1, 2) to each value, but we can also create our own custom labels.</p><pre>calories = [420, 380, 390]<br><br># Creating a Series with custom labels instead of 0, 1, 2<br>my_series = pd.Series(calories, index=[&quot;Day 1&quot;, &quot;Day 2&quot;, &quot;Day 3&quot;])<br>print(my_series)</pre><h4><strong>The DataFrame (2-D)</strong></h4><p>This is the star of the show. A DataFrame is essentially a collection of Series glued together. It is a 2-D data structure, exactly like a table with rows and columns. This grid format is exactly how a machine learning model expects to receive its training data.</p><pre># We can build a DataFrame using a Python dictionary<br>data = {<br>  &quot;calories&quot;: [420, 380, 390],<br>  &quot;duration&quot;: [50, 40, 45]<br>}<br><br># Loading the data into a DataFrame object<br>df = pd.DataFrame(data) <br>print(df)</pre><h3>2. Loading Files (Bringing Data In)</h3><p>In the real world, we rarely build datasets from scratch by typing them out. We pull them from company databases or download them from open-source sites like Kaggle. Pandas shines when loading massive external files. It can read almost anything:</p><ul><li><strong>CSVs (Comma Separated Values):</strong> The most common format. df = pd.read_csv(&#39;data.csv&#39;)</li><li><strong>JSON (Plain text object format):</strong> Great for web data. df = pd.read_json(&#39;data.json&#39;)</li><li><strong>SQL Databases:</strong> We can query databases directly into a DataFrame.</li></ul><pre>import sqlite3<br>conn = sqlite3.connect(&quot;my_database.db&quot;)<br># Write your SQL query right inside Pandas<br>df = pd.read_sql(&quot;SELECT * FROM my_table&quot;, conn)</pre><blockquote><strong>Do You Know?</strong> Pandas can also output your cleaned data back into a file! If you finish fixing a messy dataset, just use df.to_csv(&quot;clean_data.csv&quot;, index=False) to save your hard work and share it with your team.</blockquote><h3>3. Viewing &amp; Understanding: The Data Detective</h3><p>Before feeding data to an algorithm, We must know its shape and its flaws. ML algorithms will crash if they encounter unexpected text or missing numbers. Pandas gives us a built-in magnifying glass to inspect our data.</p><ul><li><strong>Quick Peeks:</strong> df.head() method returns the top 5 rows, giving us a quick sense of the data. df.tail() method gives us the bottom 5.</li><li><strong>The Blueprint:</strong> df.shape method returns a simple tuple (rows, columns) telling us exactly how massive our dataset is.</li><li><strong>The Summary:</strong> df.info() method is arguably our most important command. It tells us the data types of our columns and, crucially, how many <em>Non-Null</em> (non-empty) values exist.</li><li><strong>The Math Summary:</strong> df.describe() instantly calculates the count, mean, standard deviation, min, and max for all our numeric columns. It&#39;s a great way to spot crazy <strong>outliers.</strong></li></ul><p><strong>Outliers</strong> are extreme values that lie far away from the majority of the data points in a dataset. These values do not follow the general pattern of the data and can occur due to various reasons like data entry mistakes, measurement errors or natural variations.</p><pre>import pandas as pd<br><br># 1. Let&#39;s create a quick sample DataFrame to inspect<br>data = {<br>    &quot;Name&quot;: [&quot;Alice&quot;, &quot;Bob&quot;, &quot;Charlie&quot;, &quot;David&quot;, &quot;Eve&quot;, &quot;Frank&quot;, &quot;Grace&quot;],<br>    &quot;Age&quot;: [25, 30, 35, 40, 22, 28, 32],<br>    &quot;Sales&quot;: [250, 300, 150, 400, 200, 350, 500]<br>}<br>df = pd.DataFrame(data)<br># Returns the top 5 rows by default<br>print(df.head()) <br><br># Returns the bottom 5 rows (you can also pass a number like df.tail(2))<br>print(df.tail()) <br><br># Returns a tuple of (rows, columns). Note: no parentheses because it&#39;s an attribute, not a function!<br>print(f&quot;Shape of the dataset: {df.shape}&quot;) <br><br># Prints information about the index, column datatypes, and non-null values<br>df.info() <br><br># Calculates count, mean, standard deviation, min, max, and percentiles for numeric columns<br>print(df.describe())</pre><h3>4. Cleaning the Mess: Data Janitor Work</h3><p>In Machine Learning, 80% of our time is spent cleaning data. “Bad data” means empty cells, wrong formats, or duplicates. If we feed bad data to a model, we get bad predictions (Garbage In, Garbage Out).</p><h4><strong>Handling Empty Cells (Nulls):</strong></h4><p>We have two main choices when a cell is empty: drop it, or fill it. Dropping rows means losing valuable training data, so we often try to fill them intelligently.</p><ul><li><strong>Drop the Row:</strong> df.dropna(inplace=True) removes any row with an empty cell entirely.</li><li><strong>Fill It In:</strong> A smart way to guess a missing value without skewing our dataset is to fill it with the average (mean) of that specific column.</li></ul><pre># Step 1: Calculate the average duration<br>avg_duration = df[&quot;duration&quot;].mean()<br><br># Step 2: Fill empty cells in the &#39;duration&#39; column with that average<br>df.fillna({&quot;duration&quot;: avg_duration}, inplace=True)</pre><h4><strong>Removing Duplicates</strong></h4><p>Duplicate rows can trick our ML model into favoring certain outcomes. Find them using df.duplicated() and remove them instantly with df.drop_duplicates(inplace=True).</p><p><em>(Note: </em><em>inplace=True is a magic command in Pandas. It tells the library to make the changes directly to our original DataFrame memory, rather than creating a temporary, memory-hogging copy).</em></p><h3>5. Filtering &amp; Locating: Intermediate Magic</h3><p>Sometimes we only want to look at specific subsets of our data to train a specific type of model.</p><h4><strong>Locating Rows (</strong><strong>loc vs </strong><strong>iloc)</strong></h4><p>This trips up a lot of beginners!</p><ul><li>df.loc[0] looks at the <strong>name (label)</strong> of the row.</li><li>df.iloc[0] looks at the strict mathematical <strong>integer index</strong> (0, 1, 2), regardless of what the row is named.</li></ul><h4><strong>Conditional Filtering</strong></h4><p>Want to only see rows where the workout duration was over 40 minutes? We don’t need to write a clunky for loop. Just pass the condition directly into the DataFrame.</p><pre># Creates a new DataFrame containing only the long workouts<br>long_workouts = df[df[&quot;duration&quot;] &gt; 40] <br>print(long_workouts)</pre><h3>6. Feature Engineering: Crafting Better Clues</h3><p>In Machine Learning, a “feature” is just a column of data. <strong>Feature Engineering</strong> is the art of using our existing columns to create new, more helpful columns. Sometimes, the raw data isn’t enough for an ML model to find a pattern, so we have to give it better clues.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2Y4ick9pA9ldSy1_ZcWLXA.png" /></figure><h4><strong>Renaming Columns</strong></h4><p>Before engineering, we want clean names. If a column is named something messy like Patient_Gluc_Level_mg, we can easily rename it:</p><pre># Pass a dictionary of {&#39;old_name&#39;: &#39;new_name&#39;}<br>df.rename(columns={&#39;Patient_Gluc_Level_mg&#39;: &#39;Glucose&#39;}, inplace=True)</pre><h4><strong>Creating Mathematical Features</strong></h4><p>Often, the relationship between two columns is more important than the columns themselves. For example, in a medical dataset, raw glucose and insulin levels are fine, but their <em>ratio</em> might be a massive indicator of health. Let’s create a “Glucose-to-Insulin Ratio” (GIR) column:</p><pre># Pandas lets you do math on entire columns at once!<br>df[&#39;GIR&#39;] = df[&#39;Glucose&#39;] / df[&#39;Insulin&#39;]</pre><h4><strong>Creating Categorical Features (Binning)</strong></h4><p>Sometimes, ML models perform better when continuous numbers are turned into categories. Let’s say we want to flag workouts as “Intense” or “Normal”:</p><pre># If calories are over 400, label it &#39;Intense&#39;, otherwise &#39;Normal&#39;<br>df.loc[df[&#39;calories&#39;] &gt; 400, &#39;Workout_Type&#39;] = &#39;Intense&#39;<br>df.loc[df[&#39;calories&#39;] &lt;= 400, &#39;Workout_Type&#39;] = &#39;Normal&#39;</pre><blockquote><strong>Do You Know?</strong> Feature engineering is where data scientists spend the most time! A simple model with <em>excellent</em> engineered features will almost always outperform a complex AI model that is fed raw, unoptimized data.</blockquote><h3>7. Grouping &amp; Correlations: Finding Patterns</h3><p>Before we hand our data over to an algorithm, we need to know what story the data is telling. Pandas gives us high-level statistical tools to find these patterns in seconds.</p><h4>Counting Categories (value_counts())</h4><p>Want to know the distribution of categories in a column, value_counts() is our best friend. For example, finding out how many patients fall into different outcomes:</p><pre># Returns the total count of each unique value in the column<br>print(df[&#39;Outcome&#39;].value_counts())<br><br># Pro-Tip: Add normalize=True to see the percentages instead of raw counts!<br>print(df[&#39;Outcome&#39;].value_counts(normalize=True))</pre><h4><strong>Grouping</strong></h4><p>Grouping is how we aggregate data based on specific categories. It works in three steps: <strong>Split</strong> the data into groups, <strong>Apply</strong> a function (like mean or sum), and <strong>Combine</strong> the results.</p><pre># Simple Grouping: What is the average of ALL columns, grouped by Outcome?<br>print(df.groupby(&#39;Outcome&#39;).mean())<br><br># Multi-Level Grouping: Group by Team, then by Position!<br>print(df.groupby([&#39;Team&#39;, &#39;Position&#39;]).mean())</pre><h4><strong>Multiple Aggregations (</strong><strong>.agg)</strong></h4><p>What if we want the <em>average</em> of one column, but the <em>sum</em> of another? We don&#39;t need to do them separately. We can pass a dictionary into the .agg() method:</p><pre># We want the average of Sales, but the total sum of Quantity<br>summary = df.groupby(&#39;City&#39;).agg({<br>    &#39;Sales&#39;: &#39;mean&#39;, <br>    &#39;Quantity&#39;: &#39;sum&#39;<br>})<br>print(summary)<br># the grouped column &#39;City&#39; becomes your new index</pre><blockquote><strong>Do You Know?</strong> If you filter or drop rows, your index numbers will have gaps (e.g., 0, 1, 3, 7). You can easily reset them to be perfectly sequential again using <em>df.reset_index(drop=True, inplace=True)</em>.</blockquote><h4><strong>Correlations (</strong><strong>corr)</strong></h4><p>How are our columns related? df.corr() calculates the relationship between every numeric column, returning a value from -1 to 1. In ML, if two columns have a perfect 1.0 correlation, they are giving the model the exact same information, and we can often drop one to make our model run faster!</p><ul><li><strong>1.0:</strong> Perfect relationship (If A goes up, B goes up exactly).</li><li><strong>-1.0:</strong> Perfect inverse relationship (If A goes up, B goes down exactly).</li><li><strong>0.0:</strong> Absolutely no relationship.</li></ul><h3>Wrapping Up</h3><p>With Pandas, we’ve successfully moved from raw math to powerful data management. We can now load massive datasets, clean out the junk, engineer new features, and find initial patterns.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*F4oV9_s9UIAq-0fJuvtizA.png" /></figure><p>We have the engine (NumPy) and we have the data manager (Pandas). Next time, we’ll start looking at how to visualize this data, or perhaps take our first steps into actually training a model!</p><p><strong>Stay curious, and keep coding!</strong> 🐼🤖</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f26e93e01656" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[NumPy — The Mathematical Engine | Learning Log #2]]></title>
            <link>https://medium.com/@sannysingh1/numpy-the-mathematical-engine-learning-log-2-ed1943d629d9?source=rss-dd796636e530------2</link>
            <guid isPermaLink="false">https://medium.com/p/ed1943d629d9</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[numpy]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[python-libraries]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[SannySingh]]></dc:creator>
            <pubDate>Thu, 09 Apr 2026 20:03:40 GMT</pubDate>
            <atom:updated>2026-04-09T20:03:40.542Z</atom:updated>
            <content:encoded><![CDATA[<p>We explored the “What” and “Why” of Machine Learning (ML). But to actually do ML, we need to talk about how computers see data. To a machine, everything — from a cat photo to a stock price — is just a giant pile of numbers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FUrVWIozlb3CK9MR11HITA.png" /></figure><p>To handle these numbers without losing our minds (or crashing our computers), we need <strong>NumPy</strong>.</p><p><strong>Why Python for Machine Learning?</strong></p><ul><li><strong>Strong Data Handling Support</strong></li><li><strong>Machine Learning Algorithm</strong></li><li><strong>Deep Learning Support</strong></li><li><strong>Model Evaluation &amp; Tuning</strong></li></ul><h3><strong>Why NumPy for ML?</strong></h3><p>Python is the go-to language for ML because its syntax is simple and its libraries are incredibly powerful. While Python “lists” are great for general use, they are slow and inefficient for the heavy math needed to train models.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uwlRdzwy1eH35AvWwZ3Czw.png" /></figure><p><strong>NumPy</strong> (Numerical Python) changes the game. Unlike lists, NumPy arrays are stored in a continuous block of memory. This allows the computer to access and manipulate data lightning-fast — which is exactly what you need when processing thousands of data points.</p><blockquote><strong>Do You Know?</strong> NumPy is so fundamental that almost every major ML library — like <strong>Pandas</strong> for data cleaning, <strong>Scikit-learn</strong> for model building, and <strong>TensorFlow</strong> for neural networks — is built on top of it!</blockquote><h4><strong>Getting Started: Arrays (ndarrays)</strong></h4><p>First, you’ll need to install it via command prompt/terminal: “<em>pip install numpy”</em> Once installed, we always import it with its industry-standard nickname, <em>np .</em></p><p>The most common way to create a NumPy array (called an <strong>ndarray</strong>) is by converting a standard Python list:</p><pre>import numpy as np<br>my_list = [3, 5, 8, 6]<br>arr = np.array(my_list) # Transforming a list into a NumPy array</pre><h4><strong>Deep Dive into Dimensions</strong></h4><p>In ML, “Dimension” just refers to how “deep” your array goes.</p><ul><li><strong>0-D Array:</strong> A single point or scalar.</li><li><strong>1-D Array:</strong> A linear list of numbers.</li><li><strong>2-D Array:</strong> An array of arrays (like a table with rows and columns). These are often called <strong>Matrices</strong>.</li><li><strong>3-D Array:</strong> An array of 2nd-order tensors (think of it as a cube of data)</li></ul><pre>import numpy as np<br><br># 0-D Array (Scalar)<br>arr_0d = np.array(42)<br>print(arr_0d)<br><br># 1-D Array (Vector)<br>arr_1d = np.array([1, 2, 3, 4, 5])<br>print(arr_1d)<br><br># 2-D Array (Matrix)<br>arr_2d = np.array([[1, 2, 3], [4, 5, 6]])<br>print(arr_2d)<br><br># 3-D Array (Tensor)<br>arr_3d = np.array([[[1, 2, 3], [4, 5, 6]], [[7, 8, 9], [10, 11, 12]]])<br>print(arr_3d)<br><br># Checking the dimensions<br>print(arr_3d.ndim)</pre><blockquote><strong>Tip:</strong> If you’re ever confused about what you’re looking at, just use ‘print(arr.ndim)’ to see the number of dimensions.</blockquote><h4><strong>Numpy Array Indexing</strong></h4><p>Access array elements -&gt; by referring to its index in ‘[ ]’.</p><pre>print(arr[0])</pre><p>For 2-D array : use comma separated integer representing dimension &amp; index of element.</p><p>Thinking of 2-D array as Table then, dimension - row &amp; index - column</p><p>1st row, 2nd column: “<em>arr[0,1]”</em></p><p>2nd row, 5th column: “<em>arr[1,4]”</em></p><p>Similarly, for 3-D array too. 3rd element of 2nd array of 1st array: “<em>array[0,1,2]</em>”. Here we need three indices i.e., “<em>array[Depth, Row, Column]</em>”</p><h4><strong>NumPy Array Slicing</strong></h4><p>Extract a range of elements using the format: [start:end:step]</p><pre>print(arr[1:5]) #From index 1 to 5: <br><br>print(arr[4:]) #From index 4 to end</pre><p>For Multi-Dimensional Array,<strong> </strong>slicing can be applied to each dimension separately which allows to extract submatrices or smaller blocks of data.</p><pre>print(arr[1,1:4]) #From 2nd elements, from index 1 to 4(excl): <br>print(arr[1:1:4]) #From both elements, slice index 1 to 4: </pre><blockquote><strong>Do You Know?</strong> There is a huge difference between a <strong>Copy</strong> and a <strong>View</strong>. A <strong>Copy</strong> is a new, independent array. A <strong>View</strong> is just a “peek” at the original data — if you change the View, the original data changes too!</blockquote><h4><strong>Shaping &amp; Reshaping</strong></h4><p><strong>Shaping: </strong>Number of elements in each dimension. Attribute called <strong>shape</strong> returns a tuple with each index having the number of corresponding elements.</p><p>“<em>print(arr.shape)</em>”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mO_wWXOSCUu1mk6zwah7oQ.png" /></figure><p><strong>Reshaping: </strong>Modifying the dimension of an existing array without changing its data. By reshaping we can add or remove dimension or change number of elements in each dimension.</p><pre>newarr = arr.reshape(4,3) #From 1-D to 2-D array: <br>newarr = arr.reshape(2,3,2) #From 1-D to 3-D array: <br># Elements in both array should be equal.</pre><p><strong>Flattening the array: </strong>Converting multidimensional into 1-D array, use ‘<em>reshape(-1)</em>’</p><h4><strong>NumPy Joining Array</strong></h4><p>Putting content of two or more array in a single array.</p><ul><li><strong>concatenate()</strong>:- pass a sequence of arrays to join along with axis, default axis 0.</li></ul><pre>arr1=np.array([1,2,3])<br>arr2=np.array([4,5,6])<br>arr=np.concatenate((arr1,arr2))<br>print(arr)</pre><ul><li><strong>stack()</strong>:- pass a sequence of array to join along with axis default axis is 0.</li></ul><pre>arr1=np.array([1,2,3,])<br>arr2=np.array([4,5,6])<br>arr=np.stack((arr1,arr2), axis=0)<br>print(arr)</pre><p>Other stacking techniques:</p><ul><li><strong>hstack()</strong>:- stack along rows</li><li><strong>vstack():- </strong>stack along column</li><li><strong>dstack()</strong>:- stack along height</li></ul><blockquote><strong>Do You Know?</strong> A standard Python list of 1 million integers takes up about 28MB of RAM, while a NumPy array of the same size only takes about 4MB. This 7x difference is exactly why NumPy is the backbone of “Big Data” processing!</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MQ9H2SLlSO7z6GzcMhkwrg.png" /></figure><h4>NumPy Splitting Array</h4><p>Breaks one array into multiple smaller sub-arrays. Numpy allows splitting arrays along different directions (rows,column,depth).</p><p>some important terms to know:</p><p><em>Axis</em><strong>:- </strong>direction along which the array is split (0 for rows, 1 for column).</p><p><em>Sub-array</em>:- Smaller arrays created after splitting the original array.</p><p>Methods to split an array:</p><ul><li><strong>array_split(): </strong>for splitting arrays, pass the array &amp; no. of splits.</li></ul><pre>arr=np.array([1,2,3,4,5,6])<br>newarr = np.array_split(arr,3)<br>print(newarr) <br>#return value is list containing 3 arrays. If no. of element is less than required it’ll adjust from end.</pre><ul><li><strong>split():</strong> used to divide array into equal-sized subarrays. The no. of split must perfectly divide the size of array along chosen axis.</li></ul><pre>newarr = np.split(arr,2)<br>print(newarr)</pre><p>other splitting techniques:</p><ul><li><strong>vsplit(): </strong>performs vertical splitting, it divides matrix row-wise (along axis=0).</li><li><strong>hsplit(): </strong>performs horizontal splitting, divide array column-wise (axis=1). Helpful when separating features columns in dataset.</li><li><strong>dsplit(): </strong>used for 3D arrays, split the array along the third axis (axis = 2). useful when working with stacked matrices, image or multi-channel data.</li></ul><h4><strong>Universal Function (ufunc)</strong></h4><p>used to implement vectorization in NumPy, faster way than iterating over elements. In standard Python, to add two lists, we might use a slow “for” loop. In Numpy, we use <strong>Universal Functions (ufuncs)</strong>. This implements <strong>Vectorization</strong>, which allows the computer to perform the operation on the entire array at once.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*C2DegZCPJWen0zFIEPY_KA.png" /></figure><blockquote><strong>Vectorization : </strong>Converting iterative statement into a vector based operation is called vectorization.</blockquote><p>Takes additional arguments like:</p><p><strong>where: </strong>Boolean array or condition defining where the operation should.</p><p><strong>dtype: </strong>defining the return type of elements.</p><p><strong>out: </strong>output array where the return value should be copied.</p><h4><strong>Mathematical Operations</strong></h4><p><strong>Arithmetic Operation: </strong>used for numerical computation &amp; can perform them on array using Numpy.</p><ul><li><strong>add():</strong> add corresponding two elements of two arrays.</li><li><strong>substract(): </strong>substract two arrays element-wise.</li><li><strong>divide(): </strong>element-wise division of elements, divide each element of 1st array with corresponding element of 2nd array.</li><li><strong>power(): </strong>allows to raise each element to a specified power/exponent.</li></ul><pre>import numpy as np<br># Creating two 1-D arrays<br>a = np.array([10, 20, 30, 40])<br>b = np.array([1, 2, 3, 4])<br><br># Addition<br>add_res = np.add(a, b)       # or: a + b<br>print(add_res)<br><br># Subtraction<br>sub_res = np.subtract(a, b)  # or: a - b<br>print(sub_res)<br><br># Multiplication<br>mul_res = np.multiply(a, b)  # or: a * b<br>print(mul_res)<br><br># Division<br>div_res = np.divide(a, b)    # or: a / b<br>print(div_res)<br><br># 5. Power (a raised to the power of b)<br>pow_res = np.power(a, b)     # or: a ** b<br>print(pow_res)</pre><p><strong>Aggregation Functions</strong></p><ul><li><strong>sum(): </strong>used to calculate sum of array elements. can sum values across the entire array or along a specific axis. It also allows controlling the output data type, initial value and shape of result. <strong>Syntax:</strong> “<em>numpy.sum(arr, axis=None, dtype=None, out=None, initial=0, keepdims=False)</em>”</li><li><strong>mean(): </strong>used to calculate average of numerical values. compute mean row-wise &amp; column-wise for multi-dimensional arrays</li><li><strong>maximun(): </strong>compares two array element-wise and return a new array containing max value at each position.</li><li><strong>minimum(): </strong>compares two arrays element-wise &amp; return a new array containing minimum value at each position</li></ul><h4><strong>Creating Own ufunc</strong></h4><p>for creating own ufunc, define a function, like normally in python then add it to Numpy ufunc library with “<em>frompyfunc()”</em> method.</p><p>define a function -&gt; add to numpy ufunc library (using “<em>frompyfunc()</em>” method).</p><p>“<em>frompyfunc()</em>” method takes following arguments:</p><p>“<em>np.frompyfunc(</em><strong><em>function</em></strong><em>, </em><strong><em>inputs</em></strong><em>, </em><strong><em>outputs</em></strong><em>)</em>”</p><p><strong>i) function:</strong> name of the function</p><p><strong>ii) inputs: </strong>number of input arguments</p><p><strong>iii) output:</strong> number of output arrays</p><pre>#Create ufunc for addition.<br>def myadd(x,y):<br>    return x+y<br>myadd = np.frompyfunc(myadd,2,1)<br>print(myadd([1,2,3,4],[5,6,7,8]))</pre><h4><strong>Numpy Array Iteration</strong></h4><p>Going through each element one by one. Iterate on 1-D array:</p><pre>arr =np.array([1,2,3])<br>for x in arr:<br>    print(x)</pre><p>But It’ll not iterate on a multidimensional array. To return scaler values we’ve to iterate the arrays in each dimension using nested for loops. Example:</p><pre>arr=np.array([[1,2,3],[4,5,6]])<br>for x in arr:<br>    for y in x:<br>        print(y)</pre><p>similarly for 3-D array we’ve to use 3 nested for loops. Another method to iterate a array is using <strong>nditer().</strong></p><blockquote><strong>Do You Know?</strong> Every digital image is actually just a 3-D NumPy array! The first two dimensions are the height and width (pixels), and the third dimension holds the color values (Red, Green, and Blue). When you apply a filter to a photo, you’re actually just doing NumPy math!</blockquote><h4><strong>Iterating Array Using nditer()</strong></h4><p>Used from basic to advanced iteration.</p><pre>arr = np.array([[[1,2],[3,4]],[[5,6],[7,8]]])<br>for x in np.nditer(arr):<br>    print(x)</pre><h3>Wrapping Up</h3><p>By learning NumPy, we’ve just built the engine for your ML journey. It allows us to prepare data , train models , and evaluate how well they are performing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JaQWqvWXa0OVqU0XnSGCAg.png" /></figure><p>Next time, we’ll look at <strong>Pandas</strong> — the library that takes these NumPy arrays and turns them into organized, searchable datasets.</p><p><strong>Keep learning, one array at a time!</strong> 🤖✨</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ed1943d629d9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From 0 to ML: My Learning Log]]></title>
            <link>https://medium.com/@sannysingh1/from-0-to-ml-my-learning-log-76157ecb8471?source=rss-dd796636e530------2</link>
            <guid isPermaLink="false">https://medium.com/p/76157ecb8471</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[education]]></category>
            <category><![CDATA[computer-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[SannySingh]]></dc:creator>
            <pubDate>Sun, 05 Apr 2026 18:34:22 GMT</pubDate>
            <atom:updated>2026-04-05T18:35:16.074Z</atom:updated>
            <content:encoded><![CDATA[<p>Hey! I’ve been diving into the world of Machine Learning and GenAI lately, and honestly, it’s a bit like learning a new language. It feels like the entire tech world is shifting, and while it’s exciting, it can also feel like a lot to take in at once.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fF0iFpbX7eQ0_TF9eGrkeA.png" /></figure><p>I’ve been keeping a “learning diary” to make sense of everything, and I wanted to share a simple &amp; friendly breakdown of what I’ve gathered so far. Whether you’re just curious or looking to start your own journey, let’s walk through the basics together!</p><h3><strong>So, What Exactly is Machine Learning?</strong></h3><p>Think of traditional programming like a strict recipe — you give the computer exact steps to follow. <strong>Machine Learning (ML)</strong> is a bit different. It’s a technique that allows computers to learn from data and make decisions or predictions without being explicitly programmed for every single scenario.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Kdfis_kBNnhEalloHWtywQ.png" /></figure><p>In short, Machine Learning is a way of teaching computers to find patterns in data. Instead of us writing every rule, the machine looks at examples and learns how to make its own decisions.</p><blockquote><strong>Do You Know!</strong> Every time Netflix suggests a show you actually end up liking, or Spotify finds that perfect song for your mood, that’s ML in action! These systems “learn” your taste based on your viewing history.</blockquote><h3><strong>The Big Three: Types of ML</strong></h3><p>When you start out, you’ll mostly hear about these three main ways a machine learns:</p><ol><li><strong>Supervised Learning (The Teacher-Student Model)</strong></li></ol><p>This is like learning with a teacher who has an answer key. You use a “Labelled Dataset,” meaning you give the machine both the input (like an image) and the correct output (the tag for that image).</p><p>In short, this is when we train a model using data that is already tagged with the right answer. The machine learns by comparing its guesses to the actual answers until it gets it right.</p><ul><li><strong>Classification:</strong> Putting things into categories (like “Spam” vs “Not Spam”).</li><li><strong>Regression:</strong> Predicting a continuous number (like what a house might cost next year).</li></ul><p><strong>2. Unsupervised Learning (The Self-Learner)</strong></p><p>Here, there are no labels and no “correct” answers provided. The machine just looks at a pile of data and tries to find groups or patterns on its own.</p><p>In short, in this type, the machine works with “unlabeled” data. Since there’s no teacher to give it the answer, it has to find hidden patterns or group similar items together on its own.</p><ul><li><strong>Clustering:</strong> This is huge in marketing for grouping similar customers together based on how they shop.</li></ul><p><strong>3.</strong> <strong>Reinforcement Learning (Trial and Error)</strong></p><p>Imagine training a dog with treats. An “agent” interacts with an environment and gets a reward for a good move or a penalty for a bad one. Over time, it learns the best way to get the most rewards.</p><p>In short, this is learning through trial and error. A system performs an action, sees if the result is positive or negative, and adjusts its behavior to get the best possible outcome over time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*m85bL4932ciblpWx_glBWg.png" /></figure><h3><strong>The Modern Approach: Semi &amp; Self-Supervised</strong></h3><p>As I dug deeper, I found two “bonus” types that are behind a lot of the cool tech we see today:</p><ul><li><strong>Semi-Supervised Learning:</strong> A middle ground where you use a small amount of labelled data and a huge amount of unlabelled data. It’s perfect when hiring experts to label data is too expensive.</li><li><strong>Self-Supervised Learning (The GenAI Powerhouse):</strong> This is where models generate their own labels from raw data. It’s how models like GPT learn language — by “hiding” a word in a sentence and trying to guess what it was.</li></ul><blockquote><strong>Do You Know!</strong> Models like <strong>BERT</strong> and <strong>GPT</strong> use Self-Supervised learning to understand the nuances of human language without a human having to label every single word for them!</blockquote><h3><strong>Why Does This Matter?</strong></h3><p>We’re living in a time where traditional coding just can’t keep up with the massive amounts of data we create every single day. Machine Learning is the tool that helps us make sense of it all — whether its helping doctors detect diseases earlier or just making our daily apps feel a bit more personal and helpful.</p><p>If you’re just starting out, don’t feel like you need to master every complex algorithm overnight. Just focus on understanding how the data flows and why we choose different types of learning for different problems. We’re all students in this field, so just keep building, stay curious, and take it one step at a time. It’s a pretty exciting journey to be on!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=76157ecb8471" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>