<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Muhammad Faheem on Medium]]></title>
        <description><![CDATA[Stories by Muhammad Faheem on Medium]]></description>
        <link>https://medium.com/@limtlesleg8988?source=rss-7725a1518a25------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*gHYp8uIcM0YrvmCzZt868Q.png</url>
            <title>Stories by Muhammad Faheem on Medium</title>
            <link>https://medium.com/@limtlesleg8988?source=rss-7725a1518a25------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 21:52:44 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@limtlesleg8988/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[One-Hot vs. Ordinal Encoding: How to Prepare Categories for Machine Learning]]></title>
            <link>https://medium.com/@limtlesleg8988/one-hot-vs-ordinal-encoding-how-to-prepare-categories-for-machine-learning-3fdbd81b4a5d?source=rss-7725a1518a25------2</link>
            <guid isPermaLink="false">https://medium.com/p/3fdbd81b4a5d</guid>
            <category><![CDATA[scikit-learn]]></category>
            <category><![CDATA[encoding]]></category>
            <category><![CDATA[data-preprocessing]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Muhammad Faheem]]></dc:creator>
            <pubDate>Tue, 03 Nov 2026 13:20:05 GMT</pubDate>
            <atom:updated>2026-11-03T13:20:05.392Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Choose an encoding that matches what your categories mean, and keep its mapping consistent from training data to new examples.</em></p><p>A dataset may describe a city, a color, an education level, or a product brand. These values are useful to us, but many machine learning algorithms expect numerical inputs. The simplest solution — replacing every category with a number — can accidentally tell the model that one category is greater than another.</p><p>The right encoding depends on one question: <strong>Do these categories have a meaningful order?</strong></p><p><strong>Start by distinguishing nominal and ordinal categories</strong></p><p><strong>Nominal categories have no natural order.</strong> Cities such as Karachi and Lahore, or colors such as Yellow and Blue, are different labels. There is no meaningful statement that Lahore is greater than Karachi.</p><p><strong>Ordinal categories do have an order.</strong> Education levels such as School, Undergraduate, and Postgraduate can have a meaningful progression. Review levels such as Poor, Average, and Good do too.</p><p>This distinction guides the encoding choice:</p><p>- Use <strong>one-hot encoding</strong> for nominal categories.<br>- Use <strong>ordinal encoding</strong> when an input category has a real order, and specify that order.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tbEKFVbYnwYfR1S28p_r-g.png" /></figure><p><strong><em>The numbers used for an ordinal feature represent its order. One-hot columns represent category membership.</em></strong></p><p><strong>Why arbitrary numbers can mislead a model</strong></p><p>Imagine replacing colors with Yellow = 0, Blue = 1, and Red = 2. A model may treat those values as if Red is greater than Blue, and Blue is greater than Yellow. That relationship came from our arbitrary codes; it is not part of the color data.</p><p>Ordinal encoding is different when we use it for an ordered feature. We can tell the encoder which order to use, such as School &lt; UG &lt; PG:</p><pre>import pandas as pd<br>from sklearn.model_selection import train_test_split<br>from sklearn.preprocessing import OrdinalEncoder<br>education = pd.DataFrame({<br> &quot;Education&quot;: [&quot;UG&quot;, &quot;School&quot;, &quot;PG&quot;, &quot;UG&quot;, &quot;School&quot;]<br>})<br>education_train, education_test = train_test_split(<br> education, test_size=0.4, random_state=0<br>)<br>education_order = [[&quot;School&quot;, &quot;UG&quot;, &quot;PG&quot;]]<br>ordinal = OrdinalEncoder(categories=education_order)<br>X_train_education = ordinal.fit_transform(education_train[[&quot;Education&quot;]])<br>X_test_education = ordinal.transform(education_test[[&quot;Education&quot;]])</pre><p>The encoder uses the order we supplied, so the resulting numbers carry the intended ranking. If the feature has no genuine order, ordinal codes can create a misleading comparison.</p><p><strong>One-hot encoding for categories without an order</strong></p><p>One-hot encoding creates an indicator column for each category. A row gets a 1 in the column for its category and 0 in the other category columns. For example, a color column might become `Color_Blue`, `Color_Red`, and `Color_Yellow`.</p><p>With scikit-learn, fit the encoder on training rows and reuse it to transform test rows:```python</p><pre>import pandas as pd<br>from sklearn.model_selection import train_test_split<br>from sklearn.preprocessing import OneHotEncoder<br>colors = pd.DataFrame({<br> &quot;Color&quot;: [&quot;Yellow&quot;, &quot;Blue&quot;, &quot;Red&quot;, &quot;Yellow&quot;, &quot;Green&quot;]<br>})<br>color_train, color_test = train_test_split(<br> colors, test_size=0.4, random_state=0<br>)<br>one_hot = OneHotEncoder(handle_unknown=&quot;ignore&quot;)<br>X_train_color = one_hot.fit_transform(color_train[[&quot;Color&quot;]])<br>X_test_color = one_hot.transform(color_test[[&quot;Color&quot;]])</pre><p>`fit_transform()` learns the categories and creates the training matrix. `transform()` applies that learned mapping to later data. This keeps the output columns consistent. With `handle_unknown=”ignore”`, a category not seen during fitting does not cause the transform to fail; its encoded row is all zeros for that feature.</p><p><strong>What about the target column?</strong></p><p>The target, usually called `y`, is different from an input feature, `X`. In a classification task, labels such as Yes and No can be encoded with `LabelEncoder`:</p><pre>from sklearn.preprocessing import LabelEncoder<br>label_encoder = LabelEncoder()<br>y_train_encoded = label_encoder.fit_transform(y_train)<br>y_test_encoded = label_encoder.transform(y_test)</pre><p>Fit on the training labels, then use the same encoder for test labels. Avoid applying `LabelEncoder` to nominal input columns: it assigns a single integer to each category, which can imply an order that does not exist.</p><p><strong>When one-hot encoding creates many columns</strong></p><p>If a feature has <strong>k</strong> categories, one-hot encoding can create <strong>k</strong> indicator columns. This is manageable for a small set of categories, but a column with many distinct values — such as brands — can create a wide matrix.</p><p>One option is to group infrequent categories into a shared value such as `uncommon`. This reduces the number of distinct categories before encoding. Choose the frequency cutoff using the training data, then apply the same grouping rule to validation and test data. That way, those later rows do not influence the preprocessing decision.</p><p>For exploration, Pandas offers `pd.get_dummies()`. In a repeatable train/test workflow, a fitted encoder is useful because it remembers the categories and output columns it learned. Scikit-learn’s `ColumnTransformer` can later combine encoding with other column-specific transformations in one workflow.</p><p><strong>The dummy-variable question</strong></p><p>With <strong>k</strong> one-hot columns, the indicators for each row add up to one. In a linear model with an intercept, this creates an exact dependency among the columns: one indicator can be determined from the others. This is sometimes called the <strong>dummy-variable trap</strong>.</p><p>Dropping one category removes that exact dependency. It can be useful for some linear-model setups, but it is not a universal requirement for every model. Choose `drop=”first”` when it suits the model and interpretation; otherwise, keep all indicators.</p><p><strong>A practical checklist</strong></p><p>Before encoding a categorical feature, ask:</p><p>1. Does this feature have a real order, or are its categories just different labels?<br>2. If it is ordered, have I supplied the intended order explicitly?<br>3. Did I split the data before fitting the encoder?<br>4. Will the fitted encoder handle categories that appear later?<br>5. Could a high-cardinality feature create too many columns?</p><p>Encoding is not just converting words to numbers. It is a way of representing what a column means. One-hot encoding represents membership without inventing a ranking; ordinal encoding represents a ranking when one actually exists. Keeping that distinction clear makes the model inputs more faithful to the data.</p><p><strong>Further Learning:</strong></p><p><a href="https://github.com/Muhammadfaheem8988/machine_learning-theory-to-code"><strong>Github Repository</strong></a><strong><br></strong><a href="https://www.kaggle.com/code/muhammadf4h33m/07-feature-scaling-and-encodings"><strong>Kaggle Notebook</strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3fdbd81b4a5d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Preparing Features for Machine Learning with Python: Scaling and Encoding]]></title>
            <link>https://medium.com/@limtlesleg8988/preparing-features-for-machine-learning-with-python-scaling-and-encoding-68f40e9ba055?source=rss-7725a1518a25------2</link>
            <guid isPermaLink="false">https://medium.com/p/68f40e9ba055</guid>
            <category><![CDATA[feature-encoding]]></category>
            <category><![CDATA[data-preprocessing]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[feature-scaling]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Muhammad Faheem]]></dc:creator>
            <pubDate>Mon, 02 Nov 2026 11:09:20 GMT</pubDate>
            <atom:updated>2026-11-02T11:09:20.253Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Learn how to put numeric features on comparable scales and turn categories into model-ready values without losing their meaning.</em></p><p>A dataset can contain useful information and still be difficult for a machine learning model to use. One column might measure age in years, another salary in tens of thousands of dollars, and a third might contain categories such as color or education level.</p><p>Before fitting a model, we need to prepare those columns. Two common steps are <strong>feature scaling</strong> for numeric values and <strong>encoding</strong> for categories. The key is to choose a transformation that fits both the data and the algorithm.</p><p><strong>Feature preparation begins with the columns</strong></p><p>Feature engineering is the work of preparing or creating useful model inputs. It can include transforming existing values, constructing a new feature, selecting columns, or extracting a smaller representation such as principal components.</p><p>A small example of feature construction comes from the Titanic data. `SibSp` counts siblings and spouses aboard, while `Parch` counts parents and children. We can combine them, and include the passenger, to create a simple family-size feature:</p><pre>import pandas as pd<br>titanic = pd.read_csv(&quot;train.csv&quot;)<br>titanic[&quot;FamilySize&quot;] = titanic[&quot;SibSp&quot;] + titanic[&quot;Parch&quot;] + 1</pre><p>Whether a new feature helps depends on the problem. We create it using domain knowledge, then evaluate it as part of a sound modeling workflow.</p><p>This article focuses on two other preparation steps: scaling numeric columns and encoding categorical ones.</p><p><strong>Why put numeric features on comparable scales?</strong></p><p>Some algorithms use distances between data points. K-nearest neighbors (KNN), for example, finds observations that are close to a new observation. Euclidean distance between two points is:</p><p>d = √((x₂ − x₁)² + (y₂ − y₁)²)</p><p>Suppose one point represents a person who is 27 years old and earns 448,000, while another represents someone who is 50 and earns $83,000. The age difference is 23. The salary difference is 35,000. When squared as part of Euclidean distance, the salary term is much larger simply because dollars use larger numbers than years.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*f9L1_w-au6Ky0QWBnztg0g.png" /></figure><p><strong><em>When features use very different units, the larger numbers can dominate distance calculations.</em></strong></p><p>Scaling changes the numeric ranges so a feature’s measurement units are less likely to overwhelm another feature. It does not make every feature equally informative, and it is not equally important for every algorithm. Scaling commonly matters for KNN, K-Means, PCA, and many methods trained with gradient-based optimization. Tree-based models split on thresholds, so scaling usually does not change their decisions.</p><p><strong>Standardization: center, then scale</strong></p><p>Standardization, also called z-score scaling, subtracts the mean and divides by the standard deviation:</p><p><strong>z = (x − μ) / σ</strong></p><p>Here, <strong>μ</strong> is the mean and <strong>σ</strong> is the standard deviation. After standardization, each feature is centered around zero and has a standard deviation around one, based on the data used to fit the scaler. The data’s shape is not turned into a bell curve, and standardization does not remove outliers.</p><p>The important workflow detail is to split the data first. Fit the scaler using only the training features, then use that same fitted scaler to transform both the training and test features:</p><pre>import pandas as pd<br>from sklearn.model_selection import train_test_split<br>from sklearn.preprocessing import StandardScaler<br>ads = pd.read_csv(&quot;Social_Network_Ads.csv&quot;)<br>X = ads[[&quot;Age&quot;, &quot;EstimatedSalary&quot;]]<br>y = ads[&quot;Purchased&quot;]<br>X_train, X_test, y_train, y_test = train_test_split(<br> X, y, test_size=0.30, random_state=0, stratify=y<br>)<br>scaler = StandardScaler()<br>X_train_scaled = scaler.fit_transform(X_train)<br>X_test_scaled = scaler.transform(X_test)</pre><p>`fit_transform()` learns the training-set mean and standard deviation and applies them. `transform()` reuses those learned values. If we fit on the test set too, information from the test data can leak into preprocessing. Keeping the test set separate gives us a more honest evaluation of how the workflow handles unseen data.</p><p>The scaler changes values, not the number of rows or columns. The same fitted-scaler pattern applies to other preprocessing steps too.</p><p><strong>Min-Max scaling and other choices</strong></p><p>Min-Max scaling maps values using the training minimum and maximum. A common target range is from zero to one:</p><p><strong>x′ = (x − x_min) / (x_max − x_min)</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2f-nKKY8__a02I7aZAfy_Q.png" /></figure><p><strong><em>The training data’s minimum and maximum map to 0 and 1.</em></strong></p><pre>from sklearn.preprocessing import MinMaxScaler<br>minmax = MinMaxScaler()<br>X_train_minmax = minmax.fit_transform(X_train)<br>X_test_minmax = minmax.transform(X_test)</pre><p>This method is useful when a bounded range is appropriate. But an extreme training value can stretch the range and squeeze most observations into a small part of it. Test values beyond the training minimum or maximum can also transform to values below zero or above one; Min-Max scaling does not clip them by default.</p><p>Other scalers can be useful in particular situations:</p><p>- <strong>RobustScaler</strong> centers using the median and scales using the interquartile range, so extreme values have less influence on the scaling parameters.<br>- <strong>MaxAbsScaler</strong> divides by the largest absolute value and keeps zeros as zeros, which is useful for sparse data.<br>- <strong>Mean normalization</strong> subtracts the mean and divides by the range; it can be calculated directly when needed.</p><p>There is no single best scaler for every dataset. Standardization is a common starting point; use validation data to compare alternatives that make sense for your problem. If outliers matter, remember that scaling alone does not identify or remove them.</p><p><strong>Encode categories according to their meaning</strong></p><p>Models generally work with numerical inputs, so categories also need preparation. First ask whether the categories have a meaningful order.</p><p><strong>Nominal categories have no natural order.</strong> Examples include city names, colors, or fuel types. Coding cities as Karachi = 0 and Lahore = 1 would imply a numerical relationship that does not exist. For nominal inputs, one-hot encoding creates an indicator column for each category:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tbEKFVbYnwYfR1S28p_r-g.png" /></figure><p><strong><em>Use one-hot encoding for unordered categories and ordinal encoding only when the order is meaningful.</em></strong></p><p><strong>Ordinal categories do have an order.</strong> Education levels such as School, Undergraduate, and Postgraduate can be represented with an explicitly supplied order:</p><pre>import pandas as pd<br>from sklearn.model_selection import train_test_split<br>from sklearn.preprocessing import OrdinalEncoder<br>education = pd.DataFrame({&quot;Education&quot;: [&quot;UG&quot;, &quot;School&quot;, &quot;PG&quot;, &quot;UG&quot;, &quot;School&quot;]})<br>education_train, education_test = train_test_split(<br> education, test_size=0.4, random_state=0<br>)<br>education_order = [[&quot;School&quot;, &quot;UG&quot;, &quot;PG&quot;]]<br>ordinal = OrdinalEncoder(categories=education_order)<br>X_train_education = ordinal.fit_transform(education_train[[&quot;Education&quot;]])<br>X_test_education = ordinal.transform(education_test[[&quot;Education&quot;]])</pre><p>The assigned numbers now represent the order we specified. Use this approach only when that order is meaningful for the problem.</p><p>One-hot encoding is commonly handled with scikit-learn’s `OneHotEncoder`. Fit it on training rows, then reuse it to transform later rows:</p><pre>import pandas as pd<br>from sklearn.model_selection import train_test_split<br>from sklearn.preprocessing import OneHotEncoder<br>colors = pd.DataFrame({&quot;Color&quot;: [&quot;Yellow&quot;, &quot;Blue&quot;, &quot;Red&quot;, &quot;Yellow&quot;, &quot;Green&quot;]})<br>color_train, color_test = train_test_split(<br> colors, test_size=0.4, random_state=0<br>)<br>ohe = OneHotEncoder(handle_unknown=&quot;ignore&quot;)<br>X_train_color = ohe.fit_transform(color_train[[&quot;Color&quot;]])<br>X_test_color = ohe.transform(color_test[[&quot;Color&quot;]])</pre><p>The encoder keeps a consistent mapping of categories to output columns. `handle_unknown=”ignore”` lets the transform run if a later row contains a category that was not seen during fitting. For high-cardinality columns with many rare values, grouping infrequent categories can reduce the number of resulting columns. Learn any grouping rule from the training set, too.</p><p>A one-hot encoding with <strong>k</strong> categories can create <strong>k</strong> columns. In some linear-model setups with an intercept, one of those columns is redundant because the indicators sum to one for each row. Dropping a category can remove that exact dependence; whether to do so depends on the model and workflow, so it is not a universal requirement.</p><p>One more distinction: `LabelEncoder` is intended for encoding target labels (`y`) in classification, not for assigning arbitrary numbers to nominal input columns. For input features, use one-hot encoding or an ordinal encoding that reflects a genuine order.</p><p><strong>Put the steps together carefully</strong></p><p>A practical preparation workflow follows a few simple rules:</p><p>1. Decide what each column means and which columns are numeric, nominal, or ordinal.<br>2. Split the data into training and test sets.<br>3. Fit scalers and encoders on the training data only.<br>4. Transform the training and test data using the fitted objects.<br>5. Compare reasonable preprocessing choices using validation data.</p><p>Scaling and encoding are small steps, but they affect what a model receives as input. A distance-based algorithm can be distracted by mismatched units; an encoding can invent an order that the categories never had. When the transformations preserve the meaning of the columns, the model has a more useful representation to learn from.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=68f40e9ba055" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Exploratory Data Analysis with Python: How to Explore Data Before Machine Learning]]></title>
            <link>https://medium.com/@limtlesleg8988/exploratory-data-analysis-with-python-how-to-explore-data-before-machine-learning-317388975b04?source=rss-7725a1518a25------2</link>
            <guid isPermaLink="false">https://medium.com/p/317388975b04</guid>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[data-analyst]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[exploratory-data-analysis]]></category>
            <dc:creator><![CDATA[Muhammad Faheem]]></dc:creator>
            <pubDate>Tue, 27 Oct 2026 12:47:49 GMT</pubDate>
            <atom:updated>2026-10-27T12:47:49.441Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Use a few focused questions, summaries, and visualizations to understand what a dataset can — and cannot — tell you before modeling.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*woqu9vIKqh3IqZZGMuabbg.png" /></figure><p><strong>Start with questions, not plots</strong></p><p>Before building a machine learning model, I want to know what the data looks like. How many rows and columns are there? Which values are missing? Are there duplicate rows? What kinds of variables do I have, and how do they relate?</p><p>This process is called <strong>exploratory data analysis (EDA)</strong>. It helps us become familiar with a dataset and notice issues or patterns that deserve a closer look. In this article, I’ll use the Titanic passenger data from Notebook 06 to follow a practical EDA workflow, from basic checks to relationships between variables.</p><p>EDA does not prove why a pattern exists, and a plot does not make a dataset representative. It gives us a clearer starting point for preparing data and asking better questions.</p><p><strong>Begin with the shape and quality of the data</strong></p><p>Start with a few basic questions. Pandas has a direct way to answer each one:</p><pre>import pandas as pd<br>df = pd.read_csv(&quot;train.csv&quot;)<br>df.columns = df.columns.str.lower()<br>print(&quot;Shape:&quot;, df.shape)<br>print(df.head())<br>print(df.sample(5, random_state=42))<br>df.info()</pre><p>`shape` reports the number of rows and columns. `head()` shows the first few rows, while `sample()` gives a quick look at rows from different parts of the dataset. `info()` shows column types and how many non-missing values each column contains.</p><p>Next, count missing values and duplicate rows:</p><pre>missing = df.isna().sum().sort_values(ascending=False)<br>missing_percent = (df.isna().mean() * 100).sort_values(ascending=False)<br>print(pd.DataFrame({<br> &quot;missing values&quot;: missing,<br> &quot;missing %&quot;: missing_percent.round(1),<br>}))<br>print(&quot;Duplicate rows:&quot;, df.duplicated().sum())</pre><p>In this <a href="https://www.kaggle.com/code/muhammadf4h33m/06-exploratory-data-analysis">Titanic file</a>, `cabin` is missing for about 77% of passengers, while `age` is missing for about 20%. A high missing percentage is a reason to investigate, not an automatic instruction to delete a column. Consider what the column represents, why values may be missing, and whether missingness itself carries useful information. The 60–70% threshold in my notes is a rough decision aid, not a universal rule.</p><p>The same care applies to duplicate rows. First understand whether they are accidental duplicates or repeated records that make sense for the data.</p><p><strong>Summarize one variable at a time</strong></p><p><strong>Univariate analysis</strong> means studying one variable at a time. First check whether the variable is categorical or numerical; the type helps determine which summary or plot will be useful.</p><p>For a categorical variable such as `survived`, count plots and bar charts show how many rows belong to each category:</p><pre>import matplotlib.pyplot as plt<br>import seaborn as sns<br>sns.countplot(data=df, x=&quot;survived&quot;)<br>plt.xlabel(&quot;Survived (0 = no, 1 = yes)&quot;)<br>plt.ylabel(&quot;Passengers&quot;)<br>plt.show()</pre><p>For a numerical variable such as `age`, a histogram groups values into <strong>bins</strong>, or ranges. Changing the number of bins changes the level of detail, so it is useful to check whether the chosen view hides the distribution’s shape.</p><pre>sns.histplot(data=df, x=&quot;age&quot;, bins=30, kde=True)<br>plt.xlabel(&quot;Age&quot;)<br>plt.show()</pre><p>The bars show counts within ranges. The optional smooth curve is a <strong>kernel density estimate (KDE)</strong>, an estimate of the distribution’s shape. It is not the probability of one exact age.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*As338duTfVsJLgbhqm1eag.png" /></figure><p>The shape can help us notice skew. A right-skewed distribution has a longer tail to the right; a left-skewed one has a longer tail to the left. You can calculate a skewness summary with `df[“age”].skew()`, but look at the plot too: a single number does not show every detail in a distribution.</p><p>For a numerical summary, `describe()` reports statistics such as the count, mean, standard deviation, quartiles, and maximum. When a distribution is skewed, the median can describe a typical value better than the mean. Use the summaries and plot together rather than relying on just one.</p><p><strong>Use box plots to investigate unusual values</strong></p><p>A box plot gives a compact view of the median and the middle half of the data. That middle range is the <strong>interquartile range</strong>:</p><p><strong>IQR = Q₃ − Q₁</strong></p><p>The usual box-plot fences are `Q₁ − 1.5 × IQR` and `Q₃ + 1.5 × IQR`. Values beyond them are drawn as individual points. These points are candidates to inspect — not proof that the data is wrong and not a reason to remove them automatically.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8Wj8Bu0lZA5kbr4sFwdf9A.png" /></figure><p>In Seaborn, the age distribution can be viewed with:</p><pre>sns.boxplot(data=df, y=&quot;age&quot;)<br>plt.ylabel(&quot;Age&quot;)<br>plt.show()</pre><p>The whiskers end at the furthest observed values within the fences, so they do not necessarily show the minimum and maximum. When a point lies beyond a whisker, ask whether it is a valid but unusual observation, a data-entry problem, or something that needs more context.</p><p><strong>Compare variables to look for relationships</strong></p><p>Once we have looked at individual columns, we can compare them. <strong>Bivariate analysis</strong> studies two variables; <strong>multivariate analysis</strong> considers several together.</p><p>For two numerical variables, a scatter plot can reveal a pattern, such as an upward or downward association. Color, marker style, and point size can add other variables:</p><pre>sns.scatterplot(<br> data=df,<br> x=&quot;age&quot;,<br> y=&quot;fare&quot;,<br> hue=&quot;survived&quot;,<br> alpha=0.7,<br>)<br>plt.xlabel(&quot;Age&quot;)<br>plt.ylabel(&quot;Fare&quot;)<br>plt.show()</pre><p>Each point represents a passenger. The colors let us compare the survival groups, but overlapping points and missing ages affect what we can see. A scatter plot can suggest an association; it cannot explain why it exists.</p><p>For a numerical outcome across categories, a bar plot can summarize group values. Because `survived` is coded as 0 or 1, its mean within a group is the share marked as survived:</p><pre>survival_by_class = df.groupby(&quot;pclass&quot;)[&quot;survived&quot;].mean().mul(100)<br>print(survival_by_class)</pre><p>In this sample, the survival rates were about 63% in first class, 47% in second, and 24% in third. These are descriptive rates in this dataset. They do not show that passenger class alone caused the difference.</p><p>Categorical variables can also be compared with a <strong>crosstab</strong>, which counts combinations of categories. A heatmap makes those counts easier to scan:</p><pre>class_survival = pd.crosstab(df[&quot;pclass&quot;], df[&quot;survived&quot;])<br>sns.heatmap(class_survival, annot=True, fmt=&quot;d&quot;, cmap=&quot;Blues&quot;)<br>plt.xlabel(&quot;Survived (0 = no, 1 = yes)&quot;)<br>plt.ylabel(&quot;Passenger class&quot;)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/1*jhHtc197akdsbL-GDGq81g.png" /></figure><p>The heatmap shows <strong>counts</strong>, so a larger group can have larger cells simply because it contains more passengers. If the goal is to compare rates fairly across groups, normalize each row or compare the group means instead:</p><pre>class_percent = pd.crosstab(<br> df[&quot;pclass&quot;], df[&quot;survived&quot;], normalize=&quot;index&quot;<br>).mul(100)<br>class_percent.round(1)</pre><p>Other plots answer other questions. A pair plot creates a grid of plots for several numerical variables. A line plot is useful when one variable is time: in the notebook’s flights example, passenger counts are grouped by year before they are plotted. A heatmap of a month-by-year pivot table can make seasonal patterns easier to see.</p><p>Correlation tables offer another compact summary for numeric columns. A positive correlation means two values tend to move in the same direction; a negative one means they tend to move in opposite directions. Correlation summarizes linear association, not cause and effect. A small correlation alone is not a sound reason to remove a feature.</p><p><strong>Let profiling help, then check its results</strong></p><p>Automated profiling can collect summaries, missing-value counts, distributions, and relationships in one report. In the notebook, a profiling library creates an HTML report with sections such as overview, variables, interactions, correlations, missing values, and sample rows.</p><p>That can save time, but an automated report does not decide how to clean the data for you. Treat flags and patterns as prompts for inspection. You still need to consider the task and what each field means.</p><p><strong>Connect EDA to feature engineering</strong></p><p>EDA helps us understand the raw data; <strong>feature engineering</strong> uses domain knowledge to create useful inputs from it. For example, a team might combine or transform existing fields when that better represents the problem it is trying to model. The right choice depends on the data and the task.</p><p>The notebook’s workflow places feature engineering inside data preparation, before modeling and evaluation. Those steps often repeat: evaluation may show that the team needs to revisit its data preparation or features. After a model is deployed, monitoring can reveal changes that call for further investigation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AI6aC5MN7SI8QPbswvSEdg.png" /></figure><p><strong>A practical EDA checklist</strong></p><p>Before building a model, ask:</p><p>1. How many rows and columns do I have, and what do sample rows look like?<br>2. What types of values do the columns contain?<br>3. Where are values missing, and are there duplicate rows?<br>4. What do numeric summaries and single-variable plots show?<br>5. Which relationships are useful to inspect, and are counts or rates more appropriate?<br>6. What should I investigate before deciding to change or remove data?</p><p>The purpose is not to create every possible chart. It is to understand what is in the dataset, choose summaries and visualizations that fit the questions, and make careful decisions before modeling.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=317388975b04" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Where Does Machine Learning Data Come From? APIs, Files, SQL, and Web Scraping]]></title>
            <link>https://medium.com/@limtlesleg8988/where-does-machine-learning-data-come-from-apis-files-sql-and-web-scraping-7a23bdc640b1?source=rss-7725a1518a25------2</link>
            <guid isPermaLink="false">https://medium.com/p/7a23bdc640b1</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[api]]></category>
            <category><![CDATA[web-scraping]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Muhammad Faheem]]></dc:creator>
            <pubDate>Mon, 26 Oct 2026 06:42:59 GMT</pubDate>
            <atom:updated>2026-10-26T06:50:31.614Z</atom:updated>
            <content:encoded><![CDATA[<p>Before choosing a model, find out where the data lives, whether it covers the problem you care about, and how to turn it into a usable table.</p><p><strong>Think before you code</strong></p><p>A machine learning project does not begin with an algorithm. It begins with questions about the data.</p><p>Are the columns you need available? Does the data apply to the region or country you care about? If the project is about customers in Pakistan, for example, a dataset from another market may tell only part of the story.</p><p>These checks sound simple, but they shape the whole project. Before coding, decide what information you need and where it might come from. Then choose a way to collect it.</p><p><strong>Choose a source that fits the data</strong></p><p>Data can come in several forms:</p><ul><li><strong>CSV or TSV files:</strong> rows and columns saved as text. CSV values are separated by commas; TSV values are separated by tabs.<br>- <strong>JSON:</strong> a structured format often used to exchange data between software systems.<br>- <strong>SQL databases:</strong> tables you can query to select the rows and columns you need.<br>- <strong>APIs:</strong> interfaces that let software systems request and exchange data.<br>- <strong>Web pages:</strong> if an API is unavailable and the site permits it, you may be able to extract information from its HTML.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lMw-tZ19uzF6IeCntYtqvg.png" /></figure><p>Pandas can load delimited files directly into a DataFrame:</p><pre>import pandas as pd<br>csv_data = pd.read_csv(&quot;students.csv&quot;)<br>tsv_data = pd.read_csv(&quot;students.tsv&quot;, sep=&quot;\t&quot;)</pre><p>JSON data can also become a DataFrame once you find the list of records you need. SQL follows a similar idea: write a query for the fields you need, then read the results into a DataFrame. The exact extraction step depends on the structure of the file, response, or database.</p><p><strong>An API is a way for systems to communicate</strong></p><p>Think about booking a flight. If one station books a seat, other booking stations need to know that seat is no longer available. They do not need direct access to the same internal database; they can communicate through an API.</p><p>An API is the agreed interface between software systems. One system sends a request, and the other returns a response. Many APIs return JSON, but the format and structure depend on the API. Inspect the response before deciding which fields to extract.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*Ef1lw2RrZB5SCtW-h_MynQ.png" /></figure><p><strong>From an API response to a DataFrame</strong></p><p>The data-gathering flow is:</p><p><strong>Request → JSON response → selected columns → DataFrame</strong></p><p><a href="https://www.kaggle.com/code/muhammadf4h33m/05-data-gathering-apis-and-scraping">Notebook 05</a> uses the public TVmaze shows endpoint to demonstrate pagination. Its show index uses a page number in the URL. The [TVmaze API documentation](<a href="https://www.tvmaze.com/api">https://www.tvmaze.com/api</a>) describes the `/shows?page=:num` endpoint and its page parameter.</p><p>The example below requests two pages, keeps a few records from each page, and selects only the columns needed for the table:</p><pre>import json<br>import pandas as pd<br>from urllib.request import Request, urlopen<br>shows = []<br>for page in range(2):<br> url = f&quot;https://api.tvmaze.com/shows?page={page}&quot;<br> request = Request(<br> url,<br> headers={&quot;User-Agent&quot;: &quot;ML-notebook-learning-example&quot;},<br> )<br>with urlopen(request, timeout=5) as response:<br> print(f&quot;Page {page}: HTTP {response.status}&quot;)<br> page_records = json.loads(response.read().decode(&quot;utf-8&quot;))<br> shows.extend(page_records[:3])<br>shows_data = pd.DataFrame(shows)<br>columns = [<br> column<br> for column in [&quot;id&quot;, &quot;name&quot;, &quot;language&quot;, &quot;premiered&quot;]<br> if column in shows_data.columns<br>]<br>shows_data = shows_data[columns]<br>print(shows_data.head())</pre><p>The loop visits page `0` and page `1`. In Python, `range(2)` stops before 2, so it produces `0, 1`. APIs do not all start their page numbering at the same value; follow the specific API’s documentation.</p><p>The response structure matters too. This endpoint returns a list of show records, so each record becomes a row. Other APIs may put records inside a dictionary key such as `results`. Inspect the keys and sample values, then select the fields that match your project. When combining pages, `pd.concat()` can combine DataFrames; save the final table with `to_csv()` if you need to reuse it.</p><p><strong>When an API is not available: inspect the HTML</strong></p><p>Some sites do not provide an API for the information you need. If their terms and access rules allow collection, web scraping can extract fields from the HTML they return.</p><p>Start by inspecting the page. Find one repeated container that holds a record, such as a company card. Then identify the tags and classes inside it: a heading may contain the company name, while paragraphs with different classes may contain the rating and review count.</p><p>Here is a small HTML example following the company-card pattern in <a href="https://www.kaggle.com/code/muhammadf4h33m/05-data-gathering-apis-and-scraping">Notebook 05</a>:</p><pre>from lxml import html<br>import pandas as pd<br>page_html = &quot;&quot;&quot;<br>&lt;div class=&quot;company-card&quot;&gt;<br> &lt;h2&gt;North Star Analytics&lt;/h2&gt;<br> &lt;p class=&quot;rating&quot;&gt;4.2&lt;/p&gt;<br> &lt;p class=&quot;reviews&quot;&gt;120 reviews&lt;/p&gt;<br>&lt;/div&gt;<br>&lt;div class=&quot;company-card&quot;&gt;<br> &lt;h2&gt;Blue River Labs&lt;/h2&gt;<br> &lt;p class=&quot;rating&quot;&gt;3.8&lt;/p&gt;<br> &lt;p class=&quot;reviews&quot;&gt;86 reviews&lt;/p&gt;<br>&lt;/div&gt;<br>&quot;&quot;&quot;<br>page = html.fromstring(page_html)<br>company_rows = []<br>for card in page.xpath(<br> &#39;//div[contains(concat(&quot; &quot;, normalize-space(@class), &quot; &quot;), &quot; company-card &quot;)]&#39;<br>):<br> name = card.xpath(&quot;.//h2/text()&quot;)<br> rating = card.xpath(<br> &#39;.//p[contains(concat(&quot; &quot;, normalize-space(@class), &quot; &quot;), &quot; rating &quot;)]/text()&#39;<br> )<br> reviews = card.xpath(<br> &#39;.//p[contains(concat(&quot; &quot;, normalize-space(@class), &quot; &quot;), &quot; reviews &quot;)]/text()&#39;<br> )<br>company_rows.append({<br> &quot;company&quot;: name[0].strip() if name else None,<br> &quot;rating&quot;: rating[0].strip() if rating else None,<br> &quot;reviews&quot;: reviews[0].strip() if reviews else None,<br> })<br>companies = pd.DataFrame(company_rows)<br>print(companies)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sjDTps7qemKXc7cEcviYkw.png" /></figure><p>The important part is the order: select each `company-card` first, then read the heading and paragraphs inside that card. This keeps a company’s name, rating, and review count together. If you search the whole page for every heading first and every rating second, it becomes easier to mismatch values from different records.</p><p>The handwritten notes introduce the same idea with BeautifulSoup: `find_all()` returns multiple matching elements, while `find()` returns the first one. The notebook uses `lxml` and XPath for its demonstration, but the principle is the same — inspect the HTML structure and select the right repeated element and its fields.</p><p>If a site returns <strong>403 Forbidden</strong>, it has refused the request. That status is not always caused by `robots.txt`, and adding a browser-style header does not grant permission. Check the site’s rules or use an approved API or dataset; do not try to disguise a blocked request.</p><p><strong>Collecting data is part of the problem-solving</strong></p><p>The goal is not to gather the largest possible dataset. It is to gather data that has the fields and coverage your problem needs, from a source you are allowed to use.</p><p><strong>Before training a model, ask:</strong></p><p>1. What fields do I need, and do they exist in this source?<br>2. Does the data represent the region and population I care about?<br>3. What structure will I receive: rows, nested JSON, or HTML elements?<br>4. How will I collect every page and keep only useful columns?<br>5. Am I allowed to access and reuse this data?</p><p>Think through these questions first. Then the code has a clear job: read the source, extract the right information, and place it into a form the next step can use.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7a23bdc640b1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How Much Memory Does Video Data Need? A Tensor Shape Example]]></title>
            <link>https://medium.com/@limtlesleg8988/how-much-memory-does-video-data-need-a-tensor-shape-example-78a2355866a5?source=rss-7725a1518a25------2</link>
            <guid isPermaLink="false">https://medium.com/p/78a2355866a5</guid>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Muhammad Faheem]]></dc:creator>
            <pubDate>Sat, 24 Oct 2026 05:21:13 GMT</pubDate>
            <atom:updated>2026-10-24T05:21:13.323Z</atom:updated>
            <content:encoded><![CDATA[<p><em>A short calculation shows how video frames, resolution, color channels, batches, and data type determine a tensor’s in-memory size.</em></p><p><strong>A video is more than one image</strong></p><p>An image is a grid of pixels. A video is a sequence of images, called frames. When we arrange several videos for machine learning, the data can become a large tensor with separate axes for clips, frames, height, width, and color channels.</p><p>Let’s use the example from Notebook 04: four 60-second clips, each recorded at 30 frames per second, with a frame size of 480 by 720 pixels and three color channels.</p><p><strong>Work out the tensor shape</strong></p><p>First, calculate the number of frames in one clip:</p><p><strong>60 seconds × 30 frames per second = 1,800 frames</strong></p><p>One clip can then be represented with shape (1800, 480, 720, 3):</p><p>- 1800 frames,<br>- 480 pixel rows per frame,<br>- 720 pixel columns per frame, and<br>- 3 color channels per pixel.</p><p>The four clips add a batch axis at the beginning, so the full shape is (4, 1800, 480, 720, 3). This is a 5D tensor: clips, frames, height, width, and channels.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ETdCL4dwMsQnx_9oX0EnPQ.png" /></figure><p><strong>Count the values</strong></p><p>To estimate memory, multiply the lengths of all five axes:</p><p><strong>4 × 1,800 × 480 × 720 × 3 = 7,464,960,000 values</strong></p><p>That is the number of numbers in the tensor. The shape tells us how many values there are, but it does not yet tell us the storage size. For that, we also need the data type.</p><p><strong>Add the data type</strong></p><p>The <a href="https://www.kaggle.com/code/muhammadf4h33m/04-multi-dimensional-tensor-data-representation">notebook</a> uses `float32`. Each value takes 32 bits, or 4 bytes. So the array’s storage, before any processing overhead, is:</p><p><strong>7,464,960,000 values × 4 bytes = 29,859,840,000 bytes</strong></p><p>Using 1,024 bytes per KiB, 1,024 KiB per MiB, and 1,024 MiB per GiB gives about <strong>27.81 GiB</strong>.</p><p>Here is the same calculation in Python:</p><pre>import math<br>import numpy as np<br>video_shape = (4, 1800, 480, 720, 3)<br>value_count = math.prod(video_shape)<br>bytes_per_value = np.dtype(np.float32).itemsize<br>total_bytes = value_count * bytes_per_value<br>total_gib = total_bytes / (1024 ** 3)<br>print(f&quot;Values: {value_count:,}&quot;)<br>print(f&quot;Bytes: {total_bytes:,}&quot;)<br>print(f&quot;GiB: {total_gib:.2f}&quot;)</pre><p>The notebook’s calculation gets the same approximate result. The handwritten page calls it “<strong>GB</strong>,” but because it divides by 1,024 at each step, <strong>GiB</strong> is the more precise label for the result.</p><p><strong>Why the estimate matters</strong></p><p>The number 27.81 GiB is the size of this uncompressed `float32` tensor by itself. A real training run may need additional memory for labels, model parameters, intermediate calculations, and other data. That means the estimate is a starting point for planning, not a promise that a machine with exactly this much memory will be sufficient.</p><p>Video is often kept in compressed file formats on disk. When the video is decoded into an array for processing, the in-memory representation depends on the chosen shape and data type. A compressed file size and an uncompressed tensor size answer different questions.</p><p>If the array does not fit in memory, the tensor shape suggests several things to consider: process fewer clips at a time, use a smaller frame size or fewer frames, or choose a representation that uses fewer bytes per value when the task allows it. Each choice changes what information the model receives, so it should be made deliberately.</p><p><strong>The takeaway</strong></p><p>To estimate the storage for a tensor, multiply all the shape entries to count its values, then multiply by the bytes used by one value. For video, the axes describe the clips, frames, pixels, and channels. The data type completes the estimate.</p><p>That simple calculation turns a shape such as `(4, 1800, 480, 720, 3)` into a practical memory estimate — and can reveal a hardware constraint before training begins.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=78a2355866a5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Tensor Shape, Rank, and Size Explained With NumPy]]></title>
            <link>https://medium.com/@limtlesleg8988/tensor-shape-rank-and-size-explained-with-numpy-d799c41f9ed4?source=rss-7725a1518a25------2</link>
            <guid isPermaLink="false">https://medium.com/p/d799c41f9ed4</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[tensor]]></category>
            <category><![CDATA[numpy]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Muhammad Faheem]]></dc:creator>
            <pubDate>Fri, 23 Oct 2026 05:46:50 GMT</pubDate>
            <atom:updated>2026-10-23T05:46:50.109Z</atom:updated>
            <content:encoded><![CDATA[<p>Start with one number, then build up to the arrays machine learning uses. A few small NumPy examples make tensor rank, shape, and size easier to read.</p><p><strong>A tensor is a container for numbers</strong></p><p>Machine learning data often starts as rows and columns in a table. Before an algorithm can use it, a library stores those numbers in an array-like structure called a <strong>tensor</strong>. You can think of a tensor as a container for numbers, arranged along one or more axes.</p><p>Let’s build one from the smallest case.</p><p><strong>Start with a scalar</strong></p><p>A single value, such as `<strong>4</strong>`, is a scalar. It has no axes, so its rank is 0. NumPy shows this directly:</p><pre>import numpy as np<br>scalar = np.array(4)<br>print(scalar.ndim) # 0: number of axes, also called rank<br>print(scalar.shape) # (): no axes<br>print(scalar.size) # 1: one value</pre><p>There is one number, but no row or column axis yet.</p><p><strong>Put values along one axis</strong></p><p>Now combine values into a vector:</p><pre>vector = np.array([1, 2, 3, 4])<br>print(vector.ndim) # 1<br>print(vector.shape) # (4,)<br>print(vector.size) # 4</pre><p>The vector has one axis, and that axis contains four values. Its <strong>rank</strong> is 1, its <strong>shape</strong> is `(4,)`, and its <strong>size</strong> is 4.</p><p>This example also shows why the word “dimension” can be confusing. A tensor’s dimension often means its number of axes: this vector is 1D. In mathematics, the dimension of the vector can mean its number of values: this vector has four. When working with NumPy, `ndim` gives the number of axes, while `shape` tells you the length of each axis.</p><p><strong>Stack vectors into a matrix</strong></p><p>If we stack vectors of the same length, we get a matrix:</p><pre>matrix = np.array([<br> [1, 2, 3],<br> [4, 5, 6],<br> [7, 8, 9],<br>])<br>print(matrix.ndim) # 2: rows and columns<br>print(matrix.shape) # (3, 3): 3 rows, 3 values in each row<br>print(matrix.size) # 9: total number of values</pre><p>The first shape entry counts rows; the second counts columns. A matrix with shape `(2, 3)` has two rows and three columns, for six values. A matrix with shape `(4, 2)` has eight values.</p><p>The simple rule is:</p><p>- <strong>Rank</strong> tells you how many axes there are.<br>- <strong>Shape</strong> tells you how many items are on each axis.<br>- <strong>Size</strong> counts all the values in the tensor.</p><p>For a matrix, size is rows multiplied by columns. For higher-rank tensors, multiply all the shape entries.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZfrmIGTDuvzUQvhiEulPyA.png" /></figure><p><strong>Add axes when the data needs them</strong></p><p>The same idea extends beyond matrices. A 3D tensor can be viewed as a stack of matrices. For example, shape `(4, 3, 3)` describes four matrices, each with three rows and three columns. It has three axes and contains `4 × 3 × 3 = 36` values.</p><p>Higher rank does not mean “more important.” It means the data needs more axes to describe its structure. A batch of matrices adds an axis for which matrix you are looking at. A sequence of vectors adds an axis for sequence position.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yBtm6FjVRokzWci9kEFWyw.png" /></figure><p><strong>Read a shape as a description of the data</strong></p><p>Suppose each student is described by three input values: CGPA, IQ, and a numeric state code. One student can be represented as a vector such as `[8.1, 91, 0]`. The three numbers are the features for that student.</p><p>Stack many students and you get a 2D tensor. If there are 10,000 students and each has three inputs, the feature tensor has shape `(10000, 3)`. The first axis selects a student; the second selects one of that student’s features. If placement is the target, its labels live separately in a 1D tensor with one value per student.</p><pre>X = np.array([<br> [8.1, 91, 0],<br> [7.2, 81, 1],<br>])<br>y = np.array([1, 1])<br>print(X.shape) # (2, 3): 2 students, 3 input features each<br>print(y.shape) # (2,): one placement label per student<br>print(X[0]) # first student&#39;s three features<br>print(X[:, 1]) # IQ value for every student</pre><p>Once you know what each axis means, the shape becomes a short description of how the data is organized.</p><p><strong>Keep these three questions in mind</strong></p><p>When you see a NumPy array or tensor, ask:</p><p>1. How many axes does it have? Check `array.ndim` for the rank.<br>2. What does each axis represent? Read `array.shape` in that order.<br>3. How many values are stored? Check `array.size` or multiply the shape entries.</p><p>These questions help you catch shape mistakes before they reach a model. A tensor is not just a large list of numbers; its axes tell the model how those numbers belong together.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d799c41f9ed4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Turn a Business Goal Into a Machine Learning Problem]]></title>
            <link>https://medium.com/@limtlesleg8988/how-to-turn-a-business-goal-into-a-machine-learning-problem-46845b6b6483?source=rss-7725a1518a25------2</link>
            <guid isPermaLink="false">https://medium.com/p/46845b6b6483</guid>
            <category><![CDATA[business]]></category>
            <category><![CDATA[business-problem-framing]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[customer-experience]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Muhammad Faheem]]></dc:creator>
            <pubDate>Tue, 20 Oct 2026 08:59:23 GMT</pubDate>
            <atom:updated>2026-10-20T09:04:56.918Z</atom:updated>
            <content:encoded><![CDATA[<p>How to Turn a Business Goal Into a Machine Learning Problem</p><p><em>A customer churn example shows how to move from a broad business goal to a useful prediction task — and why a prediction alone is not a solution.</em></p><p><strong>Start with the business goal</strong></p><p>Imagine a streaming service wants to increase revenue. It could try to attract more customers, charge existing customers more, or reduce the number who leave. In the example from Notebook 03, the team chooses to focus on churn: customers ending their subscriptions.</p><p>That is a business direction, but it is not yet a machine learning problem. Before choosing an algorithm, the team needs to make the goal specific enough to guide its work.</p><p>For example, the <a href="https://www.kaggle.com/code/muhammadf4h33m/03-mldlc-problem-framing-and-toy-pipeline">notebook</a> uses a hypothetical goal of lowering churn from 4% to 3.75%. Those figures illustrate how to frame a target; they are not claims about a real company.</p><p><strong>Define churn before modeling it</strong></p><p>Churn rate measures the share of customers who leave during a period, relative to the customers at the start of that period:</p><p><strong>Churn rate = customers lost during the period ÷ customers at the start of the period × 100%</strong></p><p>If 100 customers start a month and 2% churn, about 98 remain at the end, assuming no new customers join during the month.</p><p>The rate describes what happened across a group. A model needs a more specific target for each example: did this customer leave during the period we care about, or not?</p><p><strong>Turn the goal into a prediction task</strong></p><p>There are two different questions a team might ask:</p><p>- <strong>Who is likely to leave soon?</strong> This is a near-term prediction task. The model assigns each customer a likely-leave or stay label, often using an estimated probability.<br>- <strong>Why are customers leaving?</strong> This calls for a deeper investigation into reasons such as the user experience or the content being recommended. The notebook treats this as a longer study, separate from the immediate prediction task.</p><p>Keeping these questions separate makes the first project more concrete. For each historical customer, the team needs to know what information was available at prediction time and whether the customer later left within the chosen period. Those past outcomes provide the labels for supervised learning.</p><p>Because the near-term target is a leave-or-stay label, this is a <strong>classification</strong> problem. The churn rate itself is a percentage, but that does not make the individual leave-or-stay task regression. Regression would be appropriate if the target were a numeric value, such as predicting a future churn rate.</p><p><strong>Decide what information the model can use</strong></p><p>The next question is what data might help identify changing customer behavior. Notebook 03 suggests signals such as:</p><p>- how long a customer watches,<br>- searches that do not find a result,<br>- content stopped before the end, and<br>- recommendations the customer clicks.</p><p>These are candidate signals, not proof that a person will leave. The team should work with people who understand the existing service and with data engineers who can locate the relevant fields. Then it can prepare a dataset that connects each customer’s information with the outcome the model is meant to predict.</p><p>It is also useful to check whether a solution already exists. Talking with the people who manage the current process can reveal what works, what does not, and what a new model would need to improve.</p><p><strong>Decide how to measure success</strong></p><p>Before comparing models, agree on how predictions will be evaluated and what a useful result means. Compare the model’s predictions with what customers actually did during the chosen period.</p><p>But model performance is only one part of success. The business goal is to reduce churn, so the team also needs to evaluate the action it takes based on the predictions.</p><p>In the notebook’s example, the proposed policy offers a larger discount to customers with a higher predicted chance of leaving and a smaller discount to those with a lower chance. That is a policy to test — not a result guaranteed by the model. A prediction can identify a group to target; it does not prove that a discount will change their decision.</p><p><strong>Plan how predictions will be used</strong></p><p>The team must decide when predictions are needed. If customer behavior changes quickly — for example, around holidays or during a major disruption — processing new activity more frequently may be useful. If frequent processing is too difficult or costly, the team could score a batch of customers on a schedule instead.</p><p>This is a serving decision: how often the system receives data and returns predictions. It does not automatically mean that the model is retrained after every new customer event. Prediction frequency and model training frequency are separate choices.</p><p>The deployment plan should also make clear where a prediction goes and what happens next. A score that never reaches the people or system responsible for taking action cannot help with the business goal.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YjCYbb6DM8ekxpgxgAm7Eg.png" /></figure><p><em>The churn example connects a business goal to a prediction task and a policy to evaluate.</em></p><p><strong>A practical framing checklist</strong></p><p>Before training a model, answer these questions:</p><p>1. <strong>What business outcome should improve?</strong> State it in measurable terms.<br>2. <strong>What exactly should the model predict?</strong> Define the customer, outcome, and time period.<br>3. <strong>What information is available at prediction time?</strong> Identify candidate fields and how they will be prepared.<br>4. <strong>How will predictions be evaluated?</strong> Compare them with known outcomes and choose a measure before comparing models.<br>5. <strong>What action follows a prediction?</strong> Describe the policy and test whether it helps achieve the business goal.<br>6. <strong>When does the system need a prediction?</strong> Choose an online or scheduled batch process based on how quickly the data changes and the operational constraints.</p><p>Framing the problem this way turns “reduce churn” into a sequence of decisions a team can investigate and build. It also prevents a common planning mistake: treating a model’s prediction as if it had already solved the business problem.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=46845b6b6483" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Overfitting vs. Underfitting: Understanding Model Complexity with Python]]></title>
            <link>https://medium.com/@limtlesleg8988/overfitting-vs-underfitting-understanding-model-complexity-with-python-d8ec75cfaf85?source=rss-7725a1518a25------2</link>
            <guid isPermaLink="false">https://medium.com/p/d8ec75cfaf85</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[overfitting]]></category>
            <category><![CDATA[underfitting]]></category>
            <dc:creator><![CDATA[Muhammad Faheem]]></dc:creator>
            <pubDate>Sun, 18 Oct 2026 06:52:23 GMT</pubDate>
            <atom:updated>2026-10-18T06:52:23.238Z</atom:updated>
            <content:encoded><![CDATA[<p>A simple polynomial regression example shows why fitting training data closely does not always lead to better predictions.</p><p><strong>How complex should a model be?</strong></p><p>Imagine drawing a curve through a set of data points. A straight line is easy to understand, but it may miss a curved pattern. A very flexible curve can pass close to every point, but it may also follow random bumps that will not appear in new data.</p><p>That is the balance behind <strong>underfitting</strong> and <strong>overfitting</strong>. A useful model needs enough flexibility to learn the pattern, without treating every detail in its training examples as a rule.</p><p><strong>Underfitting: when the model is too simple</strong></p><p>An underfit model is not flexible enough to capture the pattern in the data. It makes substantial errors on the examples it learned from, and often on new examples too.</p><p>For example, if the relationship between an input and an output is curved, a straight line may miss the trend — even for data the model has already seen.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*10q25RJ1wFp2lz5CNGVp9g.png" /></figure><p><em>A straight line is too simple to follow this curved pattern.</em></p><p><strong>Overfitting: when the model follows the training data too closely</strong></p><p>An overfit model follows its training examples so closely that it can learn accidental details and noise along with the underlying pattern. It may look accurate on the data it learned from but make poor predictions for examples it has not seen.</p><p>The handwritten notes give a memorable example: if one ticket in Karachi is expensive, concluding that all of Karachi is expensive would be a hasty generalization from one observation. An overfit model makes a similar mistake: it treats details in its sample as if they were a general rule.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*l2tvt3gvgQBXmCKh8GW2rg.png" /></figure><p><em>The jagged curve follows individual sample details; the smoother curve shows the broader pattern.</em></p><p><strong>Use training and test error to compare models</strong></p><p>We can make the difference more concrete by checking a model on two sets of examples:</p><p>- <strong>Training data</strong> is used to fit the model.<br>- <strong>Test data</strong> is held aside during fitting. It helps us see how the model handles examples it did not learn from.</p><p>For regression, we can measure prediction error with <strong>mean squared error (MSE)</strong>. For each example, find the difference between the actual value and the model’s prediction, square that difference, then average the squared differences:</p><p><strong>MSE = average of (actual value − predicted value)²</strong></p><p>Squaring makes larger errors count more. A lower MSE means predictions are closer to the actual values on average for the examples being measured.</p><p>We calculate MSE separately for the training set and the test set. If both are high, the model may be too simple to capture the pattern. If training MSE is low but test MSE is much higher, the model may have learned details that do not carry over to new examples.</p><p><strong>A polynomial regression example in Python</strong></p><p>Let’s compare polynomial models of three degrees on a noisy sine-wave dataset. A degree-1 polynomial is a straight line. Degree 4 and degree 12 models can make more complex curves.</p><p>In a polynomial model, each higher degree adds another power of the input: x, x², x³, and so on. That gives the model more ways to bend around the training data. The question is whether that extra flexibility helps it predict new examples.</p><p>The code below uses the same fixed training and test split as Notebook 02. It fits each polynomial to the training set, then measures its error on both sets:</p><p>import numpy as np</p><p># Create a noisy sine-wave dataset.<br>data_rng = np.random.default_rng(7)<br>x = np.sort(data_rng.uniform(-3, 3, 50))<br>y = np.sin(1.5 * x) + data_rng.normal(0, 0.3, size=len(x))</p><p># Keep a separate test set for checking generalization.<br>split_rng = np.random.default_rng(2)<br>indices = split_rng.permutation(len(x))<br>test_indices = indices[:13]<br>train_indices = indices[13:]<br>x_train, y_train = x[train_indices], y[train_indices]<br>x_test, y_test = x[test_indices], y[test_indices]</p><p>print(f”{‘Degree’:&lt;10}{‘Training MSE’:&lt;18}{‘Test MSE’}”)<br>for degree in (1, 4, 12):<br> coefficients = np.polyfit(x_train, y_train, degree)<br> train_predictions = np.polyval(coefficients, x_train)<br> test_predictions = np.polyval(coefficients, x_test)<br> train_mse = np.mean((y_train — train_predictions) ** 2)<br> test_mse = np.mean((y_test — test_predictions) ** 2)<br> print(f’{degree:&lt;10}{train_mse:&lt;18.3f}{test_mse:.3f}’)</p><p><a href="https://www.kaggle.com/code/muhammadf4h33m/02-ml-challenges-and-industry-applications">Notebook 02</a> reports these results for the fixed split:</p><p>| Polynomial degree | Training MSE | Test MSE |<br>| — -: | — -: | — -: |<br>| 1 | 0.598 | 0.332 |<br>| 4 | 0.126 | 0.092 |<br>| 12 | 0.042 | 0.608 |</p><p>The degree-1 model is too simple for this curved pattern, so its error remains relatively high. The degree-4 model captures the shape better and has the lowest test MSE in this example. The degree-12 model has the lowest training MSE, but its test MSE is much higher. It fits the training examples closely without predicting the held-out examples as well.</p><p>These numbers describe one dataset and one fixed split. They are not universal thresholds, and a degree-12 polynomial will not always overfit. The useful lesson is to compare how a model performs on the examples it learned from with how it performs on examples held aside.</p><p><strong>The takeaway</strong></p><p>Underfitting happens when a model is too simple to capture the pattern. Overfitting happens when it follows training details too closely. In both cases, the model may fail to make good predictions on new data.</p><p>Model complexity is a choice, not a goal by itself. In this example, the most complex model fits the training set best, while the middle-degree model predicts the test set best. Comparing training and test MSE helps us see that difference.</p><p># Machine Learning #Python #Data Science #Artificial Intelligence #Programming</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d8ec75cfaf85" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>