<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Aashish Singh on Medium]]></title>
        <description><![CDATA[Stories by Aashish Singh on Medium]]></description>
        <link>https://medium.com/@aashish.singh2k8?source=rss-9a88942c15ca------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*xMrcQcOJjy5TjJsDLhSvdA.jpeg</url>
            <title>Stories by Aashish Singh on Medium</title>
            <link>https://medium.com/@aashish.singh2k8?source=rss-9a88942c15ca------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 04:43:15 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@aashish.singh2k8/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Building an Interactive Data Analysis & Visualization App with Crew AI Multi-Agent System]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/@aashish.singh2k8/building-an-interactive-data-analysis-visualization-app-with-crew-ai-multi-agent-system-803e08390906?source=rss-9a88942c15ca------2"><img src="https://cdn-images-1.medium.com/max/773/1*jnRfinfA0I_pT5n2Uo28FA.png" width="773"></a></p><p class="medium-feed-snippet">In today&#x2019;s data-driven world, unlocking insights from vast datasets can be a game changer. Yet, manual querying, analysis, and&#x2026;</p><p class="medium-feed-link"><a href="https://medium.com/@aashish.singh2k8/building-an-interactive-data-analysis-visualization-app-with-crew-ai-multi-agent-system-803e08390906?source=rss-9a88942c15ca------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://medium.com/@aashish.singh2k8/building-an-interactive-data-analysis-visualization-app-with-crew-ai-multi-agent-system-803e08390906?source=rss-9a88942c15ca------2</link>
            <guid isPermaLink="false">https://medium.com/p/803e08390906</guid>
            <category><![CDATA[agentic-ai]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[generative-ai-tools]]></category>
            <category><![CDATA[retrieval-augmented-gen]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Aashish Singh]]></dc:creator>
            <pubDate>Thu, 06 Feb 2025 16:08:42 GMT</pubDate>
            <atom:updated>2025-02-06T16:08:42.749Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Building a Multimodal Retrieval-Augmented Generation (RAG) Application with Streamlit]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/@aashish.singh2k8/building-a-multimodal-retrieval-augmented-generation-rag-application-with-streamlit-b5be87f3a9d9?source=rss-9a88942c15ca------2"><img src="https://cdn-images-1.medium.com/max/760/1*OcIW_dWOVEIVZXs6T9duGQ.png" width="760"></a></p><p class="medium-feed-snippet">In today&#x2019;s data-rich environment, organizations face the challenge of efficiently extracting insights from diverse types of content&#x200A;&#x2014;&#x200A;text&#x2026;</p><p class="medium-feed-link"><a href="https://medium.com/@aashish.singh2k8/building-a-multimodal-retrieval-augmented-generation-rag-application-with-streamlit-b5be87f3a9d9?source=rss-9a88942c15ca------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://medium.com/@aashish.singh2k8/building-a-multimodal-retrieval-augmented-generation-rag-application-with-streamlit-b5be87f3a9d9?source=rss-9a88942c15ca------2</link>
            <guid isPermaLink="false">https://medium.com/p/b5be87f3a9d9</guid>
            <category><![CDATA[retrieval-augmented-gen]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[innovation]]></category>
            <category><![CDATA[generative-ai-tools]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Aashish Singh]]></dc:creator>
            <pubDate>Sun, 02 Feb 2025 06:08:51 GMT</pubDate>
            <atom:updated>2025-02-02T06:08:51.629Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Data Transformations in Machine Learning: A Deep Dive with the Breast Cancer Dataset]]></title>
            <link>https://medium.com/@aashish.singh2k8/data-transformations-in-machine-learning-a-deep-dive-with-the-breast-cancer-dataset-982af21a1e46?source=rss-9a88942c15ca------2</link>
            <guid isPermaLink="false">https://medium.com/p/982af21a1e46</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[visualization]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[data-transformation]]></category>
            <dc:creator><![CDATA[Aashish Singh]]></dc:creator>
            <pubDate>Tue, 20 Aug 2024 10:13:22 GMT</pubDate>
            <atom:updated>2024-08-20T10:14:19.133Z</atom:updated>
            <content:encoded><![CDATA[<p>Data transformation is a critical step in the machine learning pipeline, ensuring that data is in the optimal format for modeling. In this article, we will explore the importance of data transformations, different types of transformations, and how they can be applied to the Breast Cancer dataset using Python. We will also include visualizations to illustrate the effects of these transformations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/788/1*FIsJs01h2gNYwUllMwSuZA.png" /><figcaption>Overview of Data Transforms</figcaption></figure><h4>Why Data Transformation is Important in Machine Learning</h4><p>Data transformation is vital in machine learning for several reasons:</p><ol><li><strong>Model Performance</strong>: Many machine learning algorithms, such as linear regression or neural networks, perform better when the input data is normalized, standardized, or transformed to a specific distribution.</li><li><strong>Model Interpretability</strong>: Transformed data can lead to models that are easier to interpret, especially when dealing with features on different scales.</li><li><strong>Algorithm Compatibility</strong>: Some algorithms require specific data formats (e.g., categorical data needs to be encoded as integers or one-hot vectors).</li><li><strong>Handling Non-Linearity</strong>: Polynomial transformations and other non-linear transforms allow for capturing relationships that a linear model might miss.</li></ol><h3>1. Loading and Understanding the California Housing Dataset</h3><p>The California Housing dataset includes features such as median income, median house value, and other socio-economic indicators for various districts in California. This dataset is widely used for regression tasks.</p><pre>from sklearn.datasets import fetch_california_housing<br>import pandas as pd<br><br># Load dataset<br>housing = fetch_california_housing()<br>df = pd.DataFrame(housing.data, columns=housing.feature_names)<br>df[&#39;target&#39;] = housing.target<br><br># Display the first few rows<br>df.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/882/1*LlSKPGQvGGv7kDklw5tH6A.png" /><figcaption>Dataset</figcaption></figure><h4>1. Numeric Data Type Transformations</h4><p>Numeric data types include integers and floating-point values, which often require scaling or normalization to enhance model performance.</p><p><strong>1.1 Normalization Transform</strong></p><ul><li><strong>Purpose</strong>: Scale numerical data to a range between 0 and 1. This is particularly useful for algorithms like neural networks where data in different scales can slow down or skew learning.</li><li><strong>When to Use</strong>: Apply when the features have different scales or when using algorithms sensitive to the magnitude of data.</li></ul><pre>from sklearn.preprocessing import MinMaxScaler<br>import matplotlib.pyplot as plt<br><br># Normalize data<br>scaler = MinMaxScaler()<br>normalized_data = scaler.fit_transform(df[[&#39;MedInc&#39;, &#39;AveOccup&#39;]])<br><br># Visualizing the original and normalized data for &#39;MedInc&#39;<br>plt.figure(figsize=(12, 6))<br>plt.subplot(1, 2, 1)<br>plt.hist(df[&#39;MedInc&#39;], bins=30, color=&#39;blue&#39;, alpha=0.7)<br>plt.title(&#39;Original Median Income&#39;)<br><br>plt.subplot(1, 2, 2)<br>plt.hist(normalized_data[:, 0], bins=30, color=&#39;green&#39;, alpha=0.7)<br>plt.title(&#39;Normalized Median Income&#39;)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WFhYPO5SEyt-2wd5L97iLA.png" /></figure><p><strong>1.2 Standardization Transform</strong></p><ul><li><strong>Purpose</strong>: Adjust data to have a mean of 0 and a standard deviation of 1. This is helpful when features have different units or when the data follows a Gaussian distribution.</li><li><strong>When to Use</strong>: Useful when dealing with Gaussian distributions or when your algorithm assumes normally distributed data.</li></ul><pre>from sklearn.preprocessing import StandardScaler<br><br># Standardize data<br>scaler = StandardScaler()<br>standardized_data = scaler.fit_transform(df[[&#39;MedInc&#39;, &#39;AveOccup&#39;]])<br><br># Visualizing the original and standardized data for &#39;MedInc&#39;<br>plt.figure(figsize=(12, 6))<br>plt.subplot(1, 2, 1)<br>plt.hist(df[&#39;MedInc&#39;], bins=30, color=&#39;blue&#39;, alpha=0.7)<br>plt.title(&#39;Original Median Income&#39;)<br><br>plt.subplot(1, 2, 2)<br>plt.hist(standardized_data[:, 0], bins=30, color=&#39;red&#39;, alpha=0.7)<br>plt.title(&#39;Standardized Median Income&#39;)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*k4q_FgxUjfdwqEkAnAORqg.png" /></figure><p><strong>1.3 Power Transform</strong></p><ul><li><strong>Purpose</strong>: Make the data more Gaussian-like, which can improve the performance of models that assume a Gaussian distribution.</li><li><strong>When to Use</strong>: When your data is skewed and you want to apply a transform that stabilizes variance.</li></ul><pre>from sklearn.preprocessing import PowerTransformer<br><br># Power transform<br>pt = PowerTransformer()<br>power_data = pt.fit_transform(df[[&#39;MedInc&#39;, &#39;AveOccup&#39;]])<br><br># Visualizing the original and power transformed data for &#39;MedInc&#39;<br>plt.figure(figsize=(12, 6))<br>plt.subplot(1, 2, 1)<br>plt.hist(df[&#39;MedInc&#39;], bins=30, color=&#39;blue&#39;, alpha=0.7)<br>plt.title(&#39;Original Median Income&#39;)<br><br>plt.subplot(1, 2, 2)<br>plt.hist(power_data[:, 0], bins=30, color=&#39;purple&#39;, alpha=0.7)<br>plt.title(&#39;Power Transformed Median Income&#39;)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*34-kxnAIxFsvmWZiio8atQ.png" /></figure><p><strong>1.4 Polynomial Transformation</strong></p><p><strong>Purpose</strong>: Polynomial Transformation is used to create new features by taking existing features and raising them to a specified power (e.g., squared, cubed). This process allows machine learning models to capture more complex, non-linear relationships between features that might not be apparent when using the original features alone.</p><p>By expanding the feature set to include polynomial terms (e.g., x2x²x2, xyxyxy, etc.), polynomial transformations can enhance the model’s ability to fit the data more closely, particularly when the relationship between the features and the target variable is non-linear. This is especially useful in linear models where the original features do not fully capture the underlying patterns in the data.</p><p><strong>When to Use</strong>:</p><ul><li><strong>Non-Linear Relationships</strong>: Use polynomial transformations when you suspect or know that there are non-linear relationships between your features and the target variable. For instance, if a scatter plot of the data suggests a curved relationship, polynomial features can help the model fit the curve.</li></ul><pre>from sklearn.preprocessing import PolynomialFeatures<br><br># Polynomial transformation (degree 2)<br>poly = PolynomialFeatures(degree=2, include_bias=False)<br>poly_data = poly.fit_transform(df[[&#39;MedInc&#39;, &#39;AveRooms&#39;]])<br><br># Visualizing the relationship between original features and transformed features<br>plt.figure(figsize=(12, 6))<br>plt.scatter(df[&#39;MedInc&#39;], df[&#39;AveRooms&#39;], c=df[&#39;target&#39;], cmap=&#39;viridis&#39;, alpha=0.5, label=&#39;Original Data&#39;)<br>plt.scatter(poly_data[:, 0], poly_data[:, 1], c=df[&#39;target&#39;], cmap=&#39;plasma&#39;, alpha=0.5, label=&#39;Polynomial Transformed Data&#39;)<br>plt.title(&#39;Polynomial Transformation: Median Income vs Average Rooms&#39;)<br>plt.xlabel(&#39;Median Income&#39;)<br>plt.ylabel(&#39;Average Rooms&#39;)<br>plt.legend()<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oKuOTkfC0iGxIqBj2FvUhg.png" /></figure><p><strong>1.5 Quantile Transform</strong></p><ul><li><strong>Purpose</strong>: Imposes a specific probability distribution (e.g., uniform or Gaussian) on the data.</li><li><strong>When to Use</strong>: When you need to force a specific distribution on a variable, especially in cases with non-Gaussian distributions.</li></ul><pre>from sklearn.preprocessing import QuantileTransformer<br><br># Quantile transform<br>qt = QuantileTransformer(output_distribution=&#39;normal&#39;)<br>quantile_data = qt.fit_transform(df[[&#39;MedInc&#39;, &#39;AveOccup&#39;]])<br><br># Visualizing the original and quantile transformed data for &#39;MedInc&#39;<br>plt.figure(figsize=(12, 6))<br>plt.subplot(1, 2, 1)<br>plt.hist(df[&#39;MedInc&#39;], bins=30, color=&#39;blue&#39;, alpha=0.7)<br>plt.title(&#39;Original Median Income&#39;)<br><br>plt.subplot(1, 2, 2)<br>plt.hist(quantile_data[:, 0], bins=30, color=&#39;orange&#39;, alpha=0.7)<br>plt.title(&#39;Quantile Transformed Median Income&#39;)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*o25gnAHc2dcpLV4SfkXa7Q.png" /></figure><h4>2. Categorical Data Type Transformations</h4><p>Categorical data often needs to be encoded in a way that machine learning algorithms can process, such as converting categories into numbers.</p><p><strong>2.1 Ordinal Transform</strong></p><ul><li><strong>Purpose</strong>: Convert categorical variables into ordinal (ranked) integers.</li><li><strong>When to Use</strong>: When your categorical variables have a meaningful order.</li></ul><pre>import numpy as np<br>import pandas as pd<br>from sklearn.datasets import fetch_california_housing<br>from sklearn.preprocessing import OrdinalEncoder<br>import matplotlib.pyplot as plt<br><br># Load the dataset<br>housing = fetch_california_housing()<br>df = pd.DataFrame(housing.data, columns=housing.feature_names)<br><br># Create a categorical variable based on &#39;MedInc&#39;<br>df[&#39;Income_Category&#39;] = pd.cut(df[&#39;MedInc&#39;], bins=[0, 2, 5, np.inf], labels=[&#39;Low&#39;, &#39;Medium&#39;, &#39;High&#39;])<br><br># Ordinal Encoding<br>encoder = OrdinalEncoder(categories=[[&#39;Low&#39;, &#39;Medium&#39;, &#39;High&#39;]])<br>df[&#39;Income_Category_Ordinal&#39;] = encoder.fit_transform(df[[&#39;Income_Category&#39;]])<br><br># Visualization<br>plt.bar(df[&#39;Income_Category&#39;], df[&#39;Income_Category_Ordinal&#39;])<br>plt.title(&#39;Ordinal Transform of Income Category&#39;)<br>plt.xlabel(&#39;Income Category&#39;)<br>plt.ylabel(&#39;Ordinal Value&#39;)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*PymZoGe31dN0fmEzViSIqQ.png" /></figure><p><strong>2.2 One Hot Transform</strong></p><p><strong>Purpose</strong>: Convert categorical variables into a series of binary variables.</p><p><strong>When to Use</strong>: When there is no intrinsic order in the categorical variables.</p><pre>from sklearn.preprocessing import OneHotEncoder<br><br># One Hot Encoding<br>onehot_encoder = OneHotEncoder(sparse=False)<br>onehot_data = onehot_encoder.fit_transform(df[[&#39;Income_Category&#39;]])<br><br># Visualization<br>plt.imshow(onehot_data, cmap=&#39;viridis&#39;, aspect=&#39;auto&#39;)<br>plt.title(&#39;One Hot Transform of Income Category&#39;)<br>plt.xlabel(&#39;Income Category&#39;)<br>plt.ylabel(&#39;Sample Index&#39;)<br>plt.colorbar(label=&#39;Binary Encoding&#39;)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/725/1*NPoFnY6iiDFRyOEgZY3mtA.png" /></figure><p><strong>Visualization Explanation</strong>:</p><ul><li>The heatmap shows how each category (‘Low’, ‘Medium’, ‘High’) is represented as a binary vector. Each column corresponds to one category, and each row corresponds to a sample in the dataset.</li></ul><p><strong>2.3 Discretization Transform</strong></p><p><strong>Purpose</strong>: Convert continuous numeric data into discrete bins, effectively converting it into ordinal data.</p><p><strong>When to Use</strong>: When you want to segment continuous data into categories.</p><pre>from sklearn.preprocessing import KBinsDiscretizer<br><br># Discretization<br>discretizer = KBinsDiscretizer(n_bins=3, encode=&#39;ordinal&#39;, strategy=&#39;uniform&#39;)<br>discretized_data = discretizer.fit_transform(df[[&#39;MedInc&#39;]])<br><br># Visualization<br>plt.plot(df[&#39;MedInc&#39;], label=&#39;Original Median Income&#39;, alpha=0.5)<br>plt.plot(discretized_data, label=&#39;Discretized Income&#39;, alpha=0.5)<br>plt.legend()<br>plt.title(&#39;Discretization Transform of Median Income&#39;)<br>plt.xlabel(&#39;Sample Index&#39;)<br>plt.ylabel(&#39;Income Value / Discretized Value&#39;)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/692/1*GgzWFm-2wqL98Q2kLs2EnQ.png" /></figure><p><strong>Visualization Explanation</strong>:</p><ul><li>The line plot compares the original MedInc values with their discretized versions, showing how continuous income data is segmented into ordinal bins.</li></ul><h3>Conclusion</h3><p>Data transformations, particularly for categorical data, are essential steps in preparing data for machine learning models. Properly encoding categorical variables into ordinal, one-hot, or discretized formats can significantly influence the performance and interpretability of the models. The examples above demonstrate practical implementations using the California Housing dataset, providing a clear understanding of when and how to apply these transformations effectively.</p><p>These visualizations illustrate the impact of different categorical data transformations, making it easier to decide which transformation to use based on the data’s characteristics and the model’s requirements.</p><p>Stay Tuned!!</p><p>If you liked the article please do clap and follow for more!!</p><p><em>Thank you for reading!</em></p><p><em>If you’d like, </em><a href="https://www.linkedin.com/in/aashish-singh-8b6ab6a0/"><strong><em>add me on Linkedin</em></strong><em>!</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=982af21a1e46" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Unveiling the Art of Feature Selection in Machine Learning]]></title>
            <link>https://medium.com/@aashish.singh2k8/unveiling-the-art-of-feature-selection-in-machine-learning-f006b321fc2f?source=rss-9a88942c15ca------2</link>
            <guid isPermaLink="false">https://medium.com/p/f006b321fc2f</guid>
            <category><![CDATA[statistical-analysis]]></category>
            <category><![CDATA[feature-selection]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-visualization]]></category>
            <dc:creator><![CDATA[Aashish Singh]]></dc:creator>
            <pubDate>Fri, 16 Aug 2024 08:47:43 GMT</pubDate>
            <atom:updated>2024-08-16T08:47:43.678Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/697/1*3zvAJyUwXzUZ0weI0UlxeQ.png" /><figcaption>Source: Google</figcaption></figure><p>“Predictive modeling is not just about using advanced machine learning algorithms, but more about data exploration and feature engineering.”</p><p>This quote perfectly encapsulates the essence of feature selection, a crucial step in the machine learning pipeline that can significantly impact model performance and interpretability.</p><h3>What is Feature Selection?</h3><p>Feature selection is the process of identifying and selecting a subset of the most relevant features (input variables) to include in a model. This technique improves model performance by removing irrelevant or redundant features, reducing overfitting, and enhancing generalization. The goal is to develop models using only the data required to make accurate predictions, favoring simpler, well-performing models.</p><h3>Why Feature Selection?</h3><ul><li><strong>Improves Model Performance</strong>: By focusing on the most relevant features, models become more accurate in predicting new, unseen data.</li><li><strong>Reduces Computational Complexity</strong>: Fewer features mean faster training and inference times, which is crucial for real-time or large-scale applications.</li><li><strong>Enhances Interpretability</strong>: Simplifying models by removing noise makes it easier to understand how predictions are made, which is vital in sectors like healthcare and finance.</li></ul><h3>Feature Selection Techniques</h3><p>Feature selection methods are generally grouped into two categories: <strong>supervised</strong> and <strong>unsupervised</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/820/1*Ch2JFHfGEngXF_urieBz8g.png" /><figcaption>Overview of Feature Selection Techniques</figcaption></figure><h3>Supervised Feature Selection</h3><p>Supervised feature selection methods use the target variable to guide the selection of the most relevant features. Let’s dive into three popular categories: Filter, Wrapper, and Embedded methods.</p><p>First, lets load the iris dataset which we will be using throughout the article.</p><pre>import pandas as pd<br>from sklearn.datasets import load_iris<br>from sklearn.feature_selection import SelectKBest, chi2<br>import matplotlib.pyplot as plt<br><br># Load Iris dataset<br>iris = load_iris()<br>X = pd.DataFrame(iris.data, columns=iris.feature_names)<br>y = iris.target</pre><h4>1. Filter Methods</h4><p>Filter methods rely on statistical measures to evaluate the importance of features independently of the model. They are generally fast and easy to implement.</p><p><strong>1.1 Chi-Square Test</strong></p><p>The Chi-Square test is used to measure the association between categorical features and the target variable. It works well when the target variable is also categorical.</p><p><strong>When to use:</strong> Use the Chi-Square test when both the feature and the target variable are categorical.</p><p>Right now for the sake of understanding we are taking the dataset as it is.</p><pre># Apply Chi-Square test<br>chi2_selector = SelectKBest(chi2, k=2)<br>X_kbest = chi2_selector.fit_transform(X, y)<br><br># Visualize selected features<br>selected_features = X.columns[chi2_selector.get_support()]<br>plt.bar(selected_features, chi2_selector.scores_[chi2_selector.get_support()])<br>plt.title(&#39;Chi-Square Test Feature Importance&#39;)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/687/1*H9_6NDQ0PriqcRsSF_xdwg.png" /></figure><p><strong>Explanation of the Graph:</strong> The bar chart shows the Chi-Square scores for the selected features. Higher scores indicate a stronger association between the feature and the target variable. In this example, we see which features are most important in distinguishing between the classes in the Iris dataset.</p><p><strong>1.2 ANOVA (Analysis of Variance)</strong></p><p>ANOVA measures the variance between different groups and compares it to the variance within groups. It is used to identify whether there are any statistically significant differences between the means of different features concerning the target variable.</p><p><strong>When to use:</strong> Use ANOVA when the target variable is categorical, and the features are numerical.</p><pre>from sklearn.feature_selection import f_classif<br><br># Apply ANOVA<br>anova_selector = SelectKBest(f_classif, k=2)<br>X_kbest_anova = anova_selector.fit_transform(X, y)<br><br># Visualize selected features<br>selected_features_anova = X.columns[anova_selector.get_support()]<br>plt.bar(selected_features_anova, anova_selector.scores_[anova_selector.get_support()])<br>plt.title(&#39;ANOVA Feature Importance&#39;)<br>plt.xlabel(&#39;Features&#39;)<br>plt.ylabel(&#39;F-Values&#39;)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/717/1*RHM25quHNq2vAVKz2eq1OQ.png" /></figure><p><strong>Explanation of the Graph:</strong> The bar chart displays the F-values for the selected features, showing the features with the most significant variance with respect to the target variable. Higher F-values indicate that the feature contributes more to the classification task.</p><p><strong>1.3 Correlation Coefficient</strong></p><p>Correlation coefficients measure the linear relationship between features and the target variable. The Pearson correlation coefficient is commonly used to evaluate continuous variables.</p><p><strong>When to use:</strong> Use the correlation coefficient when both the feature and target variable are continuous.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/790/1*RuBVmwlumPxSbc_heAPO7Q.png" /></figure><p><strong>Explanation of the Graph:</strong> The heatmap displays the correlation coefficients between each pair of features. Values close to 1 or -1 indicate strong linear relationships, while values near 0 suggest weak or no linear relationship. This visualization helps identify highly correlated features that might be redundant.</p><h4>2. Wrapper Methods</h4><p>Wrapper methods evaluate different combinations of features and select the best subset based on model performance. While these methods are computationally intensive, they often yield superior results.</p><p><strong>2.1 Recursive Feature Elimination (RFE)</strong></p><p>RFE recursively removes the least important features and builds the model with the remaining features, iterating until the optimal subset is found.</p><p><strong>When to use:</strong> Use RFE when model interpretability is essential, and you can afford the computational cost.</p><pre>from sklearn.feature_selection import RFE<br>from sklearn.ensemble import RandomForestClassifier<br><br># Apply RFE<br>rfe_selector = RFE(estimator=RandomForestClassifier(), n_features_to_select=2)<br>X_rfe = rfe_selector.fit_transform(X, y)<br><br># Visualize selected features<br>selected_features_rfe = X.columns[rfe_selector.get_support()]<br>plt.bar(selected_features_rfe, rfe_selector.ranking_[rfe_selector.get_support()])<br>plt.title(&#39;RFE Selected Features&#39;)<br>plt.xlabel(&#39;Features&#39;)<br>plt.ylabel(&#39;Ranking&#39;)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/705/1*QGjc4lOUkcaSq03a9JzP1A.png" /></figure><p><strong>Explanation of the Graph:</strong> This bar chart illustrates the ranking of features after applying RFE. Features with lower rankings are more important, as RFE prioritizes them during selection. The graph highlights the most critical features for the model.</p><p><strong>2.2 Forward Selection</strong></p><p>Forward selection starts with no features and adds them one by one, based on their contribution to improving the model’s performance.</p><p><strong>When to use:</strong> Use Forward Selection when you need a simple and interpretable model, especially when dealing with a manageable number of features.</p><pre>from mlxtend.feature_selection import SequentialFeatureSelector as SFS<br>from sklearn.neighbors import KNeighborsClassifier<br><br># Apply Forward Selection<br>sfs = SFS(KNeighborsClassifier(), k_features=2, forward=True, scoring=&#39;accuracy&#39;, cv=5)<br>sfs.fit(X, y)<br><br># Visualize selected features<br>selected_features_fs = X.columns[list(sfs.k_feature_idx_)]<br>plt.bar(selected_features_fs, range(len(selected_features_fs)))<br>plt.title(&#39;Forward Selection Features&#39;)<br>plt.xlabel(&#39;Features&#39;)<br>plt.ylabel(&#39;Selected Order&#39;)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/703/1*khUBrJRcvLeN899t-3wtCw.png" /></figure><p><strong>Explanation of the Graph:</strong> This bar chart shows the order in which features were selected during the Forward Selection process. The first features added have the highest impact on improving the model’s accuracy, as seen from their early selection.</p><p><strong>2.3 Backward Elimination</strong></p><p>Backward elimination begins with all features and removes the least significant ones based on their p-values, refining the model iteratively.</p><p><strong>When to use:</strong> Use Backward Elimination when you prefer a more gradual reduction of features, allowing for detailed examination of each feature’s impact.</p><pre>from sklearn.feature_selection import RFECV<br><br># Apply Backward Elimination using RFECV<br>rfecv_selector = RFECV(estimator=RandomForestClassifier(), step=1, cv=5, scoring=&#39;accuracy&#39;)<br>X_rfecv = rfecv_selector.fit_transform(X, y)<br><br># Visualize selected features<br>selected_features_rfecv = X.columns[rfecv_selector.get_support()]<br>plt.bar(selected_features_rfecv, rfecv_selector.cv_results_[&#39;mean_test_score&#39;][rfecv_selector.get_support()])<br>plt.title(&#39;Backward Elimination Features&#39;)<br>plt.xlabel(&#39;Features&#39;)<br>plt.ylabel(&#39;Cross-Validation Scores&#39;)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/705/1*DANuXghtKCFjPdDEY8VrYQ.png" /></figure><p><strong>Explanation of the Graph:</strong> The bar chart displays cross-validation scores for each feature selected by Backward Elimination. The features with higher scores contribute more to the model’s performance, indicating their importance in the final feature set.</p><h4>3. Embedded Methods</h4><p>Embedded methods perform feature selection during the model training process. These methods are typically faster and offer a balance between performance and computation.</p><p><strong>3.1 Lasso Regression</strong></p><p>Lasso Regression adds a penalty to the model for large coefficients, effectively shrinking some coefficients to zero, thereby performing feature selection.</p><p><strong>When to use:</strong> Use Lasso Regression when you want to handle multicollinearity and perform feature selection simultaneously.</p><pre>from sklearn.linear_model import Lasso<br><br># Apply Lasso Regression<br>lasso = Lasso(alpha=0.1)<br>lasso.fit(X, y)<br><br># Visualize selected features<br>plt.bar(X.columns, lasso.coef_)<br>plt.title(&#39;Lasso Regression Feature Importance&#39;)<br>plt.xlabel(&#39;Features&#39;)<br>plt.ylabel(&#39;Coefficients&#39;)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/717/1*Ot6iYj7MyIzta8TWHZoPDQ.png" /></figure><p><strong>Explanation of the Graph:</strong> The bar chart shows the coefficients assigned to each feature by Lasso Regression. Features with coefficients close to zero are less important and may be eliminated, highlighting the most influential features in the model.</p><p><strong>3.2 Tree-based Methods</strong></p><p>Tree-based methods like Random Forests provide feature importance scores directly, which can be used to identify the most critical features.</p><p><strong>When to use:</strong> Use Tree-based methods when you need a robust model that can handle both classification and regression tasks, and when interpretability is less of a concern.</p><pre># Apply Random Forest<br>rf = RandomForestClassifier()<br>rf.fit(X, y)<br><br># Visualize feature importance<br>plt.bar(X.columns, rf.feature_importances_)<br>plt.title(&#39;Random Forest Feature Importance&#39;)<br>plt.xlabel(&#39;Features&#39;)<br>plt.ylabel(&#39;Importance&#39;)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/706/1*V86iYaCFi13RYmojT9D9Lg.png" /></figure><p><strong>Explanation of the Graph:</strong> This bar chart displays the importance scores assigned to each feature by the Random Forest model. Features with higher importance scores contribute more to the model’s predictions, helping to identify the most valuable features.</p><h3>Unsupervised Feature Selection</h3><p>Unsupervised feature selection methods are used when there is no target variable available. These methods are often employed for dimensionality reduction.</p><h4>1. Principal Component Analysis (PCA)</h4><p>PCA reduces the dimensionality of the data by transforming the features into a set of uncorrelated principal components, capturing as much variance as possible.</p><p><strong>When to use:</strong> Use PCA when you want to reduce the number of features while retaining most of the variance in the data.</p><pre>from sklearn.decomposition import PCA<br><br># Apply PCA<br>pca = PCA(n_components=2)<br>X_pca = pca.fit_transform(X)<br><br># Visualize PCA components<br>plt.scatter(X_pca[:, 0], X_pca[:, 1], c=y, cmap=&#39;viridis&#39;)<br>plt.title(&#39;PCA Components&#39;)<br>plt.xlabel(&#39;Principal Component 1&#39;)<br>plt.ylabel(&#39;Principal Component 2&#39;)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/717/1*mEIyswBh2zFouchdk4nFvw.png" /></figure><p><strong>Explanation of the Graph:</strong> The scatter plot visualizes the data in the reduced feature space created by PCA. The two principal components shown capture the maximum variance in the data, allowing us to see the separation between different classes in a lower-dimensional space.</p><h4>2. Independent Component Analysis (ICA)</h4><p>ICA separates a multivariate signal into additive, independent components. It’s useful when the features are assumed to be mixtures of independent sources.</p><p><strong>When to use:</strong> Use ICA when the features are non-Gaussian, and you believe they are mixed signals from different sources.</p><pre>from sklearn.decomposition import FastICA<br><br># Apply ICA<br>ica = FastICA(n_components=2)<br>X_ica = ica.fit_transform(X)<br><br># Visualize ICA components<br>plt.scatter(X_ica[:, 0], X_ica[:, 1], c=y, cmap=&#39;plasma&#39;)<br>plt.title(&#39;ICA Components&#39;)<br>plt.xlabel(&#39;Independent Component 1&#39;)<br>plt.ylabel(&#39;Independent Component 2&#39;)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/722/1*3fu9onZcRI9uqohZIfJ2IA.png" /></figure><p><strong>Explanation of the Graph:</strong> The scatter plot displays the data projected onto the independent components identified by ICA. This transformation helps in uncovering the underlying sources that contribute to the observed data, often revealing structure not apparent in the original feature space.</p><h3>Conclusion</h3><p>Feature selection is an essential technique in machine learning, helping to improve model performance, reduce complexity, and enhance interpretability. In this article, we explored a variety of supervised and unsupervised feature selection methods using Python, including practical examples with the Iris dataset. Each method has its strengths and applications, and understanding when to use each can significantly impact your model’s success.</p><p>Whether you’re dealing with labeled or unlabeled data, the right feature selection approach can make all the difference in your analysis. Experiment with these techniques to find the best fit for your specific dataset and problem.</p><p>Feel free to share your thoughts, questions, and experiences with these methods in the comments. Happy coding!</p><p>Stay Tuned!!</p><p>If you liked the article please do clap and follow for more!!</p><p><em>Thank you for reading!</em></p><p><em>If you’d like, </em><a href="https://www.linkedin.com/in/aashish-singh-8b6ab6a0/"><strong><em>add me on Linkedin</em></strong><em>!</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f006b321fc2f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Identify and Remove Outliers in Your Dataset: A Guide for Data Scientists]]></title>
            <link>https://medium.com/@aashish.singh2k8/how-to-identify-and-remove-outliers-in-your-dataset-a-guide-for-data-scientists-57b07aea7fa8?source=rss-9a88942c15ca------2</link>
            <guid isPermaLink="false">https://medium.com/p/57b07aea7fa8</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[data-preprocessing]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Aashish Singh]]></dc:creator>
            <pubDate>Thu, 15 Aug 2024 10:45:29 GMT</pubDate>
            <atom:updated>2024-08-15T10:45:29.242Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/585/1*JDY9mTi7rw1E4oFvoUh2VA.png" /><figcaption>Source:Google</figcaption></figure><p>Data is the lifeblood of any machine learning model. The quality of your data directly impacts the performance of your models. But what happens when your data is contaminated with outliers — those pesky, unusual observations that can throw off your results? In this article, we’ll dive deep into the world of outliers, exploring what they are, why they matter, and how you can identify and remove them to improve your model’s accuracy.</p><h3>What Are Outliers?</h3><p>Imagine you’re conducting a survey to measure the average height of adults in a city. You gather thousands of data points, but in the midst of your data, you find a few people who are over 7 feet tall. While these values are possible, they are far from typical. These data points are what we call outliers — observations that are significantly different from the majority of your data.</p><p>Outliers can occur for various reasons:</p><ol><li><strong>Measurement errors</strong>: Sometimes, data entry mistakes or faulty measuring instruments can produce outliers.</li><li><strong>Data corruption</strong>: Errors in data transmission or storage can lead to outlier values.</li><li><strong>Genuine extreme values</strong>: In some cases, outliers represent real, albeit rare, phenomena. For example, a company’s revenue might spike due to a viral product.</li></ol><p>Ignoring outliers can skew your results and lead to inaccurate models. However, blindly removing them can also mean losing valuable insights. The key is to handle outliers with care, ensuring that you strike the right balance between data accuracy and integrity.</p><h3>Why Should You Care About Outliers?</h3><p>Outliers can significantly affect the performance of machine learning models. Here’s why:</p><ol><li><strong>Model Accuracy</strong>: Outliers can distort statistical measures like mean and standard deviation, leading to inaccurate predictions. For example, in a linear regression model, outliers can skew the regression line, making it less representative of the overall data.</li><li><strong>Model Training</strong>: Some machine learning algorithms, like k-nearest neighbors and clustering algorithms, are sensitive to outliers. They can overfit or underfit if outliers are present in the training data, leading to poor generalization on unseen data.</li><li><strong>Data Visualization</strong>: Outliers can make it difficult to interpret visualizations like box plots or scatter plots, as they can stretch the scale and obscure trends.</li></ol><p>Given these challenges, it’s crucial to identify and manage outliers effectively.</p><h3>A Simple Approach to Outlier Detection</h3><p>To make this discussion more practical, let’s consider a simple dataset. Imagine you have a dataset of 10,000 random numbers drawn from a Gaussian distribution with a mean of 50 and a standard deviation of 5. In this dataset, most values will cluster around the mean, but some will deviate significantly — these are your potential outliers.</p><p>Creating a dataset:</p><p>Now we will create the dataset which will be used throughout the article.</p><pre>import numpy as np<br>import pandas as pd<br>import matplotlib.pyplot as plt<br>import seaborn as sns<br><br># Generating random data with outliers<br>np.random.seed(42)<br>data = np.random.normal(50, 5, 10000)<br>data_with_outliers = np.append(data, [100, 105, 110])  # Adding outliers<br><br># Creating a DataFrame<br>df = pd.DataFrame(data_with_outliers, columns=[&#39;Value&#39;])<br><br></pre><h3>1. Visual Inspection</h3><p>One of the simplest ways to detect outliers is by visual inspection. Box plots, scatter plots, and histograms can reveal outliers as points that fall outside the expected range.</p><p><strong>Histogram of Data with Outliers</strong></p><pre># Plotting the histogram<br>plt.figure(figsize=(10, 6))<br>sns.histplot(df[&#39;Value&#39;], bins=50, kde=True)<br>plt.title(&#39;Histogram of Data with Outliers&#39;)<br>plt.xlabel(&#39;Value&#39;)<br>plt.ylabel(&#39;Frequency&#39;)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ugi3s_YVvqCamnSsNolUUg.png" /></figure><p><strong>Explanation:</strong> This histogram shows the distribution of your data. Most data points cluster around the mean (50), creating a bell-shaped curve typical of a normal distribution. The outliers, however, appear as small, isolated bars far to the right of the main cluster, indicating values significantly higher than the rest of the data.</p><p><strong>Box Plot of Data with Outliers</strong></p><pre># Box plot<br>plt.figure(figsize=(10, 6))<br>sns.boxplot(x=df[&#39;Value&#39;])<br>plt.title(&#39;Box Plot of Data with Outliers&#39;)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/992/1*s7FfH579Fc46iwHv8Qshyw.png" /></figure><p><strong>Explanation:</strong> A box plot is a useful tool for visualizing the spread and skewness of your data. The box represents the interquartile range (IQR), with the line inside the box showing the median. The “whiskers” extend to 1.5 times the IQR, beyond which any points are considered potential outliers. In this plot, the outliers are clearly visible as individual points far from the main body of the data.</p><p><strong>Scatter Plot of Data with Outliers</strong></p><pre># Scatter plot<br>plt.figure(figsize=(10, 6))<br>sns.scatterplot(x=np.arange(len(df)), y=df[&#39;Value&#39;])<br>plt.title(&#39;Scatter Plot of Data with Outliers&#39;)<br>plt.xlabel(&#39;Index&#39;)<br>plt.ylabel(&#39;Value&#39;)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hVU5J-i7-9l7Fp9MTtvApQ.png" /></figure><p><strong>Explanation:</strong> The scatter plot displays each data point against its index. This allows you to see how the values are distributed over the entire dataset. The outliers stand out as points that deviate significantly from the rest of the data, making them easy to identify.</p><h3>2. Statistical Methods</h3><p>For a more rigorous approach, you can use statistical methods to identify outliers. Here are two common techniques:</p><h4>Z-Score</h4><p>The Z-score measures how many standard deviations a data point is from the mean. A common threshold is a Z-score greater than 3 or less than -3, which might indicate an outlier.</p><pre>from scipy.stats import zscore<br><br># Calculate Z-scores<br>df[&#39;Z-Score&#39;] = zscore(df[&#39;Value&#39;])<br><br># Identify outliers<br>outliers = df[np.abs(df[&#39;Z-Score&#39;]) &gt; 3]<br><br># Plot Z-Scores<br>plt.figure(figsize=(10, 6))<br>sns.scatterplot(x=np.arange(len(df)), y=df[&#39;Z-Score&#39;])<br>plt.axhline(y=3, color=&#39;r&#39;, linestyle=&#39;--&#39;, label=&#39;Z = 3&#39;)<br>plt.axhline(y=-3, color=&#39;r&#39;, linestyle=&#39;--&#39;)<br>plt.title(&#39;Z-Score Plot for Outlier Detection&#39;)<br>plt.xlabel(&#39;Index&#39;)<br>plt.ylabel(&#39;Z-Score&#39;)<br>plt.legend()<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yl2rVmG5F69YRkasvwalCA.png" /></figure><p><strong>Explanation:</strong> The Z-score method identifies outliers based on how many standard deviations a point is from the mean. In this plot, points with Z-scores greater than 3 or less than -3 are potential outliers. These points are far from the majority of the data and are marked by red lines, helping you easily spot them.</p><h4>IQR (Interquartile Range)</h4><p>The IQR is the range between the first and third quartiles (Q1 and Q3) of your data. Data points that fall below Q1–1.5<em>IQR or above Q3 + 1.5</em>IQR are often considered outliers.</p><pre># Calculate IQR<br>Q1 = df[&#39;Value&#39;].quantile(0.25)<br>Q3 = df[&#39;Value&#39;].quantile(0.75)<br>IQR = Q3 - Q1<br><br># Identify outliers<br>outliers_iqr = df[(df[&#39;Value&#39;] &lt; (Q1 - 1.5 * IQR)) | (df[&#39;Value&#39;] &gt; (Q3 + 1.5 * IQR))]<br><br># Plot IQR outliers<br>plt.figure(figsize=(10, 6))<br>sns.boxplot(x=df[&#39;Value&#39;])<br>plt.title(&#39;IQR Method: Outliers in the Data&#39;)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/997/1*WAvlM6epwgBgxnbvCl1LUg.png" /></figure><p><strong>Explanation:</strong> This is the same box plot as before, but now we focus on the IQR method for outlier detection. The IQR is calculated by subtracting the first quartile (Q1) from the third quartile (Q3). Points that lie outside 1.5 times the IQR from Q1 or Q3 are flagged as outliers. In the box plot, these are the points outside the whiskers, clearly indicating outliers.</p><h3>3. Machine Learning Models</h3><p>For large or complex datasets, machine learning models can automatically detect outliers. Here are a few popular techniques:</p><h4>Isolation Forest</h4><pre>from sklearn.ensemble import IsolationForest<br><br># Fit the model<br>iso_forest = IsolationForest(contamination=0.01)<br>df[&#39;Anomaly&#39;] = iso_forest.fit_predict(df[[&#39;Value&#39;]])<br><br># Identify outliers<br>outliers_iso = df[df[&#39;Anomaly&#39;] == -1]<br><br># Plot Isolation Forest results<br>plt.figure(figsize=(10, 6))<br>sns.scatterplot(x=np.arange(len(df)), y=df[&#39;Value&#39;], hue=df[&#39;Anomaly&#39;], palette={1: &#39;blue&#39;, -1: &#39;red&#39;})<br>plt.title(&#39;Isolation Forest: Outliers in the Data&#39;)<br>plt.xlabel(&#39;Index&#39;)<br>plt.ylabel(&#39;Value&#39;)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TTKxhejb0SHn1iXY6G7yqg.png" /></figure><p><strong>Explanation:</strong> Isolation Forest is a machine learning algorithm that isolates observations by randomly selecting a feature and then randomly selecting a split value between the maximum and minimum values of the selected feature. Outliers, which are isolated quickly, are identified and colored red in this plot, clearly distinguishing them from the rest of the data.</p><p><strong>Local Outlier Factor (LOF)</strong></p><pre>from sklearn.neighbors import LocalOutlierFactor<br><br># Fit the model<br>lof = LocalOutlierFactor(n_neighbors=20, contamination=0.01)<br>df[&#39;LOF_Score&#39;] = lof.fit_predict(df[[&#39;Value&#39;]])<br><br># Identify outliers<br>outliers_lof = df[df[&#39;LOF_Score&#39;] == -1]<br><br># Plot LOF results<br>plt.figure(figsize=(10, 6))<br>sns.scatterplot(x=np.arange(len(df)), y=df[&#39;Value&#39;], hue=df[&#39;LOF_Score&#39;], palette={1: &#39;blue&#39;, -1: &#39;red&#39;})<br>plt.title(&#39;Local Outlier Factor: Outliers in the Data&#39;)<br>plt.xlabel(&#39;Index&#39;)<br>plt.ylabel(&#39;Value&#39;)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EJGhzjzS1vjcbJK2wIw5UQ.png" /></figure><p><strong>Explanation:</strong> The Local Outlier Factor (LOF) is another machine learning algorithm that identifies outliers by comparing the local density of a point to that of its neighbors. Points with significantly lower density are considered outliers and are marked red in the plot. This visualization helps you see how the LOF algorithm isolates outliers based on local density differences.</p><h4>Autoencoders</h4><p>Autoencoders are neural networks that aim to reconstruct their input. If a data point cannot be well reconstructed, it may be an outlier. Due to the complexity of this method, we’ll keep the implementation simple and focus on the previous methods for this article.</p><h3>When Should You Remove Outliers?</h3><p>Now that you know how to identify outliers, the next question is: should you remove them? The answer depends on the context of your analysis.</p><h3>Remove Outliers If:</h3><ul><li>They are caused by measurement errors or data corruption.</li><li>They distort your model’s accuracy and predictive power.</li><li>You have a large enough dataset that removing a few outliers won’t significantly affect your results.</li></ul><h3>Keep Outliers If:</h3><ul><li>They represent a rare but important phenomenon that you want to study.</li><li>Removing them would result in a loss of critical information.</li><li>Your model is robust enough to handle outliers without being skewed.</li></ul><p>In some cases, instead of removing outliers, you might consider <strong>transforming</strong> them. For example, you could apply a logarithmic transformation to reduce the impact of extreme values.</p><h3>Conclusion</h3><p>Outliers are an inevitable part of working with real-world data. While they can complicate your analysis and modeling, they also hold valuable information that shouldn’t be ignored lightly. By carefully identifying, analyzing, and deciding how to handle outliers, you can ensure that your models are both accurate and insightful.</p><p>Remember, there’s no one-size-fits-all approach to dealing with outliers. Each dataset and problem is unique, and the right strategy will depend on the specific context of your analysis. So, the next time you encounter outliers in your data, take a step back, assess the situation, and decide on the best course of action.</p><p>Happy data wrangling! And if you found this guide helpful, be sure to follow me for more insights on data science and machine learning. Let’s continue the conversation — how do you handle outliers in your datasets?</p><p>Stay Tuned!!</p><p>If you liked the article please do clap and follow for more!!</p><p><em>Thank you for reading!</em></p><p><em>If you’d like, </em><a href="https://www.linkedin.com/in/aashish-singh-8b6ab6a0/"><strong><em>add me on Linkedin</em></strong><em>!</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=57b07aea7fa8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Deep Dive into Optimizers in Deep Learning: Roles, Mathematics, Applications and Pseudo Python…]]></title>
            <link>https://medium.com/@aashish.singh2k8/a-deep-dive-into-optimizers-in-deep-learning-roles-mathematics-applications-and-pseudo-python-5d8ea1b566dd?source=rss-9a88942c15ca------2</link>
            <guid isPermaLink="false">https://medium.com/p/5d8ea1b566dd</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[innovation]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[optimization]]></category>
            <dc:creator><![CDATA[Aashish Singh]]></dc:creator>
            <pubDate>Wed, 14 Aug 2024 10:10:37 GMT</pubDate>
            <atom:updated>2024-08-14T10:10:37.985Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/852/1*gupuhVAoEtkuceKsIAXaEw.png" /><figcaption>Source:Google</figcaption></figure><h3>A Deep Dive into Optimizers in Deep Learning: Roles, Mathematics, Applications and Pseudo Python Code</h3><p>In the realm of deep learning, optimizers play a crucial role in training neural networks. They are the algorithms that adjust the weights of the network in order to minimize the loss function. The choice of optimizer can significantly impact the model’s performance, speed of convergence, and overall effectiveness. In this article, we will explore the most commonly used optimizers, delve into the mathematics behind them, discuss their limitations, and highlight where they are best applied.</p><h3>1. The Role of Optimizers in Deep Learning</h3><p>Optimizers are algorithms designed to iteratively update the weights of a neural network to minimize the loss function. The loss function measures how well the neural network’s predictions match the actual data, and the optimizer’s job is to find the set of weights that minimize this loss.</p><p><strong>Quote:</strong> <em>“Optimization is at the heart of machine learning. It’s the driving force behind the training of models that can learn and make decisions on their own.”</em> — Andrew Ng</p><h3>2. Gradient Descent: The Foundation of Optimization</h3><p>Before diving into advanced optimizers, let’s understand the foundation upon which they are built: Gradient Descent.</p><h4>Mathematics Behind Gradient Descent</h4><p>Gradient Descent is an iterative optimization algorithm used to minimize a function by adjusting the parameters in the opposite direction of the gradient of the function. The key formula is:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/302/1*ar-Z6EI2QljS-iTmdS90Sg.png" /><figcaption>Update Equations</figcaption></figure><p>Where:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/771/1*deDqiGetwkCFpS4L13sawg.png" /></figure><pre>import numpy as np<br>import matplotlib.pyplot as plt<br>from mpl_toolkits.mplot3d import Axes3D<br><br># Loss function: simple quadratic<br>def loss_function(x, y):<br>    return x**2 + y**2<br><br># Gradient of the loss function<br>def gradient(x, y):<br>    return np.array([2*x, 2*y])<br><br># Function for Gradient Descent optimizer<br>def gradient_descent(alpha, iterations):<br>    params = np.array([2.0, 2.0])  # Starting point<br>    path = [np.append(params, loss_function(*params))]<br>    for _ in range(iterations):<br>        grad = gradient(*params)<br>        params = params - alpha * grad<br>        path.append(np.append(params, loss_function(*params)))<br>    return np.array(path)<br><br># Create a grid for plotting the loss surface<br>x = np.linspace(-3, 3, 50)<br>y = np.linspace(-3, 3, 50)<br>X, Y = np.meshgrid(x, y)<br>Z = loss_function(X, Y)<br><br># Plotting the 3D graph<br>fig = plt.figure(figsize=(10, 7))<br>ax = fig.add_subplot(111, projection=&#39;3d&#39;)<br>ax.plot_surface(X, Y, Z, cmap=&#39;viridis&#39;, alpha=0.6)<br>path = gradient_descent(0.1, 50)<br>ax.plot(path[:, 0], path[:, 1], path[:, 2], marker=&#39;o&#39;, color=&#39;red&#39;)<br>ax.set_title(&quot;Gradient Descent&quot;)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*WmLKDjSIr1tkMetWNALOhA.png" /></figure><h4>Limitations of Gradient Descent</h4><ul><li><strong>Convergence to Local Minima</strong>: Gradient Descent may converge to local minima or saddle points in non-convex functions.</li><li><strong>Sensitive to Learning Rate</strong>: A high learning rate can cause the algorithm to overshoot the minimum, while a low learning rate can make convergence slow.</li><li><strong>Requires Properly Scaled Data</strong>: The performance can degrade if the data is not scaled properly.</li></ul><h4>Applications</h4><ul><li><strong>Linear Regression</strong>: Often used in simple linear models where the cost function is convex.</li><li><strong>Logistic Regression</strong>: Employed in binary classification tasks.</li></ul><h3>3. Stochastic Gradient Descent (SGD): Faster Convergence</h3><p>Stochastic Gradient Descent (SGD) is a variant of Gradient Descent where instead of computing the gradient over the entire dataset, it computes the gradient for a single data point (or a mini-batch) at each iteration.</p><h4>Mathematics Behind SGD</h4><p>The update rule for SGD is similar to Gradient Descent, but it operates on a single sample:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/497/1*VsmDc6Lt3WBir5x6eqENFA.png" /></figure><pre># SGD optimizer<br>def sgd(alpha, iterations):<br>    params = np.array([2.0, 2.0])  # Starting point<br>    path = [np.append(params, loss_function(*params))]<br>    for _ in range(iterations):<br>        grad = gradient(*params)<br>        params = params - alpha * grad + np.random.normal(0, 0.1, 2)  # Adding noise<br>        path.append(np.append(params, loss_function(*params)))<br>    return np.array(path)<br><br># Plotting the 3D graph for SGD<br>fig = plt.figure(figsize=(10, 7))<br>ax = fig.add_subplot(111, projection=&#39;3d&#39;)<br>ax.plot_surface(X, Y, Z, cmap=&#39;viridis&#39;, alpha=0.6)<br>path = sgd(0.1, 50)<br>ax.plot(path[:, 0], path[:, 1], path[:, 2], marker=&#39;o&#39;, color=&#39;red&#39;)<br>ax.set_title(&quot;Stochastic Gradient Descent&quot;)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*eHcgBT5u8KQZDDw1hnv_mw.png" /></figure><h4>Limitations of SGD</h4><ul><li><strong>Noisy Updates</strong>: The updates are noisy due to the use of single data points, which can cause the loss function to fluctuate.</li><li><strong>Requires Tuning</strong>: The learning rate and batch size need careful tuning for optimal performance.</li></ul><h4>Applications</h4><ul><li><strong>Online Learning</strong>: Suitable for scenarios where data arrives in a stream and the model needs to be updated continuously.</li><li><strong>Deep Neural Networks</strong>: Often used in conjunction with momentum to stabilize updates.</li></ul><h3>4. Momentum: Overcoming Oscillations</h3><p>Momentum is an extension to SGD that helps in accelerating gradients vectors in the right directions, thus leading to faster converging.</p><h4>Mathematics Behind Momentum</h4><p>Momentum adds a fraction of the previous update to the current update:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/622/1*jbsXhpaRRC8GYBF7YFRAyg.png" /></figure><pre># Momentum optimizer<br>def momentum(alpha, beta, iterations):<br>    params = np.array([2.0, 2.0])  # Starting point<br>    velocity = np.array([0.0, 0.0])<br>    path = [np.append(params, loss_function(*params))]<br>    for _ in range(iterations):<br>        grad = gradient(*params)<br>        velocity = beta * velocity + alpha * grad<br>        params = params - velocity<br>        path.append(np.append(params, loss_function(*params)))<br>    return np.array(path)<br><br># Plotting the 3D graph for Momentum<br>fig = plt.figure(figsize=(10, 7))<br>ax = fig.add_subplot(111, projection=&#39;3d&#39;)<br>ax.plot_surface(X, Y, Z, cmap=&#39;viridis&#39;, alpha=0.6)<br>path = momentum(0.1, 0.9, 50)<br>ax.plot(path[:, 0], path[:, 1], path[:, 2], marker=&#39;o&#39;, color=&#39;red&#39;)<br>ax.set_title(&quot;Momentum&quot;)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/586/1*O-lEZ2DBjPWR8JN5-u7DoA.png" /></figure><h4>Limitations of Momentum</h4><ul><li><strong>Hyperparameter Tuning</strong>: Requires an additional hyperparameter (momentum factor) to be tuned.</li><li><strong>Sensitivity to Initialization</strong>: Performance can be sensitive to the initial settings of the learning rate and momentum.</li></ul><h4>Applications</h4><ul><li><strong>Image Classification</strong>: Often used in Convolutional Neural Networks (CNNs) to speed up convergence.</li><li><strong>Natural Language Processing</strong>: Applied in Recurrent Neural Networks (RNNs) to stabilize training.</li></ul><h3>5. Adaptive Learning Rate Methods: RMSprop and Adagrad</h3><p>Adaptive learning rate methods adjust the learning rate during training to improve performance and convergence.</p><h4>Adagrad</h4><p>Adagrad adapts the learning rate for each parameter based on the historical gradients:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/602/1*vpGYMbP4yV5G1RHNKsPVzg.png" /></figure><h4>RMSprop</h4><p>RMSprop addresses the diminishing learning rates issue in Adagrad by using an exponentially decaying average of squared gradients:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/340/1*0Hkw25HlkopfXjEAdWDMag.png" /></figure><pre># RMSprop optimizer<br>def rmsprop(alpha, beta, iterations):<br>    params = np.array([2.0, 2.0])  # Starting point<br>    grad_square_sum = np.array([0.0, 0.0])<br>    path = [np.append(params, loss_function(*params))]<br>    for _ in range(iterations):<br>        grad = gradient(*params)<br>        grad_square_sum = beta * grad_square_sum + (1 - beta) * grad**2<br>        params = params - (alpha / (np.sqrt(grad_square_sum) + 1e-8)) * grad<br>        path.append(np.append(params, loss_function(*params)))<br>    return np.array(path)<br><br># Plotting the 3D graph for RMSprop<br>fig = plt.figure(figsize=(10, 7))<br>ax = fig.add_subplot(111, projection=&#39;3d&#39;)<br>ax.plot_surface(X, Y, Z, cmap=&#39;viridis&#39;, alpha=0.6)<br>path = rmsprop(0.1, 0.9, 50)<br>ax.plot(path[:, 0], path[:, 1], path[:, 2], marker=&#39;o&#39;, color=&#39;red&#39;)<br>ax.set_title(&quot;RMSprop&quot;)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*27LZ79zt9bZE8ALqwhMmgw.png" /></figure><h4>Limitations</h4><ul><li><strong>Decay Parameter Sensitivity</strong>: The performance is sensitive to the choice of the decay parameter.</li><li><strong>Difficulty in Escaping Local Minima</strong>: Like other optimizers, these methods may still get stuck in local minima.</li></ul><h4>Applications</h4><ul><li><strong>Sparse Data</strong>: Adagrad is particularly effective in handling sparse data, such as text data in NLP.</li><li><strong>Recurrent Neural Networks</strong>: RMSprop is widely used in training RNNs due to its stability.</li></ul><h3>6. Adam: The Best of Both Worlds</h3><p>Adam (Adaptive Moment Estimation) combines the benefits of both Momentum and RMSprop. It is one of the most popular and effective optimization algorithms in deep learning.</p><h4>Mathematics Behind Adam</h4><p>Adam maintains two moving averages, one for the gradients (momentum) and one for the squared gradients (RMSprop):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/562/1*on-6BToId74LQSm0plP7GQ.png" /></figure><pre># Adam optimizer<br>def adam(alpha, beta1, beta2, iterations):<br>    params = np.array([2.0, 2.0])  # Starting point<br>    m, v = np.array([0.0, 0.0]), np.array([0.0, 0.0])<br>    path = [np.append(params, loss_function(*params))]<br>    for t in range(1, iterations + 1):<br>        grad = gradient(*params)<br>        m = beta1 * m + (1 - beta1) * grad<br>        v = beta2 * v + (1 - beta2) * grad**2<br>        m_hat = m / (1 - beta1**t)<br>        v_hat = v / (1 - beta2**t)<br>        params = params - (alpha / (np.sqrt(v_hat) + 1e-8)) * m_hat<br>        path.append(np.append(params, loss_function(*params)))<br>    return np.array(path)<br><br># Plotting the 3D graph for Adam<br>fig = plt.figure(figsize=(10, 7))<br>ax = fig.add_subplot(111, projection=&#39;3d&#39;)<br>ax.plot_surface(X, Y, Z, cmap=&#39;viridis&#39;, alpha=0.6)<br>path = adam(0.1, 0.9, 0.999, 50)<br>ax.plot(path[:, 0], path[:, 1], path[:, 2], marker=&#39;o&#39;, color=&#39;red&#39;)<br>ax.set_title(&quot;Adam&quot;)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/621/1*-AoXix6SPHR_6gdByeMliw.png" /></figure><h4>Limitations of Adam</h4><ul><li><strong>Non-Convergence</strong>: Adam may sometimes fail to converge to the optimal solution, particularly in the case of sparse or noisy data.</li><li><strong>Overfitting</strong>: Due to its adaptive nature, Adam can lead to overfitting if not carefully regularized.</li></ul><h4>Applications</h4><ul><li><strong>General Deep Learning</strong>: Adam is widely used across various deep learning applications, from image recognition to NLP.</li><li><strong>Transfer Learning</strong>: Particularly effective in fine-tuning pre-trained models on new tasks.</li></ul><h3>7. The Future of Optimization in Deep Learning</h3><p>The field of optimization in deep learning is ever-evolving. Recent advances include optimizers like Nadam (a combination of Nesterov momentum and Adam) and LAMB (Layer-wise Adaptive Moments for Batch training), which are tailored for large-scale deep learning models like Transformers.</p><p><strong>Quote:</strong> <em>“Optimization is not just about finding a solution — it’s about finding the best possible solution in the least amount of time.”</em> — Geoffrey Hinton</p><h3>Conclusion</h3><p>Understanding the role, mathematics, and limitations of various optimizers is crucial for effectively training deep learning models. The choice of optimizer should be guided by the specific problem at hand, the architecture of the model, and the nature of the data. While Gradient Descent laid the foundation, advanced optimizers like Adam and RMSprop have become indispensable tools in the deep learning toolkit.</p><p>Each optimizer has its unique strengths and weaknesses, making it important to experiment and validate which one works best for your specific application. As the field advances, new optimizers and techniques will continue to emerge, pushing the boundaries of what’s possible in deep learning.</p><p>Stay Tuned!!</p><p>If you liked the article please do clap and follow for more!!</p><p><em>Thank you for reading!</em></p><p><em>If you’d like, </em><a href="https://www.linkedin.com/in/aashish-singh-8b6ab6a0/"><strong><em>add me on Linkedin</em></strong><em>!</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5d8ea1b566dd" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Unpacking Word Embeddings: A Journey Through Modern NLP Techniques]]></title>
            <link>https://medium.com/@aashish.singh2k8/unpacking-word-embeddings-a-journey-through-modern-nlp-techniques-8a2ca21f1e46?source=rss-9a88942c15ca------2</link>
            <guid isPermaLink="false">https://medium.com/p/8a2ca21f1e46</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[innovation]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[naturallanguageprocessing]]></category>
            <dc:creator><![CDATA[Aashish Singh]]></dc:creator>
            <pubDate>Tue, 13 Aug 2024 07:52:04 GMT</pubDate>
            <atom:updated>2024-08-13T07:52:04.868Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/777/1*0FrPF7O_-RG5AMYLDB74SA.png" /></figure><p>In the world of Natural Language Processing (NLP), word embeddings have revolutionized the way we understand and process text. Whether you’re building chatbots, recommendation systems, or sentiment analysis tools, word embeddings form the backbone of many modern NLP applications.</p><p>In this article, we’ll dive deep into what word embeddings are, explore different techniques to create them, and provide Python code examples for each. By the end, you’ll have a solid understanding of how to leverage these powerful tools in your own projects.</p><h3>“Words are like living beings. The closer you observe, the more nuanced they become.”</h3><h3>What Are Word Embeddings?</h3><p>Word embeddings are dense vector representations of words that capture their meanings, syntactic properties, and relationships with other words. Unlike traditional one-hot encoding, where words are represented as sparse vectors, word embeddings map words into a continuous vector space where semantically similar words are closer together.</p><p>Imagine a 3D space where the word “king” is near “queen,” and “man” is near “woman.” Word embeddings allow for such meaningful relationships to be represented in a form that algorithms can easily process.</p><h3>Different Techniques for Creating Word Embeddings</h3><h3>1. Word2Vec</h3><p>Word2Vec is one of the most popular techniques for creating word embeddings. Developed by Google, it uses a shallow, two-layer neural network to learn word associations from a large corpus of text.</p><p>Word2Vec has two main approaches:</p><ul><li><strong>Skip-gram:</strong> Predicts surrounding words given a current word.</li><li><strong>Continuous Bag of Words (CBOW):</strong> Predicts the current word given surrounding words.</li></ul><pre>import gensim<br>from gensim.models import Word2Vec</pre><pre># Sample sentences<br>sentences = [[&quot;machine&quot;, &quot;learning&quot;, &quot;is&quot;, &quot;fun&quot;], <br>             [&quot;python&quot;, &quot;is&quot;, &quot;a&quot;, &quot;great&quot;, &quot;language&quot;], <br>             [&quot;deep&quot;, &quot;learning&quot;, &quot;is&quot;, &quot;a&quot;, &quot;subset&quot;, &quot;of&quot;, &quot;machine&quot;, &quot;learning&quot;]]<br># Train Word2Vec model<br>model = Word2Vec(sentences, vector_size=100, window=5, min_count=1, sg=0)  # sg=0 for CBOW, sg=1 for Skip-gram<br># Get word embeddings<br>vector = model.wv[&#39;machine&#39;]<br>print(vector)</pre><p><strong>Diagrammatic Representation:</strong></p><p>To visualize the relationships between words using Word2Vec, you can use t-SNE to reduce the dimensionality of the word vectors.</p><pre>from sklearn.manifold import TSNE<br>import matplotlib.pyplot as plt</pre><pre># Sample words to visualize<br>words = [&#39;machine&#39;, &#39;learning&#39;, &#39;python&#39;, &#39;language&#39;, &#39;deep&#39;, &#39;fun&#39;, &#39;great&#39;, &#39;subset&#39;]<br># Get vectors for the words<br>word_vectors = [model.wv[word] for word in words]<br># Apply t-SNE to reduce dimensions to 2D for visualization<br>tsne = TSNE(n_components=2, random_state=42)<br>word_vecs_2d = tsne.fit_transform(word_vectors)<br># Plotting<br>plt.figure(figsize=(10, 6))<br>plt.scatter(word_vecs_2d[:, 0], word_vecs_2d[:, 1], marker=&#39;o&#39;)<br>for i, word in enumerate(words):<br>    plt.annotate(word, xy=(word_vecs_2d[i, 0], word_vecs_2d[i, 1]))<br>plt.title(&#39;2D Visualization of Word Embeddings using t-SNE&#39;)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/871/0*cqUtH-WaJ9z8Fp8-" /><figcaption>Relationship between Words</figcaption></figure><p><strong>Graph Explanation:</strong></p><ul><li><strong>Nodes:</strong> Represent words in the vocabulary(Unique words in corpus).</li><li><strong>Edges:</strong> Represent relationships between words based on their co-occurrences in the corpus.</li></ul><p><strong>Real-World Application:</strong></p><ul><li><strong>Search Engines:</strong> Improving query results by understanding the context of search terms.</li></ul><h3>2. GloVe (Global Vectors for Word Representation)</h3><p>GloVe is another widely-used method for word embeddings. Developed by Stanford, GloVe captures global statistical information by leveraging word co-occurrence matrices. Unlike Word2Vec, which focuses on local context, GloVe looks at the global context of words in a corpus.</p><pre>import numpy as np<br>import gensim.downloader as api</pre><pre># Load GloVe pre-trained model<br>glove_model = api.load(&quot;glove-wiki-gigaword-100&quot;)<br># Get word embedding<br>vector = glove_model[&#39;learning&#39;]<br>print(vector)</pre><p><strong>Diagrammatic Representation:</strong></p><p>You can also visualize GloVe embeddings similarly using t-SNE.</p><pre># Sample words to visualize<br>words = [&#39;king&#39;, &#39;queen&#39;, &#39;man&#39;, &#39;woman&#39;, &#39;apple&#39;, &#39;orange&#39;, &#39;fruit&#39;, &#39;computer&#39;]</pre><pre># Get vectors for the words<br>word_vectors = [glove_model[word] for word in words]<br># Apply t-SNE to reduce dimensions to 2D for visualization<br>tsne = TSNE(n_components=2, random_state=42)<br>word_vecs_2d = tsne.fit_transform(word_vectors)<br># Plotting<br>plt.figure(figsize=(10, 6))<br>plt.scatter(word_vecs_2d[:, 0], word_vecs_2d[:, 1], marker=&#39;o&#39;)<br>for i, word in enumerate(words):<br>    plt.annotate(word, xy=(word_vecs_2d[i, 0], word_vecs_2d[i, 1]))<br>plt.title(&#39;2D Visualization of GloVe Embeddings using t-SNE&#39;)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/847/0*Xodd3NKSGiGdjnhQ" /><figcaption>Glove Embeddings</figcaption></figure><p><strong>Graph Explanation:</strong></p><ul><li><strong>Co-occurrence Matrix:</strong> Shows how often each word appears with every other word in the corpus.</li><li><strong>Vector Representation:</strong> Derived from matrix factorization, capturing global word relationships.</li></ul><p><strong>Real-World Application:</strong></p><ul><li><strong>Recommendation Systems:</strong> Understanding user preferences by analyzing reviews or feedback.</li></ul><h3>3. FastText</h3><p>FastText, developed by Facebook, is an extension of Word2Vec that considers subword information, making it robust to rare words or misspellings. FastText represents words as bags of character n-grams, which means it can generate embeddings for out-of-vocabulary (OOV) words by using the n-grams that compose them.</p><pre>from gensim.models import FastText</pre><pre># Sample sentences<br>sentences = [[&quot;machine&quot;, &quot;learning&quot;, &quot;is&quot;, &quot;fun&quot;], <br>             [&quot;python&quot;, &quot;is&quot;, &quot;a&quot;, &quot;great&quot;, &quot;language&quot;], <br>             [&quot;deep&quot;, &quot;learning&quot;, &quot;is&quot;, &quot;a&quot;, &quot;subset&quot;, &quot;of&quot;, &quot;machine&quot;, &quot;learning&quot;]]<br># Train FastText model<br>model = FastText(sentences, vector_size=100, window=5, min_count=1)<br># Get word embedding<br>vector = model.wv[&#39;learning&#39;]<br>print(vector)</pre><p><strong>Diagrammatic Representation:</strong></p><p>To visualize FastText embeddings, you can again use t-SNE.</p><pre># Sample words to visualize<br>words = [&#39;machine&#39;, &#39;learning&#39;, &#39;python&#39;, &#39;language&#39;, &#39;deep&#39;, &#39;fun&#39;, &#39;great&#39;, &#39;subset&#39;]</pre><pre># Get vectors for the words<br>word_vectors = [model.wv[word] for word in words]<br># Apply t-SNE to reduce dimensions to 2D for visualization<br>tsne = TSNE(n_components=2, random_state=42)<br>word_vecs_2d = tsne.fit_transform(word_vectors)<br># Plotting<br>plt.figure(figsize=(10, 6))<br>plt.scatter(word_vecs_2d[:, 0], word_vecs_2d[:, 1], marker=&#39;o&#39;)<br>for i, word in enumerate(words):<br>    plt.annotate(word, xy=(word_vecs_2d[i, 0], word_vecs_2d[i, 1]))<br>plt.title(&#39;2D Visualization of FastText Embeddings using t-SNE&#39;)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/873/0*onK3B7APBcuUrpey" /><figcaption>FastText Embeddings</figcaption></figure><p><strong>Graph Explanation:</strong></p><ul><li><strong>Character n-grams:</strong> Represent subword components that form the basis for word embeddings.</li><li><strong>Word Vector Composition:</strong> Derived from the sum or average of n-gram vectors.</li></ul><p><strong>Real-World Application:</strong></p><ul><li><strong>Spell Checkers:</strong> Providing suggestions for misspelled words based on learned embeddings.</li></ul><h3>4. BERT (Bidirectional Encoder Representations from Transformers)</h3><p>BERT, developed by Google, takes word embeddings to the next level by providing context-aware embeddings. Unlike previous models, BERT understands the context of a word in a sentence by looking at both the left and right sides of the word.</p><pre>from transformers import BertTokenizer, BertModel<br>import torch</pre><pre># Load pre-trained BERT model and tokenizer<br>tokenizer = BertTokenizer.from_pretrained(&#39;bert-base-uncased&#39;)<br>model = BertModel.from_pretrained(&#39;bert-base-uncased&#39;)<br># Sample sentence<br>sentence = &quot;Machine learning is fascinating.&quot;<br># Tokenize and encode the sentence<br>input_ids = torch.tensor(tokenizer.encode(sentence)).unsqueeze(0)<br># Get BERT embeddings<br>with torch.no_grad():<br>    outputs = model(input_ids)<br>    last_hidden_states = outputs.last_hidden_state<br>print(last_hidden_states)</pre><p><strong>Diagrammatic Representation:</strong></p><p>To visualize BERT embeddings, we’ll use t-SNE after pooling the word embeddings.</p><pre>import numpy as np<br>from sklearn.manifold import TSNE<br>import matplotlib.pyplot as plt</pre><pre># Sample sentences<br>sentences = [&quot;Machine learning is fascinating.&quot;, <br>             &quot;Deep learning models require large amounts of data.&quot;,<br>             &quot;Natural language processing is a subfield of AI.&quot;]<br># Tokenize and encode sentences<br>tokens = [tokenizer.encode(sentence, add_special_tokens=True) for sentence in sentences]<br># Pad sequences to the same length<br>max_len = max([len(token) for token in tokens])<br>padded_tokens = [token + [0]*(max_len - len(token)) for token in tokens]<br># Convert to tensor<br>input_ids = torch.tensor(padded_tokens)<br># Get BERT embeddings<br>with torch.no_grad():<br>    outputs = model(input_ids)<br>    last_hidden_states = outputs.last_hidden_state<br># Pool the last hidden states<br>sentence_embeddings = torch.mean(last_hidden_states, dim=1).numpy()<br># Apply t-SNE to reduce dimensions to 2D for visualization<br>tsne = TSNE(n_components=2, random_state=42)<br>sentence_vecs_2d = tsne.fit_transform(sentence_embeddings)<br># Plotting<br>plt.figure(figsize=(10, 6))<br>plt.scatter(sentence_vecs_2d[:, 0], sentence_vecs_2d[:, 1], marker=&#39;o&#39;)<br>for i, sentence in enumerate(sentences):<br>    plt.annotate(f&quot;Sentence {i+1}&quot;, xy=(sentence_vecs_2d[i, 0], sentence_vecs_2d[i, 1]))<br>plt.title(&#39;2D Visualization of BERT Sentence Embeddings using t-SNE&#39;)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/887/0*fKVjMagAwMK5A5U1" /><figcaption>Bert Embeddings Representation</figcaption></figure><p><strong>Graph Explanation:</strong></p><ul><li><strong>Transformer Architecture:</strong> Processes input text by considering the context from all directions.</li><li><strong>Dynamic Embeddings:</strong> Captures context-dependent meanings for words.</li></ul><p><strong>Real-World Application:</strong></p><ul><li><strong>Chatbots:</strong> Understanding user intent by interpreting the context of words in sentences.</li></ul><h3>Conclusion</h3><p>Word embeddings have transformed NLP by providing a way to capture the semantic meaning of words in a form that machines can understand. From Word2Vec to BERT, each technique offers unique strengths and applications. Whether you’re working on search engines, recommendation systems, or chatbots, understanding and leveraging word embeddings can significantly enhance the performance of your models.</p><p>Feel free to experiment with the Python code provided in this article to create your own word embeddings and visualizations. Happy coding!</p><p><strong>Final Thoughts</strong></p><p>“The true power of language lies in its ability to connect words and meanings seamlessly. Word embeddings unlock this potential for machines, bringing us closer to truly intelligent systems.”</p><p>Stay Connected!!</p><p>Happy Learning!</p><p>Have a query: <a href="https://www.linkedin.com/in/aashish-singh-8b6ab6a0/"><strong>Follow on Linkedin</strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8a2ca21f1e46" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Demystifying Hypothesis Testing: A Guide for Data Enthusiasts]]></title>
            <link>https://medium.com/@aashish.singh2k8/demystifying-hypothesis-testing-a-guide-for-data-enthusiasts-b26d145f2e9f?source=rss-9a88942c15ca------2</link>
            <guid isPermaLink="false">https://medium.com/p/b26d145f2e9f</guid>
            <category><![CDATA[statistical-learning]]></category>
            <category><![CDATA[hypothesis-testing]]></category>
            <category><![CDATA[statistical-analysis]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[statistics]]></category>
            <dc:creator><![CDATA[Aashish Singh]]></dc:creator>
            <pubDate>Mon, 12 Aug 2024 10:19:29 GMT</pubDate>
            <atom:updated>2024-08-12T10:19:29.006Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/897/1*mHL-oTsT7tz-i-2RvdQzGQ.png" /><figcaption>Source: Google</figcaption></figure><p>Hypothesis testing is a fundamental aspect of statistical analysis that helps us make data-driven decisions. Whether you’re analyzing business performance, evaluating the effectiveness of a new marketing strategy, or understanding scientific phenomena, hypothesis testing provides a structured approach to draw conclusions from data.</p><p>In this article, we’ll explore the basics of hypothesis testing, dive into the different types of tests, and walk through real-world examples with Python code. Let’s make this journey interactive and fun!</p><h3>“In God we trust. All others must bring data.” — W. Edwards Deming</h3><h3>What is Hypothesis Testing?</h3><p>Hypothesis testing is a method used to decide whether there is enough evidence in a sample of data to infer that a certain condition is true for the entire population.</p><h3>Key Concepts:</h3><ol><li><strong>Null Hypothesis (H₀):</strong> The statement being tested, usually a statement of no effect or no difference.</li><li><strong>Alternative Hypothesis (H₁):</strong> The statement we want to test for, often suggesting some effect or difference.</li><li><strong>p-value:</strong> The probability of obtaining results at least as extreme as the observed results, under the assumption that the null hypothesis is true.</li><li><strong>Significance Level (α):</strong> The threshold for rejecting the null hypothesis, commonly set at 0.05.</li><li><strong>Type I Error:</strong> Rejecting the null hypothesis when it is actually true.</li><li><strong>Type II Error:</strong> Failing to reject the null hypothesis when it is actually false.</li></ol><h3>Types of Hypothesis Tests</h3><h3>1. Z-Test</h3><p>The Z-Test is used to determine if there is a significant difference between the means of two groups when the population variance is known and the sample size is large (n &gt; 30).</p><p><strong>When to use:</strong></p><ul><li>Comparing the means of two independent samples.</li><li>The population variance is known.</li></ul><pre>import numpy as np<br>import matplotlib.pyplot as plt<br>from scipy import stats</pre><pre># Sample data<br>sample1 = np.array([2.5, 3.1, 2.8, 3.6, 3.0])<br>sample2 = np.array([3.5, 2.9, 3.2, 3.8, 3.3])</pre><pre># Known population standard deviation<br>pop_std = 0.5</pre><pre># Z-test calculation<br>z_stat, p_val = stats.ttest_ind(sample1, sample2)</pre><pre># Plotting the distributions<br>plt.figure(figsize=(10, 6))<br>x = np.linspace(-4, 4, 1000)<br>y1 = stats.norm.pdf(x, np.mean(sample1), pop_std)<br>y2 = stats.norm.pdf(x, np.mean(sample2), pop_std)</pre><pre>plt.plot(x, y1, label=&#39;Sample 1 Distribution&#39;, color=&#39;blue&#39;)<br>plt.plot(x, y2, label=&#39;Sample 2 Distribution&#39;, color=&#39;orange&#39;)<br>plt.axvline(np.mean(sample1), color=&#39;blue&#39;, linestyle=&#39;dashed&#39;)<br>plt.axvline(np.mean(sample2), color=&#39;orange&#39;, linestyle=&#39;dashed&#39;)<br>plt.title(&#39;Z-Test: Comparing Two Sample Means&#39;)<br>plt.xlabel(&#39;Value&#39;)<br>plt.ylabel(&#39;Density&#39;)<br>plt.legend()<br>plt.grid(True)<br>plt.show()</pre><pre>print(f&quot;Z-Statistic: {z_stat}, P-Value: {p_val}&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/886/0*QAnp_NXKuAY9IuqF" /></figure><pre>Z-Statistic: -1.4419211804559553, P-Value: 0.18730219201099</pre><p><strong>Graph Explanation:</strong></p><ul><li><strong>Blue Curve:</strong> Represents the probability distribution of Sample 1.</li><li><strong>Orange Curve:</strong> Represents the probability distribution of Sample 2.</li><li><strong>Dashed Lines:</strong> Indicate the mean of each sample.</li></ul><p>The graph helps visualize the difference between the means of the two samples. If the distributions overlap significantly, it suggests that the means might not be significantly different. The Z-Test and the resulting p-value help quantify this difference.</p><p><strong>Real-World Application:</strong></p><ul><li><strong>Marketing:</strong> Determining if a new advertisement campaign leads to a significant increase in sales compared to the previous campaign.</li></ul><h3>2. T-Test</h3><p>The T-Test is similar to the Z-Test but is used when the population variance is unknown and the sample size is small (n &lt; 30).</p><p><strong>Types of T-Tests:</strong></p><ul><li><strong>One-Sample T-Test:</strong> Compare the sample mean to a known value.</li><li><strong>Independent T-Test:</strong> Compare the means of two independent samples.</li><li><strong>Paired T-Test:</strong> Compare means from the same group at different times (e.g., before and after treatment).</li></ul><pre>import numpy as np<br>import matplotlib.pyplot as plt<br>from scipy import stats</pre><pre># Sample data<br>sample1 = np.array([2.9, 3.1, 2.8, 3.6, 3.0])<br>sample2 = np.array([3.7, 2.8, 3.2, 3.9, 3.1])</pre><pre># T-test calculation<br>t_stat, p_val = stats.ttest_ind(sample1, sample2)</pre><pre># Plotting the distributions<br>plt.figure(figsize=(10, 6))<br>x = np.linspace(-4, 4, 1000)<br>y1 = stats.norm.pdf(x, np.mean(sample1), np.std(sample1))<br>y2 = stats.norm.pdf(x, np.mean(sample2), np.std(sample2))</pre><pre>plt.plot(x, y1, label=&#39;Sample 1 Distribution&#39;, color=&#39;blue&#39;)<br>plt.plot(x, y2, label=&#39;Sample 2 Distribution&#39;, color=&#39;orange&#39;)<br>plt.axvline(np.mean(sample1), color=&#39;blue&#39;, linestyle=&#39;dashed&#39;)<br>plt.axvline(np.mean(sample2), color=&#39;orange&#39;, linestyle=&#39;dashed&#39;)<br>plt.title(&#39;T-Test: Comparing Two Sample Means&#39;)<br>plt.xlabel(&#39;Value&#39;)<br>plt.ylabel(&#39;Density&#39;)<br>plt.legend()<br>plt.grid(True)<br>plt.show()</pre><pre>print(f&quot;T-Statistic: {t_stat}, P-Value: {p_val}&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/855/0*6soBwjxZfIueX0MK" /></figure><pre>T-Statistic: -1.061445555206043, P-Value: 0.3194774919862749</pre><p><strong>Graph Explanation:</strong></p><ul><li><strong>Blue Curve:</strong> Represents the distribution of Sample 1.</li><li><strong>Orange Curve:</strong> Represents the distribution of Sample 2.</li><li><strong>Dashed Lines:</strong> Show the mean of each sample.</li></ul><p>The T-Test graph shows how the distributions overlap or differ, giving a visual representation of the difference between the sample means. The T-statistic and p-value further help in deciding whether the means are statistically different.</p><p><strong>Real-World Application:</strong></p><ul><li><strong>Healthcare:</strong> Assessing the effectiveness of a new drug compared to a placebo.</li></ul><h3>3. Chi-Square Test</h3><p>The Chi-Square Test is used to determine if there is a significant association between categorical variables.</p><p><strong>Types of Chi-Square Tests:</strong></p><ul><li><strong>Chi-Square Goodness of Fit Test:</strong> Determine if a sample matches the population.</li><li><strong>Chi-Square Test of Independence:</strong> Determine if there is a relationship between two categorical variables.</li></ul><pre>import numpy as np<br>import matplotlib.pyplot as plt<br>from scipy.stats import chi2_contingency</pre><pre># Contingency table<br>observed = np.array([[30, 10], [20, 40]])</pre><pre># Chi-square test calculation<br>chi2_stat, p_val, dof, expected = chi2_contingency(observed)</pre><pre># Plotting the contingency table<br>labels = [&#39;Group 1&#39;, &#39;Group 2&#39;]<br>categories = [&#39;Category 1&#39;, &#39;Category 2&#39;]</pre><pre>fig, ax = plt.subplots()<br>im = ax.imshow(observed, cmap=&quot;Blues&quot;)</pre><pre>ax.set_xticks(np.arange(len(labels)))<br>ax.set_yticks(np.arange(len(categories)))<br>ax.set_xticklabels(labels)<br>ax.set_yticklabels(categories)</pre><pre># Rotate the tick labels and set their alignment.<br>plt.setp(ax.get_xticklabels(), rotation=45, ha=&quot;right&quot;,<br>         rotation_mode=&quot;anchor&quot;)</pre><pre># Loop over data dimensions and create text annotations.<br>for i in range(len(categories)):<br>    for j in range(len(labels)):<br>        text = ax.text(j, i, observed[i, j],<br>                       ha=&quot;center&quot;, va=&quot;center&quot;, color=&quot;black&quot;)</pre><pre>ax.set_title(&quot;Chi-Square Test: Observed Frequencies&quot;)<br>fig.tight_layout()<br>plt.show()</pre><pre>print(f&quot;Chi-Square Statistic: {chi2_stat}, P-Value: {p_val}&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/450/0*sByg_4bKPCATOSDA" /></figure><pre>Chi-Square Statistic: 15.041666666666666, P-Value: 0.00010516355403363098</pre><p><strong>Graph Explanation:</strong></p><ul><li><strong>Heatmap:</strong> Shows the observed frequencies in each category.</li><li><strong>Text Annotations:</strong> Display the actual frequency values within the cells of the heatmap.</li></ul><p>The heatmap visualizes the relationship between two categorical variables, allowing you to see if certain combinations occur more frequently. The Chi-Square test then determines if the observed frequencies differ significantly from what would be expected if the variables were independent.</p><p><strong>Real-World Application:</strong></p><ul><li><strong>Retail:</strong> Analyzing the relationship between customer age groups and their purchasing preferences.</li></ul><h3>4. ANOVA (Analysis of Variance)</h3><p>ANOVA is used when comparing the means of three or more groups to see if at least one mean is different.</p><p><strong>When to use:</strong></p><ul><li>When comparing the means of multiple groups.</li><li>For determining the effect of one or more independent variables on a dependent variable.</li></ul><pre>import numpy as np<br>import matplotlib.pyplot as plt<br>from scipy import stats</pre><pre># Sample data<br>group1 = [22, 23, 24, 25, 24]<br>group2 = [30, 31, 29, 33, 32]<br>group3 = [27, 26, 28, 27, 29]</pre><pre># ANOVA calculation<br>f_stat, p_val = stats.f_oneway(group1, group2, group3)</pre><pre># Boxplot visualization<br>plt.figure(figsize=(10, 6))<br>data = [group1, group2, group3]<br>plt.boxplot(data, patch_artist=True, labels=[&#39;Group 1&#39;, &#39;Group 2&#39;, &#39;Group 3&#39;])<br>plt.title(&#39;ANOVA: Comparison of Three Groups&#39;)<br>plt.ylabel(&#39;Values&#39;)<br>plt.grid(True)<br>plt.show()</pre><pre>print(f&quot;F-Statistic: {f_stat}, P-Value: {p_val}&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/845/0*CqNym8_DRtXTKVtb" /></figure><pre>F-Statistic: 40.2745098039215, P-Value: 4.751782361009403e-06</pre><p><strong>Graph Explanation:</strong></p><ul><li><strong>Boxplots:</strong> Show the distribution of values in each group.</li><li><strong>Whiskers:</strong> Indicate the range within 1.5 times the interquartile range (IQR) from the first and third quartile.</li><li><strong>Boxes:</strong> Represent the IQR, with the line inside the box showing the median.</li></ul><p>The boxplot visualization in ANOVA helps you compare the distribution of multiple groups simultaneously. ANOVA tests whether the differences between these groups’ means are statistically significant.</p><p><strong>Real-World Application:</strong></p><ul><li><strong>Education:</strong> Comparing the test scores of students taught with three different teaching methods to determine which is most effective.</li></ul><h3>When Data Talks, We Listen</h3><p>Hypothesis testing is a powerful tool that allows us to make informed decisions based on data. Whether you’re in marketing, healthcare, retail, or education, understanding and applying these tests can significantly enhance your ability to draw meaningful insights.</p><h3>Visualize the Results</h3><p>Data visualization is an essential part of hypothesis testing. For instance, plotting the distribution of sample data or the results of a test can help in interpreting the outcomes. Use libraries like matplotlib, seaborn, or plotly to create engaging visuals.</p><h3>Final Thoughts</h3><p>Mastering hypothesis testing is not just about crunching numbers — it’s about developing a data-driven mindset. Whether you’re testing a new hypothesis in the lab, making a business decision, or trying to predict market trends, these tools will help you navigate the world of uncertainty.</p><h3>“Without data, you’re just another person with an opinion.” — W. Edwards Deming</h3><p>By applying these techniques and integrating them into your workflows, you can ensure that your decisions are grounded in solid evidence, making you not just a thinker, but a true data scientist.</p><h3>Stay Curious and Keep Testing!</h3><p>Hypothesis testing is an evolving field, and the more you practice, the more intuitive it becomes. Use the code examples provided as a starting point, and don’t hesitate to explore further.</p><p>Happy testing!</p><p>Stay Tune for More!!</p><p>Have queries: Reach out on <a href="https://www.linkedin.com/in/aashish-singh-8b6ab6a0/"><strong>Linkedin</strong></a><strong>!!</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b26d145f2e9f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Unlocking the Power of Confidence Intervals: Why They Matter and How to Use Them]]></title>
            <link>https://medium.com/@aashish.singh2k8/unlocking-the-power-of-confidence-intervals-why-they-matter-and-how-to-use-them-c0ad48891e1a?source=rss-9a88942c15ca------2</link>
            <guid isPermaLink="false">https://medium.com/p/c0ad48891e1a</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[visualization]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Aashish Singh]]></dc:creator>
            <pubDate>Sun, 11 Aug 2024 05:28:01 GMT</pubDate>
            <atom:updated>2024-08-11T05:29:22.834Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/472/1*wkywMJ-ydLwILrlJbePfyg.png" /><figcaption>Source: Google</figcaption></figure><p>In the world of data science and statistics, confidence intervals (CIs) are like a compass, guiding us through the uncertainty of data. They provide a range of plausible values for an unknown parameter, offering insights into the reliability of our estimates. In this article, we’ll explore what confidence intervals are, why they matter, and how you can apply them using both parametric and non-parametric techniques, all while keeping things interactive and engaging.</p><h3>What Exactly is a Confidence Interval?</h3><p><strong>“A confidence interval doesn’t just tell us where we are; it tells us how far we might be from where we want to be.</strong>”</p><p>Imagine you’re trying to estimate the average height of all adults in your city. You can’t measure everyone, so you take a sample. But how accurate is your estimate? A confidence interval gives you a range of values that likely includes the true population average. For example, if you calculate a 95% confidence interval of 165 cm to 175 cm, you can be 95% confident that the true average height lies within this range.</p><p>We’ll start with a bell curve that represents a normal distribution. We’ll highlight a 95% confidence interval on the curve.</p><p><strong>Visualization: Confidence Interval on a Bell Curve</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/867/1*L91grMPmiQEAN9bVMzRSzw.png" /><figcaption>Bell Curve Highlighting Confidence Interval</figcaption></figure><p><em>Explanation: The highlighted region on the bell curve shows the 95% confidence interval, illustrating how much of the data falls within this range.</em></p><h3>Why Should You Care About Confidence Intervals?</h3><p><strong>“In statistics, a confidence interval is the difference between guessing and knowing.</strong>”</p><p>Confidence intervals are crucial because they quantify uncertainty. They don’t just give you a single estimate; they give you a range, which is more informative. Here are a few reasons why CIs are important:</p><ul><li><strong>Business Decisions</strong>: Confidence intervals help businesses make data-driven decisions, like estimating the average customer satisfaction score and understanding the margin of error.</li><li><strong>Medical Research</strong>: In clinical trials, confidence intervals show the effectiveness of new drugs, providing a range of possible outcomes.</li><li><strong>Quality Control</strong>: In manufacturing, confidence intervals help ensure product reliability by predicting the lifespan of products.</li></ul><p><strong>Visualization: Real-World Application of Confidence Intervals</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/866/1*wTVJZuiVaXCrt6BWc1rfgw.png" /><figcaption>Real World Applications of Confidence Interval</figcaption></figure><p><em>Explanation: This chart demonstrates how confidence intervals might be used in finance, healthcare, and manufacturing.</em></p><h3>Interactive Exploration: Calculating Confidence Intervals</h3><p><strong>“Data may speak for itself, but confidence intervals give it the language to convey its uncertainty.</strong>”</p><p>Let’s dive into some hands-on examples using Python to calculate confidence intervals. We’ll start with a parametric approach and then explore a non-parametric method using bootstrap aggregation.</p><h3>Parametric Confidence Intervals</h3><p>Parametric techniques assume that your data follows a specific distribution, usually a normal distribution. Let’s calculate a 95% confidence interval for the mean of a sample.</p><pre>import numpy as np<br>import scipy.stats as stats<br><br># Sample data<br>data = np.array([12, 15, 14, 10, 18, 21, 13, 17, 19, 11])<br><br># Sample mean and standard error<br>mean = np.mean(data)<br>se = stats.sem(data)<br><br># Confidence interval<br>confidence = 0.95<br>ci = stats.t.interval(confidence, len(data)-1, loc=mean, scale=se)<br><br>print(&quot;95% Confidence Interval:&quot;, ci)</pre><p>In this code, we calculate the 95% confidence interval for the mean of our sample. The result gives us a range where we can be confident the true mean lies.</p><pre>95% Confidence Interval: (12.387885906284497, 17.612114093715505)</pre><p>Here’s a visual representation of the sample data along with the 95% confidence interval for the mean. The blue dot represents the sample mean, and the error bar shows the range of the confidence interval. The purple dots are the individual data points from the sample. This plot illustrates how the confidence interval gives us a range within which we expect the true population mean to fall.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/890/1*FpRKyjNFD8cSvxgp9XQDmQ.png" /><figcaption>95% CI for Mean</figcaption></figure><h3>Non-Parametric Confidence Intervals: Bootstrap Aggregation</h3><p><strong>“Bootstrap gives data a voice when the assumptions are silent.</strong>”</p><p>What if your data doesn’t follow a normal distribution? That’s where non-parametric methods like bootstrap aggregation come in. Bootstrap involves resampling your data with replacement to create many new samples. You then calculate the statistic (like the median) for each sample and determine the confidence interval from the distribution of these statistics.</p><p>Here’s how you can compute a bootstrap confidence interval:</p><pre>import numpy as np<br><br># Sample data<br>data = np.array([12, 15, 14, 10, 18, 21, 13, 17, 19, 11])<br><br># Bootstrap<br>n_iterations = 1000<br>n_size = len(data)<br>medians = []<br><br>for i in range(n_iterations):<br>    sample = np.random.choice(data, size=n_size, replace=True)<br>    medians.append(np.median(sample))<br><br># Confidence interval<br>alpha = 0.95 <br>lower = np.percentile(medians, (1 - alpha) / 2 * 100)<br>upper = np.percentile(medians, (alpha + (1 - alpha) / 2) * 100)<br><br>print(f&quot;{int(alpha * 100)}% Confidence Interval:&quot;, (lower, upper))</pre><pre>95% Confidence Interval: (12.0, 18.0)</pre><p>This method doesn’t rely on the assumption of normality, making it more flexible and robust for different types of data.</p><h3>Visualizing Bootstrap Aggregation</h3><p>To make things more interactive, let’s visualize the bootstrap process and see how the confidence interval is formed:</p><p><strong>“Understanding variability is not just about seeing the data; it’s about seeing the possibilities.</strong>”</p><p><strong>Visualization: Bootstrap Aggregation</strong></p><pre>import matplotlib.pyplot as plt<br><br>plt.hist(medians, bins=30, edgecolor=&#39;k&#39;, alpha=0.7, color=&#39;purple&#39;)<br>plt.axvline(lower, color=&#39;red&#39;, linestyle=&#39;dashed&#39;, linewidth=2, label=f&#39;Lower Bound: {lower:.2f}&#39;)<br>plt.axvline(upper, color=&#39;red&#39;, linestyle=&#39;dashed&#39;, linewidth=2, label=f&#39;Upper Bound: {upper:.2f}&#39;)<br>plt.title(&#39;Bootstrap Aggregation - Median Distribution&#39;)<br>plt.xlabel(&#39;Median Value&#39;)<br>plt.ylabel(&#39;Frequency&#39;)<br>plt.legend()<br>plt.grid(True)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/587/1*SqsFY46V8gnu9tJ8fWYFyQ.png" /><figcaption>Histogram</figcaption></figure><p><em>Explanation: This histogram shows the distribution of medians from the bootstrap samples, with red dashed lines marking the confidence interval.</em></p><h3>Conclusion: Why Confidence Intervals Matter</h3><p><strong>“A confidence interval is not a boundary; it’s a bridge between knowledge and uncertainty.</strong>”</p><p>Confidence intervals are not just numbers; they are powerful tools for making informed decisions. Whether you’re analyzing financial data, conducting clinical trials, or ensuring product quality, confidence intervals provide a clearer picture of uncertainty and help you make better decisions.</p><p>By exploring both parametric and non-parametric methods, we’ve seen how versatile confidence intervals can be, and how they can be applied in various real-world scenarios.</p><p>Stay Tuned for More!!</p><p>Feel free to reach out: <a href="https://www.linkedin.com/in/aashish-singh-8b6ab6a0/">Follow on linkedin</a></p><p>Happy Learning!!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c0ad48891e1a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding Transformer Architecture: The Backbone of Modern AI]]></title>
            <link>https://medium.com/@aashish.singh2k8/transformers-have-revolutionized-the-field-of-natural-language-processing-nlp-and-beyond-589fe8bb2f49?source=rss-9a88942c15ca------2</link>
            <guid isPermaLink="false">https://medium.com/p/589fe8bb2f49</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[generative-ai-tools]]></category>
            <category><![CDATA[transformers]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Aashish Singh]]></dc:creator>
            <pubDate>Sat, 10 Aug 2024 08:54:01 GMT</pubDate>
            <atom:updated>2024-08-10T08:56:42.998Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/493/1*-YRC5dxSQrbnM3dS9F9o7w.png" /><figcaption>Source: Google</figcaption></figure><p>Transformers have revolutionized the field of natural language processing (NLP) and beyond. They power state-of-the-art models like GPT-4, BERT, and T5, enabling impressive feats in language understanding and generation. Let’s dive into how transformers work, from the fundamental architecture to the mathematical principles that drive them.</p><h3>1. Introduction to Transformers</h3><p>Transformers were introduced by Vaswani et al. in their 2017 paper, <em>“Attention is All You Need.”</em> Unlike RNNs or LSTMs, which process sequential data step by step, transformers handle entire sequences simultaneously, enabling parallelization and significantly speeding up training.</p><p>If you are interested in exploring the research paper, here is the link: <a href="https://arxiv.org/abs/1706.03762">Attention is all you need</a></p><p>To put it simply:</p><p><em>“In the world of AI, transformers are the key to unlocking the future.”</em></p><h3>2. The Shift from LSTM/RNN to Transformer: Why It Was Necessary</h3><p>In the evolution of natural language processing (NLP), Long Short-Term Memory (LSTM) and Recurrent Neural Networks (RNN) played crucial roles by enabling models to capture temporal dependencies in sequential data. However, as the complexity of tasks increased, these models faced limitations, particularly in handling long-range dependencies and parallelization.</p><p>Key Challenges with LSTM/RNN:</p><ul><li><strong>Sequential Processing:</strong> LSTM and RNN process data sequentially, making them slow for long sequences and difficult to parallelize. This hampers scalability, especially with large datasets.</li><li><strong>Vanishing/Exploding Gradients:</strong> As sequences get longer, these models struggle with vanishing or exploding gradients, leading to difficulties in training deep networks and losing important context over long distances.</li><li><strong>Limited Long-Range Dependency Capture:</strong> While LSTM and RNN can theoretically capture long-term dependencies, in practice, they struggle with effectively understanding relationships between distant words in a sequence.</li></ul><h3>The Emergence of Transformers:</h3><p>Transformers introduced a paradigm shift in NLP with the concept of self-attention, allowing models to attend to all parts of a sequence simultaneously. This shift addressed several limitations of LSTM/RNN models:</p><ul><li><strong>Parallelization:</strong> Unlike LSTM/RNN, transformers process sequences in parallel, greatly increasing computational efficiency and making them suitable for training on large-scale datasets.</li><li><strong>Effective Long-Range Dependencies:</strong> The self-attention mechanism enables transformers to directly model relationships between all words in a sequence, regardless of distance, leading to better contextual understanding.</li><li><strong>Scalability:</strong> Transformers can scale up efficiently, handling larger models and datasets without the training issues faced by LSTM/RNNs.</li></ul><h3>The Transformer Architecture</h3><h3>Overview</h3><p>Transformers, initially designed for sequence transduction tasks like neural machine translation, have become foundational in modern AI. These models excel at converting input sequences into output sequences, relying entirely on self-attention mechanisms rather than sequence-aligned RNNs or convolutional networks. A key feature of the transformer architecture is its encoder-decoder structure.</p><p>For instance, when used in language translation, a transformer takes a sentence in one language, such as English, and outputs its translation in another language, like French, maintaining a sophisticated understanding of context and semantics throughout the process.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*1UmJJp_qlWp5lDXD" /></figure><p>Source: Google</p><p>When we delve into the transformer architecture, we find that it consists of two primary components:</p><ol><li><strong>The Encoder:</strong> This part processes the input and converts it into a matrix representation. For example, it takes the English sentence “How are you?” and encodes it into a structured format that captures its meaning.</li><li><strong>The Decoder:</strong> The decoder then takes this encoded representation and iteratively generates the output. In our translation example, it produces the corresponding sentence in the target language, such as “¿Cómo estás?” in Spanish.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/647/0*y1udC9qBowM7Gcrt" /></figure><p>Source: Google</p><p>Both the encoder and decoder in the transformer architecture are composed of multiple layers stacked on top of each other, with each encoder and decoder layer sharing the same internal structure. The input data passes sequentially through each encoder layer before moving on to the decoders. Similarly, each decoder layer processes the output of the preceding layer.</p><p>The original transformer model consisted of 6 layers each for the encoder and decoder, but this can be expanded to any number NNN of layers, depending on the complexity required.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*wOQhscdPDwqjjjXB" /></figure><p>Source: Google</p><p>So now that we have a generic idea of the overall Transformer architecture, let’s focus on both Encoders and Decoders to understand better their working flow:</p><h3>The Encoder WorkFlow</h3><p>The encoder is a critical component of the Transformer architecture, designed to convert input tokens into contextualized representations. Unlike traditional models that process tokens in isolation, the Transformer encoder captures the context of each token relative to the entire sequence, allowing for a deeper understanding of the input data.</p><p>The encoder is composed of multiple identical layers, each with two main sub-layers: a multi-head self-attention mechanism and a position-wise fully connected feed-forward network. Additionally, each sub-layer includes residual connections and layer normalization to enhance stability and performance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*HB3XpEQbN6X8P0rI" /></figure><p>Source: Google</p><p>So let’s break its workflow into its most basic steps:</p><h3>Step 1: Input Embeddings</h3><p>The process begins in the bottom-most encoder, where input tokens — words or subwords — are transformed into numerical vectors using embedding layers. These embeddings capture the semantic meaning of the tokens and convert them into fixed-sized vectors, typically of size 512.</p><p>Each encoder receives a list of these vectors. In the initial encoder, these vectors are the word embeddings, while in subsequent encoders, they are the output from the encoder layer directly beneath them. This hierarchical structure allows for the progressive refinement of token representations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Bd33Xv4E-qzZP5P1" /></figure><p>Source: Google</p><h3>Step 2: Positional Encoding</h3><p>Transformers, unlike RNNs, lack a built-in mechanism to capture the order of tokens. To address this, positional encodings are added to the input embeddings, providing information about each token’s position within a sequence.</p><p>Researchers introduced a method using sine and cosine functions to generate positional vectors that can represent sequences of any length. Each dimension of the positional encoding corresponds to a unique frequency, with values ranging from -1 to 1, effectively encoding the position of each token in the sequence.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*vP6odVMjL509vHGf" /></figure><p>Source: Google</p><h3>Step 3: Stack of Encoder Layers</h3><p>The Transformer encoder is composed of a stack of identical layers, typically six in the original model. Each encoder layer plays a crucial role in transforming input sequences into continuous, abstract representations that encapsulate information from the entire sequence.</p><p>Each layer consists of two key sub-modules:</p><ol><li><strong>Multi-Headed Attention Mechanism</strong>: Captures different aspects of the input by focusing on various parts of the sequence simultaneously.</li><li><strong>Fully Connected Feed-Forward Network</strong>: Further processes the output of the attention mechanism, refining the representation.</li></ol><p>To enhance stability and performance, residual connections are applied around each sublayer, followed by layer normalization. This ensures that the information flows smoothly through the network while maintaining the integrity of the data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*-6jtDvyCKF5NyYxE" /></figure><p>Source: Google</p><h3>3.1 Multi-Headed Self-Attention Mechanism</h3><p>In the encoder architecture, the multi-headed attention mechanism employs a specialized form of attention known as self-attention. This mechanism allows the model to capture dependencies between words in an input sequence, effectively enabling it to relate each word to others. For instance, the model might learn to associate the word “are” with “you” within a given sentence.</p><p>Self-attention empowers the encoder to dynamically focus on different parts of the input sequence as it processes each token. This is achieved through the computation of attention scores, which are derived from three primary components:</p><ul><li><strong>Query:</strong> A vector that represents a specific word or token from the input sequence within the attention mechanism.</li><li><strong>Key:</strong> A vector associated with each word or token in the input sequence, used to determine the relevance of a particular word to the query.</li><li><strong>Value:</strong> Each value is linked to a key and is utilized to construct the output of the attention layer. When a query and a key have a high attention score, indicating a strong match, the corresponding value is given greater emphasis in the output.</li></ul><p>The self-attention mechanism allows the model to capture contextual information from the entire sequence, enabling it to better understand relationships between words. Rather than applying a single attention function, the queries, keys, and values are linearly projected multiple times, corresponding to the number of heads (h) in the mechanism. The attention function is then executed in parallel on these h projected versions, resulting in h-dimensional output values.</p><p>This multi-headed approach allows the model to attend to different aspects of the input sequence simultaneously, thereby enhancing its ability to capture intricate relationships and dependencies across the entire sequence.</p><p>The detailed architecture goes as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*zZCSoCz9FLeZWpmC" /></figure><p>Source: Google</p><h3>Matrix Multiplication (MatMul) — Dot Product of Query and Key</h3><p>Once the query, key, and value vectors are passed through a linear layer, a dot product matrix multiplication is performed between the queries and keys, resulting in the creation of a score matrix.</p><p>The score matrix establishes the degree of emphasis each word should place on other words. Therefore, each word is assigned a score in relation to other words within the same time step. A higher score indicates greater focus.</p><p>This process effectively maps the queries to their corresponding keys.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*T8tXikaUnZWBRfth" /></figure><p>Source: Google</p><h3>Reducing the Magnitude of attention scores</h3><p>The scores are then scaled down by dividing them by the square root of the dimension of the query and key vectors. This step is implemented to ensure more stable gradients, as the multiplication of values can lead to excessively large effects.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*NIS3fxp8TJe3gxAQ" /></figure><p>Source: Google</p><h3>Applying Softmax to the Adjusted Scores</h3><p>Subsequently, a softmax function is applied to the computed attention scores to derive the attention weights. This operation transforms the scores into probability values that range between 0 and 1. The softmax function accentuates higher scores while suppressing lower ones, thereby refining the model’s ability to prioritize words that should receive greater attention. This process ensures that the most relevant words are given more focus during the encoding process.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*FG8-6yKV4Vw9Cb-z" /></figure><p>Source: Google</p><p><em>.</em></p><h3>Combining Softmax Results with the Value Vector</h3><p>The next step in the attention mechanism involves combining the attention weights, derived from the softmax function, with the value vector. Specifically, the attention weights are multiplied by the corresponding value vector, resulting in an output vector that emphasizes the most relevant words based on their attention scores.</p><p>In this process, only the words with high softmax scores significantly influence the output, effectively filtering out less relevant information. The resulting output vector is then passed through a linear layer for further processing, allowing the model to refine its understanding of the input sequence.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Zjyz8FdqfPYkL_LY" /></figure><p>Source: Google</p><h3>Final Output of the Attention Mechanism</h3><p>The culmination of the attention mechanism yields the final output vector. At this point, you might wonder why it is termed “Multi-Head Attention.”</p><p>Before the attention process begins, the queries, keys, and values are divided into multiple subsets, denoted as h heads. The self-attention mechanism is then applied independently within each of these smaller subsets or ‘heads,’ allowing each head to generate its own output vector.</p><p>These individual output vectors are subsequently combined and passed through a final linear layer, which acts as a filter to fine-tune their collective output. The strength of this approach lies in the diversity of learning that occurs across the different heads, enabling the encoder model to develop a more comprehensive and nuanced understanding of the input sequence.</p><h3>3.2 Normalization and Residual Connections</h3><p>In the encoder architecture, each sub-layer is followed by a normalization step. Additionally, a residual connection is employed, where the output of each sub-layer is added to its input. This technique helps mitigate the vanishing gradient problem, facilitating the training of deeper models by ensuring that important information is preserved as it passes through the network.</p><p>This process is also applied after the Feed-Forward Neural Network, ensuring consistency in the model’s learning process and enhancing its overall stability and performance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*vTtoKkOjXXnVYsx0" /></figure><p>Source: Google</p><h3>STEP 3.3 Feed-Forward Neural Network</h3><p>The normalized residual output then progresses through a pointwise feed-forward network, a vital phase for further refinement. This network can be envisioned as two linear layers connected by a ReLU activation function, which serves as an intermediary bridge.</p><p>Once the output is processed by the feed-forward network, it undergoes a residual connection, merging with the original input of the network. This integration is followed by another normalization step, ensuring that the output is well-adjusted and harmonized, preparing it for the subsequent stages of the model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*MwlcLhk5rch6akjq" /></figure><p>Source: Google</p><h3>4. Output of the Encoder</h3><p>The output of the final encoder layer consists of a series of vectors, each providing a rich contextual representation of the input sequence. These vectors are then passed as input to the decoder in a Transformer model, where they play a crucial role in guiding the decoding process.</p><p>This meticulous encoding process ensures that the decoder can effectively focus on the relevant parts of the input sequence during translation or generation tasks. The encoder’s layered structure, which can consist of multiple stacked layers, allows each layer to explore and learn different aspects of attention. This layered approach not only enhances the model’s understanding but also significantly improves the predictive capabilities of the Transformer network.</p><h3>The Decoder Workflow</h3><p>The decoder is tasked with generating text sequences. Much like the encoder, the decoder is composed of a series of sub-layers. It includes two multi-headed attention layers, a pointwise feed-forward layer, and utilizes residual connections and layer normalization after each sub-layer. These elements work in concert to refine the decoder’s ability to generate coherent and contextually accurate text sequences.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/701/0*2n08AdBZmLdVSXh1" /></figure><p>Source: Google</p><p>These components operate similarly to the layers of the encoder, but with a distinct purpose: each multi-headed attention layer in the decoder is designed for a specific task. The final stage of the decoder involves a linear layer that acts as a classifier, followed by a softmax function to compute the probabilities of different possible words.</p><p>The Transformer decoder is architected to generate output by systematically decoding the encoded information. It functions in an autoregressive manner, beginning with a start token and using previously generated outputs as inputs, along with the encoder’s outputs, which are enriched with attention information from the original input.</p><p>This sequential decoding process continues until the decoder generates a token that signifies the end of the output sequence, completing the generation task.</p><h3>1. Output Embeddings</h3><p>At the outset of the decoder’s process, the workflow mirrors that of the encoder. The input sequence first passes through an embedding layer, converting the input tokens into dense vectors that capture semantic information.</p><h3>2. Positional Encoding</h3><p>Following the embedding layer, the input is passed through a positional encoding layer, just as in the encoder. This step introduces positional embeddings to the sequence, enabling the model to capture the order of words, which is crucial for understanding context.</p><p>These positional embeddings are then fed into the first multi-head attention layer of the decoder, where the attention scores specific to the decoder’s input are carefully calculated.</p><h3>3. Stack of Decoder Layers</h3><p>The decoder is composed of a stack of identical layers — six layers in the original Transformer model. Each layer comprises three key sub-components:</p><h3>3.1 Masked Self-Attention Mechanism</h3><p>The masked self-attention mechanism in the decoder functions similarly to the self-attention mechanism in the encoder, with one critical difference: it prevents any position in the sequence from attending to subsequent positions. This ensures that each word is only influenced by the words that have come before it in the sequence, not by any future tokens.</p><p>For example, when computing the attention scores for the word “are,” the mechanism ensures that “are” does not have access to information from “you,” which appears later in the sequence. This masking is essential for preserving the autoregressive nature of the decoder’s generation process.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*pvLDOhvMW1dGtk3q" /></figure><p>Source: Google</p><p><em>.</em></p><p>This masking mechanism ensures that predictions for any given position can only depend on known outputs from earlier positions in the sequence.</p><h3>3.2 Encoder-Decoder Multi-Head Attention (Cross-Attention)</h3><p>In the second multi-headed attention layer of the decoder, there is a distinct interaction between the components of the encoder and decoder. In this layer, the outputs from the encoder serve as the keys and values, while the outputs from the first multi-headed attention layer of the decoder act as the queries.</p><p>This configuration effectively aligns the decoder’s understanding with the encoded input, allowing the decoder to identify and emphasize the most relevant parts of the input sequence provided by the encoder.</p><p>The output from this cross-attention layer is then further processed through a pointwise feedforward layer, refining the information and enhancing the overall quality of the decoding process.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*MApAuDZIxEjhrlrA" /></figure><p>Source: Google</p><p>In this sub-layer, the queries come from the previous decoder layer, and the keys and values come from the output of the encoder. This allows every position in the decoder to attend over all positions in the input sequence, effectively integrating information from the encoder with the information in the decoder.</p><h3>STEP 3.3 Feed-Forward Neural Network</h3><p>Similar to the encoder, each decoder layer includes a fully connected feed-forward network, applied to each position separately and identically.</p><h3>STEP 4 Linear Classifier and Softmax for Generating Output Probabilities</h3><p>The journey of data through the transformer model culminates in its passage through a final linear layer, which functions as a classifier.</p><p>The size of this classifier corresponds to the total number of classes involved (number of words contained in the vocabulary). For instance, in a scenario with 1000 distinct classes representing 1000 different words, the classifier’s output will be an array with 1000 elements.</p><p>This output is then introduced to a softmax layer, which transforms it into a range of probability scores, each lying between 0 and 1. The highest of these probability scores is key,its corresponding index directly points to the word that the model predicts as the next in the sequence.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*BX2-aeZFlWr_cX4s" /></figure><p>Source: Google</p><h3>Normalization and Residual Connections</h3><p>Each sub-layer (masked self-attention, encoder-decoder attention, feed-forward network) is followed by a normalization step, and each also includes a residual connection around it.</p><h3>Output of the Decoder</h3><p>The final layer’s output is transformed into a predicted sequence, typically through a linear layer followed by a softmax to generate probabilities over the vocabulary.</p><p>The decoder, in its operational flow, incorporates the freshly generated output into its growing list of inputs, and then proceeds with the decoding process. This cycle repeats until the model predicts a specific token, signaling completion.</p><p>The token predicted with the highest probability is assigned as the concluding class, often represented by the end token.</p><p>Again remember that the decoder isn’t limited to a single layer. It can be structured with N layers, each one building upon the input received from the encoder and its preceding layers. This layered architecture allows the model to diversify its focus and extract varying attention patterns across its attention heads.</p><p>Such a multi-layered approach can significantly enhance the model’s ability to predict, as it develops a more nuanced understanding of different attention combinations.</p><p>And the final architecture is something similar like this (form the original paper)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/912/0*LszD_KrleQ1EakoI" /></figure><p>Source: Google</p><h3>Conclusion</h3><p>The transformer architecture’s ability to handle long-range dependencies and its parallel processing power make it a cornerstone of modern AI. Understanding its inner workings, from self-attention to positional encoding, is key to grasping the power and potential of today’s AI models.</p><p>Feel free to reach out for any queries!!</p><p>Stay Tuned!!</p><p><a href="https://www.linkedin.com/in/aashish-singh-8b6ab6a0/">Follow on Linkedin</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=589fe8bb2f49" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>