<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Sudip Chatterjee on Medium]]></title>
        <description><![CDATA[Stories by Sudip Chatterjee on Medium]]></description>
        <link>https://medium.com/@sc-blogs?source=rss-4f1c6e3f54f1------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*2iXhGW7xv3wXcjYuo14Sjw.png</url>
            <title>Stories by Sudip Chatterjee on Medium</title>
            <link>https://medium.com/@sc-blogs?source=rss-4f1c6e3f54f1------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 08:32:28 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@sc-blogs/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[✈️ AI Just Killed the Travel Agent]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/@sc-blogs/%EF%B8%8F-ai-just-killed-the-travel-agent-a72873783734?source=rss-4f1c6e3f54f1------2"><img src="https://cdn-images-1.medium.com/max/2600/0*G_d6xnt4oUfQHoop" width="3888"></a></p><p class="medium-feed-snippet">Not a chatbot. Not a bot. A team of AI agents just planned and booked a full trip&#x200A;&#x2014;&#x200A;start to finish&#x200A;&#x2014;&#x200A;without human help. The future of&#x2026;</p><p class="medium-feed-link"><a href="https://medium.com/@sc-blogs/%EF%B8%8F-ai-just-killed-the-travel-agent-a72873783734?source=rss-4f1c6e3f54f1------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://medium.com/@sc-blogs/%EF%B8%8F-ai-just-killed-the-travel-agent-a72873783734?source=rss-4f1c6e3f54f1------2</link>
            <guid isPermaLink="false">https://medium.com/p/a72873783734</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[agentic-ai]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[langgraph]]></category>
            <dc:creator><![CDATA[Sudip Chatterjee]]></dc:creator>
            <pubDate>Tue, 09 Sep 2025 11:12:22 GMT</pubDate>
            <atom:updated>2025-09-09T11:12:22.654Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[ Open-Source LLMs in 2025: The Rise of Open Access Intelligence]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/@sc-blogs/open-source-llms-in-2025-the-rise-of-open-access-intelligence-a996f0f5dba7?source=rss-4f1c6e3f54f1------2"><img src="https://cdn-images-1.medium.com/max/1814/1*swr_zH0U65NqVrZkRgDPEg.png" width="1814"></a></p><p class="medium-feed-snippet">In 2020, training a large language model cost millions. In 2025, you can do it on a budget laptop&#x200A;&#x2014;&#x200A;with open weights, open code, and a&#x2026;</p><p class="medium-feed-link"><a href="https://medium.com/@sc-blogs/open-source-llms-in-2025-the-rise-of-open-access-intelligence-a996f0f5dba7?source=rss-4f1c6e3f54f1------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://medium.com/@sc-blogs/open-source-llms-in-2025-the-rise-of-open-access-intelligence-a996f0f5dba7?source=rss-4f1c6e3f54f1------2</link>
            <guid isPermaLink="false">https://medium.com/p/a996f0f5dba7</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[tech-trends-2025]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[open-source]]></category>
            <dc:creator><![CDATA[Sudip Chatterjee]]></dc:creator>
            <pubDate>Fri, 20 Jun 2025 11:01:37 GMT</pubDate>
            <atom:updated>2025-06-20T11:01:37.443Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[ Rethinking AGI: Exploring Bold New Paths Beyond LLMs]]></title>
            <link>https://python.plainenglish.io/why-llms-are-not-the-only-step-in-reaching-agi-534138ce29b5?source=rss-4f1c6e3f54f1------2</link>
            <guid isPermaLink="false">https://medium.com/p/534138ce29b5</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[aı]]></category>
            <dc:creator><![CDATA[Sudip Chatterjee]]></dc:creator>
            <pubDate>Fri, 18 Apr 2025 10:05:17 GMT</pubDate>
            <atom:updated>2025-04-28T00:44:17.250Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZUC7Qnem8G-djZcqBApfwQ.png" /><figcaption>Human brain Vs AI</figcaption></figure><p>The rise of Large Language Models (LLMs) has been nothing short of revolutionary. These AI systems, trained on vast amounts of text data, can generate human-like language, answer questions, and even write code with remarkable proficiency. However, despite their impressive capabilities, LLMs are not the ultimate solution to achieving Artificial General Intelligence (AGI) — a form of AI that can perform any intellectual task a human can, across a wide range of domains. While LLMs represent a significant leap forward, they are only one piece of the puzzle. This article explores why LLMs alone are insufficient for AGI, supported by logic, real-world examples, and insights from cognitive science and AI research.</p><h3>🧠 Defining LLMs and AGI</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RseTVynXG6HKrRxPsuW6jg.png" /></figure><p>Before diving into the limitations of LLMs, it’s essential to clarify what these terms mean.</p><h4><strong>📘 Large Language Models (LLMs)</strong></h4><p>LLMs like OpenAI’s GPT-4 or Google’s Gemini are trained on massive corpora of internet text. They predict and generate text based on learned statistical patterns. Their capabilities include translation, summarization, dialogue, and creative writing.</p><h4>🧠 Artificial General Intelligence (AGI)</h4><p>AGI refers to a hypothetical form of AI that can perform any intellectual task that a human can, including reasoning, planning, learning, and adapting across a wide variety of domains.</p><p><em>🎯</em><strong><em> Key Difference:</em></strong><em> LLMs are powerful tools specialized in language, but AGI demands cognitive versatility and true generalization far beyond text.</em></p><h3>🕰️ The Current State of AGI</h3><p>Today, AGI remains a long-term goal in the field of AI. While systems like GPT-4 exhibit surprising versatility, they are still considered “narrow AI” — highly capable within constrained tasks but unable to generalize in the way humans do. Experts remain divided on when AGI will be achieved, with timelines ranging from decades to never. Research is actively exploring the necessary ingredients to move beyond the limitations of LLMs and toward true general intelligence.</p><h3>🚧 The Limitations of LLMs</h3><h4>❌ 1. Lack of True Understanding</h4><p>LLMs excel at mimicking language but lack comprehension.</p><p><strong>Example:</strong> Ask, “Can a penguin fly?” and it may answer “No.” But it doesn’t <em>understand</em> flight mechanics or penguin biology. Ask it, “What if a penguin had wings like an eagle?” and it might provide a speculative answer based on textual correlation — not grounded reasoning.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7fxhAX_WAJCbkuxWOgWOKQ.png" /></figure><h4>🔄 2. Inability to Reason Beyond Training Data</h4><p>LLMs interpolate within what they’ve seen, but can’t reliably extrapolate to unfamiliar territory.</p><p><strong>Example</strong>: Given a novel logic puzzle or a real-world scenario without clear textual precedent, an LLM often fails. For instance, asked to invent a new card game with specific constraints, it might contradict itself or reuse elements from known games.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xeV12h3Dmq6D1a9IkFxGcQ.png" /></figure><h4>📊 3. Dependence on Massive Datasets</h4><p>LLMs are data-hungry. Humans, by contrast, learn from few examples.</p><p><strong>Example</strong>: A toddler sees one zebra and learns the concept. An LLM would need thousands of data points to form a similar representation — a stark contrast in efficiency.</p><h4>🌍 4. No Real-World Experience</h4><p>LLMs have never physically interacted with the world.</p><p><strong>Example:</strong> Drop a glass on a floor and a human <em>knows</em> it breaks because they’ve seen and heard it. Ask an LLM what happens and it might get it right — but it doesn’t understand physics, only patterns.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tafGtB2Zr1QKQY7KhXr5XQ.png" /></figure><h4>⚠️ 5. Ethical and Robustness Issues</h4><p>LLMs are vulnerable to hallucination, bias, and manipulation.</p><p><strong>Example:</strong> A slight rewording of a question can lead to hallucinated facts. For instance, changing “Who was the first person on the moon?” to “Which Soviet astronaut walked on the moon first?” might result in a confidently wrong answer.</p><h3>🚀 What Else Is Needed for AGI?</h3><p>To achieve AGI, we must go beyond LLMs and build systems that integrate:</p><h4>🧍‍♂️ 1. Embodied Cognition</h4><p>Intelligence emerges from interacting with the physical world. Without sensory input and bodily experience, AI lacks intuition and common sense.</p><p><strong>Example:</strong> Boston Dynamics robots learn to navigate rough terrain through trial and error — an experience that provides deep understanding.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*i_zKcUg-P5Vf3OFsu6Cc-w.png" /></figure><h4>🧠 2. Symbolic Reasoning &amp; Neurosymbolic AI</h4><p><strong>Neurosymbolic AI</strong> = neural networks (pattern recognition) + symbolic logic (reasoning).</p><p><strong>Example:</strong> In autonomous driving, a neurosymbolic system could recognize a traffic sign visually (neural net) and understand what it legally means (symbolic logic).</p><h4>🔁 3. Efficient Learning Mechanisms</h4><p>AGI must learn from limited data like humans.</p><p><strong>Mathematical Insight:</strong> Generalization error typically decreases as training data increases:</p><p>Generalization Error ∝ 1 / √n</p><p>Meta-learning techniques aim to reduce this error even with small n, helping AI learn new tasks faster.</p><p><strong>Example:</strong> Few-shot learning enables an AI to learn from a handful of examples. Meta-learning allows it to generalize faster to new tasks by learning how to learn.</p><h4>🔄 4. World Models &amp; Causal Reasoning</h4><p>AGI needs internal simulations to understand cause-effect relationships.</p><p><strong>Mathematical Insight:</strong> Causal reasoning can be framed using Pearl’s do-calculus:</p><p>P(Y | do(X)) ≠ P(Y | X)</p><p>This distinction is essential: observing a variable (X) is not the same as intervening in it. AGI needs to understand such causal relationships to make meaningful decisions.</p><p><strong>Example:</strong> Just like AlphaZero simulates game strategies, AGI should predict the outcomes of its actions in complex environments.</p><h4>💡 5. Intrinsic Motivation and Curiosity</h4><p>Humans explore and learn out of curiosity. AGI should too.</p><p><strong>Example:</strong> An intrinsically motivated AI might explore new environments, ask questions, or learn skills without direct prompting.</p><h3>🎯 Conclusion</h3><p><strong>Mathematical Perspective:</strong> Scaling LLMs has shown impressive gains, but size alone is not enough:</p><p>lim (θ → ∞) LLM(θ) ≠ AGI</p><p>Large Language Models have ushered in an exciting new era of AI ✨ but they are <strong>not enough</strong>.</p><p>Their inability to reason beyond data, understand causality, or interact with the world limits their generality.</p><h4>🧩 To Reach AGI, We Need:</h4><ul><li>🤖 Physical embodiment</li><li>🧠 Symbolic reasoning</li><li>📚 Efficient learning</li><li>🔍 Causal inference</li><li>🚀 Intrinsic goals and curiosity</li></ul><blockquote><em>🗣️ </em>In the words of Yann LeCun: “Language is not the only path to intelligence.”</blockquote><p>The road to AGI is a mosaic. LLMs are powerful pieces, but they must be connected with many others to form the full picture of human-like intelligence.</p><p>📌 <em>Follow for more deep dives into AI, neuroscience, and the future of intelligence.</em></p><p><strong>🔎 References &amp; Further Reading:</strong></p><ul><li><a href="https://openreview.net/forum?id=BZ5a1r-kVsf"><em>Yann LeCun: A Path Towards Autonomous Machines</em></a></li><li><a href="https://www.bostondynamics.com/"><em>Embodied Cognition in Robotics</em></a></li><li><a href="https://arxiv.org/abs/1906.11468"><em>The Neurosymbolic Concept Learner</em></a></li><li><a href="https://arxiv.org/abs/2004.05439"><em>Meta-Learning Survey</em></a></li><li><a href="https://worldmodels.github.io/"><em>World Models</em></a></li></ul><h3>Thank you for being a part of the community</h3><p><em>Before you go:</em></p><ul><li>Be sure to <strong>clap</strong> and <strong>follow</strong> the writer ️👏<strong>️️</strong></li><li>Follow us: <a href="https://x.com/inPlainEngHQ"><strong>X</strong></a> | <a href="https://www.linkedin.com/company/inplainenglish/"><strong>LinkedIn</strong></a> | <a href="https://www.youtube.com/@InPlainEnglish"><strong>YouTube</strong></a> | <a href="https://newsletter.plainenglish.io/"><strong>Newsletter</strong></a> | <a href="https://open.spotify.com/show/7qxylRWKhvZwMz2WuEoua0"><strong>Podcast</strong></a> | <a href="https://differ.blog/inplainenglish"><strong>Differ</strong></a> | <a href="https://twitch.tv/inplainenglish"><strong>Twitch</strong></a></li><li><a href="https://cofeed.app/"><strong>Check out CoFeed, the smart way to stay up-to-date with the latest in tech</strong></a> <strong>🧪</strong></li><li><a href="https://differ.blog/"><strong>Start your own free AI-powered blog on Differ</strong></a> 🚀</li><li><a href="https://discord.gg/in-plain-english-709094664682340443"><strong>Join our content creators community on Discord</strong></a> 🧑🏻‍💻</li><li>For more content, visit <a href="https://plainenglish.io/"><strong>plainenglish.io</strong></a> + <a href="https://stackademic.com/"><strong>stackademic.com</strong></a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=534138ce29b5" width="1" height="1" alt=""><hr><p><a href="https://python.plainenglish.io/why-llms-are-not-the-only-step-in-reaching-agi-534138ce29b5">🌐 Rethinking AGI: Exploring Bold New Paths Beyond LLMs</a> was originally published in <a href="https://python.plainenglish.io/">Python in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How I Built a Personal AI Agent to Automate My Everyday Tasks]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://python.plainenglish.io/how-i-built-a-personal-ai-agent-to-automate-my-everyday-tasks-367c512a7558?source=rss-4f1c6e3f54f1------2"><img src="https://cdn-images-1.medium.com/max/1536/1*PAbZBkiNS63ngRAD1Rwkbg.png" width="1536"></a></p><p class="medium-feed-snippet">What if you had a digital assistant that could handle your boring chores&#x200A;&#x2014;&#x200A;writing emails, scheduling meetings, even shopping online &#x2014;&#x2026;</p><p class="medium-feed-link"><a href="https://python.plainenglish.io/how-i-built-a-personal-ai-agent-to-automate-my-everyday-tasks-367c512a7558?source=rss-4f1c6e3f54f1------2">Continue reading on Python in Plain English »</a></p></div>]]></description>
            <link>https://python.plainenglish.io/how-i-built-a-personal-ai-agent-to-automate-my-everyday-tasks-367c512a7558?source=rss-4f1c6e3f54f1------2</link>
            <guid isPermaLink="false">https://medium.com/p/367c512a7558</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[busıness]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Sudip Chatterjee]]></dc:creator>
            <pubDate>Fri, 04 Apr 2025 08:28:33 GMT</pubDate>
            <atom:updated>2025-04-07T00:56:46.153Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Machine Learning Pipelines]]></title>
            <link>https://medium.com/@sc-blogs/machine-learning-pipelines-ee5357579cf3?source=rss-4f1c6e3f54f1------2</link>
            <guid isPermaLink="false">https://medium.com/p/ee5357579cf3</guid>
            <category><![CDATA[ml-so-good]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[pipeline]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Sudip Chatterjee]]></dc:creator>
            <pubDate>Tue, 02 Aug 2022 10:18:19 GMT</pubDate>
            <atom:updated>2025-04-18T10:06:33.342Z</atom:updated>
            <content:encoded><![CDATA[<p>A quick refresher on how to build an effective ML Pipeline using Scikit Learn Library.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*BOrMUiPtLF1zmbOM" /><figcaption>Photo by <a href="https://unsplash.com/@rstar50?utm_source=medium&amp;utm_medium=referral">Roger Starnes Sr</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>Introduction</h3><p>A Pipeline in Scikit-Learn is a chain of <strong>transformers</strong> and a final <strong>estimator</strong>. Together they are called <strong>Pipeline</strong>.</p><p>It is designed in such a way that the output of one is the input of the other. In the diagram below, you will see the pipeline with 1,2,3…,n <strong>Transformers</strong> and an <strong>Estimator</strong> used to create the Pipeline.</p><p>This Pipeline when fed with data gives predictions based on model learning.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/783/1*xDynWXhwh0ygE6qo-lnXVw.png" /><figcaption>Fig-1: A simple machine learning Pipeline</figcaption></figure><p>A machine learning pipeline <strong>helps to standardize the development process and helps to build efficient and accurate models</strong>.</p><h4>Transformers</h4><p>Transformers are used to clean, filter or modify data. It&#39;s a part of data preprocessing.</p><p>Scikit-learn provides a library of transformers for different activities like</p><ol><li>cleaning data(pre-processing),</li><li>reduce the size(dimensionality reduction)</li><li>expand size(Kernel Approximation)</li><li>generate feature representation(feature extraction)</li></ol><p>These transformer classes are represented by the following methods:</p><ul><li>fit() <em>method</em>: which learns model parameters (e.g. mean and standard deviation for normalization) from a training set</li><li>transform() <em>method</em>: which applies this transformation model to unseen data</li><li>fit_transform() <em>method</em>: which may be more convenient and efficient for modeling and transforming the training data simultaneously</li></ul><h4>Estimators</h4><p>An estimator is any object that learns from data. It may be a classification, regression, clustering algorithm, or a transformer that extracts/filters useful features from raw data.</p><p>The estimator class implements the following methods:</p><ul><li>fit() method: to learn from the data</li><li>predict() method: to predict the result(regression or classification)</li></ul><h3>Sample use case</h3><p>In this use case, we will be using the <strong>Titanic</strong> dataset. In this dataset, we will apply some common Transformers on certain columns and then we will use a Decision Tree Estimator to classify whether the passenger will live or die.</p><h4>Plan</h4><p>Here is the plan outline for our use case.</p><p>To make our use case easy to understand, let us see the diagram below. This will give you a fairly good understanding of the pipeline visually.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/761/1*UdF0Ry8R_c7m3v4TgHtaTg.png" /><figcaption>Fig-2: Pipeline with 4 transformers</figcaption></figure><h4>Dataset</h4><pre># Importing necessary libraries<br>import pandas as pd<br>import numpy as np<br>from sklearn.datasets import fetch_openml<br>from sklearn.model_selection import train_test_split<br>from sklearn.compose import ColumnTransformer<br>from sklearn.impute import SimpleImputer<br>from sklearn.preprocessing import OneHotEncoder<br>from sklearn.preprocessing import MinMaxScaler<br>from sklearn.pipeline import Pipeline, make_pipeline<br>from sklearn.feature_selection import SelectKBest, chi2<br>from sklearn.tree import DecisionTreeClassifier<br>from sklearn import set_config</pre><pre># config to show pipeline visually<br>set_config(display=&#39;diagram&#39;)</pre><p>Now that we have imported all the necessary libraries, let’s fetch data from Titanic Dataset</p><pre># fetch the data<br>dataset = fetch_openml(&quot;titanic&quot;, version=1)</pre><pre># check the data<br>dataset[&#39;frame&#39;].head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xcfX4S8OgjQkrpclZjp3Hg.png" /></figure><p>We do not need all the features here, so we can safely ignore some features that are not so important to us.</p><pre># features<br>features = dataset.frame</pre><pre># drop the columns that we don&#39;t need<br>features = features.drop([&#39;name&#39;, &#39;ticket&#39;, &#39;cabin&#39;, &#39;survived&#39;, &#39;boat&#39;, &#39;body&#39;, &#39;home.dest&#39;], axis=1)</pre><pre># check the features<br>features.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*u9eyZQLz_fGO0tm1JyFFQQ.png" /></figure><pre># columns description</pre><pre>&quot;&quot;&quot;<br>pclass: passenger class<br>name: name of passenger<br>sex: male or female<br>age: age of passenger<br>sibsp: siblings of other passenger<br>parch: number of parent-child<br>fare: ticker fare<br>embarked: passenger boarded from (There are three possible boarding points: Southampton[S], Cherbourg[C], Queenstown[Q])<br>&quot;&quot;&quot;</pre><p>This dataset also comes with the target column. So let us take the target column from it.</p><pre># target<br>target = dataset.target</pre><pre># check the target<br>target.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*umf9FAJkl6THNnsiu603Rg.png" /></figure><pre># train-test split<br>X_train, X_test, y_train, y_test = train_test_split(features, target, test_size=0.2, random_state=42)</pre><pre># see the sample training on features<br>X_train.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Rx9j93v07BzM_QX5nbjMVQ.png" /></figure><pre># see the training data on target<br>y_train.head(5)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RN2M8FLT_Sx0bJuX5K7cVA.png" /></figure><h4>Transformers</h4><p>Transformers are an important part of Pipeline. This will transform the data into various forms like Cleaning, Scaling, or Feature Engineering. These are called ColumnTransformers .</p><p>Now as per the diagram below, we will implement the transformers one by one.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/761/1*UdF0Ry8R_c7m3v4TgHtaTg.png" /></figure><p>Following are the transformers used:</p><ol><li>Imputation: Here we will use SimpleImputer Transformer</li><li>One-Hot-Encoding: Here we will use OneHotEncoder Transformer</li><li>Feature Scaling: Here we will use MinMaxScaler Transformer</li><li>Feature Selection: Here we will use SelectKBest Transformer</li></ol><p>And at the end, we will use a classifier to predict our outcome. Here we will use DecisionTreeClassifier .</p><h4>SimpleImputer Transformer</h4><p>In this transformer, we will check if the dataset has missing values.</p><p>For the columns which will have missing values, we will have to apply SimpleImputer to handle missing values.</p><pre># check for missing values<br>features.isnull().sum()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wnsF25R5rgjsVHW9fycfFw.png" /></figure><p>We can see that the age column and embarked columns have missing values. on fare there is only one column missing, so we will try to ignore it here.</p><pre># Imputation tansformer<br>trfOne = ColumnTransformer(<br>[(&#39;impute_age&#39;, SimpleImputer(), [2]),<br>(&#39;impute_embarked&#39;, SimpleImputer(strategy=&#39;most_frequent&#39;), [6])],<br>remainder=&#39;passthrough&#39;)</pre><pre># check if correctly set<br>trfOne</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WRAfaoMoNSOEx6XMJRlkiw.png" /></figure><h4>OneHotEncoder Transformer</h4><p>Encodes categorical features as a one-hot numeric array.</p><p>In our dataset, we see that the sex column has categorical data(male or female) and the embarked column which has categorical columns(Q or S or C).</p><p>Our model will perform better if we supply all columns in numerical values. To do that, we will apply OneHotEncoder to the sex column and the embarked column.</p><pre># OneHotEncoding transformer<br>trfTwo = ColumnTransformer(<br>[(&#39;ohe_sex_embarked&#39;, OneHotEncoder(sparse=False,<br>handle_unknown=&#39;ignore&#39;), [1,6])],<br>remainder = &#39;passthrough&#39;)</pre><pre># check if correctly set<br>trfTwo</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KsstMLETsKH9UX9qL7Rf8Q.png" /></figure><h4>Scaling Transformer</h4><p>In this transformer, we will scale the values using Scikit-learn MinMaxScaler. This will be applied to all numerical columns. Since different columns have different values, so scaling will bring them to one uniform scale.</p><p>If you are interested to know more about scaling, I have a dedicated blog on this. Just check out the link <a href="https://medium.com/@sc050409/the-modern-rules-of-feature-scaling-in-machine-learning-236b1ce4f818">here</a>.</p><pre># Scaling Tranformer<br>trfThree = ColumnTransformer([(&#39;scale&#39;, MinMaxScaler(), slice(0,10))])</pre><pre># check the transformer<br>trfThree</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vEwcw9Y6lKGFXctMH6ExLQ.png" /></figure><p>If you closely notice, I have applied MinMaxScaler to all the columns from 0 to 10. You should be wondering when we have just 7 columns, why we choose 10 columns.</p><p>The answer is very simple, when we do OneHotEncoding, we will drop two columns sex and embarked, but then we will also add 2 more columns related to sex and 3 columns related to embarked. This is what OneHotEncoding does. Check out the details <a href="https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.OneHotEncoder.html">here</a>.</p><h4>Feature Selection Transformer</h4><p>For selecting features, we will use Scikit-Learn’s SlectKBest. As the name suggests, this will help us select the top K best features from the list of all the features that the algorithm will pick. We will also use chi2 which is the scoring algorithm to select the top features as per the score.</p><pre># feature selection<br>trfFour = SelectKBest(score_func=chi2, k=5)</pre><pre># see the details<br>trfFour</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jnPGv9dESF0gmDpRf7GJpw.png" /></figure><h4>Encoders</h4><p>Now when we are ready with all the transformers, we will feed the transformed data to our Encoders. These are nothing but the Machine learning models which will now learn from this transformed data and make predictions.</p><h4>Decision Tree Classifier</h4><p>For this problem, we will be using the Decision Tree Classifier.</p><pre># Encoder<br>trfFive = DecisionTreeClassifier()</pre><pre># details<br>trfFive</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*f-6AeazEHeY8Ox46gH5JBg.png" /></figure><h4>Complete Pipeline</h4><p>With this, we complete what we planned. Now let us look at the diagram once more, so that we know what we are doing</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/761/1*z-VM74vshAdRbj4hhzo-Kg.png" /></figure><h4>Pipeline</h4><p>Here, we will create our Pipeline with the help of all the above transformers and encoders.</p><pre># pipeline<br>pipe = Pipeline([(&#39;trfOne&#39;, trfOne),(&#39;trfTwo&#39;, trfTwo),(&#39;trfThree&#39;, trfThree),(&#39;trfFour&#39;, trfFour),(&#39;trfFive&#39;, trfFive)])</pre><pre># details<br>pipe</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HJITr0xkdIx9s4J4OTk7cQ.png" /></figure><h4>make_pipeline</h4><p>There is one more way to create a Pipeline. Instead of using Pipeline directly, we can also use make_pipeline(). Here we don’t have to declare the names of each Pipeline. We just have to call all the components inside this function.</p><pre># make_pipeline<br>pipe = make_pipeline(trfOne, trfTwo, trfThree, trfFour, trfFive)</pre><pre># check details<br>pipe</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MSOyshUGmOT05zXf4dTZnQ.png" /></figure><h4>Training</h4><p>Now that we are ready with our Pipeline, we now have to train our data using this pipeline and see if it is able to predict the desired outcome.</p><p>But before that, if we call our pipe.fit() method, it will also give us the diagram of the entire pipe. There are multiple benefits of getting a visual representation of the pipeline. For example, someone might just understand what all operations you have done on the data. Check it out.</p><pre># train the pipeline and display the graph<br>pipe.fit(X_train, y_train)</pre><pre># display the steps<br>pipe.named_steps</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OrbuFObwRW0B9hvX-Eoe1w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*T-WpwcabesEa3LNoR9g2Mw.png" /></figure><pre># display sigle step<br>pipe.named_steps[&#39;columntransformer-1&#39;]</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XZCSlnAhtHaKkV3WmSd6mQ.png" /></figure><h4>Prediction</h4><p>Now that we have trained our Pipeline on the data, it is now time to make some predictions and check if our model has learned something from the data.</p><pre># predict<br>y_pred = pipe.predict(X_test)</pre><pre># check the prediction score<br>from sklearn.metrics import accuracy_score<br>accuracy_score(y_test, y_pred)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FL4qRzFUEsNULkRX9mAMvA.png" /></figure><p>So we can see that our model is not almost 60% confident in its prediction. Since our focus is on understanding how to build a pipeline here, so we will not be much concerned about the accuracy score.</p><h3>Conclusion</h3><p>The goal of this blog is to understand the concept of Pipelines and how to build one yourself. So we can safely overlook the model scores here.</p><p>We tried to build a Pipeline step-by-step following a plan and we succeeded.</p><p>This is just to show you how much this can reduce your efforts if you work with Pipeline.</p><blockquote>The code looks neat and clean. It is also Production ready. You can just deploy this Pipeline inside your project code and start getting predictions.</blockquote><h4>Before you go…</h4><p>In this blog, we have learned the following:</p><ol><li>What is a Pipeline in Machine Learning</li><li>What are Transformer and Encoders</li><li>How to build a Pipeline</li><li>How to Train and Test it with actual data</li><li>How to deploy this into Production</li></ol><p>If you have followed me till here, I am sure you must have gained a very good idea of Pipelines in Machine learning and how to use them.</p><p><a href="https://medium.com/mlearning-ai/mlearning-ai-submission-suggestions-b51e2b130bfb">Mlearning.ai Submission Suggestions</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ee5357579cf3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[4 Pandas Functions for Data Exploration]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://python.plainenglish.io/4-pandas-functions-for-data-exploration-c0f73643fcf0?source=rss-4f1c6e3f54f1------2"><img src="https://cdn-images-1.medium.com/max/2592/0*Oew3N4M0aVoqH9XG" width="2592"></a></p><p class="medium-feed-snippet">Use them effectively in your Machine Learsing and Data Science projects.</p><p class="medium-feed-link"><a href="https://python.plainenglish.io/4-pandas-functions-for-data-exploration-c0f73643fcf0?source=rss-4f1c6e3f54f1------2">Continue reading on Python in Plain English »</a></p></div>]]></description>
            <link>https://python.plainenglish.io/4-pandas-functions-for-data-exploration-c0f73643fcf0?source=rss-4f1c6e3f54f1------2</link>
            <guid isPermaLink="false">https://medium.com/p/c0f73643fcf0</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[pandas]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Sudip Chatterjee]]></dc:creator>
            <pubDate>Fri, 03 Jun 2022 18:34:43 GMT</pubDate>
            <atom:updated>2025-04-06T05:06:25.755Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[The Modern Rules Of Feature Scaling in Machine Learning]]></title>
            <link>https://medium.com/@sc-blogs/the-modern-rules-of-feature-scaling-in-machine-learning-236b1ce4f818?source=rss-4f1c6e3f54f1------2</link>
            <guid isPermaLink="false">https://medium.com/p/236b1ce4f818</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[feature-engineering]]></category>
            <category><![CDATA[ml-so-good]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Sudip Chatterjee]]></dc:creator>
            <pubDate>Wed, 18 May 2022 10:59:47 GMT</pubDate>
            <atom:updated>2022-07-26T10:08:46.212Z</atom:updated>
            <content:encoded><![CDATA[<p><em>A complete guide to feature scaling using Scikit-Learn and Python</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Fq-gk44REHt_VLlRI7EcWA.jpeg" /><figcaption>Fruits And Vegetables of different sizes. Image source: <a href="https://unsplash.com/photos/4BRysALhyuw">Unsplash</a></figcaption></figure><h4>What is scaling?</h4><p>Scaling is an important part of Machine Learning.</p><p>Having said that, not all Machine Learning algorithms require scaling. You should know this beforehand while using these algorithms.</p><p>Algorithms that require the feature scaling:</p><ul><li>KNN (K-Nearest Neighbours)</li><li>Neural Networks</li><li>Linear Regression</li><li>Logistic Regression</li></ul><p>Algorithms that do not require feature scaling:</p><ul><li>Decision trees</li><li>Random Forest</li><li>AdaBoost</li><li>Naive Bayes</li></ul><p>Now, let’s understand why scaling is required with the help of an example.</p><p>Suppose we are going to make a Blueberry Banana Smoothie. Now, what will happen if we add one Banana and one Blueberry to the milk? You will say obviously the smoothie will be dominant with a taste of Banana, as it is large in size compared to Blueberry.</p><p>On the other hand, an expert smoothie maker will mix Blueberry and One Banana in the right proportion to get a great taste.</p><p>Scaling in machine learning is similar in nature to this analogy. We will have multiple features and they all have their own size. Now if we fit them to our model just like that, then the feature with the highest value will dominate the results.</p><p>So with scaling, we bring all the features to one scale, let us say between 0 and 1, and then feed it to the model. This way we can expect an unbiased result from our model.</p><p>The two most discussed scaling methods are:</p><ol><li>Normalizatoion</li><li>Standardization.</li></ol><p><em>Special Note — Neither Normalization nor Standardization changes the distribution of the data.</em></p><p><strong>Normalization</strong> typically rescales the values into a range of [0, 1].</p><p>It can be represented as</p><figure><img alt="Normalisation Formulae" src="https://cdn-images-1.medium.com/max/143/1*h48AKSGexXCSoNV_sKA2tw.png" /></figure><p>Here x-min and x-max are min and max values of the feature X.</p><p><strong>Standardization</strong> typically rescales data to have a mean of 0 and a standard deviation of 1.</p><p>It can be represented as</p><figure><img alt="Standardisation Formulae" src="https://cdn-images-1.medium.com/max/76/1*Bxhl0ong89YJMFD96MohUw.png" /></figure><p>where μ is the mean of the features, and it can be represented as</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/113/1*EsVgX8IuTaU1SohVWeru2A.png" /></figure><p>and σ is the standard deviation of the feature.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/168/1*_6bJ7dpKU6h4IqscR9Hiog.png" /></figure><h4>Why do we need scaling?</h4><p>When we feed the features to our machine learning model, it will predict an outcome and compare it with the actual outcome. It then corrects itself and then repeats it once again. In the process, it will gain an understanding of some hidden patterns which we can not see.</p><p>Since the model relies only on data that we provide, it becomes most important that we do proper cleaning and processing of the data before we feed it to our model.</p><h4>Use Case</h4><p>Let us walk through a practical use case, where suppose you want to predict the price of Apple stock using machine learning.</p><p>We will use <a href="https://github.com/ranaroussi/yfinance">yfinance</a> package to download the stock data from <a href="https://finance.yahoo.com/">Yahoo Finance</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vXG-yUF_g7TgoxARAu_moQ.jpeg" /></figure><p>Let’s download one year of data for the AAPL ticker. We then remove all the other columns and keep only Close and Volume columns. These will be our features.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pMFRty_iC-AnzMAvaME0QQ.jpeg" /></figure><p>Let us see the mean of the features.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hH9xpDWEC2EXPMkK2182mg.jpeg" /></figure><p>As we can see there is a big difference between the mean of Close and Volume . If we feed this data to our model, it will surely impact the performance of our model.</p><p>Let’s check the distribution of our features.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*a5h8rl9imCWhlDjv_s0DDA.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IjKFla79ji3q6LHh-giW3Q.jpeg" /></figure><p>As discussed in the beginning, scaling can be performed using</p><ul><li>Normalization or</li><li>Standardization.</li></ul><p>In sci-kit-learn, we call these libraries with following names:</p><ul><li>MinMaxScaler</li><li>StandardScaler</li></ul><p>Let us see them one by one. But before that, we need to convert our dataFrame into a NumPy array.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mWYFRMD9cuWPUyY0PZPnIQ.jpeg" /></figure><p>Now let’s see the mean</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*60MOLYQYHn4pKPXa3JILlg.jpeg" /></figure><p>and standard deviation</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xQ9bOGJcy_Jq55MuB1AtWQ.jpeg" /></figure><h4>StandardScaler</h4><p>In <a href="https://scikit-learn.org/stable/">Scikit-Learn</a> the library is called <a href="https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html">StandardScaler</a> for Standardisation.</p><blockquote>Remember that in Standardisation, the mean of the features will be 0 and the standard deviation will always be 1.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*La7QHiKzUU9xq2i2V2db_g.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*w13WPW5YkMz8Ky7UBEY3Lg.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PzNSYzP3Kf8RL64GP4qT2A.jpeg" /></figure><p>Now, after scaling, let us check the mean and standard deviation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PIENqmeL9n3FcVmowtvwuA.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6aXiLI3bGoJc0kjc05jhzg.jpeg" /></figure><p>So we have achieved the scaling using StandardScaler. But let’s see the histogram and check if the distribution has changed?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9B-rie-L0uS82QNbxuHTaQ.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5KPAtVTD2Y5LpRZvXE-ZQQ.jpeg" /></figure><p>As we can see the distribution is the same but the data is scaled in the same scale.</p><h4>MinmaxScaler</h4><p>We will not try <a href="https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.MinMaxScaler.html">MinmaxScaler</a> for Normalisation.</p><blockquote>Normalization is a process of scaling the data from the original range so that all values are within the new range of 0 and 1.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AdL5oHOyVks1N7NEes60xA.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iPsHQ042bvOgCjhxST3PNA.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gWk8vx8QfmuJAawtxqHh4A.jpeg" /></figure><p>Let us see the minimum and maximum values of our features. Ideally, it should be 1 and 0.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UnKb0wkJoJ8mmBnIL0E5Qw.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xyey31YyTvx2dFAQ-xSbIw.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pYYNPyB1sl7FgtxCrGrdMw.jpeg" /></figure><p>So we can see that using MinMaxScaler, we could split the data between 0 and 1. Now let us check the histogram of the features once again so that we become rest assured that the distribution is not changed</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AtfVFj-szb-hxrz66eBS6Q.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*u0If60bfMySfZARp1aCrmw.jpeg" /></figure><p>So we have seen that both Normalization and Standardization just bring uneven features to one scale without altering the distribution of data. These methods are really helpful to achieve better results from your mode.</p><h4>Bonus Notes</h4><h4>Using scaler:</h4><p>When you split the entire feature set into training and test datasets, it is very important to note that, you need to send the training data to the fit function once and then you can generate transformed data for both the training set and test set. If you forget to do that, you will have different datasets.</p><blockquote>StandardScaler.fit(features_train)</blockquote><blockquote>StandardScaler.transform(features_train)</blockquote><blockquote>StandardScaler.transform(features_test)</blockquote><h4>Feeding Data:</h4><p>Never fit the entire dataset into the StandardsScaler as this may cause the model to learn some features about the test and this may lead to biased outcomes. Our model should never see the test set features in anyways. This may cause data leakage.</p><h4>Remember:</h4><p>When you transform the test dataset, the features will not have an exact mean of 0 and a standard deviation of1 because the scaler used in transformation is based on the training set. This is absolutely fine.</p><h4>Finally:</h4><p>To summarize what we have learned in this article:</p><ol><li>What is feature scaling</li><li>When to use it</li><li>Main two types of scaling-Normalisation and Standardisation</li><li>Use case.</li></ol><p>I hope this article will enhance your understanding of feature scaling in machine learning. If you like the article, just leave a like and comment. I would love to hear what you think.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/780/1*OBAJEpu9Izgs6ThKjUvTeA.gif" /></figure><p><a href="https://medium.com/mlearning-ai/mlearning-ai-submission-suggestions-b51e2b130bfb">Mlearning.ai Submission Suggestions</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=236b1ce4f818" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>