<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Simone Azeglio on Medium]]></title>
        <description><![CDATA[Stories by Simone Azeglio on Medium]]></description>
        <link>https://medium.com/@simoneazeglio?source=rss-2fefde940982------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*-EelyLxxdsX6nvki1Y6aig.jpeg</url>
            <title>Stories by Simone Azeglio on Medium</title>
            <link>https://medium.com/@simoneazeglio?source=rss-2fefde940982------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 06:32:20 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@simoneazeglio/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[The Last Step in Data Preprocessing: Handling Missing Values]]></title>
            <link>https://medium.com/mljcunito/the-last-step-in-data-preprocessing-handling-missing-values-ae8f19dae309?source=rss-2fefde940982------2</link>
            <guid isPermaLink="false">https://medium.com/p/ae8f19dae309</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[mljcunito]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Simone Azeglio]]></dc:creator>
            <pubDate>Sun, 30 Aug 2020 11:59:28 GMT</pubDate>
            <atom:updated>2020-08-30T11:59:28.548Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1mLboEEMIdHjHJbI8_Z7Hg.jpeg" /><figcaption>Source: Freepik</figcaption></figure><blockquote>We are at the end of this mini-series on Data Preprocessing, the last step (if you’ve missed the previous article, you can find it <a href="https://medium.com/mljcunito/a-second-step-into-feature-engineering-feature-selection-d597619e6e2b">here</a>) to overcome is how to handle missing values. In this short article, after introducing three categories of missing data, we’ll go trough a few simple techniques which can lead us to a solution.</blockquote><p>“ <em>Missingness</em>” is almost always informative by itself, and we should tell our algorithm if a value is missing. Even if we build a model to impute our values, we are not adding any real information. We’re just reinforcing the patterns already provided by other features.</p><p>Basically, there are three categories of missing data:</p><ul><li><strong>MCAR</strong> (Missing Completely At Random) where the pattern of missinginess is statistically independent of the data record. Example: you have a data set on a piece of paper and you spill coffee on the paper destroying part of the data.</li><li><strong>MAR</strong> (Missing At Random) where the probability distribution of the pattern of missingness is functionally dependent upon the observable component in the record. MCAR is a special case of MAR. Example: if a child does not attend an educational assessment because the child is (genuinely) ill, this might be predictable from other data we have about the child’s health, but it would not be related to what we would have measured had the child not been ill.</li><li><strong>MNAR</strong> (Missing Not at Random) which is defined as the case which is NOT MAR, or when the missingness is specifically related to what is missing. Example: a person does not attend a drug test because the person took drugs the night before.</li></ul><p>Let’s see a few strategies to impute missing values, i.e. to infer them from the known part of the data.</p><h4>Univariate Feature Imputation</h4><p>We can rely on scikit-learn’s <a href="https://scikit-learn.org/stable/modules/generated/sklearn.impute.SimpleImputer.html#sklearn.impute.SimpleImputer">SimpleImputer</a> class, which provides a few strategies for imputing missing values, such as : imputing by a constant value, by using statistics (mean, median, etc).</p><p>Let’s see how we can replace missing values ( np.nan) using the mean value of the columns that contain the missing values.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ff07ed28679637968f2a512528de8a0b/href">https://medium.com/media/ff07ed28679637968f2a512528de8a0b/href</a></iframe><p>SimpleImputer can also be used in conjunction with <em>pandas</em> and in particular with data represented as strings or categoricals, by using most_frequentor constant strategy:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d22d5e9e00bbb95de8bff2294ecf89fb/href">https://medium.com/media/d22d5e9e00bbb95de8bff2294ecf89fb/href</a></iframe><h4>Multivariate Feature Imputation</h4><p>Another approach is to use the <a href="https://scikit-learn.org/stable/modules/generated/sklearn.impute.IterativeImputer.html#sklearn.impute.IterativeImputer">IterativeImputer</a> class, which models each feature with missing values as a function of other features. How does each iteration work? At each step, a feature column is designated as output yand the other feature columns are treated as inputs X. Then, the regressor is used to predict the missing values of y. This is done for each feature in an iterative fashion.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/868c6528965ecf17ed95eb5a4527022b/href">https://medium.com/media/868c6528965ecf17ed95eb5a4527022b/href</a></iframe><p>IterativeImputer is very flexible, it allows you to use a variety of estimators, if you want to further delve into this class take a look at <a href="https://scikit-learn.org/stable/auto_examples/impute/plot_iterative_imputer_variants_comparison.html#sphx-glr-auto-examples-impute-plot-iterative-imputer-variants-comparison-py">Imputing missing values with variants of IterativeImputer</a>.</p><h4>Nearest Neighbors Imputation</h4><p>The <a href="https://scikit-learn.org/stable/modules/generated/sklearn.impute.KNNImputer.html#sklearn.impute.KNNImputer">KNNImputer</a> class offers imputation for filling in missing values using the k-Nearest Neighbors approach. By default it uses an euclidean distance metric that supports missing values, nan_euclidean_distances.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*WpVMZ0_RxWc6MoOH.gif" /></figure><p>The following code snippet shows how to replace missing values using the mean feature value of the two nearest neighbors of samples with missing values:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/891626f9c49edda54fd67ced0be0c0c3/href">https://medium.com/media/891626f9c49edda54fd67ced0be0c0c3/href</a></iframe><p>Thank you for your attention, in the next mini-series of article we’re going to introduce EDA (Exploratory Data Analysis) and some techniques for data visualization.</p><p>Here it follows a short list of additional articles/ tutorials for Data Preprocessing tasks.</p><h3>References &amp; Additional Material</h3><p><strong>Preprocessing Data</strong></p><ul><li><a href="https://sebastianraschka.com/Articles/2014_about_feature_scaling.html">About Feature Scaling by Sebastian Raschka</a></li><li><a href="https://scikit-learn.org/stable/modules/preprocessing.html">Scikit-learn’s Preprocessing</a></li></ul><p><strong>Feature Selection</strong></p><ul><li><a href="https://scikit-learn.org/stable/auto_examples/feature_selection/plot_feature_selection.html#sphx-glr-auto-examples-feature-selection-plot-feature-selection-py">Univariate</a></li><li><a href="https://scikit-learn.org/stable/auto_examples/feature_selection/plot_select_from_model_diabetes.html#sphx-glr-auto-examples-feature-selection-plot-select-from-model-diabetes-py">Feature Selection using SelectFromModel and LassoCV</a></li><li><a href="https://scikit-learn.org/stable/auto_examples/feature_selection/plot_select_from_model_diabetes.html#sphx-glr-auto-examples-feature-selection-plot-select-from-model-diabetes-py">Tree-based feature importance 1</a></li><li><a href="https://scikit-learn.org/stable/auto_examples/ensemble/plot_forest_importances.html#sphx-glr-auto-examples-ensemble-plot-forest-importances-py">Tree-based feature importance 2</a></li></ul><p><strong>Encoding Categorical Features</strong></p><ul><li><a href="https://hackernoon.com/what-is-one-hot-encoding-why-and-when-do-you-have-to-use-it-e3c6186d008f">One Hot Encoding</a></li><li><a href="https://scikit-learn.org/stable/auto_examples/text/plot_hashing_vs_dict_vectorizer.html">Feature Hasher vs DictVectorizer</a></li></ul><p><strong>Feature Importance</strong></p><ul><li><a href="https://machinelearningmastery.com/calculate-feature-importance-with-python/">Calculate Feature Importance by Machinelearningmastery</a></li></ul><p><strong>Feature Selection</strong></p><ul><li><a href="https://en.wikipedia.org/wiki/Feature_selection">Feature Selection, Wikipedia</a></li><li><a href="https://machinelearningmastery.com/an-introduction-to-feature-selection/">Feature Selection 1, by Machinelearningmastery</a></li><li><a href="https://machinelearningmastery.com/feature-selection-machine-learning-python/">Feature Selection 2, by Machinelearningmastery</a></li><li><a href="https://machinelearningmastery.com/feature-selection-with-real-and-categorical-data/">Feature Selection 3, by Machinelearningmastery</a></li></ul><p><strong>Handling Missing Values</strong></p><ul><li><a href="https://scikit-learn.org/stable/modules/impute.html#marking-imputed-values">Marking Imputed Values</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ae8f19dae309" width="1" height="1" alt=""><hr><p><a href="https://medium.com/mljcunito/the-last-step-in-data-preprocessing-handling-missing-values-ae8f19dae309">The Last Step in Data Preprocessing: Handling Missing Values</a> was originally published in <a href="https://medium.com/mljcunito">MLJC</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[DeepFake Detection Challenge: an Overview]]></title>
            <link>https://medium.com/mljcunito/deepfake-detection-challenge-an-overview-a8f6fc5f6cfe?source=rss-2fefde940982------2</link>
            <guid isPermaLink="false">https://medium.com/p/a8f6fc5f6cfe</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[mljcunito]]></category>
            <category><![CDATA[social-media]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[deepfakes]]></category>
            <dc:creator><![CDATA[Simone Azeglio]]></dc:creator>
            <pubDate>Fri, 28 Aug 2020 15:01:18 GMT</pubDate>
            <atom:updated>2020-08-28T15:04:21.971Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yO1hcqKdoGYjsU6pFxdyPg.png" /><figcaption>Source: OpenDemocracy</figcaption></figure><blockquote>At MLJCUnito we deeply believe that, first of all, AI should be ethical. That’s why a few months ago we decided to take part into DeepFake Detection Challenge: a joint contribution from Amazon AWS, Facebook and Microsoft. Deepfake techniques, in short, present realistic AI-generated videos of people doing and saying fictional things. They possess the potential to have a significant impact on how people determine the legitimacy of information presented online, and, as citizens we should be well aware of this technology.</blockquote><h3>DeepFake: a threat to democracy or just a bit of fun?</h3><blockquote><em>“We are already at the point where you can’t tell the difference between deepfakes and the real thing,”</em> <strong>Professor Hao Li, University of Southern California</strong></blockquote><p><strong>Facebook</strong> <a href="https://about.fb.com/news/2020/01/enforcing-against-manipulated-media/">has announced</a> it will remove videos modified by artificial intelligence, known as deepfakes, from its platform.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FcQ54GDm1eL0%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DcQ54GDm1eL0&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FcQ54GDm1eL0%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/002d8915c14edd7968069d71086104ba/href">https://medium.com/media/002d8915c14edd7968069d71086104ba/href</a></iframe><h4><a href="http://www.kaggle.com/"><strong>Kaggle</strong></a> is an <strong>AirBnB for Data Scientists</strong></h4><p>This is where they spend their nights and weekends. It’s a crowd-sourced platform to attract, nurture, train and challenge data scientists from all around the world to solve data science, machine learning and predictive analytics problems. It has over 536,000 active members from 194 countries and it receives close to 150,000 submissions per month. Started from Melbourne, Australia Kaggle moved to Silicon Valley in 2011, raised some 11 million dollars from the likes of Hal Varian (Chief Economist at Google), Max Levchin (Paypal), Index and Khosla Ventures and then ultimately been acquired by the Google in March of 2017. Kaggle is the number one stop for data science enthusiasts all around the world who compete for prizes and boost their Kaggle rankings. There are only 94 Kaggle Grandmasters in the world to this date.</p><p>Do you know that most data scientists are only theorists and rarely get a chance to practice before being employed in the real-world? Kaggle solves this problem by giving data science enthusiasts a platform to interact and compete in solving real-life problems. The experience you get on Kaggle is invaluable in preparing you to understand what goes into finding feasible solutions for big data.</p><p>Fine, fine, fine but what do we do on Kaggle? <strong>We Learn</strong></p><p>Deepfakes are fakes generated by deep learning. So far so easy.</p><p>This usually means someone used a generative model like an <a href="http://ufldl.stanford.edu/tutorial/unsupervised/Autoencoders/"><em>AutoEncoder</em></a> or most likely a <a href="https://papers.nips.cc/paper/5423-generative-adversarial-nets.pdf"><em>Generative Adversarial Network</em></a>, short <strong>GAN</strong>. GANs are technically two networks that work against each other, illustrated below. The <em>artist</em> ( <strong>generator</strong>) draws its inspiration from a noise sample and creates a rendering of the data you are trying to generate with said GAN. The private <em>investigator</em> ( <strong>discriminator</strong>) randomly gets assigned real and fake data to investigate.</p><p>The learning process is collaborative. The generator gets better at fooling the discriminator and the discriminator gets better at figuring out which data is real and which isn’t. In mathematical terms they are learning until a <a href="https://en.wikipedia.org/wiki/Nash_equilibrium">Nash equilibrium</a> is reached, which means neither can learn new tricks and get better. They’re a really cool concept and even used in scientific simulation at <a href="https://indico.cern.ch/event/595059/contributions/2497383/attachments/1431666/2199445/gan_presentation_IML.pdf">CERN</a>.</p><p>You can probably guess that they can be tricky to train, due to so many moving parts. This has become a very popular area of research, warranting a <a href="https://github.com/hindupuravinash/the-gan-zoo">GAN Zoo</a> of all named GANs. Some important stuff you may want to check out if your interested are keywords like Wasserstein GANs, Gradient Penalization, Attention, and in this context: Style Transfer (namely face2face).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*l3VxPIN0Cum4vbmD" /></figure><p>It sounds absurd, I know. <a href="https://poloclub.github.io/ganlab/">Here you can find some more practical examples</a>, why don’t you play with them for a while?</p><h4>A few tips for the <a href="https://www.kaggle.com/c/deepfake-detection-challenge/">Official Challenge on Kaggle</a></h4><ul><li>I strongly encourage you to start first with the <a href="https://www.kaggle.com/c/deepfake-detection-challenge/overview/getting-started">official Getting Started guide here</a>.</li><li>What is the goal of the Deepfake Detection Challenge? According to the FAQ “The AI technologies that power deepfakes and other tampered media are rapidly evolving, making deepfakes so hard to detect that, at times, even human evaluators can’t reliably tell the difference. The Deepfake Detection Challenge is designed to incentivize rapid progress in this area by inviting participants to compete to create new ways of detecting and preventing manipulated media.”</li><li>In this Code Competition:</li><li>CPU Notebook &lt;= 9 hours run-time, GPU Notebook &lt;= 9 hours run-time on Kaggle’s P100 GPUs, <strong>No internet access enabled</strong></li><li>External data is allowed <strong>up to 1 GB in size</strong>. External data must be freely &amp; publicly available, including pre-trained models</li><li>This code competition’s training set is not available directly on Kaggle, as its size is prohibitively large to train in Kaggle. Instead, it’s strongly recommended that you train offline and load the externally trained model as an external dataset into Kaggle Notebooks to perform inference on the Test Set. Review Getting Started for more detailed information.</li></ul><p>Submissions are scored on <a href="http://wiki.fast.ai/index.php/Log_Loss">log loss</a>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*DDyk-PzE4fw-jhvK" /></figure><p>where:</p><ul><li><em>n</em> is the number of videos being predicted</li><li>is the predicted probability of the video being FAKE</li><li>is 1 if the video is FAKE, 0 if REAL</li><li><em>log()</em> is the natural (base e) logarithm</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d5cadcc9ef7ebb981810811c17f94c72/href">https://medium.com/media/d5cadcc9ef7ebb981810811c17f94c72/href</a></iframe><h4><a href="https://deepfakedetectionchallenge.ai/faqs">Data</a></h4><ul><li>We have a bunch of <em>.mp4 </em>files, split into compressed sets of ~10GB a piece. A <em>metadata.json</em> accompanies each set of .<em>mp4</em> files, and contains filename, label (REAL/FAKE), original and split columns, listed below under Columns.</li><li>The full training set is just over 470 GB (<strong>Yeah it’s huge !</strong>).</li></ul><p>There are 4 groups of datasets associated with this competition.</p><p><strong>Training Set:</strong> This dataset, containing labels for the target, is available for download for competitors to build their models. It is broken up into 50 files, for ease of access and download. Due to its large size, it must be accessed through a GCS bucket which is only made available to participants after accepting the competition’s rules. Please read the rules fully before accessing the dataset, as they contain important details about the dataset’s permitted use. <em>It is expected and encouraged that you train your models outside of Kaggle’s notebooks environment and submit to Kaggle by uploading the trained model as an external data source.</em></p><p><strong>Public Validation Set:</strong> When you commit your Kaggle notebook, the submission file output that is generated will be based on the small set of 400 videos/ids contained within this Public Validation Set. This is available on the Kaggle Data page as test_videos.zip</p><p><strong>Public Test Set:</strong> This dataset is completely withheld and is what Kaggle’s platform computes the public leaderboard against. When you “Submit to Competition” from the “Output” file of a committed notebook that contains the competition’s dataset, your code will be re-run in the background against this Public Test Set. When the re-run is complete, the score will be posted to the public leaderboard. If the re-run fails, you will see an error reflected in your “My Submissions” page. Unfortunately, we are unable to surface any details about your error, so as to prevent error-probing. <em>You are limited to 2 submissions per day, including submissions with errors.</em></p><p><strong>Private Test Set:</strong> This dataset is privately held outside of Kaggle’s platform, and is used to compute the private leaderboard. It contains videos with a similar format and nature as the Training and Public Validation/Test Sets, but are real, organic videos with and without deepfakes. After the competition deadline, Kaggle transfers your 2 final selected submissions’ code to the host. They will re-run your code against this private dataset and return prediction submissions back to Kaggle for computing your final private leaderboard scores.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/df73bb60368b0fe87f78056988e9e9c0/href">https://medium.com/media/df73bb60368b0fe87f78056988e9e9c0/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/78eb7d2d545a0dca7de5211bb481734c/href">https://medium.com/media/78eb7d2d545a0dca7de5211bb481734c/href</a></iframe><h4>Review of Data Files Accessible within kernel</h4><ul><li><strong>train_sample_videos.zip</strong> — a ZIP file containing a sample set of training videos and a metadata.json with labels. the full set of training videos is available through the links provided above.</li><li><strong>sample_submission.csv</strong> — a sample submission file in the correct format.</li><li><strong>test_videos.zip</strong> — a zip file containing a small set of videos to be used as a public validation set. To understand the datasets available for this competition, review the Getting Started information.</li><li><strong>filename</strong> — the filename of the video</li><li><strong>label</strong> — whether the video is REAL or FAKE</li><li><strong>original</strong> — in the case that a train set video is FAKE, the original video is listed here</li><li><strong>split</strong> — this is always equal to “train”.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e9cb642730cb558ccf3184f11f207058/href">https://medium.com/media/e9cb642730cb558ccf3184f11f207058/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/01277620235c84dba8f4320686e6baa8/href">https://medium.com/media/01277620235c84dba8f4320686e6baa8/href</a></iframe><h4>Detection Starter Kit</h4><blockquote>A <strong>quickstart guide</strong> on DeepFakes: <a href="https://arxiv.org/abs/2001.00179">“DeepFakes and Beyond: A Survey of Face Manipulation and Fake Detection</a></blockquote><p>Here it follows a Deep Fakes video<strong> EDA</strong> (<em>Exploratory Data Analysis</em>). It relies on <a href="https://www.ffmpeg.org/download.html">static FFMPEG</a> to read/extract data from videos.</p><ul><li>It extracts meta-data. They help us to know frame rate, dimensions and audio format (we can forget leak of “display_ratio” as it will be fixed).</li><li>It extracts frames of videos as <em>PNG</em>.</li><li>It extracts audio track as <em>AAC </em>(disabled).</li><li>It compares a few face detectors (OpenCV HaarCascade, MTCNN). In the end we opted for <em>RetinaFace</em> indeed.</li><li>It provides basic statistics on faces per video, face width/height and face detection confidence. It computes an average face width/height.</li></ul><p>We notice that face detection (with OpenCV currently) is far from being perfect. An additional stage to clean-up detected faces is required before training a model! Maybe some kind of <em>votes/ensemble</em> with different detectors would help.</p><p>In this kernel you will see also some interesting edge cases of face detection:</p><ul><li>Face detected on a t-shirt.</li><li>Face detected on a background board.</li><li>Face detected inside a face.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5601802593b8ed8bf693153e68710861/href">https://medium.com/media/5601802593b8ed8bf693153e68710861/href</a></iframe><p>What is <a href="https://ffmpeg.org/ffprobe.html">ffprobe</a> indeed? Basically, ffprobe gathers information from multimedia streams and prints it in human — and machine — readable fashion.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3e7a662148455d414e4cc2ccbd7b2112/href">https://medium.com/media/3e7a662148455d414e4cc2ccbd7b2112/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2164a3e69ada0b61065ba134d5b749be/href">https://medium.com/media/2164a3e69ada0b61065ba134d5b749be/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/68170ebbd5f7fa8d872e7c9f112039ab/href">https://medium.com/media/68170ebbd5f7fa8d872e7c9f112039ab/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c01873e53f7090b38212a1e1879633f7/href">https://medium.com/media/c01873e53f7090b38212a1e1879633f7/href</a></iframe><p>A few info on <a href="https://filmora.wondershare.com/video-editing-tips/what-is-video-bitrate.html">bitrate</a></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e2fe6910267802509094cb90033caa90/href">https://medium.com/media/e2fe6910267802509094cb90033caa90/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0a427e7219c19b081cab98a8ce4297a9/href">https://medium.com/media/0a427e7219c19b081cab98a8ce4297a9/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/caf995a401162663824b4b318b8fc26d/href">https://medium.com/media/caf995a401162663824b4b318b8fc26d/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ce989fdd8bfafa829b3c5d37b61a2b85/href">https://medium.com/media/ce989fdd8bfafa829b3c5d37b61a2b85/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e538c35c12c9d9860a41102642da155a/href">https://medium.com/media/e538c35c12c9d9860a41102642da155a/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c84fe49ddd4f8fa6f69c1b455beb5608/href">https://medium.com/media/c84fe49ddd4f8fa6f69c1b455beb5608/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/54eca9ba1003fae52f671d7e7d8a600d/href">https://medium.com/media/54eca9ba1003fae52f671d7e7d8a600d/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/def9eb46f7a654f3659c611843f2e2b4/href">https://medium.com/media/def9eb46f7a654f3659c611843f2e2b4/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d0ace17cbd2b17f3dc9bd6135f4f4b69/href">https://medium.com/media/d0ace17cbd2b17f3dc9bd6135f4f4b69/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f021cfbe47b4582fcdcf1951d296ce2a/href">https://medium.com/media/f021cfbe47b4582fcdcf1951d296ce2a/href</a></iframe><p>Try to run the code by yourself, you can find a Jupyter notebook file on <a href="https://github.com/sazio/DeepFakeChallenge/blob/master/DeepFake_Overview.ipynb">my Github</a>. If you explore the same repository you’ll find the code we used for the whole pipeline, soon we’re going to convert it in a series of articles.</p><blockquote>Thank you for your attention, if you like our projects follow us on Medium and Github. Soon we’ll be out with more articles.</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a8f6fc5f6cfe" width="1" height="1" alt=""><hr><p><a href="https://medium.com/mljcunito/deepfake-detection-challenge-an-overview-a8f6fc5f6cfe">DeepFake Detection Challenge: an Overview</a> was originally published in <a href="https://medium.com/mljcunito">MLJC</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Second Step into Feature Engineering: Feature Selection]]></title>
            <link>https://medium.com/mljcunito/a-second-step-into-feature-engineering-feature-selection-d597619e6e2b?source=rss-2fefde940982------2</link>
            <guid isPermaLink="false">https://medium.com/p/d597619e6e2b</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[mljcunito]]></category>
            <dc:creator><![CDATA[Simone Azeglio]]></dc:creator>
            <pubDate>Fri, 21 Aug 2020 12:10:33 GMT</pubDate>
            <atom:updated>2020-08-21T15:56:55.915Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>We are ready to start with the second part of Feature Engineering (if you’ve missed the previous article, you can find it <a href="https://medium.com/mljcunito/an-introduction-to-feature-engineering-feature-importance-7e8265eb3a36">here</a>). In this short article we’ll go trough a few simple techniques in Feature Selection and Extraction.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XHHToil9E5EFeEh0H0rnjA.jpeg" /></figure><blockquote>Not all features are created equal</blockquote><blockquote>Zhe Chen</blockquote><h3>Feature Selection</h3><p>There would always be some features which are less important with respect to a specific problem. Those irrelevant features need to be removed. <em>Feature selection</em> addresses these problems by automatically selecting a subset that is most useful to the problem.</p><p>Most of the times the reduction in the number of input variables shrinks the computational cost of modeling, but sometimes it might happen that it also improves the performance of the model.</p><p>Among the large amount of feature selection methods we’ll focus mainly on statistical-based ones. They involve evaluating the relationship between each input variable and the target variable using statistics. These methods are usually fast and effective, the only issue is that statistical measures depends on the data type of both input and output variables.</p><p>The classes in the sklearn.feature_selection module can be used for feature selection/dimensionality reduction on sample sets.</p><p>Whenever you want to go for a simple approach, there’s always a <em>threshold</em> involved. VarianceThreshold is a simple baseline approach to select features. It removes all features whose variance doesn&#39;t reach a certain threshold.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4b812d22d663cc33cebca2bafd83c3fe/href">https://medium.com/media/4b812d22d663cc33cebca2bafd83c3fe/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/cef1c50d785388c8a3fbf16586f58320/href">https://medium.com/media/cef1c50d785388c8a3fbf16586f58320/href</a></iframe><h4>Univariate Feature Selection</h4><p>Univariate feature selection examines each feature individually to determine the strength of the relationship of the feature with the response variable.</p><p>There are a few different options for univariate selection:</p><p>We can perform chi-squared (𝝌²) test to the samples to retrieve only the two best features:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6c3ed19ec548eda19b1834fa6a028c1d/href">https://medium.com/media/6c3ed19ec548eda19b1834fa6a028c1d/href</a></iframe><p>We have different scoring functions for regression and classification, some of them are listed here:</p><ul><li>Regression: <a href="https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.f_regression.html#sklearn.feature_selection.f_regression">f_regression</a>, <a href="https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.mutual_info_regression.html#sklearn.feature_selection.mutual_info_regression">mutual_info_regression</a></li><li>Classification: <a href="https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.chi2.html#sklearn.feature_selection.chi2">chi2</a>, <a href="https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.f_classif.html#sklearn.feature_selection.f_classif">f_classif</a>, <a href="https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.mutual_info_classif.html#sklearn.feature_selection.mutual_info_classif">mutual_info_classif</a></li></ul><h4>Recursive Feature Elimination</h4><p>Recursive Feature Elimination (RFE) as its name suggests recursively removes features, builds a model using the remaining attributes and calculates model accuracy. RFE is able to work out the combination of attributes that contribute to the prediction on the target variable.</p><p>Given an external estimator that assigns weights to features (e.g., the coefficients of a linear model), the goal of recursive feature elimination (RFE) is to select features by recursively considering smaller and smaller sets of features.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/aeb79017aa3e8e5a968b6e25a2b5e41b/href">https://medium.com/media/aeb79017aa3e8e5a968b6e25a2b5e41b/href</a></iframe><h3>Feature Extraction (Bonus)</h3><blockquote>Feature extraction is very different from Feature selection: the former consists in transforming arbitrary data, such as text or images, into numerical features usable for machine learning. The latter is a machine learning technique applied on these features.</blockquote><p>We’ve decided to show you a standard technique from <em>sklearn.</em></p><h4>Loading Features from Dicts</h4><p>The class <a href="https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.DictVectorizer.html#sklearn.feature_extraction.DictVectorizer">DictVectorizer</a> transforms lists of feature-value mappings to vectors.</p><p>In particular, it turns lists of mappings (dict-like objects) of feature names to feature values into <em>Numpy</em> arrays or <em>scipy.sparse</em> matrices for use with scikit-learn estimators.</p><p>While not particularly fast to process, Python’s dict has the advantages of being convenient to use, being sparse (absent features need not be stored) and storing feature names in addition to values.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5bfb633d57a1329d3d3e932d9f988e1b/href">https://medium.com/media/5bfb633d57a1329d3d3e932d9f988e1b/href</a></iframe><p>DictVectorizer is also a useful representation transformation for training sequence classifiers in Natural Language Processing (NLP).</p><h4>Feature Hashing</h4><p>Named as one of the best hacks in Machine Learning, <a href="https://scikit-learn.org/stable/modules/feature_extraction.html#feature-hashing">Feature Hashing</a> is a fast and space-efficient way of vectorizing features, i.e. turning arbitrary features into indices in a vector or matrix. For this topic, sklearn’s documentation is exhaustive, you can find it in the link above.</p><h3>Feature Construction</h3><p>There’s no strict recipe for Feature Construction, I personally consider it as 99% creativity. We’re gonna take a look at some use cases in the next lectures though.</p><p>So far, you should take a look at the Feature Extraction part of t<a href="https://www.kaggle.com/gaborfodor/from-eda-to-the-top-lb-0-367">his marvellous notebook</a> from <em>Beluga,</em> one of the best Competitions Grandmasters on Kaggle.</p><p>In the last two articles we’ve been introducing Feature Engineering as a subsequent step to Feature Processing. As you can see, we’re building a data processing pipeline, indeed, the next step would be finding a way to deal with <em>missing values. </em>Stay tuned for the next article and don’t forget to take a look at our <a href="https://github.com/MLJCUnito/ProjectX2020">Github page</a>, you’ll find the code related to this series of articles.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d597619e6e2b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/mljcunito/a-second-step-into-feature-engineering-feature-selection-d597619e6e2b">A Second Step into Feature Engineering: Feature Selection</a> was originally published in <a href="https://medium.com/mljcunito">MLJC</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[An Introduction to Feature Engineering: Feature Importance]]></title>
            <link>https://medium.com/mljcunito/an-introduction-to-feature-engineering-feature-importance-7e8265eb3a36?source=rss-2fefde940982------2</link>
            <guid isPermaLink="false">https://medium.com/p/7e8265eb3a36</guid>
            <category><![CDATA[mljcunito]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Simone Azeglio]]></dc:creator>
            <pubDate>Tue, 18 Aug 2020 12:39:48 GMT</pubDate>
            <atom:updated>2020-08-18T12:39:48.930Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vWsOfiQ78s7f6HZ6wpLoHg.png" /><figcaption>Applying Feature Engineering techniques is like building a lego house</figcaption></figure><blockquote>After having read the <a href="https://medium.com/mljcunito/data-spectrometry-or-how-to-preprocess-your-data-2a6593ee5dbb">first article</a> of this series, devoted to figure out a way to tackle a Machine Learning (ML) competition, we are now ready to go on with Feature Engineering, and, in particular Feature Importance.</blockquote><p>The real deal is that nobody explicitly tells you what <strong>feature engineering</strong> is, in some way, you are expected to understand for yourself what are good features.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*aiC1GF0XWxz_bHQl.jpg" /></figure><blockquote>Feature engineering is another topic which doesn’t seem to merit any review papers or books, or even chapters in books, but it is absolutely vital to ML success. […] Much of the success of machine learning is actually success in engineering features that a learner can understand.</blockquote><blockquote>(Scott Locklin, in “Neglected machine learning ideas”)</blockquote><p>Let’s try to figure out what feature engineering is.</p><p>In solving such problems, our goal is to get the best possible result from a model. In order to achieve that, we need to extract useful information and get the most from what we have. On one side, this includes getting the best possible result from the algorithms we are employing. On the other side, it also involves getting the most out of the available data.</p><p><em>How do we get the most out of our data for predictive modeling?</em></p><p>Feature engineering tries to find an answer to this question.</p><blockquote>Actually, the success of all Machine Learning algorithms depends on how you present the data.</blockquote><blockquote>( Mohammad Pezeshki, answer to “What are some general tips on feature selection and engineering that every data scientist should know?”)</blockquote><h3>Feature Importance</h3><p>Feature importance refers to a bunch of techniques that assign a score to input features based on how useful they are at predicting a target variable. These scores play an important role in predictive modeling, they usually provide useful insights into the dataset and the basis for dimensionality reduction and feature selection.</p><p>Feature importance scores can be calculated both for regression and classification problems.</p><p>These scores can be used in a range of situations, such as:</p><ul><li><em>Better understanding the data</em>: the relative scorse can highlight which features may be most relevant to the target, and on the other side, which are least relevant. This could be a useful notion for a domain expert and could be used as a basis for gathering more or different data.</li><li><em>Better understanding a model</em>: inspecting the importance score provides insights into the specific model we’re using and which features are the most important to the model when elaborating a prediction.</li><li><em>Reducing the number of input features</em>: we can use the importance scores to select those features to delete (lowest score) and those to keep (highest scores).</li></ul><p>Now let’s jot down a few lines of code in order to grasp this topic in a better way. In order to explore feature importance scores, we’ll import a few test datasets directly from <em>sklearn</em>.</p><p><strong>Classification Dataset</strong></p><p>Easy peasy, we can use the <a href="https://scikit-learn.org/stable/modules/generated/sklearn.datasets.make_classification.html"><em>make_classification()</em></a> function to create a test binary classification dataset.</p><p>We can specify the number of samples and the number of features, some of them are going to be informative and the remaining redundant. (<em>Tip</em>: you should fix the <em>random seed</em>, in this way you’ll get a reproducible result)</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/75bdd5962b4c5cad6a0c80ce35dc1d1d/href">https://medium.com/media/75bdd5962b4c5cad6a0c80ce35dc1d1d/href</a></iframe><p><strong>Regression Dataset</strong></p><p>In a parallel fashion, we’ll use the <a href="https://machinelearningmastery.com/calculate-feature-importance-with-python/#:~:text=Feature%20importance%20refers%20to%20techniques,at%20predicting%20a%20target%20variable."><em>make_regression()</em></a> function to create a regression dataset.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f86af911813405717db1c9b7f33d7944/href">https://medium.com/media/f86af911813405717db1c9b7f33d7944/href</a></iframe><h3>Coefficients as Feature Importance</h3><p>When we think about linear machine learning algorithms, we always fit a model where the prediction is the weighted sum of the input values (e.g. linear regression, logistic regression, ridge regression etc..)</p><p>These coefficients can be used directly as naive feature importance scores. Firstly we’ll fit a model on the dataset to find the coefficients, then summarize the importance scores for each input feature and create a bar chart to get an idea of the relative importance.</p><p><strong>Linear Regression Feature Importance</strong></p><p>It’s time to fit a <a href="https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LinearRegression.html"><em>LinearRegression()</em></a> model on the regression dataset and get the <em>coef_</em> property that contains the coefficients. The only assumption is that the input variables have the same scale or have been scaled prior to fitting the model.</p><p>This same approach can be used with regularized linear models, such as Ridge and ElasticNet.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/df81365e34034cd55c21f064352eea0e/href">https://medium.com/media/df81365e34034cd55c21f064352eea0e/href</a></iframe><p><strong>Logistic Regression Feature Importance</strong></p><p>In a similar fashion, we can do the same to fit a <a href="https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html"><em>LogisticRegression()</em></a> model.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0f1633396526b4fdb77281282c91df1f/href">https://medium.com/media/0f1633396526b4fdb77281282c91df1f/href</a></iframe><p>Recall that this is a classification problem with classes 0 and 1 (binary). Notice that the coefficients are both positive and negative, positive scores indicate a feature that predicts class 1 while negative scores indicate a feature that predicts class 0.</p><p><em>Why can’t we analyze a regression problem with Logistic Regression?</em> (A pretty naive question, try to answer tho)</p><h3>Decision Tree Feature Importance</h3><p>Decision Tree algorithms like <strong>C</strong>lassification <strong>A</strong>nd <strong>R</strong>egression <strong>T</strong>rees ( <strong>CART</strong>) offer importance scores based on the reduction in the criterion used to select split points, like Gini or Entropy. This approach can be also used for ensembles of decision trees, such as Random Forest and Gradient Boositng algorithms.</p><p>We can directly use the CART algorithm for feature importance implemented in Scikit-Learn as the <a href="https://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeRegressor.html"><em>DecisionTreeRegressor</em></a> and <a href="https://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html"><em>DecisionTreeClassifier</em></a>.</p><p>The model provides a <em>feature_importances_</em> property that tells us the relative importance scores for each feature.</p><p><strong>CART Regression Feature Importance</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c71810588dce0e4dbff3fd9f6d755b9b/href">https://medium.com/media/c71810588dce0e4dbff3fd9f6d755b9b/href</a></iframe><p><strong>CART Classification Feature Importance</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e5d047de2f3fe3ded86d6ab9c9901c87/href">https://medium.com/media/e5d047de2f3fe3ded86d6ab9c9901c87/href</a></iframe><h3>Random Forest Feature Importance</h3><p>Analogously, we can use the RandomForest algorithm for feature importance implemented in scikit-learn as the <a href="https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestRegressor.html"><em>RandomForestRegressor</em></a> and <a href="https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html"><em>RandomForestClassifier</em></a>.</p><p>As above, the model provides a <em>feature_importances_</em> property.</p><p><strong>Random Forest Regression Feature Importance</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5da269773457f8b117bae6d556eb92e3/href">https://medium.com/media/5da269773457f8b117bae6d556eb92e3/href</a></iframe><p><strong>Random Forest Classification Feature Importance</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5019cbcdcd1bfc105da1ddfd308e6996/href">https://medium.com/media/5019cbcdcd1bfc105da1ddfd308e6996/href</a></iframe><h3>XGBoost Feature Importance</h3><p><a href="https://xgboost.readthedocs.io/en/latest/index.html">XGBoost</a> is a Python library that provides an efficient implementation of the stochastic gradient boostig algorithm. (For an introduction to Boosted Trees, you can take a look <a href="https://xgboost.readthedocs.io/en/latest/tutorials/model.html">here</a>)</p><p>This algorithm can be integrated with Scikit-Learn via the <em>XGBRegressor</em> and <em>XGBClassifier</em> classes.</p><p>Even in this one, we can find the <em>feature_importances_</em> property.</p><p>First, let’s install the XGBoost library, with pip:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1336fea87c083ebf21d576ad8f96c3a4/href">https://medium.com/media/1336fea87c083ebf21d576ad8f96c3a4/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7a74f4e38e89f963ecfe192ac4799142/href">https://medium.com/media/7a74f4e38e89f963ecfe192ac4799142/href</a></iframe><p>Now, let’s take a look at an example of XGBoost for feature importance.</p><p><strong>XGBoost Regression Feature Importance</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/af8d9ca335d5f64dd637f9ac3410656a/href">https://medium.com/media/af8d9ca335d5f64dd637f9ac3410656a/href</a></iframe><p><strong>XGBoost Classification Feature Importance</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4a1d111fdf88e43f8c604bfabfa51262/href">https://medium.com/media/4a1d111fdf88e43f8c604bfabfa51262/href</a></iframe><p>Permutation feature importance is a technique for calculating relative importance scores that is independent of the model used. It measures the increase in the prediction error of the model after we permuted the feature’s values, which breaks the relationship between the feature and the true outcome.</p><p>The concept is really straightforward: we measure the importance of a feature by calculating the increase in the model’s prediction error after permuting the feature. A feature is “important” if shuffling its values increases the model error, because in this case the model relied on the feature for the prediction. A feature is “unimportant” if shuffling its values leaves the model error unchanged, because in this case the model ignored the feature for the prediction.</p><p>Permutation feature selection can be used via the <a href="https://scikit-learn.org/stable/modules/generated/sklearn.inspection.permutation_importance.html"><em>permutation_importance()</em></a> function that take a fit model, a dataset and a scoring function.</p><p>Let’s try this approach with an algorithm that doesn’t support feature selection natively, <a href="https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm">KNN</a> (K-Nearest Neighbors).</p><p><strong>Permutation Feature Importance for Regression</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/cae12aeb1112fce6ae8f851043726846/href">https://medium.com/media/cae12aeb1112fce6ae8f851043726846/href</a></iframe><p><strong>Permutation Feature Importance for Classification</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/fda780618f72afab51796e4a631282ff/href">https://medium.com/media/fda780618f72afab51796e4a631282ff/href</a></iframe><p><strong>Feature Selection with Importance</strong></p><p>Feature importance scores can be used to find useful insights and interpret the data, but they can also be used directly to help rank and select features that are most useful. This procedure is usually referred as <em>Feature Selection</em>, and we’ll look at it in more detail soon.</p><p>In our case, we can show how is possible to find redundant features by using the previously shown techniques.</p><p>Firstly, we can split the dataset into train and test sets, train a model on the training set, make predictions on the test set and evaluate the results by employing classification accuracy. We’ll use a Logistic Regression model to fit our data.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a986db71a6027559185ed21fe7ab7c8a/href">https://medium.com/media/a986db71a6027559185ed21fe7ab7c8a/href</a></iframe><p>In this case, we can see that our model achieved a classification accuracy of about <em>86.67 %</em> using all the features in the dataset.</p><p>Let’s see what happens if we select only relevant features. We could use any of the feature importance scores above, but in this case we’ll use the ones provided by random forest.</p><p>We can use the <a href="https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.SelectFromModel.html"><em>SelectFromModel</em></a> class to define both the model abd the number of features to select.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/981d44799a94f1e73f28baac02caa7a4/href">https://medium.com/media/981d44799a94f1e73f28baac02caa7a4/href</a></iframe><p>This will calculate the importance scores that can be used to rank all input features. We can then apply the method as a transform to select a subset of 5 most important features from the dataset. This transform will be applied to the training set and the test set.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/72b40c515c12d82e2da56c550049add6/href">https://medium.com/media/72b40c515c12d82e2da56c550049add6/href</a></iframe><p>We can wrap up every piece and get this code snippet.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/838c5aecb5fb274c66fcede8571ecac1/href">https://medium.com/media/838c5aecb5fb274c66fcede8571ecac1/href</a></iframe><p>In this case, we can see that the model achieves the same performance on the dataset, although with almost half of the features.</p><p>It’s time to press the pause button for now, you should take a look at the code above and try to tweak some parameters, you might find better results too. In the next article we’ll cover Feature Selection and Feature Extraction, stay tuned! 👩‍💻👨‍💻</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7e8265eb3a36" width="1" height="1" alt=""><hr><p><a href="https://medium.com/mljcunito/an-introduction-to-feature-engineering-feature-importance-7e8265eb3a36">An Introduction to Feature Engineering: Feature Importance</a> was originally published in <a href="https://medium.com/mljcunito">MLJC</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Spectrometry or How to Preprocess your Data]]></title>
            <link>https://medium.com/mljcunito/data-spectrometry-or-how-to-preprocess-your-data-2a6593ee5dbb?source=rss-2fefde940982------2</link>
            <guid isPermaLink="false">https://medium.com/p/2a6593ee5dbb</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[mljcunito]]></category>
            <category><![CDATA[projectx2020]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Simone Azeglio]]></dc:creator>
            <pubDate>Fri, 31 Jul 2020 15:09:01 GMT</pubDate>
            <atom:updated>2020-07-31T16:58:06.303Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rzyaIB4v1udby1uxloxl8g.jpeg" /><figcaption>Chad Hagen, Nonsensical Infographic №5, 2009, digital</figcaption></figure><blockquote>This is the first article of a to-be series devoted to figure out a way to tackle a Machine Learning (ML) competition. A few months ago, the MLJCUnito gained access to an international ML competition on climate change, organized by the University of Toronto: <a href="http://www.projectx2020.com/">www.projectx2020.com</a>. This competition will start on September 2020, so we decided to start gathering some useful material related to competition dynamics such as Data Preprocessing, Feature Engineering and convert them in a lecture-like format.</blockquote><p>In this first part, we’d like to tell you about some practical tricks for making <strong>gradient descent</strong> work well, in particular, we’re going to delve into feature scaling. As an introductory view, it seems reasonable to try to depict an intuition of the concept of <em>scale</em>.</p><h3>Macro, Meso, Micro-scale in Science</h3><p>As scientists, we are well aware of the effects of using a specific measurement tool in order to characterize some quantity and describe reality. As an ideal example we consider the <strong>length scale</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*mxtbVvVB3v5NTbbp.png" /><figcaption>Images of two systems (one screw and one boot) observed from the macroscopic (left), mesoscopic (center) and microscopic (right) point of view.</figcaption></figure><p>We can identify three different points of view: <em>microscopic</em>, <em>mesoscopic</em> and <em>macroscopic</em>; which are intimately related to the adopted lenght scale.</p><p>We usually deal with the <em>macroscopic scale</em> when the observer is in such a position (pretty far, in terms of distance), with respect to the object, that she/he can describe its global characteristics. Instead, we do refer to the <em>microscopic scale</em> when the observer is so close to the object that she/he can describe its atomistic details or elementary parts (e.g. molecules, atoms, quarks). Last but not least, we talk about <em>mesoscopic scale</em> everytime we are in between micro and macro.</p><p>These definitions are deliberately vague, since delineating a precise and neat explanation would be higly difficult and complex, and it’s actually far from our purposes.</p><p>On the other side, this kind of introduction is quite useful, we should take a few minutes to think about the “active” role of the observer and about the fact that, to be honest, for every length scale, there’s some specific theory, i.e. there’s no global theory for a multi-scale description of some phenomenon.</p><h3>Scaling in Data Science</h3><p>If our beloved observer (i.e. the scientist) has some kind of “privilege”, i.e. choosing the right measurement tool, which is nothing but choosing the right scale in the description of some phenomenon, we can’t really say the same for a data scientist.</p><p>It’s a sort of paradox, but a data scientist can’t really deal with data retrieval most of the times. Because of that, a data scientist is often left alone in front of data, without even knowing from which measurement tool they’re coming from. There’s no way to interact with the length scale for example.</p><p>Is there something that we can do about it? The only thing we can do is assuming that features are independent and scale these features in order to have something compatible from one to the other. This procedure is called <strong>feature scaling</strong>, and soon we’ll understand why it is useful even for ML algorithms, such as gradient descent.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*Cn7w5qo3PAd3om6t.png" /></figure><p>If you make sure that features are on similar scales, i.e. features take on similar range of values, then gradient descent can converge more quickly.</p><p>More concretely, let’s say we have a problem with two features where <strong><em>x</em></strong>₁ is the length of a football field and take values between <em>90</em> (meters) and <em>115</em> (meters) and <strong><em>x</em></strong>₂ is the radius of a ball which takes values between <em>10.5</em>(centimeters) to <em>11.5</em> (centimeters). If you plot the countours of the cost function <strong><em>J(ω)</em></strong> then you might get something similar to the <em>left plot</em>, and because of these very skewed elliptical shape, if we run gradient descent on this cost function, it may end up taking a long time and oscillating back and forth before reaching the global minimum.</p><p>In these settings, as stated previously, a useful thing to do is to scale the features. Generally, the idea is to get every feature into approximately a <em>-1</em> to <em>+1</em> range. By doing this, we get the <em>right plot</em>. In this way, you can find a much more direct path to the global minimum rather than taking a much more convoluted path where you’re sort of trying to follow a very complicated trajectory.</p><h3>Preprocessing Data</h3><blockquote>In any Machine Learning process, Data Preprocessing is that step in which the data gets transformed, or encoded, to bring it to such a state that now the machine can easily parse it. In other words, the features of the data can now be easily interpreted by the algorithm.</blockquote><p>We’re going to dive into Scikit-Learn for this section and exploit its powerful <em>processing</em> package.</p><p>We’ve been talking about scaling our data, now it’s time to understand how to put our hands on code and try to do that. Usually, as previously stated, learning algorithms benefit from standardization of the data set. If some outliers are present in the set, robust scalers or transformers might be more appropriate. (Take a look at <a href="https://scikit-learn.org/stable/auto_examples/preprocessing/plot_all_scaling.html#sphx-glr-auto-examples-preprocessing-plot-all-scaling-py">Compare the effect of different scalers on data with outliers</a>, you’ll see the behaviors of the different scalers, transformers and normalizers with outliers).</p><h3>Standardization</h3><p>Many Machine Learning estimators require <em>standardization</em> of datasets, elseways they might behave badly because data are far from a Gaussian (with zero mean and unit variance) distribution.</p><p>Most of the times, we ignore the shape of the distribution and just transform the data by subtracting the mean value of each feature, then scale by dividing features by their standard deviation.</p><p><em>Do you have in mind some models that assume that all features are centered around zero and have variance in the same order of magnitude? Can you think about possible issues related to the objective function in these cases?</em></p><p>A possible answer: many elements used in the objective function of a learning algorithm (such as the <em>RBF kernel</em> of <em>Support Vector Machines</em> or the <em>l1</em> and <em>l2</em> regularizers of linear models) assume that all features are centered around zero and have variance in the same order. If a feature has a variance that is orders of magnitude larger than others, it might dominate the objective function and make the estimator unable to learn from other features correctly as expected.</p><p>There’s a fast way to do that on a single array, by means of the <em>scale</em> function.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a57908529c2d0afe95fc3862c96e6e17/href">https://medium.com/media/a57908529c2d0afe95fc3862c96e6e17/href</a></iframe><p>The <em>preprocessing</em> module provides a utility class <a href="https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html#sklearn.preprocessing.StandardScaler"><em>StandardScaler</em></a> that compute the mean and std on a training set so as to be able to later reapply the same transform on the test set.</p><p>(You should be well aware of what <a href="https://scikit-learn.org/stable/modules/generated/sklearn.pipeline.Pipeline.html#sklearn.pipeline.Pipeline"><em>sklearn.pipeline.Pipeline</em></a> is, it’s crucial for strategies’ deployment.)</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/efa617fc0cce6c44730e9ca53f3e3449/href">https://medium.com/media/efa617fc0cce6c44730e9ca53f3e3449/href</a></iframe><p>Now we can use the scaler instance on new data, to transform them in the same way we did previously.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/906716a194713981fed547fc336dcffe/href">https://medium.com/media/906716a194713981fed547fc336dcffe/href</a></iframe><p>It is possible to disable centering or scaling by passing <em>with_mean = False</em> or <em>with_std = False</em>. The first one might be particularly useful if applied to sparse CSR or CSC matrices to avoid breaking the sparsity structure of the data.</p><p><strong>Scaling Features to a Range</strong></p><p>Another standardization is scaling features to lie between a given minimum and maximum value, or so that the maximum absolute value of each future is scaled to unit size. This can be achieved with <a href="https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.MinMaxScaler.html#sklearn.preprocessing.MinMaxScaler"><em>MinMaxScaler</em></a> or <a href="https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.MaxAbsScaler.html#sklearn.preprocessing.MaxAbsScaler"><em>MaxAbsScaler</em></a>.</p><p>Here you can see how to scale a toy data matrix to the <em>[0,1]</em> range:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1c5015f0213577f079ab72630b8117e2/href">https://medium.com/media/1c5015f0213577f079ab72630b8117e2/href</a></iframe><p>In the same way as above, the same instance of the transformer can be applied to some new test data: same scaling and shifting will be applied to be consistent.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5cf12eb830e0141ef6ad211b5f98459f/href">https://medium.com/media/5cf12eb830e0141ef6ad211b5f98459f/href</a></iframe><p>It’s pretty useful to let the scaler reveal some details about the transformation learned on the training data:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/bc007def251c302a93863374ce2d4579/href">https://medium.com/media/bc007def251c302a93863374ce2d4579/href</a></iframe><p>Can you retrieve the explicit formula for <em>MinMaxScaler</em>?</p><p>Here’s the solution</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qycZhu6uSTaACblZjut9hg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hDdvlr-A_4qXkGJPtYOR2w.png" /></figure><p><em>MaxAbsScaler</em> works in a similar fashion, the data will lie in the range <em>[-1,1]</em>. It is meant for data that is already centered at zero or sparse data.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/742401f9a3d96e2fba65fa6b873ac75d/href">https://medium.com/media/742401f9a3d96e2fba65fa6b873ac75d/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/964419f1ce87775b20caad2a77d94c03/href">https://medium.com/media/964419f1ce87775b20caad2a77d94c03/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c1071852dafdb8ecdcb1144bf119557c/href">https://medium.com/media/c1071852dafdb8ecdcb1144bf119557c/href</a></iframe><p><strong>Scaling Data with Outliers</strong></p><p>If our data contain many outliers, scaling using the mean and variance of the data is not likely to work well. In this case, we can use <a href="https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.RobustScaler.html#sklearn.preprocessing.RobustScaler"><em>RobustScaler</em></a>.</p><p>This scaler removes the <a href="https://en.wikipedia.org/wiki/Median">median</a> and scales data according to the <a href="https://en.wikipedia.org/wiki/Interquartile_range">IQR</a> (InterQuartile Range).</p><p>Centering and scaling happen independently on each feature by computing the relevant statistics on the samples in the training set.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/71a3857832f1a1a19297f4acbafe49e3/href">https://medium.com/media/71a3857832f1a1a19297f4acbafe49e3/href</a></iframe><p>Median and interquartile range are then stored to be used on later data using the <em>transform</em> method.</p><h3>Non-linear transformations</h3><p>It’s possible to generalize to non-linear transformations. We are going to talk about two types of transformations: <em>quantile transforms</em> and <em>power transforms</em>. The main take-home message is that we need <em>monotonic</em> transformations to preserve the rank of the values along each feature.</p><p>Quantile transforms smooth out unusual distributions and are less influenced by outliers than scaling methods. It does distort correlations and distances within and across features.</p><p>Power transforms are, indeed, a family of parametric transformations that aim to map data from any distribution to as close to a Gaussian distribution.</p><p><strong>Mapping to a Uniform Distribution</strong></p><p><a href="https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.QuantileTransformer.html#sklearn.preprocessing.QuantileTransformer"><em>QuantileTransformer</em></a> provides a non-parametric transformation to map the data to a uniform distribution with values between 0 and 1:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/aca7f04a650ffcd79e3ee36c6278fefd/href">https://medium.com/media/aca7f04a650ffcd79e3ee36c6278fefd/href</a></iframe><p>This feature corresponds to the sepal length in cm. Once the quantile transform is applied, those landmarks approach closely the percentiles previously defined</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/52f8511822b70b5be3531e8780d9b098/href">https://medium.com/media/52f8511822b70b5be3531e8780d9b098/href</a></iframe><p>Some more applications <a href="https://machinelearningmastery.com/quantile-transforms-for-machine-learning/">here</a>.</p><p><strong>Mapping to a Gaussian Distribution</strong></p><p>Many machine learning algorithms prefer or perform better when numerical input variables and even output variables in the case of regression have a Gaussian distribution. Power transforms are a family of parametric, monotonic transforms that aim to map data from any distribution to as close to a Gaussian distribution as possible, in order to stabilize variance and minimize skewness.</p><p><a href="https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.PowerTransformer.html#sklearn.preprocessing.PowerTransformer"><em>PowerTransformer</em></a> provides two transformations, the <em>Yeo-Johnson</em> transform:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*o49kyDielDx0_pS-s5FZWg.png" /></figure><p>and the <em>Box-Cox</em> transform:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/957/1*vVNA5FCtIaOKgyWp6yqlvQ.png" /></figure><p>Box-Cox can only be applied to strictly positive data. In both methods, the transformation is parametrized by $\lambda$, which is determined trough maximum-likelihood estimation. Here an example of using Box-Cox to map samples drawn from a lognormal distribution to a normal distribution:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/25251388abc112069683e254bd5a7cac/href">https://medium.com/media/25251388abc112069683e254bd5a7cac/href</a></iframe><p>(Some more applications <a href="https://machinelearningmastery.com/power-transforms-with-scikit-learn/">here</a>)</p><p>Below some examples of the two transforms applied to various probability distributions, <em>any comment</em>?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/0*lIAgUh67_Lgm7yqh.png" /></figure><h3>Normalization</h3><p>As scientists, we feel much more comfortable with Vector Space Models. <em>Normalization</em> is the process of scaling individual samples to have unit norm. This process might be useful if we plan to use a dot-product or some kernel to quantify similarities of pairs of samples.</p><p>The function provides a quick and easy way to perform this operation on a single array, using <a href="https://medium.com/@montjoile/l0-norm-l1-norm-l2-norm-l-infinity-norm-7a7d18a4f40c">L1 or L2 norms</a>:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b5e53bd3abc3aed11d09a953d0cced92/href">https://medium.com/media/b5e53bd3abc3aed11d09a953d0cced92/href</a></iframe><p>The <em>preprocessing</em> module provides a utility class <em>Normalizer</em> that implements the same operation using the <em>Transformer</em> API. This class is suitable for <em>sklearn.pipeline.Pipeline</em></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a11e4c4e21a93e472f491dfbf801d628/href">https://medium.com/media/a11e4c4e21a93e472f491dfbf801d628/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e206595051d49b3dd8689c38216b3572/href">https://medium.com/media/e206595051d49b3dd8689c38216b3572/href</a></iframe><h3>Encoding Categorical Features</h3><p>In many cases, features are not continous values but categorical. E.g. a person could have some features: [&quot;from Italy&quot;, &quot;from France&quot;, &quot;from Germany&quot;], [&quot;play sports&quot;, &quot;doesn&#39;t play sports&quot;], [&quot;uses Firefox&quot;, &quot;uses Opera&quot;, &quot;uses Chrome&quot;, &quot;uses Safari&quot;, &quot;uses Internet Explorer&quot;].</p><p>Such features can be efficiently coded as integers, for instance [&quot;from France&quot;, &quot;play sports&quot;, &quot;uses Chrome&quot;] could be [1,0,2].</p><p>To convert categorical features to such integer codes, we can use the <a href="https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.OrdinalEncoder.html#sklearn.preprocessing.OrdinalEncoder"><em>OrdinalEncoder</em></a>. In this way, we transform each feature to one new feature of integers <em>(0</em> to <em>n_categories-1)</em>:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c2a76a62e1c9d2d1517ebab7449d0efe/href">https://medium.com/media/c2a76a62e1c9d2d1517ebab7449d0efe/href</a></iframe><p>Some scikit-learn estimators expect continuous input, and would interpret categories as ordered, which is usually not desired.</p><p>There’s another way to convert categorical features to features that can be used with scikit-learn estimators: <em>one-hot encoding</em>. It can be obtained with the <a href="https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.OneHotEncoder.html#sklearn.preprocessing.OneHotEncoder"><em>OneHotEncoder</em></a>, which transforms each categorical feature with n_categoriespossible values into n_categories binary features, with one of them 1, and all others 0.</p><p>Let’s continue with the example above:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1d397649917632b6bde9c1e4d9bef4ba/href">https://medium.com/media/1d397649917632b6bde9c1e4d9bef4ba/href</a></iframe><p>The values each feature can take is inferred automatically from the dataset and can be found in the categories_ attribute:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f892af8f037a20b98896215f268de983/href">https://medium.com/media/f892af8f037a20b98896215f268de983/href</a></iframe><p>We are done with a brief introduction to Data Preprocessing. I’ve tried to show some of the most useful features of <em>scikit-learn, </em>which is definitely one of the cardinal libraries for ML. In the next lecture we are going to dive into Feature Engineering, which I personally consider the most fundamental part of a ML pipeline. Stay tuned for the next article !👩‍💻👨‍💻</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2a6593ee5dbb" width="1" height="1" alt=""><hr><p><a href="https://medium.com/mljcunito/data-spectrometry-or-how-to-preprocess-your-data-2a6593ee5dbb">Data Spectrometry or How to Preprocess your Data</a> was originally published in <a href="https://medium.com/mljcunito">MLJC</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Learning Problem: Comparison between Brain and Machine]]></title>
            <link>https://medium.com/mljcunito/the-learning-problem-comparison-between-brain-and-machine-1e52213a1a63?source=rss-2fefde940982------2</link>
            <guid isPermaLink="false">https://medium.com/p/1e52213a1a63</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[neuroscience]]></category>
            <dc:creator><![CDATA[Simone Azeglio]]></dc:creator>
            <pubDate>Tue, 24 Sep 2019 20:03:29 GMT</pubDate>
            <atom:updated>2019-09-24T20:07:36.618Z</atom:updated>
            <content:encoded><![CDATA[<h4>Chapter 1: The Neuron &amp; the Perceptron</h4><figure><img alt="A mixture of an Artificial Neural Network and the Brain Connectome" src="https://cdn-images-1.medium.com/max/1024/1*7rjzz-2nBK7vhOwF8qSkCw.jpeg" /><figcaption>source on the left: <a href="https://www.extremetech.com/wp-content/uploads/2016/01/connectome.jpg">https://www.extremetech.com/wp-content/uploads/2016/01/connectome.jpg</a> ; on the right: <a href="https://clusterdata.nl/wp-content/uploads/2018/01/maxresdefault-1.jpg">https://clusterdata.nl/wp-content/uploads/2018/01/maxresdefault-1.jpg</a>— modified by me</figcaption></figure><p><strong>This is the first chapter of a series devoted to analyze and decompose the learning problem in its very bases. In doing that I would like to stress the comparison between the biological and the artificial realm. Particularly, as a Physicist, I’m interested in understanding Emergence and Self-Organization phenomena, which are topics typically related to Complex Systems (e.g. the Brain or a Deep Neural Network). The central idea will thus be: showing how well-understood solutions (e.g. a single neuron) can be applied in a bottom-up manner in order to understand complex situations. With the purpose of building a solid structure for tackling this problem we’d need two tools: Mathematical Modeling and Simulation (a.k.a. Coding). I’ll go trough all of this since the very beginning, let’s get it started!</strong></p><blockquote><em>If you can’t solve a problem, then there is an easier problem you can solve: find it</em>. (<strong>George Pólya, Mathematical Discovery on Understanding, Learning, and Teaching Problem Solving)</strong></blockquote><p>Physics has been traditionally faithful to this idea: complex situations are preferentially studied on top of simple solutions. Even on the more abstract level, where issues pertaining to the general nature of constraints on cognitive processes are discussed, a physicist is likely to feel that natural language, for example, is much too complicated a subject-matter for a starting point. He is likely to try to construct a structure of increasing complexity consisting of definite realizations of simple processes possessing cognitive flavor. <em>One</em> of the main criteria for the selection of these stages is their analizability. Their properties can be studied in a non-abstract manner, avoiding mysterious conclusions which are brought by the blurriness of complexity.</p><p>In this way, in order to understand the learning problem, we should deal with its basic units. In particular we’re going to compare <em>neurons</em> and <em>perceptrons</em>. Let’s go on with the first one.</p><h4>Neurons: biological candidates for understanding complexity [1]</h4><p>A short description of the biological background is called for, even though it wouldn’t be possible, for a long time to come, to overcome <a href="https://neurology.mhmedical.com/book.aspx?bookid=1049">Eric Kandel’s</a> description of the neural sciences (you can check, therefore, this outstanding text for a general overview of the field). Only those features which are essential for the construction of the model will be summarized here.</p><p>The basic elements are, naturally, <em>neurons</em> and <em>synapses</em>. There is a fairly large variety of types of neurons in the human nervous system — variations are found in size, in structure and in function. As a choice of context, for the present purposes, we will consider a “canonical” type of neuron. If the underlying principles depend on the structure of individual neurons, it is unlikely that physics will contribute much to their clarification. Beyond a certain level, complex functions must be a result of the interaction of large numbers of simple elements.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OAupKc8QH6G36LDzKE8-9A.jpeg" /><figcaption>Fig.1A source: <a href="https://www.forbes.com/sites/andreamorris/2018/08/27/scientists-discover-a-new-type-of-brain-cell-in-humans/">https://www.forbes.com/sites/andreamorris/2018/08/27/scientists-discover-a-new-type-of-brain-cell-in-humans/</a> — modified by myself</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/834/1*MzlrxSlV-ryQDAJtOVoBLQ.jpeg" /><figcaption>Fig.1B source : <a href="https://ib.bioninja.com.au/standard-level/topic-6-human-physiology/65-neurons-and-synapses/neurons.html">https://ib.bioninja.com.au/standard-level/topic-6-human-physiology/65-neurons-and-synapses/neurons.html</a></figcaption></figure><p>A neuron is depicted in Figure1, alongside its schematic representation. The neurons communicate via synapses, which are the points along the <em>axon</em> of the <em>pre-synaptic </em>neuron at which it can communicate the outcome of the computation that has been performed in its <em>soma</em> to the <em>dendrites</em> or even directly to the soma of the <em>post-synaptic</em> neuron. The output part is the axon. Usually only one axon leaves the soma and then, downstream, it branches repeatedly, to communicate with many post-synaptic neurons.</p><p>Now, from our perspective, the <em>dynamics</em> of neurons and synapses, which is very similar to the propagation of an electric signal in a cable, is based on the following sequence:</p><ul><li>The neural axon is in an all-or-none state. In the first state it propagates a signal — <em>spike</em>, or <em>action potential — </em>based on the result of the summation, performed in the soma, of signals coming from dendrites. The shape and amplitude of the propagating signal is very stable and is replicated at branching points in the axon. Furthermore, the presence of a traveling impulse in the axon blocks the possibility of a second impulse transmission.</li><li>When the traveling signal reaches the endings of the axon it causes the secretion of <em>neuro-transmitters — </em>our<em> </em>harbingers<em> — </em>into the synapse extremity.</li><li>The neuro-transmitters arrive, across the synapse, at the post-synaptic neuron membrane and bind to <em>receptors</em>, thus causing the latter to open up and allow for the penetration of ionic current.</li><li>The post-synaptic potential (<strong>PSP</strong>) diffuses in a graded manner towards the soma where the inputs from all the pre-synaptic neurons connected to the post-synaptic one are summed. If the total sum of the PSP’s arriving within a short period surpasses a certain<em> threshold</em>, the probability for the emission of a spike becomes significant.</li></ul><p>Following the event of the emission of a spike, the neuron needs time to recover. We are going to name this amount of time as <em>absolute refractory period</em>, in which the neuron cannot emit a second spike (in this way nature sets a maximal spike frequency, <em>de facto</em> limiting the amount of information that a neuron can process in a fixed amount of time).</p><p>The previous description indicates that the only way neurons can communicate the outcome of their computations to other neurons is trough the emission of neurotransmitters. For the sake of mathematical modeling we shall make a strong assumption, namely that: <strong>sub-treshold potentials do not lead to the release of neurotransmitters</strong>. In other words, neurotransmitters are released by spikes only.</p><p>Now that we have a basic introduction of neurons functionalities we can take a look at a mathematical model, i.e. the<em> Leaky Integrate-and-Fire</em> (<strong>LIF</strong>) neuron. For the ones of you who are interested in the “heavy” math behind this model, I’m going to briefly introduce it.</p><h4>Leaky Integrate-and-Fire model, a mathematical perspective [2]</h4><p>The canonical way in order to deal with neuronal models is the <em>circuit analogy</em>. This is pretty technical, but I’ll try to keep it as low level as possible. For the ones of you who have taken an <em>Electronics</em> <em>101</em> it’ll just be a quick recap of a simple 𝑅𝐶 circuit.</p><p>We’ve been previously talking about the summation (sometimes referred as <em>integration</em>) process that happens in the soma, which is, combined with the mechanism that triggers action potentials above some critical threshold, the very core of neuronal dynamics.</p><p>Let’s now dive into math so that to build a phenomenological model of neuronal dynamics. We describe the critical voltage as a formal <em>threshold</em> 𝜽. If the<em> voltage</em> 𝑢(𝑡) (the sum of all inputs) reaches 𝜽 from below, we say that the neuron fires a spike. In this model we have two different components that are both necessary to define the dynamics: first, an equation that describes the evolution of the potential 𝑢(𝑡); and second, a mechanism to generate spikes.</p><p>The following is the simplest model in the class of integrate-and-fire models is made up of two ingredients: a linear differential equation to describe the evolution of 𝑢(𝑡) and a threshold for spike firing.</p><p>The variable 𝑢(𝑡) describes the instant value of the potential of our neuron. In the absence of any input, the potential is at its resting state 𝑣 . If the neuron receives an input (a <em>current</em>) 𝐼(𝑡), the potential 𝑢(𝑡) will be deflected from its resting value.</p><p>In order to arrive at an equation that links the momentary voltage 𝑢(𝑡) — 𝑣 to the input current 𝐼(𝑡), we use elementary laws from the theory of electricity. If a current pulse 𝐼(𝑡) is injected into the neuron, the additional electrical charge will charge the cell membrane. The cell membrane will therefore act as a <em>capacitor</em> of <em>capacity</em> 𝐶. The charge will slowly leak through the cell membrane since this ladder is not a perfect insulator. We can take this into account by adding a finite leak <em>resistance</em> 𝑅 to our model.</p><p>The basic electrical circuit representing a leaky integrate-and-fire model consists of a capacitor 𝐶 <em>in parallel </em>with a resistor 𝑅 driven by a current 𝐼(𝑡); see Figure 2</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/890/1*JvFPiUaBjkaNgPbIbuoMFw.png" /><figcaption>Figure2 — source : <a href="https://neuronaldynamics.epfl.ch/online/Ch1.S3.html">https://neuronaldynamics.epfl.ch/online/Ch1.S3.html</a> . On the left: a neuron, which is enclosed by the cell membrane (big circle), receives a (positive) input current 𝐼(𝑡) which increases the electrical charge inside the cell. The corresponding circuit is depicted at the bottom. On the right: the cell membrane reaction to a step current (top) with a smooth voltage signal (bottom)</figcaption></figure><p>With the purpose of analyzing the circuit, we use the <em>law of current conservation</em> and split the current into two components:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/152/1*sgPELoaX6or6EhbDexCCVw@2x.png" /></figure><p>The first component is the current which passes through the linear resistor 𝑅 and it can be calculated from <em>Ohm’s law</em>. The second component charges the capacitor 𝐶. Thus, by using <em>Ohm’s law</em> and the <em>current-voltage relation</em> for capacitors, we get:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/269/1*7x6bwRLR4ML5bqQzJBCNUg@2x.png" /></figure><p>Luckily, this is a linear differential equation in 𝑢(𝑡) and it is easy to solve, especially if we consider a constant input current 𝐼(𝑡) = 𝑖 which starts at 𝑡 = 0 and ends at time 𝑡 = 𝚫. For the sake of simplicity we assume that the membrane potential at time 𝑡 = 0 is at its resting value 𝑢(0) = 𝑣.</p><p>The solution for 0&lt;𝑡&lt;𝚫 is, thus:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/316/1*Enr-764r8vXOso3i8iIaQg@2x.png" /></figure><p>If the input current never stopped, the potential would approach for 𝑡→ ∞ to the asymptotic value 𝑢(∞)=𝑣 +𝑅𝑖. We can understand this result by taking a look at Figure2 right-side bottom. Once a plateau is reached, the charge on the capacitor no longer changes. All input current must then flow through the resistor. Additionally, for notation purposes we usually denote <a href="https://en.wikipedia.org/wiki/RC_time_constant"><em>RC</em> as 𝜏</a>, phisically the time constant of our circuit. Now that we have introduced our ingredients it’s time to start baking some code.</p><h4>Let’s code a Leaky Integrate-and-Fire neuron</h4><p>We’re going to use <a href="https://brian2.readthedocs.io/en/stable/index.html"><em>Brian2</em></a>, a very efficient <em>Python</em> library for simulating spiking neural networks (i.e. biological neural networks), which you can easily install by following the official <a href="https://brian2.readthedocs.io/en/stable/introduction/install.html">instructions</a>.</p><p>I often use <a href="https://jupyter.org/"><em>Jupyter notebooks</em></a><em> </em>in order to run <em>Python</em> scripts: it’s an interactive environment that let you code and integrate <em>Markdown </em>and a wide range of useful plugins specifically designed for scientific computing and machine learning.</p><p>Once you’re ready with the installation, you can start by importing <em>Brian2.</em></p><pre>import brian2 as br2</pre><p>After that, we’re going to define some global parameters of our model: <em>N</em> is the number of neurons; <em>tau </em>is the circuit time constant, previosly defined ( i.e. 𝜏 = <em>RC ); v_r </em>is the resting membrane potential (i.e. 𝑣); <em>I_c</em> is the constant input current (i.e. 𝑖); <em>v_th</em> is the spike critical voltage (i.e. threshold 𝜗)</p><pre>N = 1</pre><pre>tau  = 10 *br2.ms<br>v_r = 0 *br2.mV<br>I_c = 18 * br2.mV</pre><pre>#v_th is a string, it&#39;s going to be clear in a while<br>v_th = &quot;v &gt; 15*mV&quot;</pre><p>You should note that in defining the variables we’ve set physical units as well.</p><p>Consequently, we can sketch out the dynamics: <em>Brian2</em> let you write the equations that describe your model, you have to specify units as well (I know the current is apparently measured in<em> Volt </em>here — it’s a kick in the teeth, but everything is done for the sake of congruence)</p><pre>eqs = &#39;&#39;&#39;<br>dv/dt = -(v-I)/tau : volt<br>I : volt<br>&#39;&#39;&#39;</pre><p>By calling <a href="https://brian2.readthedocs.io/en/stable/reference/brian2.groups.neurongroup.NeuronGroup.html?highlight=NeuronGroup"><em>NeuronGroup</em></a> we’re creating a group of Neurons (here we have <em>1 </em>neuron) with the previously defined dynamics, <em>v_r </em>and<em> I_c</em> as the characteristics of our neuron. In order to record its activity we call <a href="https://brian2.readthedocs.io/en/stable/reference/brian2.monitors.statemonitor.StateMonitor.html?highlight=StateMonitor"><em>StateMonitor</em></a>, <em>v_trace</em> can be used to extract time and voltage values as we’ll soon see.</p><pre>lif = br2.NeuronGroup(N, model = eqs, threshold =v_th, reset = &#39;v = 0*mV&#39;)<br>lif.v = v_r   <br>lif.I = I_c</pre><pre>v_trace = br2.StateMonitor(lif, &#39;v&#39;, record = True)</pre><p>The last step consists in running the simulation and plotting the result. As previously said, we extract time from <em>v_trace</em> as our x-axis and voltage (<em>v[0] </em>refers to the first — and only — neuron). You have to be careful with physical units — again — and remember that for plotting we need a dimensional data.</p><pre>br2.run(0.1*br2.second)</pre><pre>br2.figure(1)<br>br2.plot(v_trace.t[:]/br2.ms, v_trace.v[0]/br2.mV)<br>br2.xlabel(&#39;Time (ms)&#39;, fontsize = 24)<br>br2.ylabel(&#39;v (mV)&#39;, fontsize = 24)<br>br2.yticks([0,4,8,12,16])<br> <br>br2.show()</pre><p>The result is thus a sequence of spikes. You should play with parameters in order to see different kinds of behaviour.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*Cnny-_CcfQElvCaTlWmgLQ.png" /></figure><p>We’ve described a fairly biological model of spiking neurons, but what’s the transition from this realm to the artificial one? Let’s talk about <em>perceptrons.</em></p><h4>Perceptrons: artificial candidates for understanding complexity [3] [4]</h4><p>By continuing with the complex systems’ paradigm, from Minsky’s and Papert’s book:</p><blockquote>Although we do not have an equally elaborated theory of ‘learning’, we can at least demonstrate that in cases where ‘learning’ or ‘adaption’ or ‘self-organization’ does occur, its occurrence can be thoroughly elucidated and carries no suggestion of mysterious little-understood principles of complex systems. Whether there are such principles we cannot know. But the perceptron provides no evidence; and our success in analyzing it adds another piece of circumstantial evidence for the thesis that cybernetic processes that work can be understood, and those that cannot be understood are suspect.</blockquote><p>As hinted by the quotation above and strongly advocated by <a href="https://www.csee.umbc.edu/courses/471/papers/turing.pdf">Turing</a>, mental phenomena are nothing but an expression of a very complex structure operating on relatively simple processes. This is the very first thought that brought to the formalization of the brain.</p><p>The perceptron is, in this way, the very first brick of Artificial Intelligence. We are going to formalize the biological neuron,<em> de facto</em> getting the perceptron and after that introduce <em>Rosenblatt’s perceptron learning algorithm.</em></p><p>We now focus on the <em>logical </em>structure of a single neuron. The description of the previous sections suggests the following scheme:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/382/1*ZFlzROot9R3M89WGCRI0vg.png" /><figcaption>Figure 3</figcaption></figure><ul><li>There is a <em>processing unit</em>, the large circle , which represents the soma.</li><li>A number of input lines connect, <em>logically</em>, to the soma, depicted with incoming arrows in Figure3. They represent dendrites and synapses.</li><li>The input channels are activated by the signals they receive from the input variables (<strong><em>x</em></strong>’s) to which they are connected. These variables are our pre-synaptic axons, and they have an intrinsic <em>logical</em> nature, since they can either activate the channel (carry a spike) or not activate it (sub-treshold activity in the pre-synaptic neuron).</li></ul><p>To each input line, we associate a parameter<em> w</em> — the subscript refers to the various input channels. The numerical value of each <em>w</em> is indeed the amount of post-synaptic potential (<strong>PSP</strong>) that would be added to the soma if the channel were activated.</p><p>Furthermore, there is a single <em>logical</em> output line (outgoing arrow in Fig3). It expresses the logical fact that our neuron produces a single relevant ouput — a spike .</p><p>We can arrange the operations of the unit in the following way:</p><ul><li>At a given moment, some of the logical inputs are activated.</li><li>The soma receives an input which is the linear sum of <strong>PSP</strong> values of the channels that were activated — the variable <em>x</em> indicates whether the channel is active (<em>x = 1</em>) or inactive (<em>x = -1</em>)</li><li>The sum of <strong>PSP</strong>’s is compared to the threshold value of the neuron and the output channel is activated if it overcomes the threshold.</li></ul><p>Formally, we’ll get:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/130/1*e8iI2MGv8PU2yBLBbAGOOg@2x.png" /></figure><p>with <em>h</em> as the PSP at our neuron and <em>n</em> as the number of pre-synaptic neurons. Mathematically <em>h</em> is nothing but a <a href="https://en.wikipedia.org/wiki/Dot_product"><em>dot product</em></a>.</p><p>Anyways, the operation implemented by our little “machine” can be expressed as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/165/1*CXKtLDLRx30UOf5wLBBuYg@2x.png" /></figure><p>with <em>sign</em> as the <a href="https://en.wikipedia.org/wiki/Sign_function">sign function</a> and 𝜽 as the threshold of our neuron. <em>H</em> is basically <em>1</em> if its argument (<em>h+</em>𝜽) is positive and -1 if negative — and, to be precise, <em>0</em> if <em>h +</em>𝜽 = 0. The variable <em>y</em> indicates, in our biological analogy, whether a spike will appear in the output axon.</p><p>Now that our perceptron has taken shape we can talk about how this mathematical structure can be involved in the learning process. Let’s introduce <em>Rosenblatt’s perceptron learning algorithm.</em></p><h4>The learning process in perceptrons [4]</h4><p>There’s a clear analogy between neurons and perceptrons, but how can we use this ladder model in order to learn ? We’re going to show that the perceptron can be used to solve <em>classification</em> problems, namely it can tell you whether, <em>if we have two sets of points</em>, a point belong to one set or another. We can say without a lack of generalizability that the problem can be thought as a binary (yes/no) decision, where the variable <em>y </em>(the <em>output value</em>) defined above is one of these two binary values ( <em>yes = 1</em>, <em>no = -1</em>) .</p><p>From the previous definition of <em>h</em> as a dot product it’s even clearer that <strong><em>x</em></strong> is formally a vector (the <em>input vector</em>). Every component of <strong><em>x </em></strong>is called <em>feature, </em>by talking in Machine Learning terms. In the same way <strong><em>w</em></strong> is a vector (the <em>weight</em> <em>vector</em>). The threshold term 𝜽 it’s usually called <em>bias term</em>. In the direction of a simplified notation we’re going to <em>augment</em> both <strong><em>x </em></strong>and<strong><em> w</em></strong> in the following way:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/321/1*CXeNkMeP5mLJ2eHx60Lukg@2x.png" /></figure><p>Basically, by extending the sum from <em>i = 0 </em>to <em>i = n</em>.</p><p>The learning algorithm we’re going to build will consider all these terms and in particular the term <em>learning</em> specifically refers on finding the best value of <strong><em>w</em></strong>’s components in order to achieve the best classification possible. Take a look at the following<em> 2-dimensional</em> (here <strong><em>x</em></strong> has only two components) problem in Figure4 and everything will be clear.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1006/1*yF4wXHHEb1zrZLy8KXUryQ.png" /><figcaption>Figure 4 — On the left: misclassified data. On the right: perfectly classified data.</figcaption></figure><p>So, in practice, classification means finding a way in order to separate the two different “kinds” of data we have. Each “kind” of data is specified by a different <em>label</em> (plus or minus, in the figure above). So to speak, this is a <em>two labels </em>classification problem.</p><p>It’s time to introduce the <em>perceptron learning algorithm</em> (<strong>PLA</strong>), which will determine what <strong><em>w</em></strong> should be, based on the data. This learning algorithm consists in a simple <em>iterative </em>method. By involving the term “iteration”, we’re going to introduce a new parameter in our model, namely, <em>time</em>. At iteration <em>t</em>, where <em>t</em> <em>= 0,1,2</em>…, there is a current value of the <em>weight vector</em>, call it <strong><em>w</em></strong><em>(t). </em>The algorithm picks a point, associated with its label, that is currently misclassified, call it (<strong>x</strong><em>(t)</em>, <em>y(t)</em>), and uses it to update <strong><em>w</em></strong><em>(t). </em>Since the example is misclassified, we have <em>y(t) ≠ sign(</em><strong><em>w</em></strong><em>(t) </em><strong><em>· x</em></strong><em>(t)), </em>where the dot represents the dot product.</p><p>The <em>weight update rule</em> is as follow:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/277/1*P7wg-Gqpf00Rni3rIPoTjA@2x.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1006/1*JpApwtU4mYTOQNdIwiSC-g.png" /><figcaption>Figure 5 — The last update</figcaption></figure><p>This rule moves the boundary in the direction to classify <strong><em>x</em></strong><em>(t)</em> correctly, as depicted in the figure above. The algorithm continues with further iterations until there are no longer misclassified points in the data set. With this intuitive view of the <strong>PLA</strong>, it’s now time to code it.</p><h4>Let’s code the Perceptron Learning Algorithm</h4><p>We’re going to implement the <strong>PLA</strong> from scratch by coding a simple <em>Python</em> script. Let’s open a new notebook with <em>Jupyter</em> and start by importing <em>matplotlib.pyplot</em> and <em>numpy</em></p><pre>import matplotlib.pyplot as plt<br>import numpy as np</pre><p>We’re going to create a 2-dimensional random dataset of <em>5</em> points (with two possible labels, <em>-1</em> and <em>+1</em>) by using <a href="https://docs.scipy.org/doc/numpy-1.15.1/reference/generated/numpy.random.rand.html"><em>numpy.random.rand(</em></a><em>).</em> In order to reproduce the result we’ll fix the random seed — you can actually choose every integer number, <a href="https://en.wikipedia.org/wiki/137_(number)#In_physics"><em>137</em> has a particular meaning for Physicists</a> though.</p><pre># Setting the random seed <br>np.random.seed(seed = 137)</pre><pre># Generate x1 and x2, coordinates of our points<br>number_of_points = 5<br>x1 = np.random.rand(number_of_points)<br>x2 = np.random.rand(number_of_points)</pre><pre># We have two labels, namely -1 and 1<br>possible_ys = np.array([-1,1])</pre><pre># We randomly build the label y to point (x1,x2) association <br>y = np.random.choice(possible_ys, number_of_points)</pre><p>Data are represented by triplets of values.</p><pre># We create data as triplets of values<br>data = []<br>for i in range(number_of_points):<br>    data.append((x1[i],x2[i],y[i]))</pre><p>You can take a look at your data by executing the following line and you’ll get:</p><pre># Taking a look at data<br>data</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/820/1*W4RkEoavMx58zuSOYV0yWA.png" /></figure><p>The next step is to plot them by denoting with a “-” the label corresponding to “-1” and by “+” the label corresponding to “1”</p><pre># Plotting our data <br>plt.plot([x1 for (x1,x2,y) in data if y==-1], [x2 for (x1,x2,y) in data if y==-1], &#39;_&#39;, mec=&#39;r&#39;, mfc=&#39;none&#39;)<br>plt.plot([x1 for (x1,x2,y) in data if y==1], [x2 for (x1,x2,y) in data if y==1], &#39;+&#39;, mec=&#39;b&#39;, mfc=&#39;none&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8_R1z2kQdioF7OQjHVBxFg.png" /></figure><p>After that we can start with the learning model. The very first thing to do is to initialize the weights, usually a “little” random value is the best solution for faster convergence.</p><pre># Initializing the weight vector<br>w = np.random.rand(3)*10e-03</pre><p>Consequently, we define the function “<em>predict”</em> which is nothing but <em>sign(</em><strong><em>w</em></strong><em>(t) </em><strong><em>· x</em></strong><em>(t))</em></p><pre>def predict(x1,x2):<br>    # w[0] is the threshold value, x0 = 1 <br>    h= w[0] + w[1]*x1 + w[2]*x2<br>    <br>    if h&lt;0:<br>        return -1<br>    else:<br>        return 1</pre><p>The last part of the learning model is the function “fit” which corresponds to the <em>weight update rule</em>. The essence is: for every point we compare the predicted label with the actual one, if these are not equal each component of the <em>weight vector</em> is updated.</p><pre>def fit(data):<br>    stop = False<br>    <br>    while stop == False:<br>        stop = True<br>        <br>        for x1,x2,y in data:        <br>            ypredict= predict(x1,x2) <br>            if y != ypredict: <br>                              <br>                stop = False<br>            <br>                w[1]= w[1] + x1*y<br>                w[2]= w[2] + x2*y<br>                w[0]= w[0] + y</pre><p>By applying the function “<em>fit</em>” to our data, the <strong>PLA</strong> will converge to a solution. If this is the case we’ll get a printed ‘SUCCESS!’ as the output of the following cell.</p><pre>fit(data)</pre><pre># Check if the model is predicting correct labels<br>for (x1, x2, y) in data:<br>    if predict(x1,x2) != y:<br>        print(&#39;FAIL&#39;)<br>        break<br>else:<br>    print(&#39;SUCCESS!&#39;)</pre><p>Last but not least, we can plot our result. “<em>f(x)</em>” is the functional form of the line given by the <strong>PLA </strong>(you can get it by applying simple algebraic calculations — <em>de facto</em> by expliciting the expression for <em>x2</em>).</p><pre>def f(x):<br>    return -(w[0] + w[1]*x)/w[2]</pre><pre>d = range(0,2)<br>plt.plot(d, [f(x) for x in d])</pre><pre>plt.plot([x1 for (x1,x2,y) in data if y==-1], [x2 for (x1,x2,y) in data if y==-1], &#39;_&#39;, mec=&#39;r&#39;, mfc=&#39;none&#39;)<br>plt.plot([x1 for (x1,x2,y) in data if y==1], [x2 for (x1,x2,y) in data if y==1], &#39;+&#39;, mec=&#39;b&#39;, mfc=&#39;none&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Y3WU7pfjyymHq-V3dozBlw.png" /></figure><p>In this way, we’ve solved a simple classification problem, i.e. we’ve been learning!</p><h4>Conclusions</h4><p>We’ve reached the end of our adventure: we’ve been traveling trough a mathematical model of the neuron and its implementation. With that in mind,we’ve been taking inspiration from it in order to formalize the neuron and build the perceptron. Be careful though, the perceptron is not a biologically plausible model. The relationship between neurons and pereptrons is much more similar to the one between birds and planes. Here, the problem of “flying” is actually the <em>learning problem</em>, but again, the road for a perfectly reliable plane it’s way longer than this.</p><p>In particular perceptrons have an <em>intrinsic problem, </em>which is insurmountable without a major change. Minsky and Papert have been pointing out that issue so clearly that research in Artificial Intelligence has been subjected to a quite long break. Therefore, I would like you to point out this serious perceptron’s limitation. I think that you can grasp an intuition of the problem by looking at the code in the previous section and tweaking <em>np.random.seed() </em>and<em> number_of_points. </em>Give it a try and let me know!</p><h4>References</h4><p>[1] Amit, D. (1989). <em>Modeling Brain Function: The World of Attractor Neural Networks</em>. Cambridge: Cambridge University Press.</p><p>[2] Wulfram Gerstner, Werner M. Kistler, Richard Naud, and Liam Paninski. 2014. Neuronal Dynamics: From Single Neurons to Networks and Models of Cognition. Cambridge University Press, New York, NY, USA. <em>Online version: </em><a href="https://neuronaldynamics.epfl.ch/online/index.html">https://neuronaldynamics.epfl.ch/online/index.html</a></p><p>[3] Marvin L. Minsky and Seymour A. Papert. 1988. Perceptrons: Expanded Edition. MIT Press, Cambridge, MA, USA.</p><p>[4]Rosenblatt, Frank. 1962. Principles of neurodynamics; perceptrons and the theory of brain mechanisms. Washington, Spartan Books</p><p>[5] Yaser S. Abu-Mostafa, Malik Magdon-Ismail, and Hsuan-Tien Lin. 2012. Learning from Data. AMLBook.</p><blockquote>Do you like our project? Your feedbacks are important to us! You can support in many ways, such as: by contributing to our contents or by submitting a small donation in form of a coffee down here👇</blockquote><figure><a href="https://www.buymeacoffee.com/m3LJoOtRw"><img alt="" src="https://cdn-images-1.medium.com/max/170/1*bk_C8Um34KavCkO2yzMCZg.png" /></a></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1e52213a1a63" width="1" height="1" alt=""><hr><p><a href="https://medium.com/mljcunito/the-learning-problem-comparison-between-brain-and-machine-1e52213a1a63">The Learning Problem: Comparison between Brain and Machine</a> was originally published in <a href="https://medium.com/mljcunito">MLJC</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Learning Problem: Comparison between Brain and Machine]]></title>
            <link>https://medium.com/analytics-vidhya/the-learning-problem-comparison-between-brain-and-machine-339c8de6635a?source=rss-2fefde940982------2</link>
            <guid isPermaLink="false">https://medium.com/p/339c8de6635a</guid>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[neuroscience]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Simone Azeglio]]></dc:creator>
            <pubDate>Tue, 24 Sep 2019 11:46:23 GMT</pubDate>
            <atom:updated>2019-09-24T11:58:46.389Z</atom:updated>
            <content:encoded><![CDATA[<h4>Chapter 1: The Neuron &amp; the Perceptron</h4><figure><img alt="A mixture of an Artificial Neural Network and the Brain Connectome" src="https://cdn-images-1.medium.com/max/1024/1*7rjzz-2nBK7vhOwF8qSkCw.jpeg" /><figcaption>source on the left: <a href="https://www.extremetech.com/wp-content/uploads/2016/01/connectome.jpg">https://www.extremetech.com/wp-content/uploads/2016/01/connectome.jpg</a> ; on the right: <a href="https://clusterdata.nl/wp-content/uploads/2018/01/maxresdefault-1.jpg">https://clusterdata.nl/wp-content/uploads/2018/01/maxresdefault-1.jpg</a>— modified by me</figcaption></figure><p><strong>This is the first chapter of a series devoted to analyze and decompose the learning problem in its very bases. In doing that I would like to stress the comparison between the biological and the artificial realm. Particularly, as a Physicist, I’m interested in understanding Emergence and Self-Organization phenomena, which are topics typically related to Complex Systems (e.g. the Brain or a Deep Neural Network). The central idea will thus be: showing how well-understood solutions (e.g. a single neuron) can be applied in a bottom-up manner in order to understand complex situations. With the purpose of building a solid structure for tackling this problem we’d need two tools: Mathematical Modeling and Simulation (a.k.a. Coding). I’ll go trough all of this since the very beginning, let’s get it started!</strong></p><blockquote><em>If you can’t solve a problem, then there is an easier problem you can solve: find it</em>. (<strong>George Pólya, Mathematical Discovery on Understanding, Learning, and Teaching Problem Solving)</strong></blockquote><p>Physics has been traditionally faithful to this idea: complex situations are preferentially studied on top of simple solutions. Even on the more abstract level, where issues pertaining to the general nature of constraints on cognitive processes are discussed, a physicist is likely to feel that natural language, for example, is much too complicated a subject-matter for a starting point. He is likely to try to construct a structure of increasing complexity consisting of definite realizations of simple processes possessing cognitive flavor. <em>One</em> of the main criteria for the selection of these stages is their analizability. Their properties can be studied in a non-abstract manner, avoiding mysterious conclusions which are brought by the blurriness of complexity.</p><p>In this way, in order to understand the learning problem, we should deal with its basic units. In particular we’re going to compare <em>neurons</em> and <em>perceptrons</em>. Let’s go on with the first one.</p><h4>Neurons: biological candidates for understanding complexity [1]</h4><p>A short description of the biological background is called for, even though it wouldn’t be possible, for a long time to come, to overcome <a href="https://neurology.mhmedical.com/book.aspx?bookid=1049">Eric Kandel’s</a> description of the neural sciences (you can check, therefore, this outstanding text for a general overview of the field). Only those features which are essential for the construction of the model will be summarized here.</p><p>The basic elements are, naturally, <em>neurons</em> and <em>synapses</em>. There is a fairly large variety of types of neurons in the human nervous system — variations are found in size, in structure and in function. As a choice of context, for the present purposes, we will consider a “canonical” type of neuron. If the underlying principles depend on the structure of individual neurons, it is unlikely that physics will contribute much to their clarification. Beyond a certain level, complex functions must be a result of the interaction of large numbers of simple elements.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OAupKc8QH6G36LDzKE8-9A.jpeg" /><figcaption>Fig.1A source: <a href="https://www.forbes.com/sites/andreamorris/2018/08/27/scientists-discover-a-new-type-of-brain-cell-in-humans/">https://www.forbes.com/sites/andreamorris/2018/08/27/scientists-discover-a-new-type-of-brain-cell-in-humans/</a> — modified by myself</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/834/1*MzlrxSlV-ryQDAJtOVoBLQ.jpeg" /><figcaption>Fig.1B source : <a href="https://ib.bioninja.com.au/standard-level/topic-6-human-physiology/65-neurons-and-synapses/neurons.html">https://ib.bioninja.com.au/standard-level/topic-6-human-physiology/65-neurons-and-synapses/neurons.html</a></figcaption></figure><p>A neuron is depicted in Figure1, alongside its schematic representation. The neurons communicate via synapses, which are the points along the <em>axon</em> of the <em>pre-synaptic </em>neuron at which it can communicate the outcome of the computation that has been performed in its <em>soma</em> to the <em>dendrites</em> or even directly to the soma of the <em>post-synaptic</em> neuron. The output part is the axon. Usually only one axon leaves the soma and then, downstream, it branches repeatedly, to communicate with many post-synaptic neurons.</p><p>Now, from our perspective, the <em>dynamics</em> of neurons and synapses, which is very similar to the propagation of an electric signal in a cable, is based on the following sequence:</p><ul><li>The neural axon is in an all-or-none state. In the first state it propagates a signal — <em>spike</em>, or <em>action potential —</em> based on the result of the summation, performed in the soma, of signals coming from dendrites. The shape and amplitude of the propagating signal is very stable and is replicated at branching points in the axon. Furthermore, the presence of a traveling impulse in the axon blocks the possibility of a second impulse transmission.</li><li>When the traveling signal reaches the endings of the axon it causes the secretion of <em>neuro-transmitters — </em>our<em> </em>harbingers<em>— </em>into the synapse extremity.</li><li>The neuro-transmitters arrive, across the synapse, at the post-synaptic neuron membrane and bind to <em>receptors</em>, thus causing the latter to open up and allow for the penetration of ionic current.</li><li>The post-synaptic potential (<strong>PSP</strong>) diffuses in a graded manner towards the soma where the inputs from all the pre-synaptic neurons connected to the post-synaptic one are summed. If the total sum of the PSP’s arriving within a short period surpasses a certain<em> threshold</em>, the probability for the emission of a spike becomes significant.</li></ul><p>Following the event of the emission of a spike, the neuron needs time to recover. We are going to name this amount of time as <em>absolute refractory period</em>, in which the neuron cannot emit a second spike (in this way nature sets a maximal spike frequency, <em>de facto</em> limiting the amount of information that a neuron can process in a fixed amount of time).</p><p>The previous description indicates that the only way neurons can communicate the outcome of their computations to other neurons is trough the emission of neurotransmitters. For the sake of mathematical modeling we shall make a strong assumption, namely that: <strong>sub-treshold potentials do not lead to the release of neurotransmitters</strong>. In other words, neurotransmitters are released by spikes only.</p><p>Now that we have a basic introduction of neurons functionalities we can take a look at a mathematical model, i.e. the<em> Leaky Integrate-and-Fire</em> (<strong>LIF</strong>) neuron. For the ones of you who are interested in the “heavy” math behind this model, I’m going to briefly introduce it.</p><h4>Leaky Integrate-and-Fire model, a mathematical perspective [2]</h4><p>The canonical way in order to deal with neuronal models is the <em>circuit analogy</em>. This is pretty technical, but I’ll try to keep it as low level as possible. For the ones of you who have taken an <em>Electronics</em> <em>101</em> it’ll just be a quick recap of a simple 𝑅𝐶 circuit.</p><p>We’ve been previously talking about the summation (sometimes referred as <em>integration</em>) process that happens in the soma, which is, combined with the mechanism that triggers action potentials above some critical threshold, the very core of neuronal dynamics.</p><p>Let’s now dive into math so that to build a phenomenological model of neuronal dynamics. We describe the critical voltage as a formal <em>threshold</em> 𝜽. If the<em> voltage</em> 𝑢(𝑡) (the sum of all inputs) reaches 𝜽 from below, we say that the neuron fires a spike. In this model we have two different components that are both necessary to define the dynamics: first, an equation that describes the evolution of the potential 𝑢(𝑡); and second, a mechanism to generate spikes.</p><p>The following is the simplest model in the class of integrate-and-fire models is made up of two ingredients: a linear differential equation to describe the evolution of 𝑢(𝑡) and a threshold for spike firing.</p><p>The variable 𝑢(𝑡) describes the instant value of the potential of our neuron. In the absence of any input, the potential is at its resting state 𝑣 . If the neuron receives an input (a <em>current</em>) 𝐼(𝑡), the potential 𝑢(𝑡) will be deflected from its resting value.</p><p>In order to arrive at an equation that links the momentary voltage 𝑢(𝑡) - 𝑣 to the input current 𝐼(𝑡), we use elementary laws from the theory of electricity. If a current pulse 𝐼(𝑡) is injected into the neuron, the additional electrical charge will charge the cell membrane. The cell membrane will therefore act as a <em>capacitor</em> of <em>capacity</em> 𝐶. The charge will slowly leak through the cell membrane since this ladder is not a perfect insulator. We can take this into account by adding a finite leak <em>resistance</em> 𝑅 to our model.</p><p>The basic electrical circuit representing a leaky integrate-and-fire model consists of a capacitor 𝐶 <em>in parallel </em>with a resistor 𝑅 driven by a current 𝐼(𝑡); see Figure 2</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/890/1*JvFPiUaBjkaNgPbIbuoMFw.png" /><figcaption>Figure2 — source : <a href="https://neuronaldynamics.epfl.ch/online/Ch1.S3.html">https://neuronaldynamics.epfl.ch/online/Ch1.S3.html</a> . On the left: a neuron, which is enclosed by the cell membrane (big circle), receives a (positive) input current 𝐼(𝑡) which increases the electrical charge inside the cell. The corresponding circuit is depicted at the bottom. On the right: the cell membrane reaction to a step current (top) with a smooth voltage signal (bottom)</figcaption></figure><p>With the purpose of analyzing the circuit, we use the <em>law of current conservation</em> and split the current into two components:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/152/1*sgPELoaX6or6EhbDexCCVw@2x.png" /></figure><p>The first component is the current which passes through the linear resistor 𝑅 and it can be calculated from <em>Ohm’s law</em>. The second component charges the capacitor 𝐶. Thus, by using <em>Ohm’s law</em> and the <em>current-voltage relation</em> for capacitors, we get:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/269/1*7x6bwRLR4ML5bqQzJBCNUg@2x.png" /></figure><p>Luckily, this is a linear differential equation in 𝑢(𝑡) and it is easy to solve, especially if we consider a constant input current 𝐼(𝑡) = 𝑖 which starts at 𝑡 = 0 and ends at time 𝑡 = 𝚫. For the sake of simplicity we assume that the membrane potential at time 𝑡 = 0 is at its resting value 𝑢(0) = 𝑣.</p><p>The solution for 0&lt;𝑡&lt;𝚫 is, thus:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/316/1*Enr-764r8vXOso3i8iIaQg@2x.png" /></figure><p>If the input current never stopped, the potential would approach for 𝑡→ ∞ to the asymptotic value 𝑢(∞)=𝑣 +𝑅𝑖. We can understand this result by taking a look at Figure2 right-side bottom. Once a plateau is reached, the charge on the capacitor no longer changes. All input current must then flow through the resistor. Additionally, for notation purposes we usually denote <a href="https://en.wikipedia.org/wiki/RC_time_constant"><em>RC</em> as 𝜏</a>, phisically the time constant of our circuit. Now that we have introduced our ingredients it’s time to start baking some code.</p><h4>Let’s code a Leaky Integrate-and-Fire neuron</h4><p>We’re going to use <a href="https://brian2.readthedocs.io/en/stable/index.html"><em>Brian2</em></a>, a very efficient <em>Python</em> library for simulating spiking neural networks (i.e. biological neural networks), which you can easily install by following the official <a href="https://brian2.readthedocs.io/en/stable/introduction/install.html">instructions</a>.</p><p>I often use <a href="https://jupyter.org/"><em>Jupyter notebooks</em></a><em> </em>in order to run <em>Python</em> scripts: it’s an interactive environment that let you code and integrate <em>Markdown </em>and a wide range of useful plugins specifically designed for scientific computing and machine learning.</p><p>Once you’re ready with the installation, you can start by importing <em>Brian2.</em></p><pre>import brian2 as br2</pre><p>After that, we’re going to define some global parameters of our model: <em>N</em> is the number of neurons; <em>tau </em>is the circuit time constant, previosly defined ( i.e. 𝜏 = <em>RC ); v_r </em>is the resting membrane potential (i.e. 𝑣); <em>I_c</em> is the constant input current (i.e. 𝑖); <em>v_th</em> is the spike critical voltage (i.e. threshold 𝜗)</p><pre>N = 1</pre><pre>tau  = 10 *br2.ms<br>v_r = 0 *br2.mV<br>I_c = 18 * br2.mV</pre><pre>#v_th is a string, it&#39;s going to be clear in a while<br>v_th = &quot;v &gt; 15*mV&quot;</pre><p>You should note that in defining the variables we’ve set physical units as well.</p><p>Consequently, we can sketch out the dynamics: <em>Brian2</em> let you write the equations that describe your model, you have to specify units as well (I know the current is apparently measured in<em> Volt </em>here — it’s a kick in the teeth, but everything is done for the sake of congruence)</p><pre>eqs = &#39;&#39;&#39;<br>dv/dt = -(v-I)/tau : volt<br>I : volt<br>&#39;&#39;&#39;</pre><p>By calling <a href="https://brian2.readthedocs.io/en/stable/reference/brian2.groups.neurongroup.NeuronGroup.html?highlight=NeuronGroup"><em>NeuronGroup</em></a> we’re creating a group of Neurons (here we have <em>1 </em>neuron) with the previously defined dynamics, <em>v_r </em>and<em> I_c</em> as the characteristics of our neuron. In order to record its activity we call <a href="https://brian2.readthedocs.io/en/stable/reference/brian2.monitors.statemonitor.StateMonitor.html?highlight=StateMonitor"><em>StateMonitor</em></a>, <em>v_trace</em> can be used to extract time and voltage values as we’ll soon see.</p><pre>lif = br2.NeuronGroup(N, model = eqs, threshold =v_th, reset = &#39;v = 0*mV&#39;)<br>lif.v = v_r   <br>lif.I = I_c</pre><pre>v_trace = br2.StateMonitor(lif, &#39;v&#39;, record = True)</pre><p>The last step consists in running the simulation and plotting the result. As previously said, we extract time from <em>v_trace</em> as our x-axis and voltage (<em>v[0] </em>refers to the first — and only — neuron). You have to be careful with physical units — again — and remember that for plotting we need a dimensional data.</p><pre>br2.run(0.1*br2.second)</pre><pre>br2.figure(1)<br>br2.plot(v_trace.t[:]/br2.ms, v_trace.v[0]/br2.mV)<br>br2.xlabel(&#39;Time (ms)&#39;, fontsize = 24)<br>br2.ylabel(&#39;v (mV)&#39;, fontsize = 24)<br>br2.yticks([0,4,8,12,16])<br> <br>br2.show()</pre><p>The result is thus a sequence of spikes. You should play with parameters in order to see different kinds of behaviour.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*Cnny-_CcfQElvCaTlWmgLQ.png" /></figure><p>We’ve described a fairly biological model of spiking neurons, but what’s the transition from this realm to the artificial one? Let’s talk about <em>perceptrons.</em></p><h4>Perceptrons: artificial candidates for understanding complexity [3] [4]</h4><p>By continuing with the complex systems&#39; paradigm, from Minsky’s and Papert’s book:</p><blockquote>Although we do not have an equally elaborated theory of ‘learning’, we can at least demonstrate that in cases where ‘learning’ or ‘adaption’ or ‘self-organization’ does occur, its occurrence can be thoroughly elucidated and carries no suggestion of mysterious little-understood principles of complex systems. Whether there are such principles we cannot know. But the perceptron provides no evidence; and our success in analyzing it adds another piece of circumstantial evidence for the thesis that cybernetic processes that work can be understood, and those that cannot be understood are suspect.</blockquote><p>As hinted by the quotation above and strongly advocated by <a href="https://www.csee.umbc.edu/courses/471/papers/turing.pdf">Turing</a>, mental phenomena are nothing but an expression of a very complex structure operating on relatively simple processes. This is the very first thought that brought to the formalization of the brain.</p><p>The perceptron is, in this way, the very first brick of Artificial Intelligence. We are going to formalize the biological neuron,<em> de facto</em> getting the perceptron and after that introduce <em>Rosenblatt’s perceptron learning algorithm.</em></p><p>We now focus on the <em>logical </em>structure of a single neuron. The description of the previous sections suggests the following scheme:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/382/1*ZFlzROot9R3M89WGCRI0vg.png" /><figcaption>Figure 3</figcaption></figure><ul><li>There is a <em>processing unit</em>, the large circle , which represents the soma.</li><li>A number of input lines connect, <em>logically</em>, to the soma, depicted with incoming arrows in Figure3. They represent dendrites and synapses.</li><li>The input channels are activated by the signals they receive from the input variables (<strong><em>x</em></strong>’s) to which they are connected. These variables are our pre-synaptic axons, and they have an intrinsic <em>logical</em> nature, since they can either activate the channel (carry a spike) or not activate it (sub-treshold activity in the pre-synaptic neuron).</li></ul><p>To each input line, we associate a parameter<em> w</em> — the subscript refers to the various input channels. The numerical value of each <em>w</em> is indeed the amount of post-synaptic potential (<strong>PSP</strong>) that would be added to the soma if the channel were activated.</p><p>Furthermore, there is a single <em>logical</em> output line (outgoing arrow in Fig3). It expresses the logical fact that our neuron produces a single relevant ouput — a spike .</p><p>We can arrange the operations of the unit in the following way:</p><ul><li>At a given moment, some of the logical inputs are activated.</li><li>The soma receives an input which is the linear sum of <strong>PSP</strong> values of the channels that were activated — the variable <em>x</em> indicates whether the channel is active (<em>x = 1</em>) or inactive (<em>x = -1</em>)</li><li>The sum of <strong>PSP</strong>’s is compared to the threshold value of the neuron and the output channel is activated if it overcomes the threshold.</li></ul><p>Formally, we’ll get:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/130/1*e8iI2MGv8PU2yBLBbAGOOg@2x.png" /></figure><p>with <em>h</em> as the PSP at our neuron and <em>n</em> as the number of pre-synaptic neurons. Mathematically <em>h</em> is nothing but a <a href="https://en.wikipedia.org/wiki/Dot_product"><em>dot product</em></a>.</p><p>Anyways, the operation implemented by our little “machine” can be expressed as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/165/1*CXKtLDLRx30UOf5wLBBuYg@2x.png" /></figure><p>with <em>sign</em> as the <a href="https://en.wikipedia.org/wiki/Sign_function">sign function</a> and 𝜽 as the threshold of our neuron. <em>H</em> is basically <em>1</em> if its argument (<em>h+</em>𝜽) is positive and -1 if negative — and, to be precise, <em>0</em> if <em>h +</em>𝜽 = 0. The variable <em>y</em> indicates, in our biological analogy, whether a spike will appear in the output axon.</p><p>Now that our perceptron has taken shape we can talk about how this mathematical structure can be involved in the learning process. Let’s introduce <em>Rosenblatt’s perceptron learning algorithm.</em></p><h4>The learning process in perceptrons [4]</h4><p>There’s a clear analogy between neurons and perceptrons, but how can we use this ladder model in order to learn ? We’re going to show that the perceptron can be used to solve <em>classification</em> problems, namely it can tell you whether, <em>if we have two sets of points</em>, a point belong to one set or another. We can say without a lack of generalizability that the problem can be thought as a binary (yes/no) decision, where the variable <em>y </em>(the <em>output value</em>) defined above is one of these two binary values ( <em>yes = 1</em>, <em>no = -1</em>) .</p><p>From the previous definition of <em>h</em> as a dot product it’s even clearer that <strong><em>x</em></strong> is formally a vector (the <em>input vector</em>). Every component of <strong><em>x </em></strong>is called <em>feature, </em>by talking in Machine Learning terms. In the same way <strong><em>w</em></strong> is a vector (the <em>weight</em> <em>vector</em>). The threshold term 𝜽 it’s usually called <em>bias term</em>. In the direction of a simplified notation we’re going to <em>augment</em> both <strong><em>x </em></strong>and<strong><em> w</em></strong> in the following way:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/321/1*CXeNkMeP5mLJ2eHx60Lukg@2x.png" /></figure><p>Basically, by extending the sum from <em>i = 0 </em>to <em>i = n</em>.</p><p>The learning algorithm we’re going to build will consider all these terms and in particular the term <em>learning</em> specifically refers on finding the best value of <strong><em>w</em></strong>’s components in order to achieve the best classification possible. Take a look at the following<em> 2-dimensional</em> (here <strong><em>x</em></strong> has only two components) problem in Figure4 and everything will be clear.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1006/1*yF4wXHHEb1zrZLy8KXUryQ.png" /><figcaption>Figure 4 — On the left: misclassified data. On the right: perfectly classified data.</figcaption></figure><p>So, in practice, classification means finding a way in order to separate the two different “kinds” of data we have. Each “kind” of data is specified by a different <em>label</em> (plus or minus, in the figure above). So to speak, this is a <em>two labels </em>classification problem.</p><p>It’s time to introduce the <em>perceptron learning algorithm</em> (<strong>PLA</strong>), which will determine what <strong><em>w</em></strong> should be, based on the data. This learning algorithm consists in a simple <em>iterative </em>method. By involving the term “iteration”, we’re going to introduce a new parameter in our model, namely, <em>time</em>. At iteration <em>t</em>, where <em>t</em> <em>= 0,1,2</em>…, there is a current value of the <em>weight vector</em>, call it <strong><em>w</em></strong><em>(t). </em>The algorithm picks a point, associated with its label, that is currently misclassified, call it (<strong>x</strong><em>(t)</em>, <em>y(t)</em>), and uses it to update <strong><em>w</em></strong><em>(t). </em>Since the example is misclassified, we have <em>y(t) ≠ sign(</em><strong><em>w</em></strong><em>(t) </em><strong><em>· x</em></strong><em>(t)), </em>where the dot represents the dot product.</p><p>The <em>weight update rule</em> is as follow:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/277/1*P7wg-Gqpf00Rni3rIPoTjA@2x.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1006/1*JpApwtU4mYTOQNdIwiSC-g.png" /><figcaption>Figure 5 — The last update</figcaption></figure><p>This rule moves the boundary in the direction to classify <strong><em>x</em></strong><em>(t)</em> correctly, as depicted in the figure above. The algorithm continues with further iterations until there are no longer misclassified points in the data set. With this intuitive view of the <strong>PLA</strong>, it’s now time to code it.</p><h4>Let’s code the Perceptron Learning Algorithm</h4><p>We’re going to implement the <strong>PLA</strong> from scratch by coding a simple <em>Python</em> script. Let’s open a new notebook with <em>Jupyter</em> and start by importing <em>matplotlib.pyplot</em> and <em>numpy</em></p><pre>import matplotlib.pyplot as plt<br>import numpy as np </pre><p>We’re going to create a 2-dimensional random dataset of <em>5</em> points (with two possible labels, <em>-1</em> and <em>+1</em>) by using <a href="https://docs.scipy.org/doc/numpy-1.15.1/reference/generated/numpy.random.rand.html"><em>numpy.random.rand(</em></a><em>).</em> In order to reproduce the result we’ll fix the random seed — you can actually choose every integer number, <a href="https://en.wikipedia.org/wiki/137_(number)#In_physics"><em>137</em> has a particular meaning for Physicists</a> though.</p><pre># Setting the random seed <br>np.random.seed(seed = 137)</pre><pre># Generate x1 and x2, coordinates of our points<br>number_of_points = 5<br>x1 = np.random.rand(number_of_points)<br>x2 = np.random.rand(number_of_points)</pre><pre># We have two labels, namely -1 and 1<br>possible_ys = np.array([-1,1])</pre><pre># We randomly build the label y to point (x1,x2) association <br>y = np.random.choice(possible_ys, number_of_points)</pre><p>Data are represented by triplets of values.</p><pre># We create data as triplets of values<br>data = []<br>for i in range(number_of_points):<br>    data.append((x1[i],x2[i],y[i]))</pre><p>You can take a look at your data by executing the following line and you’ll get:</p><pre># Taking a look at data<br>data</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/820/1*W4RkEoavMx58zuSOYV0yWA.png" /></figure><p>The next step is to plot them by denoting with a “-” the label corresponding to “-1” and by “+” the label corresponding to “1”</p><pre># Plotting our data <br>plt.plot([x1 for (x1,x2,y) in data if y==-1], [x2 for (x1,x2,y) in data if y==-1], &#39;_&#39;, mec=&#39;r&#39;, mfc=&#39;none&#39;)<br>plt.plot([x1 for (x1,x2,y) in data if y==1], [x2 for (x1,x2,y) in data if y==1], &#39;+&#39;, mec=&#39;b&#39;, mfc=&#39;none&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8_R1z2kQdioF7OQjHVBxFg.png" /></figure><p>After that we can start with the learning model. The very first thing to do is to initialize the weights, usually a “little” random value is the best solution for faster convergence.</p><pre># Initializing the weight vector<br>w = np.random.rand(3)*10e-03</pre><p>Consequently, we define the function “<em>predict”</em> which is nothing but <em>sign(</em><strong><em>w</em></strong><em>(t) </em><strong><em>· x</em></strong><em>(t))</em></p><pre>def predict(x1,x2):<br>    # w[0] is the threshold value, x0 = 1 <br>    h= w[0] + w[1]*x1 + w[2]*x2<br>    <br>    if h&lt;0:<br>        return -1<br>    else:<br>        return 1</pre><p>The last part of the learning model is the function “fit” which corresponds to the <em>weight update rule</em>. The essence is: for every point we compare the predicted label with the actual one, if these are not equal each component of the <em>weight vector</em> is updated.</p><pre>def fit(data):<br>    stop = False<br>    <br>    while stop == False:<br>        stop = True<br>        <br>        for x1,x2,y in data:        <br>            ypredict= predict(x1,x2) <br>            if y != ypredict: <br>                              <br>                stop = False<br>            <br>                w[1]= w[1] + x1*y<br>                w[2]= w[2] + x2*y<br>                w[0]= w[0] + y </pre><p>By applying the function “<em>fit</em>” to our data, the <strong>PLA</strong> will converge to a solution. If this is the case we’ll get a printed ‘SUCCESS!’ as the output of the following cell.</p><pre>fit(data) </pre><pre># Check if the model is predicting correct labels<br>for (x1, x2, y) in data:<br>    if predict(x1,x2) != y:<br>        print(&#39;FAIL&#39;)<br>        break<br>else:<br>    print(&#39;SUCCESS!&#39;)</pre><p>Last but not least, we can plot our result. “<em>f(x)</em>” is the functional form of the line given by the <strong>PLA </strong>(you can get it by applying simple algebraic calculations — <em>de facto</em> by expliciting the expression for <em>x2</em>).</p><pre>def f(x):<br>    return -(w[0] + w[1]*x)/w[2]</pre><pre>d = range(0,2)<br>plt.plot(d, [f(x) for x in d])</pre><pre>plt.plot([x1 for (x1,x2,y) in data if y==-1], [x2 for (x1,x2,y) in data if y==-1], &#39;_&#39;, mec=&#39;r&#39;, mfc=&#39;none&#39;)<br>plt.plot([x1 for (x1,x2,y) in data if y==1], [x2 for (x1,x2,y) in data if y==1], &#39;+&#39;, mec=&#39;b&#39;, mfc=&#39;none&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Y3WU7pfjyymHq-V3dozBlw.png" /></figure><p>In this way, we’ve solved a simple classification problem, i.e. we’ve been learning!</p><h4>Conclusions</h4><p>We’ve reached the end of our adventure: we’ve been traveling trough a mathematical model of the neuron and its implementation. With that in mind,we’ve been taking inspiration from it in order to formalize the neuron and build the perceptron. Be careful though, the perceptron is not a biologically plausible model. The relationship between neurons and pereptrons is much more similar to the one between birds and planes. Here, the problem of “flying” is actually the <em>learning problem</em>, but again, the road for a perfectly reliable plane it’s way longer than this.</p><p>In particular perceptrons have an <em>intrinsic problem, </em>which is insurmountable without a major change. Minsky and Papert have been pointing out that issue so clearly that research in Artificial Intelligence has been subjected to a quite long break. Therefore, I would like you to point out this serious perceptron’s limitation. I think that you can grasp an intuition of the problem by looking at the code in the previous section and tweaking <em>np.random.seed() </em>and<em> number_of_points. </em>Give it a try and let me know!</p><h4>References</h4><p>[1] Amit, D. (1989). <em>Modeling Brain Function: The World of Attractor Neural Networks</em>. Cambridge: Cambridge University Press.</p><p>[2] Wulfram Gerstner, Werner M. Kistler, Richard Naud, and Liam Paninski. 2014. Neuronal Dynamics: From Single Neurons to Networks and Models of Cognition. Cambridge University Press, New York, NY, USA. <em>Online version: </em><a href="https://neuronaldynamics.epfl.ch/online/index.html">https://neuronaldynamics.epfl.ch/online/index.html</a></p><p>[3] Marvin L. Minsky and Seymour A. Papert. 1988. Perceptrons: Expanded Edition. MIT Press, Cambridge, MA, USA.</p><p>[4]Rosenblatt, Frank. 1962. Principles of neurodynamics; perceptrons and the theory of brain mechanisms. Washington, Spartan Books</p><p>[5] Yaser S. Abu-Mostafa, Malik Magdon-Ismail, and Hsuan-Tien Lin. 2012. Learning from Data. AMLBook.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=339c8de6635a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/the-learning-problem-comparison-between-brain-and-machine-339c8de6635a">The Learning Problem: Comparison between Brain and Machine</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Dive Into Machine Learning | Chapter 4]]></title>
            <link>https://medium.com/mljcunito/a-dive-into-machine-learning-chapter-4-85d774227b82?source=rss-2fefde940982------2</link>
            <guid isPermaLink="false">https://medium.com/p/85d774227b82</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Simone Azeglio]]></dc:creator>
            <pubDate>Fri, 06 Sep 2019 23:50:58 GMT</pubDate>
            <atom:updated>2019-09-07T08:05:14.785Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qJohz9jivSphmhQcNfCXcg.jpeg" /><figcaption>Hold your breath and dive into Machine Learning</figcaption></figure><p>Hello Learners and Welcome Back!</p><p>After going trough <em>Scientific Programming </em>and exploring different libraries in the <a href="https://medium.com/mljcunito/scientific-programming-chapter-2-1-kung-fu-pandas-a6e715e0753d">previous posts</a>, we’re finally ready to take you to the next level and introduce you to <em>Machine Learning</em>.</p><p>In <a href="https://github.com/MLJCUnito/SummerCoding/tree/master/(4)TutorialMachineLearning">these notebooks</a> you’ll find a broad dissertation of Machine Learning foundations with hands on coding: starting from Exploratory Data Analysis (EDA) and basic visualization to the very basics of Deep Learning (<a href="https://ujjwalkarn.me/2016/08/11/intuitive-explanation-convnets/">CNN</a>, <a href="https://towardsdatascience.com/recurrent-neural-networks-d4642c9bc7ce">RNN</a>)</p><p>Take your time to go trough these notebooks, it’s a pretty long ride to explore every chunk of code.</p><p>With this last bunch of notebooks you’ll be able to further investigate Machine Learning projects by yourself. Our introductory work is pretty much done: it’s time to go on with something differente. In particular we’re thinking about expanding the Machine Learning Journal Club and starting a brand new collaborative coding playground experience. We’ll keep you updated in the next days!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JyOrzJm4Hl6IehTrXCR5rA.jpeg" /><figcaption>MLJC Building is under renovation — work in progress!</figcaption></figure><p>Simone, MLJC</p><p>P.S. If you find any bug/error or you have some problems with the code feel free to contact us at <a href="/service/mailto:mljcunito@gmail.com">mljcunito@gmail.com</a></p><blockquote>Do you like our project? Your feedbacks are important to us! You can support in many ways, such as: by contributing to our contents or by submitting a small donation in form of a coffee down here👇</blockquote><figure><a href="https://www.buymeacoffee.com/m3LJoOtRw"><img alt="" src="https://cdn-images-1.medium.com/max/170/1*bk_C8Um34KavCkO2yzMCZg.png" /></a></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=85d774227b82" width="1" height="1" alt=""><hr><p><a href="https://medium.com/mljcunito/a-dive-into-machine-learning-chapter-4-85d774227b82">A Dive Into Machine Learning | Chapter 4</a> was originally published in <a href="https://medium.com/mljcunito">MLJC</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Scientific Programming | Chapter 2.1 : Kung-Fu Pandas]]></title>
            <link>https://medium.com/mljcunito/scientific-programming-chapter-2-1-kung-fu-pandas-a6e715e0753d?source=rss-2fefde940982------2</link>
            <guid isPermaLink="false">https://medium.com/p/a6e715e0753d</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[pandas]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Simone Azeglio]]></dc:creator>
            <pubDate>Tue, 20 Aug 2019 02:13:30 GMT</pubDate>
            <atom:updated>2019-08-20T03:50:44.632Z</atom:updated>
            <content:encoded><![CDATA[<h3>Scientific Programming | Chapter 2.1 : Kung-Fu Pandas</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Imt2iaHVmAjVfrSKQbAuKA.jpeg" /></figure><p>Hello Learners and Welcome Back!</p><p>Have you ever been in trouble by trying to manage a <em>huge amount of data</em>? Have you ever wanted to get rid of Excel and <em>automate all the boring stuff</em>?</p><p>Luckily, somebody is here to lend you a hand! Our hero is not <strong>Po</strong> from Kung Fu Panda but rather <a href="https://pandas.pydata.org/">Pandas</a>, the canonical library for data structures in Python<em>.</em></p><p>You already had a little taste of Pandas in the previous <a href="https://medium.com/mljcunito/scientific-programming-chapter-2-data-handling-and-data-visualisation-6b51ec29b5a5">article</a>, but here we would like to go in-depth and take you to the next level: <em>Machine Learning.</em></p><p>Take your time to go trough <a href="https://github.com/MLJCUnito/SummerCoding/tree/master/(3)TutorialPandas">this set of notebooks</a>, these contents are very important. In fact, being able to manipulate your data is the very first step for creating a successful model.</p><p>Let’s make a quick recap of what we have learned so far:</p><ul><li><a href="https://medium.com/mljcunito/scientific-programming-chapter-1-ipython-and-python3-3d2ab728be85"><strong>Chapter 1</strong></a>: Python3 and IPython</li><li><a href="https://medium.com/mljcunito/scientific-programming-chapter-2-data-handling-and-data-visualisation-6b51ec29b5a5"><strong>Chapter 2</strong></a> and<strong> 2.1</strong>: data handling and visualization + Pandas in-depth</li><li><a href="https://medium.com/mljcunito/scientific-programming-chapter-3-numpy-scipy-and-matplotlib-8e215b4ffe99"><strong>Chapter 3</strong></a>: Numpy, Scipy and Matplotlib in detail</li><li><strong>Chapter 4</strong>: the next level</li></ul><p>Now that we’ve completely covered scientific programming we’re almost ready to reach the rooftop and dive into Machine Learning.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/770/1*oyXQdC9t9JKMCYjq3CETyQ.jpeg" /></figure><p>Have fun!</p><p>Simone, MLJC</p><p>P.S. If you find any bug/error or you have some problems with the code feel free to contact us at <a href="/service/mailto:mljcunito@gmail.com">mljcunito@gmail.com</a></p><blockquote>Do you like our project? Your feedbacks are important to us! You can support in many ways, such as: by contributing to our contents or by submitting a small donation in form of a coffee down here👇</blockquote><figure><a href="https://www.buymeacoffee.com/m3LJoOtRw"><img alt="" src="https://cdn-images-1.medium.com/max/170/1*bk_C8Um34KavCkO2yzMCZg.png" /></a></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a6e715e0753d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/mljcunito/scientific-programming-chapter-2-1-kung-fu-pandas-a6e715e0753d">Scientific Programming | Chapter 2.1 : Kung-Fu Pandas</a> was originally published in <a href="https://medium.com/mljcunito">MLJC</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Scientific Programming | Chapter 3 : Numpy, Scipy and Matplotlib]]></title>
            <link>https://medium.com/mljcunito/scientific-programming-chapter-3-numpy-scipy-and-matplotlib-8e215b4ffe99?source=rss-2fefde940982------2</link>
            <guid isPermaLink="false">https://medium.com/p/8e215b4ffe99</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Simone Azeglio]]></dc:creator>
            <pubDate>Sun, 11 Aug 2019 20:29:13 GMT</pubDate>
            <atom:updated>2019-08-11T20:29:13.358Z</atom:updated>
            <content:encoded><![CDATA[<h3>Scientific Programming | Chapter 3 : Numpy, Scipy and Matplotlib</h3><p><em>Hello Learners</em> and Welcome Back!</p><p>There’s a new chapter in town! We’re glad to introduce you to our new friends: <a href="https://www.numpy.org/">NumPy</a>, <a href="https://www.scipy.org/">SciPy</a> and <a href="https://matplotlib.org/">Matplotlib</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OTQHk3rsuzwdidO9zgSOfA.png" /><figcaption>Populating the environment</figcaption></figure><p>Together with Pandas (go back to<a href="https://medium.com/mljcunito/scientific-programming-chapter-2-data-handling-and-data-visualisation-6b51ec29b5a5"> Chapter 2</a> for a brief introduction) they constitute the very foundations of scientific programming in <em>Python</em>.</p><p>As we told you before, scientific programming is considered as an appetizer for Machine Learning, so take your time to complete these notebooks.</p><p>You can find them on our <a href="https://github.com/MLJCUnito/SummerCoding">Github</a>, remember to follow us there for updates.</p><p>Have fun!</p><p>Simone, MLJC</p><p>P.S. If you find any bug / error or you have some problems with the code feel free to contact us at <a href="/service/mailto:mljcunito@gmail.com">mljcunito@gmail.com</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8e215b4ffe99" width="1" height="1" alt=""><hr><p><a href="https://medium.com/mljcunito/scientific-programming-chapter-3-numpy-scipy-and-matplotlib-8e215b4ffe99">Scientific Programming | Chapter 3 : Numpy, Scipy and Matplotlib</a> was originally published in <a href="https://medium.com/mljcunito">MLJC</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>