<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Mukul Malik on Medium]]></title>
        <description><![CDATA[Stories by Mukul Malik on Medium]]></description>
        <link>https://medium.com/@mukulmalik?source=rss-bd3e39707a79------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*VV0J8NCXZzTRvNJwuJs4VA.jpeg</url>
            <title>Stories by Mukul Malik on Medium</title>
            <link>https://medium.com/@mukulmalik?source=rss-bd3e39707a79------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 04:00:09 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@mukulmalik/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Don’t Be Overwhelmed by NLP]]></title>
            <link>https://pub.towardsai.net/dont-be-overwhelmed-by-nlp-c174a8b673cb?source=rss-bd3e39707a79------2</link>
            <guid isPermaLink="false">https://medium.com/p/c174a8b673cb</guid>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[research]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Mukul Malik]]></dc:creator>
            <pubDate>Fri, 21 Aug 2020 00:01:04 GMT</pubDate>
            <atom:updated>2020-08-21T04:31:28.488Z</atom:updated>
            <content:encoded><![CDATA[<h4><a href="https://towardsai.net/p/category/nlp">Natural Language Processing</a></h4><h3>Don’t Be Overwhelmed by NLP Research</h3><h4>How to cope with the volume of ongoing research in NLP which is probably infeasible for you</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*odADLNKmfYa2KmNe9Eod5Q.png" /><figcaption>-by <a href="http://geek-and-poke.com/">geek-and-poke.com</a> under <a href="https://creativecommons.org/licenses/by/3.0/">CC-BY-3.0</a></figcaption></figure><h3>What is going on?</h3><blockquote>NLP is the new Computer Vision</blockquote><p>With enormous amount go textual datasets available; giants like Google, Microsoft, Facebook etc have diverted their focus towards NLP.</p><p>Models using <strong>thousands</strong> of super-costly TPUs/GPUs, making them infeasible for most.</p><blockquote><strong>This gave me anxiety!</strong> (we’ll come back to that)</blockquote><p>Let’s these Tweets put things into perspective:</p><p><strong>Tweet 1:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1417qPcET9nQZsOOFV87Dg.png" /></figure><p><strong>Tweet 2:</strong> <em>(read the trailing tweet)</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UirseD4gVp5YpPotd0OJPA.png" /></figure><h3>Consequences?</h3><p>In about last one-year following knowledge became mainstream:</p><ul><li>Transformers was followed by Reformer, Longformer, GTrXL, Linformer, and others.</li><li>BERT was followed by XLNet, RoBERTa, AlBERT, Electra, BART, T5, Big Bird, and others.</li><li>Model Compression was extended by DistilBERT, TinyBERT, BERT-of-Theseus, Huffman Coding, Movement Pruning, PrunBERT, MobileBERT, and others.</li><li>Even new tokenizations were introduced: Byte-Pair encoding (BPE), Word-Piece Encoding (WPE), Sentence-Piece Encoding (SPE), and others.</li></ul><p>This is barely the tip of the iceberg.</p><blockquote>So while you were trying to understand and implement a model, a bunch of new lighter and faster models were already available.</blockquote><h3>How to Cope with it?</h3><p>The answer is short:</p><blockquote>you don’t need to know it all, know only what is necessary and use what is available</blockquote><h3>Reason</h3><p>I read them all to realize <strong>most of the research is re-iteration of similar concepts</strong>.</p><p>At the end of the day (vaguely speaking):</p><ul><li>the reformer is hashed version of the transformers and longfomer is a convolution-based counterpart of the transformers</li><li>all compression techniques are trying to consolidate information</li><li>everything from BERT to GPT3 is just a language model</li></ul><h3>Priorities -&gt; Pipeline over Accuracy</h3><blockquote>Learn to use what’s available, efficiently, before jumping on to what else can be used</blockquote><p>In practice, these models are a <em>small part of a much bigger pipeline</em>.</p><blockquote>Your first instinct should not be of competeing with Tech Giants’ in-terms of training a better model.</blockquote><blockquote>Instead, Your first instinct should be to use the availbale models to build an end-to-end application which<strong> solves a practical problem</strong>.</blockquote><p>Now if you feel that the model is the performance bottleneck of your application; re-train that model or switch to another model.</p><p>Consider the following:</p><ul><li>Huge deep learning models usually take thousands for GPU hours just to train.</li><li>This increases 10x when you consider hyper-parameter tuning (HP Tuning).</li><li>HP Tuning something as efficient as an Electra model can also take a week or two.</li></ul><h3>Practical Scenario -&gt; The Real Speedup</h3><p>Take an example of Q&amp;A Systems. Given millions of documents, for this task, something like ElasticSearch is way more essential to the pipeline than a new Q&amp;A model (comparatively).</p><p>In production success of your pipeline will not (only) be determined by how awesome are your Deep Learning models but also by:</p><ul><li><strong>the latency of the inference time</strong></li><li><strong>predictability of the results and boundary cases</strong></li><li>the ease of fine-tuning</li><li>the ease of reproducing the model on a similar dataset</li></ul><p>Something like DistilBERT can be scaled to handle Billion queries as beautifully mentioned in this <a href="https://blog.roblox.com/2020/05/scaled-bert-serve-1-billion-daily-requests-cpus/">blog</a> by Robolox.</p><blockquote>While new models can decrease the inference time by <strong>2x-5x</strong>.</blockquote><blockquote>Techniques like <a href="https://www.mathworks.com/company/newsletters/articles/what-is-int8-quantization-and-why-is-it-popular-for-deep-neural-networks.html">quantization</a>, pruning and using <a href="https://medium.com/microsoftazure/accelerating-model-training-with-the-onnx-runtime-519d75a97166#:~:text=ONNX%20Runtime%20is%20a%20performance,in%20high%2Dscale%20production%20scenarios.">Onnx</a> can decrease the inference time by <strong>10x-40x</strong>!</blockquote><h3>Personal Experience</h3><p>I was working on an Event Extraction pipeline, which used:</p><ul><li>4 different transformer-based models</li><li>1 RNN-based model</li></ul><p>But. At the heart of the entire pipeline were:</p><ul><li>WordNet</li><li>FrameNet</li><li>Word2Vec</li><li>Regular-Expressions</li></ul><p>And. Most of my team’s focus was on:</p><ul><li><strong>Extraction of text from PPTs, images &amp; tables</strong></li><li><strong>Cleaning &amp; preprocessing text</strong></li><li><strong>Visualization of results</strong></li><li>Optimization of ElasticSearch</li><li>Format of info for Neo4J</li></ul><h3>Conclusion</h3><p>It is more essential to have an average performing pipeline than to have a non-functional pipeline with a few brilliant modules.</p><p>Neither Christopher Manning nor Andrew NG knows it all. They just know what is required and when it is required; well enough.</p><p>So, have realistic expectations of yourself.</p><blockquote>Thank you!</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c174a8b673cb" width="1" height="1" alt=""><hr><p><a href="https://pub.towardsai.net/dont-be-overwhelmed-by-nlp-c174a8b673cb">Don’t Be Overwhelmed by NLP</a> was originally published in <a href="https://pub.towardsai.net/">Towards AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Art of Doing More in Less Time]]></title>
            <link>https://medium.com/@mukulmalik/the-art-of-doing-more-in-less-time-1d98e73700d3?source=rss-bd3e39707a79------2</link>
            <guid isPermaLink="false">https://medium.com/p/1d98e73700d3</guid>
            <dc:creator><![CDATA[Mukul Malik]]></dc:creator>
            <pubDate>Tue, 04 Aug 2020 22:15:43 GMT</pubDate>
            <atom:updated>2020-08-04T22:15:43.865Z</atom:updated>
            <content:encoded><![CDATA[<h4>from being a cognitive miser to being a cognitive aficionado</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aHDkaZpAFtfnXanzWRFH3w.png" /></figure><h3>Now read the following carefully:</h3><blockquote>work expands so as to fill the time available for its completion</blockquote><p>This is called <a href="https://en.wikipedia.org/wiki/Parkinson&#39;s_law"><strong>Parkinson’s law</strong></a><strong>.</strong></p><h3>Now we’ll simplify it:</h3><p>Have you ever forgotten the deadline for any exam/project/assignment until the last moment? (engineers, I feel you)</p><blockquote><em>You might not have aced those exams but the amount of work and information you were able to efficiently crunch in those days, it must have been obnoxiously vast.</em></blockquote><p>While on a normal day, you’d probably never be able to do the same.</p><p>This is so common actually a term for that, <a href="https://en.wikipedia.org/wiki/Student_syndrome"><strong>Student syndrome</strong></a>.</p><h3>Student Syndrome during Planning:</h3><p>We predict the timeline for completion of work with an optimism bias (a belief that one is less likely to experience a negative event) and underestimate the time needed.</p><p>Due to Optimism Bias, we allocate a large proportion of the available time for exploration and acquisition of new ideas while being confident that the remaining time allocated for execution of task would be sufficient.</p><p>Hence, we let our over-optimism (ignorance towards the potential obstacles) overwhelm our ration choice of maintaining a margin/buffer of time.</p><p>This is known as the <a href="https://en.wikipedia.org/wiki/Planning_fallacy"><strong>Planning Fallacy</strong></a>.</p><h3>Student Syndrome during execution:</h3><p>Even in the initial phase of execution, when time might be available in abundance, we fail to strike a balance between the need to acquire new knowledge or to utilize the existing knowledge to get the work done.</p><p>We end up giving disproportionate weight to trivial issues, random thoughts and vague ideas. Therefore we end up wasting most of our time. This is known as the<strong> </strong><a href="https://en.wikipedia.org/wiki/Law_of_triviality"><strong>Law of triviality</strong></a>.</p><blockquote>This has been humorously summarised in the ninety-ninety rule:</blockquote><blockquote>The first 90 percent of the code accounts for the first 90 percent of the development time. The remaining 10 percent of the code accounts for the other 90 percent of the development time.</blockquote><blockquote>Note: Joke is that this adds up to 180% or 80% more than the estimated time.</blockquote><h3>Why We Commit this Fallacy:</h3><p>Kahneman and Tversky (authors) originally explained that the planners commit this fallacy because <strong>they fail to utilise their full experience</strong> of how much time similar tasks require. This root from the fact that <strong>humans are </strong><a href="https://en.wikipedia.org/wiki/Cognitive_miser"><strong>cognitive misers</strong></a>.</p><h3>On Being a Cognitive Miser:</h3><p>I won’t get into details of neuro-psychology and keep it short.</p><p>The human brain is considered to be a <strong>cognitive miser</strong> due to the tendency of people to think and solve problems in simpler and less effortful ways rather than in more sophisticated and more effortful ways, <strong>regardless of intelligence</strong>.</p><p>Which is a fancy way of saying <strong>our brain is damn lazy</strong>!</p><h3>Creativity, the Royal Flush</h3><p>Creativity always comes as a surprise to us; therefore we can never count on it and we dare not believe in it until it has happened.</p><p>Hence, the only way in which we can bring our creative resources fully into play is by misjudging the nature of the task, assuming it to be much simpler than it actually is.</p><p>This is when panic and sense of urgency kicks in our brain comes out of ‘miser’ mode and use its actual potential based upon its entire experience.</p><p>This is called <a href="https://en.wikipedia.org/wiki/Hiding_hand_principle"><strong>Hiding Hand Principle</strong></a>.</p><h3>Be a Pygmalion, Not a Miser</h3><p><strong>Cognitive Restructuring</strong> is an effective behavioural therapy, which helps counter our natural instinct of being a cognitive miser.</p><p>This is done by distorting the current thought process and creating an environment where being innovative and original fetches you some rewards that would push to be more cognitively rigorous.</p><p><strong>Feynman Technique of learning</strong> worked exceptionally for me for creating that environment. Pick up a bunch of people you can teach, the more creative and original you are, the more you will grasp the attention of appreciation of the audience.</p><p>The net time when you face the audience, their expectations are much higher than what they were the last time. So you’ll have to outperform yourself this time.</p><p>This, in turn, will raise your expectations from yourself and you’ll begin to utilize more and more of your brain’s potential to think, imagine and create!</p><p>And Yes, there is a term for this as well, it is called <a href="https://en.wikipedia.org/wiki/Pygmalion_effect"><strong>Pygmalion Effect</strong></a> and this leads to <a href="https://en.wikipedia.org/wiki/Self-fulfilling_prophecy"><strong>Self-Fulfilling Prophecy</strong></a>.</p><p>Basically, <strong>learn to love your work, cherish it</strong>.</p><h3>Be a Cognitive Philanthropist</h3><p>Be sceptical and cynical (not pessimistic) about your ability to complete work in a given duration of time. While still being brutally optimistic about your will to succeed (keep pessimism at bay).</p><p>Prepare for the worse, so you don’t lose your calm or are caught off guard if things take a turn for worse.</p><p>Avoid assumptions, the main cause of optimism bias. Do <a href="https://medium.com/towards-artificial-intelligence/ensure-success-of-every-machine-learning-project-dbe1ea187597"><strong>Marr’s Level of Analysis</strong></a> for every task at hand.</p><blockquote><em>A skeptic is someone who is reluctant to believe;<br>A cynic is someone who refuses to believe;<br>A pessimist is someone who has given up on believing.</em></blockquote><blockquote><em>— by Mark Goulston M.D., F.A.P.A.</em></blockquote><h3>Know when to Relax!</h3><p>Working continuously can saturate the mind’s comprehension abilities. This is because our bodies go through an <strong>ultradian sleep cycle </strong>(cycle repeated throughout day) of sleep.</p><p>Vaguely speaking, if you do any activity continuously for 1–2 hours, your mind would go to sleep. So take sufficient breaks.</p><h3>The Subtle Art of Not Giving a Fuck</h3><p>Manage the expectations of others from you, don’t get overwhelmed by them because nothing matters more than your sanity.</p><blockquote>Thank you for the read!</blockquote><blockquote>Clap if you liked it!</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1d98e73700d3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Ensure Success of Every Machine Learning Project]]></title>
            <link>https://pub.towardsai.net/ensure-success-of-every-machine-learning-project-dbe1ea187597?source=rss-bd3e39707a79------2</link>
            <guid isPermaLink="false">https://medium.com/p/dbe1ea187597</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[project-management]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Mukul Malik]]></dc:creator>
            <pubDate>Mon, 27 Apr 2020 12:01:01 GMT</pubDate>
            <atom:updated>2020-05-05T11:00:49.325Z</atom:updated>
            <content:encoded><![CDATA[<h4>Ensuring essential debates and preventing the project from imminent failure.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bigUzsh7YeK1IvAZTZmGmA.png" /><figcaption>-by <a href="http://geek-and-poke.com/">geek-and-poke.com</a> under <a href="https://creativecommons.org/licenses/by/3.0/">CC-BY-3.0</a></figcaption></figure><blockquote>Machine Learning projects can convince even an atheist that ‘the devil is in the detail’.</blockquote><p><em>Allegedly</em>, a report from Gartner predicted that <strong>85% of the Machine Learning projects will fail</strong>. Well, I am not too surprised.</p><p>But. There is a cure, as we’ll explore later in this blog.</p><h3>The Problem</h3><p>The problem isn’t that there aren’t enough discussions among and within the teams.</p><p>Rather, the problem is that:</p><ul><li><em>people with different backgrounds understand different languages</em></li><li><em>a common language is almost always absent when defining an ML project</em></li><li><em>most discussions soon turn into the game of Chinese whisper</em></li><li>ambiguity, arising from assumptions, accumulates</li><li>completion within the span of the original timeline becomes infeasible</li></ul><blockquote>Long story short, it can be a mess!</blockquote><blockquote>Why?</blockquote><blockquote>Simple Answer: Assumptions</blockquote><h3>The Solution</h3><p>The paper <a href="https://arxiv.org/abs/2004.05107">Levels of Analysis for Machine Learning</a> from DeepMind discusses how Marr’s Level of Analysis can be leveraged:</p><ul><li>to minimize assumptions at different levels of the project</li><li>by providing a common language for people from different backgrounds</li><li>and encouraging relevant discussions/debates</li><li>to avoid last-minute damage control</li></ul><h3>The Background</h3><p>Originally a concept from Neuroscience, Marr’s <em>levels of analysis </em>promotes the idea that complex systems such as the brain, a computer, or human behavior should be understood at different levels.</p><p>While the levels have traditionally been used to reverse-engineer biological behavior, Marr’s framework proved to be an elegant and popular way of reasoning about complex systems.</p><h3>The Desired Outcome</h3><p>The desired outcome of Marr’s framework is:</p><ul><li>break the solution into different levels</li><li>to encourage discussions for each level</li><li>disagreements are the desired outcomes of these discussions</li><li>forcing any differing assumption out into the open</li><li>exploring and comparing different solutions at each level</li><li>yielding a mutually agreed-upon solution</li></ul><h3>The ML Implementation</h3><p>Marr (1982) identified a three-layer hierarchy for describing and analyzing a computational system:</p><h4>Computational level:</h4><ul><li>What is the goal of the project?</li><li>What is the intended contribution of each model towards the goal of the project?</li><li>What are the various formats of data?</li><li>What is the desired output of an ML model?</li><li>How do we formulate the efficiency of an ML model?</li><li>What is the size and quality of the training set?</li><li>To what extent does the dev/test set mimic the inputs to the ML model when in production?</li><li>What is the expected velocity of the incoming data in production?</li><li>What are the hardware options at disposal?</li><li>And others</li></ul><h4>Algorithmic level:</h4><ul><li>What are the candidate algorithms for creating the ML model?</li><li>What are the pros and cons of each algorithm?</li><li>How scalable is each algorithm inherently?</li><li>What is the sample efficiency of each algorithm?</li><li>What is the generalization capacity of each algorithm on different domains?</li><li>How efficiently can an algorithm be parallelized over a distributed system?</li><li>How resource-efficient is each algorithm?</li><li>And others</li></ul><h4>Implementation level:</h4><ul><li>What language to be used?</li><li>What framework to be used?</li><li>How to optimize the algorithm for the provided hardware?</li><li>How to ensure minimum latency?</li><li>How to continuously update the model?</li><li>How to scale the algorithm?</li><li>And others</li></ul><h3>Note</h3><blockquote>This hierarchy of discussion also enables us to select which people are required for which discussions.</blockquote><blockquote>A lot of time during projects is wasted because people end up attending meetings they have no contribution to.</blockquote><h3>Conclusion</h3><p>The reason that these levels work so well is that they allow everyone to approach discussion using a common frame-of-reference that highlights different choices or assumptions, streamlining discussion while also encouraging scepticism.</p><blockquote>Thanks for the read! Have a good day!</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dbe1ea187597" width="1" height="1" alt=""><hr><p><a href="https://pub.towardsai.net/ensure-success-of-every-machine-learning-project-dbe1ea187597">Ensure Success of Every Machine Learning Project</a> was originally published in <a href="https://pub.towardsai.net/">Towards AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Distillation of Knowledge in Neural Networks]]></title>
            <link>https://medium.com/data-science/distillation-of-knowledge-in-neural-networks-cc02f79698b6?source=rss-bd3e39707a79------2</link>
            <guid isPermaLink="false">https://medium.com/p/cc02f79698b6</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[towards-data-science]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[transfer-learning]]></category>
            <dc:creator><![CDATA[Mukul Malik]]></dc:creator>
            <pubDate>Sun, 26 Jan 2020 14:51:02 GMT</pubDate>
            <atom:updated>2020-01-26T15:04:10.762Z</atom:updated>
            <content:encoded><![CDATA[<h4>State-of-the-art performance on much Smaller &amp; Faster models</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nDQRMa_l1WqeKT-q_ysvTQ.png" /></figure><blockquote>Distillation of Knowledge (in machine learning) is an architecture agnostic approach for generalization of knowledge (consolidating the knowledge) within a neural network to train another neural network.</blockquote><h3>Importance</h3><p>Currently, especially in NLP, very large scale models are being trained. A large portion of those can’t even fit on an average person’s hardware. Plus, due to the Law of diminishing returns, a great increase in the size of model barely maps to a small increase in the accuracy.</p><p>These models can barely run on commercial servers, let alone on a smartphone.</p><p>Using distillation, one could reduce the size of models like BERT by 87% and still retain 96% of its performance.</p><p>Basically, distillation enables one to:</p><ul><li>get State-of-the-Art accuracies</li><li>with a fraction of size</li><li>within a fraction of response time</li><li>within a fraction of fine-tuning time</li><li>fit an ensemble on single GPU</li><li>could run models on CPUs</li><li>enables federation learning</li><li>etc</li></ul><h3>Problem with Normal Neural Networks</h3><blockquote>The goal of every learner is to optimize its performance on training data. This doesn’t exactly translate as a generalization of knowledge within the dataset.</blockquote><p>Take an example of MNIST dataset. Let’s pick a sample picture of number 3.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/172/1*sk3dzRqi-rH_EWa_py6gLg.jpeg" /></figure><p>In training data, the number 3 translates to a corresponding one-hot-vector:</p><p><strong>0 0 0 1 0 0 0 0 0 0</strong></p><p>This vector simply tells that the number in that image in 3 but</p><blockquote>fails to explicitly mention anything about the shape of number 3. Like the shape of 3 is similar to 8.</blockquote><p>Hence:</p><blockquote>neural network is never explicitly being asked to learn the generalized undersdtanding of the training data. The degree of generalization is the implicit ability of the neural network.</blockquote><p>As a result, in a normally trained neural network, information (detected feature) within each neuron is not equally significant with respect to the desired output.</p><p>Simply put normally trained neural networks carry a lot of dead weight in the form of neurons which never learned to generalize data and hence result in lowering the accuracy over the test data.</p><h3>Distillation</h3><p>Distillation enables us to train another neural network using a pre-trained network, without the dead weight of the original neural network.</p><blockquote>Enabling us to compress the size of the network without much loss of accuracy.</blockquote><blockquote>Hence distilled models have higher accuracies than their normally trained counterparts.</blockquote><blockquote>Note: Distillation of knowledge can be done from any form of a learner (logistic regression, SVM, neural networks etc) to any other form of a learner.</blockquote><p>Though for the simplicity of this blog, I’ll be taking references of only neural networks.</p><h3>Generalization of Information</h3><p>Let us take a step back and revise the goal of a neural network:</p><blockquote>predict the output for samples that the network had never seen during training by generalizing the knowledge within the training data.</blockquote><p>Taking an example of a discriminative neural network whose objective is to identify the relevant class for a given input.</p><blockquote><em>Now the neural network returns distribution of probabilities across all classes, even the wrong ones.</em></blockquote><p><em>This tells us a lot about the capability of the network to generalize over the concepts within the training data.</em></p><h3>Measure of Generalization</h3><p>For a decently trained neural network on MNIST following observations would be true:</p><ul><li>even though the probability for number 3 is significantly greater than the probability for the number 8 and number 0</li><li>Probability of 8 and 0 are comparable</li><li>still higher probability of 8 and 0 is comparatively higher than other numbers</li></ul><p>So, the neural network is able to identify that the shape of the number in that image is 3 but <strong>the neural network also suggests that the shape of 3 is quite similar to the shape of numbers 8 and 0 (all are quite curvy)</strong>.</p><h3>Process of Generalization of Information</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*XzVBiFQv4_OiSBpSry8EMg.png" /><figcaption>No, you don’t actually need all this equipment. This is just funny and relevant.</figcaption></figure><p>Let’s start with the obvious, train an enormously gigantic neural network (that your hardware can support) in a normal manner. We’ll be referring to this network as the <strong>cumbersome network</strong>.</p><blockquote>Note: This cumbersome model can very well be an ensemble of multiple normally trained models.</blockquote><p>References:</p><ul><li><strong>soft targets</strong>: network’s probability/weight distribution across all classes</li><li><strong>hard targets</strong>: one-hot vector representation within the original training data</li></ul><p>When the cumbersome model is not a single model but an ensemble of multiple models, the arithmetic/geometric mean of their outputs as the soft-targets.</p><p>The new model can be trained on the same dataset as the original model or on a different dataset called the ‘transfer set’.</p><blockquote>Transfer-Set: Pass the data through the cumbersome model and use its output (probability distribution) as the respective truth values. It can consist of the dataset used to train the original model, new dataset or both.</blockquote><h3>Temperature &amp; Entropy</h3><p>By adjusting the temperature of the soft-targets, it is possible to adjust the size of this transfer-set.</p><p>When soft-targets have high entropy, they give much more information per-training sample than hard-targets (let’s get back to that in a minute).</p><p>This leads to:</p><ul><li>smaller loss, hence smaller correction gradient (backpropagation)</li><li>less variation between the gradients of different training examples</li></ul><p>As a result:</p><ul><li>a greater learning rate can be used to train the model</li><li>a smaller dataset can be used to train the model</li></ul><h3>More on Temperature</h3><p>Something we learned in Physics that applies here ‘<strong>Entropy</strong> increases with <strong>temperature’</strong>.</p><p>Let us use an analogy:</p><p>Imagine a box of balls well stacked on top of each other. If we increase the entropy of the box by shaking it a bit, the balls don’t fall out of the box but rather spread a bit.</p><blockquote>The resulting distribution has a shape similar to the original distribution but the magnitude of its peaks change.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sZ7W5390tlqCMpcHyqY1VA.png" /></figure><p>However, at higher <strong>temperatures</strong>, a certain amount of heat added to the system causes a smaller change in <strong>entropy</strong> than the same amount of heat at a lower <strong>temperature</strong>.</p><p>Following is an example of probability distribution comparing a system with low temperature and high temperature.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UhCn-Q4fdEFQd2pmD31hyQ.png" /></figure><p>Now the formula we use for temperature for transferring knowledge from one neural network to another is:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/808/1*B_czduk1sf0CN1IaRCsThA.png" /></figure><ul><li>q ᵢ : resulting probability</li><li>z ᵢ : logit of a class</li><li>z ⱼ : other logits</li><li>T : temperature</li></ul><blockquote><a href="https://developers.google.com/machine-learning/glossary/#logits">Logits</a>: vector of raw (non-normalized) predictions that a classification model generates, which is ordinarily then passed to a normalization function. If the model is solving a multi-class classification problem, logits typically become an input to the softmax function.</blockquote><p>Well, imagine the following scenarios:</p><p>Imagine an MNIST data-point for number 3 (yes, I like the number 3) but let us concentrate on only 3 (yes yes) classes:</p><ul><li><strong>0</strong></li><li><strong>3</strong></li><li><strong>8</strong></li></ul><blockquote><strong>Dataset Values</strong>: 0 , 1 , 0</blockquote><blockquote><strong>Logits (NN output)</strong>: 0.1 , 0.7 , 0.2</blockquote><blockquote><strong>Temp 0.5:</strong> 0.1831659252 , 0.5939903214 , 0.2228437534</blockquote><blockquote><strong>Temp 1:</strong> 0.254628528 , 0.463963428 , 0.281408044</blockquote><blockquote><strong>Temp 2:</strong> 0.294019937 , 0.3968854015 , 0.3090946615</blockquote><blockquote><strong>Temp 5:</strong> 0.3176924658 , 0.3581972549 , 0.3241102793</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pp1c-qYHDps6MowOOZ6Kbg.png" /></figure><p>Higher temperature results in a softer probability distribution over classes.</p><p>Now, let’s visualize the smoothness of probability distribution as we increase the temperature:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yWUpJW2LFkytjR4Y7rRKSQ.png" /></figure><h3>Training the Distil Model</h3><p>The simplest form of distillation is training a model using the soft targets generated by a cumbersome model with high temperature and are distilled into another model with the same temperature.</p><blockquote>After the training, the temperature of the distilled model is set to 1.</blockquote><p>By now we have established that the distilled network can be trained on a transfer set consisting of soft targets.</p><p>BUT.</p><p>We can also leverage the truth values or the hard targets which are known for all or some of the data.</p><p>One of the most efficient methods of doing this is by using 2 objective functions:</p><ul><li>cross-entropy with soft targets using a high-temperature cumbersome model</li><li>cross-entropy with hard targets using the same cumbersome model but with the temperature set to 1</li></ul><blockquote>Note: The magnitude of soft-targets is scaled to i/T² times, whereas hard-targets undergo no such scaling. So we multiply the soft-targets with T² to normalize the impact of soft-targets and hard-targets.</blockquote><h3>Code</h3><p>HuggingFace actually provides scripts to train your own <strong>DistilBERT </strong>&amp;<strong> DistilRoBERTa</strong>, which are 40% smaller, 60% faster while retaining 99% accuracy of the original model.</p><p>To get the repository use:</p><pre>git clone <a href="https://github.com/huggingface/transformers.git">https://github.com/huggingface/transformers.gi</a>t</pre><p>then</p><pre>cd transformers/examples/distillation</pre><p>First, we will binarize the data, i.e. tokenize the data and convert each token in an index in our model’s vocabulary.</p><pre>python scripts/binarized_data.py \<br>    --file_path data/dump.txt \<br>    --tokenizer_type bert \<br>    --tokenizer_name bert-base-uncased \<br>    --dump_file data/binarized_text</pre><p>HuggingFace follows <a href="https://github.com/facebookresearch/XLM">XLM</a>’s one and smoothes the probability of masking with a factor that put more emphasis on rare words. Thus count the occurrences of each token in the data:</p><pre>python scripts/token_counts.py \<br>    --data_file data/binarized_text.bert-base-uncased.pickle \<br>    --token_counts_dump data/token_counts.bert-base-uncased.pickle \<br>    --vocab_size 30522</pre><p>Training with distillation is really simple once you have pre-processed the data:</p><pre>python train.py \<br>    --student_type distilbert \<br>    --student_config training_configs/distilbert-base-uncased.json \<br>    --teacher_type bert \<br>    --teacher_name bert-base-uncased \<br>    --alpha_ce 5.0 --alpha_mlm 2.0 --alpha_cos 1.0 --alpha_clm 0.0 --mlm \<br>    --freeze_pos_embs \<br>    --dump_path serialization_dir/my_first_training \<br>    --data_file data/binarized_text.bert-base-uncased.pickle \<br>    --token_counts data/token_counts.bert-base-uncased.pickle \<br>    --force # overwrites the `dump_path` if it already exists.</pre><h3>Conclusion</h3><p>By distilling neural networks, we obtain a smaller model that bears a lot of similarities with the original model while being lighter, smaller and faster to run.</p><p><strong>Distilled models are thus an interesting option to put the large-scaled neural networks into production.</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cc02f79698b6" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/distillation-of-knowledge-in-neural-networks-cc02f79698b6">Distillation of Knowledge in Neural Networks</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Ace Deep Learning in a Service-Based Organization]]></title>
            <link>https://medium.com/data-science/ace-deep-learning-in-a-service-based-organization-5cc61265bb6f?source=rss-bd3e39707a79------2</link>
            <guid isPermaLink="false">https://medium.com/p/5cc61265bb6f</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[strategy]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Mukul Malik]]></dc:creator>
            <pubDate>Wed, 11 Sep 2019 03:52:17 GMT</pubDate>
            <atom:updated>2019-09-11T07:24:57.216Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ToImGFzSERw4s-SgADEjzg.jpeg" /><figcaption>From: <a href="https://newsroom.cisco.com/feature-content?type=webcontent&amp;articleId=1895218">https://newsroom.cisco.com/feature-content?type=webcontent&amp;articleId=1895218</a></figcaption></figure><p>Let us begin with a motivational quote:</p><blockquote>Between now and 2030, it [Deep Learning] will create an estimated $13 trillion of GDP growth. — Andrew Ng</blockquote><h3>Present Scenario</h3><p><em>Every tech titan; be it Alphabet, Facebook, Microsoft; sees Data Science as the key to ruling the market.</em></p><p>Jokes like <em>“You are either working for Google or competing against it”</em>, now, feel like nightmares.</p><p>With the tech titans open-sourcing their frequently updated research. So almost every other startup is able to build upon that to create their own Deep Learning solution.</p><blockquote>These startups, due to their low burning rate, are able to offer Deep Learning solutions at much lower rates.</blockquote><p><em>Note: Henceforth, I will be using Deep Learning and Data Science interchangeably.</em></p><h4>Here are my 2 cents on how a Service-Based Organization (SBO) can compete and ace in this environment</h4><h3>MVP over POC</h3><p>Most service-based companies suffer from a phobia of the word “Product”. So service-based companies develop POCs instead.</p><p>Let’s first understand what is what:</p><blockquote><strong>Product</strong> is an article or substance that is manufactured or <strong>refined</strong> for sale.</blockquote><blockquote><strong>Proof of Concept (POC)</strong> is a miniature representation of the end-product, with <strong>a few working features</strong>, that aims of verifying that some concept has practical potential.</blockquote><blockquote><strong>Minimum Viable Product (MVP)</strong> is a development technique in which a new product or website is developed with <strong>sufficient features to satisfy early adopters</strong>.</blockquote><p>Not being a product-based company is no excuse to create half-baked solutions in the name of POCs.</p><p><em>Remember the </em><strong><em>bottom line is always to create what people want</em></strong><em>. While POCs focus over a few working features, MVP stresses over the fact that </em><strong><em>a few working features are useless unless they are satisfying the customer</em></strong><em>.</em></p><p>Often, predominantly in Deep Learning, the working of the solution has this cloud of uncertainty around its end-result. This makes the client nervous about investing in the proposed solution.</p><p>Repeatedly, you will encounter clients who already had a few bad experiences before they approached your organization. Showcasing them something they have a hard time wrapping their head around, especially when they are already frustrated, would frankly be stupid.</p><blockquote>A controlled-demo or a command-line demo; doesn’t make the client any more comfortable with the work.</blockquote><blockquote>The client needs an intuitive and interactive interface; something he can play around with; to satisfy his curiosity and hence overcome the reluctance.</blockquote><h3>Data-Efficient Systems</h3><p>Client requirements are mostly too specific to a niche domain. Hence the availability of a huge high-quality dataset is not available.</p><p>Client-based companies are usually working on multiple projects even under the same client. Their resources are much more distributed across different tasks and even expertise.</p><p>Also,</p><blockquote>Depending upon the client for data, especially for POCs / POVs/MVPs, is a lost cause.</blockquote><p>To circumvent this, companies initially show the demo upon a rather large but irrelevant (to client’s actual ask) dataset with magnificent results. This sets a disastrous precursor, leading to over-commitment but under-delivery.</p><p><em>Data-efficient systems which can produce somewhat satisfactory results over small datasets make way more sense as ‘small’ high-quality dataset is rather easy to find or even build.</em></p><h3>Exact Client Requirements</h3><p><em>Decentralized distribution of resources over multiple clients </em>allows service-based companies to penetrate deep into the requirements of individual clients.</p><p>This allows service-based companies to tailor <em>(modify, not build)</em> the solution to the client’s exact requirements.</p><p><em>Note: This does not decrease the scope of the project, simply an expansion of scope to include clients’ additional requirements.</em></p><p>This allows service-based companies to move away from academia’s SOTA (State-Of-The-Art) ideology <em>(iff and) when required</em> and simply <strong>customize <em>(again, not build)</em> a system that works for that one specific client or type of client.</strong></p><h3>Domain Expertise</h3><p>Service-based companies usually hire domain experts who work upon different projects from the same domain for different clients.</p><p>Their niche but solid expertise results in the <strong><em>assurance of deliverability of quality with minimum risk</em></strong>. This assurance is what the big clients value the most, especially the banks and trading corporations.</p><h3>Out of the Box Solutions</h3><p>In Data Science, it has become a standard practice to develop solutions that require manual efforts on client-side to train systems before using them.</p><p>This is an easy but ineffective way to circumvent developing data-efficient solutions because:</p><blockquote>Data Science projects are usually the necessity of a non-technical team on the client’s side (financial team or business team).</blockquote><blockquote>Training of such solutions is usually too cumbersome for the the personnel of these non-technical teams.</blockquote><p>So the client usually allocates a set of ‘technical personnel’ to train these solutions.</p><p>Hence, even before the client has seen a glimpse of the system, he has to wait for:</p><ul><li><em>education of its technical personnel about the use-case</em></li><li><em>education of its technical personnel about the technology</em></li><li><em>its technical personnel to train of the system</em></li></ul><p><em>On the other hand, </em><strong><em>Out of the Box Solutions</em></strong><em> are the systems that can </em><strong><em>directly be used without any pre-training</em></strong><em>.</em></p><blockquote><em>Therefore, without much ado, the client has something substantial.</em></blockquote><blockquote><em>This provides the client with the an early assurance and confidence in capablities of the service-based company.</em></blockquote><h3>Flexible Architecture</h3><p>Though solutions should be tailored to the client’s requirement, this does<strong><em> not translate to developing use-and-throw solutions</em></strong>.</p><p><em>Avoid developing solutions with a scope too narrow to accommodate any future requirements.</em></p><p>Developing an immutable black-box solution which can be barely modified, usually leads to conflicts with clients and messy patch-up work.</p><p>Enters the <strong>Micro-Service Architecture</strong>.</p><p>Micro-Service is an architectural style that defines a project a collection of services which are:</p><ul><li>highly maintainable and testable</li><li>loosely coupled</li><li>independently deployable</li></ul><blockquote><em>This architecture ensures that the </em><strong><em>client can alter the working of the solution by applying a different permutation, omission or addition of services with minimal effort, as per its satisfaction</em></strong><em>.</em></blockquote><h3>Centralised R&amp;D</h3><p>Service-based companies are usually <em>scared to allocate ‘sufficient’ separate funds for </em><strong><em>non-client-specific R&amp;D</em></strong>.</p><p>Usually while developing a client-specific Deep Learning solution, the scope of the R&amp;D is limited to the ‘current’ requirements of the client and not the domain of that client.</p><p><em>The Deep Learning projects from the same domain (NLP, Computer Vision etc) can have substantial overlap of pipelines, learners, pre-processing steps etc.</em></p><p>In the absence of a centralized team, there is going to be a lot of <strong>redundant efforts</strong>.</p><blockquote>The presence of a centralized team is the key to ensure that client-specific teams have an existing solution to improve upon, rather than building a new solution every time.</blockquote><h3>Educational Tie-Ups</h3><p>When facing a unique non-traditional client requirement or maintaining an effective R&amp;D; experience plays a crucial role.</p><p>Let’s say, you want to design a slot tagger that works well on 20 examples.</p><p>One of the architectures that would work:</p><blockquote>Attention Augmented Bi-Directional Gated Convolution Network using Scattering Wavelets and Fourier Transformation</blockquote><p>Now this architecture, one won’t find in any tutorial nor in any blog.</p><p>This architecture is not an obvious choice until you have researched <em>Information Theory, Signal Processing and Deep Learning</em> for quite some time. The amount of time that is generally infeasible.</p><blockquote>Professors have decades of experience, by virtue of which, they gain tremendous knowledge around a specific field. Their capability to pinpoint and resolve/circumvent issues is unparalleled.</blockquote><h3>Benchmarking</h3><p>Everyone is cynical about pushing blackbox solutions to production.</p><p>None has been able to predict the behaviour of a neural network especially a deep on under diverse scenarios.</p><blockquote>Extensive benchmarking ensures everyone from the developer to the client, is aware of the capabilities and hence the limits of those capabilities for a given solution under different scenarios.</blockquote><h3>Patience</h3><p><strong><em>Deep Learning in the simplest of forms is a few random numbers converging to a certain configuration where they start to learn the relations between the input and the output.</em></strong></p><p>Unlike in traditional software solution, one can <strong><em>at max have a hunch about what might work and what might not</em></strong>.</p><p>Hence creating robust solutions can take more time and effort than what was expected at the beginning.</p><blockquote>A hard definite timeline should never be declared or expected.</blockquote><blockquote>Stakeholders should maintain a buffer on their fronts to incorporate the delays.</blockquote><h4>Special thanks to Deepak Saini &amp; Mayank Saini.</h4><h3>Cheers!</h3><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5cc61265bb6f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/ace-deep-learning-in-a-service-based-organization-5cc61265bb6f">Ace Deep Learning in a Service-Based Organization</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Machine Learning of Human Brain]]></title>
            <link>https://medium.com/data-science/machine-learning-of-human-brain-739ab0419612?source=rss-bd3e39707a79------2</link>
            <guid isPermaLink="false">https://medium.com/p/739ab0419612</guid>
            <category><![CDATA[brain]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[neuroscience]]></category>
            <dc:creator><![CDATA[Mukul Malik]]></dc:creator>
            <pubDate>Tue, 08 May 2018 01:44:00 GMT</pubDate>
            <atom:updated>2018-05-08T01:44:18.356Z</atom:updated>
            <content:encoded><![CDATA[<h4>Learning of Highest Order</h4><p><strong>Q) Why Algorithmic leaps can be better than Hardware leaps?</strong></p><p><strong>Ans)</strong> Hardware constraints create bottlenecks that are hard to tackle as uncertainty of physics at small scale (nano-meters and less) come into play (<em>electrons start jumping around</em>).</p><p><em>At this point, ideas (algorithms) can be used to unleash full potential of the feasible hardware.</em></p><p><strong>Q) Why these improvements are so beneficial for Machine Learning?</strong></p><p><strong>Ans)</strong> Machine Learning (ML) programs can easily involve:</p><ul><li><em>Billions of computations</em></li><li><em>Similar equations</em> with <em>hundreds to millions degree of exponents</em></li><li>Most <em>computations are independent</em> of each other (can be parallelised or distributed)</li></ul><p>Hence even a small optimisation can have very visible effects, saving you from a few hours to months!</p><p><strong>Q) Why refer to the brain for optimisations?</strong></p><p><strong>Ans)</strong> Short answer, without the below mentioned evolution in human brain, we might have looked like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*VDf09EA8Kki047jQ8COR1Q.jpeg" /></figure><h4>The Memory-Prediction Framework</h4><p><strong>Q) What is Neocortex?</strong></p><p><strong><em>Ans) Neocortex</em></strong><em> is the largest part of the cerebral cortex </em>which is the outer layer of the <strong>cerebrum</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/740/1*hviicG4VP2997DdAHBCeMA.jpeg" /></figure><p><strong>Q) Why is neocortex important?</strong></p><p><strong>Ans) Neocortex</strong>, also called the <strong>isocortex</strong>, is the part of the mammalian brain involved in <em>higher-order brain functions</em> such as <em>sensory perception</em>, <em>cognition</em>, <em>generation of motor commands</em>,<em> spatial reasoning</em> and <em>language </em>(<em>partially</em>).</p><blockquote>Extra : The neocortex is further subdivided into the <strong>true isocortex</strong> and the <strong>proisocortex</strong>. More about working of Neocortex can be found <a href="https://www.quora.com/How-do-the-layers-of-the-neocortex-work-What-is-the-neurological-difference-between-them-Are-they-concerned-in-different-operations/answer/Jens-Mowatt?srid=JEbN">here</a>, <a href="http://www.nibb.ac.jp/brish/Gallery/cortexE.html">here</a> and <a href="http://wondergressive.com/neocortex-how-human-memory-works/">here</a> (<em>curiosity might just make you open the links</em>).</blockquote><p><strong>Q) How is neocortex relevant to machine learning?</strong></p><p><strong>Ans) Hierarchical temporal memory</strong> (<strong>HTM</strong>) is based on <strong>interaction of pyramidal </strong><a href="https://en.wikipedia.org/wiki/Neuron"><strong>neurons</strong></a><strong> in the neocortex</strong> of the human brain.</p><p><em>At the core of HTM:</em></p><ul><li><strong><em>Learning algorithms that can store, learn, infer and recall high-order sequences</em></strong></li><li><strong><em>Learn time-based patterns in unlabeled data on a continuous basis</em></strong><em>.</em></li><li><strong><em>Robust to noise and high capacity, meaning that it can learn multiple patterns simultaneously</em></strong></li></ul><p>When applied to computers,<em> HTM is well suited for prediction, anomaly detection, classification and ultimately sensorimotor applications.</em></p><p>The <em>HTM has a hierarchical topology</em> on account of hierarchies observed in some naturally occurring neural networks, such as those observed in the brain.</p><p>Human Memory hierarchy can be really really complex!</p><p>Example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*uTOwrZdCJirdI-tWUT5HaA.png" /></figure><p>Or in Simplified form:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*ICQ4rUOvVsKUAjWs5r3NWw.png" /></figure><p>As one moves up the hierarchy, representations have increased:</p><ul><li><em>Extent: for example, larger areas of the visual field, or more extensive tactile regions.</em></li><li><em>Temporal stability: lower-level entities change quickly, whereas, higher-level percepts (mental-concepts) tend to be more stable.</em></li><li><em>Abstraction: through the process of successive extraction of invariant features, increasingly abstract entities are recognised.</em></li><li><em>Less resource intensive: require much less resources especially memory.</em></li></ul><blockquote><em>Human brain’s algorithmic efficiency is almost petrifyingly high!</em></blockquote><h3>Philosphy’s Treasure, The Introspection</h3><p><strong>Q) What is Introspection?</strong></p><p><strong>Asn) Introspection </strong>or<strong> metacognition</strong>, is self-awareness about one’s thinking or ‘ability to think about the thinking’. A high-level mental process (very high).</p><p>Accurate introspection ables discrimination between correct decisions and incorrect ones.</p><p><em>Basically, a learner developing a conscious. Being self-able to reject or encourage patterns during training. </em><strong><em>Hence a self-optimising learner, the very necessity of General Learning.</em></strong></p><blockquote>Example:</blockquote><blockquote><strong>Bistable Stimuli</strong> (two pictures in binocular vision, one from each eye) are one of the most popular approaches to studying the neural mechanism of conscious visual perception.</blockquote><blockquote>Such stimuli contain conflicting information, which the visual system cannot integrate into a unified percept.</blockquote><blockquote>This causes the perceptual (ability to interpret) state of the observer to change every few seconds trying to figure out correct interpretation. All this while the physical stimulus remains the same.</blockquote><blockquote>In this case the frontal lobe of cerebrum makes a self aware (introspective) decision to reject either information or modify each information to eliminate conflict.</blockquote><blockquote>Details of this can be found in this <a href="https://www.google.co.in/url?sa=t&amp;rct=j&amp;q=&amp;esrc=s&amp;source=web&amp;cd=1&amp;cad=rja&amp;uact=8&amp;ved=0ahUKEwj_v8mY2NzUAhUVTo8KHclUARAQFggoMAA&amp;url=http%3A%2F%2Fjournal.frontiersin.org%2Farticle%2F10.3389%2Ffnhum.2014.00527%2Fpdf&amp;usg=AFQjCNHbtCyawSIOUx0VkfkbtpMlEh18Wg">paper</a> by Natalia Zaretskaya and Marine Narinyan</blockquote><p><strong>Q) What is gray matter?</strong></p><p><strong>Ans)</strong> The interior of the <a href="https://en.wikipedia.org/wiki/Central_nervous_system">central nervous system</a> is organised into <em>gray and white matter</em>.</p><p><em>Gray matter</em> consists of nerve cells embedded in <a href="https://en.wikipedia.org/wiki/Neuroglia"><em>neuroglia</em></a><em> (nervous tissue made up of large number of nerve cells)</em>; it has a gray color.</p><p><strong>Q) Why is gray matter important?</strong></p><p><strong>Ans) </strong>In humans the gray-matter volume might help clarify the extent to which a person’s confidence about his introspective abilities is supported. Also how accurate his introspective abilities are.</p><p>More gray matter usually signifies more intelligence.</p><p>As Einstein once said “Gray Matter bitches!” ….. no, he actually didn’t.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/798/1*I8SHpvmfSWKf9z_8TuwW9w.png" /></figure><p><strong>Q) Then why don’t we directly implement the working of gray matter?</strong></p><p><strong>Ans)</strong> Simply because the working of gray matter is very very complex. We do not know the exact correlation between ‘<em>confidence in</em>’ and ‘<em>accuracy of</em>’ one’s introspective abilities.</p><p><strong>Q) So what do we know?</strong></p><p><strong>Ans) </strong>Introspection is still (after millions years of evolution) a rare abilities even for complex organisms with most organisms lacking this ability completely. This has been one of the most recent development in evolution.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GoHDN3E8KoJKZNKCbx-O-Q.jpeg" /></figure><p><strong>Q) Which part of the brain is responsible for introspection?</strong></p><p><strong>Ans)</strong> <strong>Prefrontal cortex</strong> in cerebrum.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/1*QuzXJnO1XGcUt3i4o-qCqQ.jpeg" /></figure><p><strong>Prefrontal cortex</strong> (<strong>PFC</strong>) is the cerebral cortex which covers the front part of the frontal lobe.</p><p>Two parts of this area are of great interest:</p><ul><li><em>Anterior prefrontal cortex</em></li><li><em>Medial prefrontal cortex</em></li></ul><p><strong>Q) What’s special about Anterior prefrontal cortex?</strong></p><p><strong>Ans) Anterior prefrontal cortex</strong> has been associated with top-level processing abilities that are thought to set humans apart from other animals.</p><p><em>Most of the gray-matter which plays the role in introspection exists in this region</em> (right anterior prefrontal cortex, to be precise). The structure of neighbouring white matter plays some (unclear) role in introspective capabilities too.</p><p>This brain region has been implicated in planning <strong>complex cognitive behaviour</strong>, <strong>personality expression</strong>, <strong>decision making</strong>, and <strong>moderating social behaviour</strong>. The basic activity of this brain region is considered to be <strong>orchestration of thoughts and actions in accordance with internal goals</strong>.</p><p><strong>Q) What does that mean in terms of machine learning?</strong></p><p><strong>Ans)</strong> I’ll answer this in short point</p><ul><li>Learner that is<em> aware of it’s decision</em></li><li>Leaner that<em> adapts to it’s environment</em></li><li>Learner that can<em> set it’s goals on the go</em></li><li>Learner that can <em>Generalise massive information in just a few patterns</em></li></ul><p><strong>Q) And what’s special about Medial prefrontal cortex?</strong></p><p><strong>Ans) Medial prefrontal cortex</strong> <strong>(mPFC)</strong> is considered to be a part of the <strong>brain’s reward system</strong>.</p><p>The mPFC is part of the <em>mesocorticolimbic dopaminergic</em> system (<strong><em>basically it generates dopamine, the ‘happy enzyme’</em></strong>).</p><p>Evidence for the involvement of the mPFC in reward-related mechanisms comes mainly from three types of studies:</p><ul><li><strong>Conditioned Place Preference (CPP)</strong> : measure the motivational effects of objects or experiences</li><li><strong>Intracranial Self-Stimulation (ICSS)</strong> : what conditions trigger brain reward system</li><li><strong>Self-Administration</strong> : which external drug (like insulin for diabetes) the brain wants</li></ul><p>What’s interesting about this reward system is:</p><ul><li><em>Different subareas of the mPFC appear to be differently involved in the ‘rewarding actions’ of different drugs</em>. This indicates existence of <strong>multiple reward systems</strong>.</li><li><em>Some drugs can produce reward directly within the mPFC, and that some drugs, even though not having direct rewarding effects within the mPFC, depend on the function of the mPFC for the mediation of their rewarding effects.</em> Which can be represented as <strong>multi-tier reward process</strong>.</li></ul><p><strong>Q) What does that mean in terms of machine learning?</strong></p><p><strong>Ans)</strong> Even shorter answer:</p><ul><li><em>WHOLE OF REINFORCEMENT LEARNING</em></li></ul><p><strong>Q) What’s reinforcement learning?</strong></p><p><strong>Ans)</strong> A type of machine learning that involves:</p><ul><li><em>No Supervision</em> (trial and error on bases of rewards)</li><li><em>Good or Bad is decided a few steps later, not instantaneously</em></li><li><em>Dynamic system so time really matters, sequence of steps really matters</em></li><li><em>Agent gets to take action and alter/influence it’s environment</em></li><li>For <em>every action the Agent gets a reward</em> and it’s <em>job is to maximise it</em></li></ul><p>It works something like this:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FL4KBBAwF_bE%3Fstart%3D26%26feature%3Doembed%26start%3D26&amp;url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DL4KBBAwF_bE&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FL4KBBAwF_bE%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/63795f8f23479e32692fc0e326c3a62d/href">https://medium.com/media/63795f8f23479e32692fc0e326c3a62d/href</a></iframe><p><strong>Q) Confused?</strong></p><p><strong>Ans) </strong><em>‘</em>Reward’ is what enables to prioritise one decision over other.</p><p>It is a scalar feedback, quantifying how well agent is adapting/interacting with environment.</p><p>Like you give treat to your dog if he sits first else no treat (why u no gud boi?!).</p><p>The reinforcement learner simply follows the steps which maximise the cumulative reward.</p><p><em>Multiple reward systems</em> essentially mean that brain has inbuilt <strong>multi </strong><a href="https://papers.nips.cc/paper/1786-actor-critic-algorithms.pdf"><strong>actor-critic</strong></a> reinforcement learning mechanism.</p><p>In simple words, actor-critic learning involves an <strong>actor who takes an action</strong> and <strong>critic finds pros and cons (criticises) in that action</strong>.</p><p><strong>Q) Role of reward system sounds very similar to introspection??</strong></p><p><strong>Ans)</strong> Well that’s because reward system does play a major part in one’s introspective abilities. This relation, though, is REALLY COMPLEX.</p><blockquote>“Reinforcemennt learning is simply Science of Decision Making. That’s what makes it so General.” — David Silver</blockquote><p><strong><em>So Prefrontal Lobe is frontier in field of General Intelligence!</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/1*BsFDEUquCZX5_eLyceSnQg.jpeg" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=739ab0419612" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/machine-learning-of-human-brain-739ab0419612">Machine Learning of Human Brain</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Information Theory of Neural Networks]]></title>
            <link>https://medium.com/data-science/information-theory-of-neural-networks-ad4053f8e177?source=rss-bd3e39707a79------2</link>
            <guid isPermaLink="false">https://medium.com/p/ad4053f8e177</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[information-theory]]></category>
            <dc:creator><![CDATA[Mukul Malik]]></dc:creator>
            <pubDate>Wed, 04 Apr 2018 14:31:46 GMT</pubDate>
            <atom:updated>2018-07-31T03:25:50.402Z</atom:updated>
            <content:encoded><![CDATA[<h4>Opening the Black Box …. Somewhat</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/794/1*cdrrLhNd5KrQWv9IS7i8Kw.png" /></figure><blockquote>“Information: the negative reciprocal value of probability.” — Claude Shannon</blockquote><p>Aim of this blog is not to understand the underlying mathematical concepts behind Neural Network but to visualise Neural Networks in terms of information manipulation.</p><h3>Encoder-Decoder</h3><p>Before we start:</p><blockquote>Encoder-Decoder is not two CNNs/RNNs combined together! Neither have to be neural network in fact!</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ICOovMPf3eR-HscOWjXR6Q.jpeg" /></figure><p>Originally, a concept of information theory. <a href="https://www.cs.toronto.edu/~hinton/science.pdf">Encoder is simply compresses the information and decoder expands the encoded information.</a></p><p>In case of machine learning, both encoding and decoding are both lose-full processes i.e. some information is always lost.</p><p>The encoded output of the encoder is called the<em> context vector</em> and this is the input for the decoder.</p><p>There are two ways to setup an encoder-decoder setting:</p><ol><li>The Decoder in inverse function of Encoder. This way the decoder tries to reproduce the original information. This is used to de-noise the data. This setting has a special name, called auto-encoder.</li><li>The Encoder is a compression algorithm and decoder is generating algorithm. This helps to transfer context from one format to another.</li></ol><blockquote>Example Applications:</blockquote><blockquote>Auto-Encoder: Encoder compressing English text into a vector. Decoder generating original English text from the vector.</blockquote><blockquote>Encoder-Decoder: Encoder compressing English text into a vector. Decoder generating French translation of the original text from the vector.</blockquote><blockquote>Encoder-Decoder: Encoder compressing English text into a vector. Decoder generating image from the content of text.</blockquote><h3>Information Theory</h3><p>Now, if I say every neural network, itself, is an encoder-decoder setting; it would sound absurd to most.</p><p><strong>Let’s re-imagine the neural networks.</strong></p><p>Let input layer be X and their real tags/classes (present in the training set) be Y. Now we already know Neural Networks find the underlying function between X and Y.</p><p><em>So X can be seen as High-Entropy distribution of Y. High entropy because X contains the information Y but it also a lot of other information.</em></p><blockquote>Example:</blockquote><blockquote>“This boy is good.” contains enough information to tell us about it’s ‘positive’ sentiment.</blockquote><blockquote>But.</blockquote><blockquote>It also contains things like:</blockquote><blockquote>1. It’s a specific Boy</blockquote><blockquote>2. It’s just one Boy</blockquote><blockquote>3. Tense of sentence is present</blockquote><blockquote>Now no-entropy version of this sentence would be ‘positive’. Yes that’s also the output. We’ll come back to that in a while.</blockquote><p>Now imagine every hidden layer as a single variable H ( so layers will be named as H0, H1 ….. H(n-1) )</p><p>Now every layer becomes a variable and the neural network becomes a <a href="https://brilliant.org/wiki/markov-chains/">Markov Chain</a>. Markov Chain because each variable is only dependent upon the previous layer only.</p><p><em>So essentially each layer forms a partisan of information.</em></p><p>Following is a visualisation of a Neural Network as a Markov Chain.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xomkoSgT4-Icsu8NnqU0mg.jpeg" /></figure><p>The last layer Y_ is supposed to result in least entropy output (with respect to ‘Y’, the original tag/class).</p><p><em>This process of obtaining Y_ is squeezing the information in X layer as it flows through H layers and retaining only the information most relevant to Y. This is the information bottleneck.</em></p><h3>Mutual Information</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GXaA5KmLrl4P-FDSJdtrdA.jpeg" /></figure><p>I(X,Y) = H(X) — H(X|Y)</p><p>H -&gt; Entropy</p><p>H(X) -&gt; Entropy of X</p><p>H(X|Y) -&gt; Conditional Entropy of X given Y</p><p>In other words, H(X|Y) signifies how much uncertainty is removed form X if Y is known.</p><h4>Properties of Mutual Information</h4><ol><li>When you move along the Markov Chain the mutual information only decreases</li><li>Mutual Information is invariant to reparameterization i.e. shuffling values in a layer won’t change the output</li></ol><h3>Revisiting Bottleneck</h3><p>In the Markov Representation of Neural Network, every layer becomes a partition of information.</p><p>In Information Theory these partitions are known as Successive Refinement of Relevant Information. You don’t have to worry about the details.</p><p>Another way of seeing this is the input being encoded and decoded into the output.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fyOYhqxXJNrzN7SxSuKiJA.jpeg" /></figure><p>So, for enough hidden layers:</p><ol><li>the Sample Complexity of deep neural network is determined by encoded Mutual Information at Last Hidden Layer</li><li>the Accuracy is determined by Decoded Mutual Information of Last Hidden Layer</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/558/1*dbVNyK7nacJHhvutuzyEdg.jpeg" /></figure><p><em>Sample Complexity is the number and variety of examples one needs to receive certain accuracy.</em></p><h3>Mutual Information along the Training Phase</h3><p>We calculate Mutual Information between</p><ol><li>Layer and Input</li><li>Layer and Output</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*1rm8So6vBDdN5oWFJeDrHA.jpeg" /><figcaption>Initial Conditions</figcaption></figure><p>Initially,weights are randomly initialised. So barely anything is known about the correct output. With successive layers the mutual information about input decreases and the information in hidden layers about the output is low as well.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*-_YbjhJ9h_6sZR1a6Yj5yQ.jpeg" /><figcaption>Compression Phase</figcaption></figure><p>As we train the Neural Network the plots start moving up, signifying gain of information about the output.</p><p>But.</p><p>Plots also start shifting towards the right side, signifying increase of information in latter layers about the input.</p><p>This his the longest phase. Here the density of the plots is maximum and plots are concentrated at the top right. This signifies compression of information about the input in relation to the output.</p><p>This is called the compression phase.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*JXNRt_1D2zbXK5l6ejVNnQ.jpeg" /><figcaption>Expansion Phase</figcaption></figure><p>After the compression phase, the plots starts shifting towards the top but also to left side.</p><p>This signifies, with successive layers, there is loss of information about the input and what’s retained in the last layer is the <em>lowest entropy information </em>about the output.</p><h3>Virtualisation</h3><p>The Markov Chain version of Neural Network highlights one more point, learning happens from layer to layer. A layer has all the information it needs to predict the output (plus some noise).</p><p>So we use each layer to predict the output. This helps us peep into the layer-wise knowledge of the so called black box.</p><p>This gives us a perspective about how many layers are required to make an accurate enough prediction of output. If saturation is achieved at earlier layers, the layers succeeding this layer can be pruned/dropped.</p><p>These layers are usually very of hundreds or thousands of dimensions. Our evolution doesn’t allow us to visualise anything beyond the 3-dimensions. So we use dimension reduction techniques.</p><p>There are various methods of performing dimension reduction. Cristopher Olah has a brilliant <a href="http://colah.github.io/posts/2014-10-Visualizing-MNIST/">blog</a> explaining those methods. I won’t go into the details of t-SNE, you can check this <a href="https://distill.pub/2016/misread-tsne/">blog</a> for details.</p><p>To keep it short, t-SNE tries to reduce dimension retaining the neighbours from higher dimensions in lower dimensions. So this results in pretty accurate 2D and 3D plots.</p><p>Following are the layer plots of a language model with 2 layers.</p><p>About the plots:</p><ol><li>Selected 16 words</li><li>Used the final language model to find N synonyms of the above 16 words (N=200 for 2D and N=50 for 3D)</li><li>Find the representation vector for each word at each layer</li><li>Find 2D and 3D reduced representation of each of above selected words and their synonyms using t-SNE</li><li>Plot the reduced representations</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/883/1*TkvVA6cxM6rcl0GHGps4ZQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/883/1*7HqPTclRdGWCm2t_49WxZQ.png" /><figcaption>2D plots of Layer 1 and Layer 2</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PbY873h94s7r5qKm7GiSiA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9M3jIfQPi2kLhc3IwVAuFg.png" /><figcaption>3D plots of Layer 1 and Layer 2</figcaption></figure><h3>Summary</h3><ol><li>Most every deep neural network acts like an encoder-decoder</li><li>Most of the training time is spent in compression phase</li><li>Layers are learned bottom-up</li><li>Beyond compression phase, more the neural network forgets about the input, the more accurate it gets (wiping out the irrelevant part of input).</li></ol><p>I have excluded mathematics from this blog. If you are comfortable enough with the mathematics of Information Theory, Game Theory, Learning Theory etc then do watch this <a href="https://www.youtube.com/watch?v=bLqJHjXihK8&amp;t=856s"><strong>video</strong></a> of the Mastero <strong>Naftali Tishby</strong>.</p><blockquote>Thank You \(-_- )/</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ad4053f8e177" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/information-theory-of-neural-networks-ad4053f8e177">Information Theory of Neural Networks</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Information Theory of Neural Networks]]></title>
            <link>https://medium.com/hackernoon/information-theory-of-neural-networks-c96a0f0a8d9?source=rss-bd3e39707a79------2</link>
            <guid isPermaLink="false">https://medium.com/p/c96a0f0a8d9</guid>
            <category><![CDATA[information-theory]]></category>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[theory-of-neural-networks]]></category>
            <dc:creator><![CDATA[Mukul Malik]]></dc:creator>
            <pubDate>Tue, 03 Apr 2018 16:26:07 GMT</pubDate>
            <atom:updated>2019-05-08T08:38:28.517Z</atom:updated>
            <content:encoded><![CDATA[<h4>Opening the Black Box …. Somewhat</h4><blockquote>“Information: the negative reciprocal value of probability.” — Claude Shannon</blockquote><p>Aim of this blog is not to understand the underlying mathematical concepts behind Neural <a href="https://hackernoon.com/tagged/network">Network</a> but to visualise Neural Networks in terms of information manipulation.</p><h3>Encoder-Decoder</h3><p>Before we start:</p><blockquote>Encoder-Decoder is not two CNNs/RNNs combined together! Neither have to be neural network in fact!</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ICOovMPf3eR-HscOWjXR6Q.jpeg" /></figure><p>Originally, a concept of information theory. <a href="https://www.cs.toronto.edu/~hinton/science.pdf">Encoder is simply compresses the information and decoder expands the encoded information.</a></p><p>In case of machine <a href="https://hackernoon.com/tagged/learning">learning</a>, both encoding and decoding are both lose-full processes i.e. some information is always lost.</p><p>The encoded output of the encoder is called the<em> context vector</em> and this is the input for the decoder.</p><p>There are two ways to setup an encoder-decoder setting:</p><ol><li>The Decoder in inverse function of Encoder. This way the decoder tries to reproduce the original information. This is used to de-noise the data. This setting has a special name, called auto-encoder.</li><li>The Encoder is a compression algorithm and decoder is generating algorithm. This helps to transfer context from one format to another.</li></ol><blockquote>Example Applications:</blockquote><blockquote>Auto-Encoder: Encoder compressing English text into a vector. Decoder generating original English text from the vector.</blockquote><blockquote>Encoder-Decoder: Encoder compressing English text into a vector. Decoder generating French translation of the original text from the vector.</blockquote><blockquote>Encoder-Decoder: Encoder compressing English text into a vector. Decoder generating image from the content of text.</blockquote><h3>Information Theory</h3><p>Now, if I say every neural network, itself, is an encoder-decoder setting; it would sound absurd to most.</p><p><strong>Let’s re-imagine the neural networks.</strong></p><p>Let input layer be X and their real tags/classes (present in the training set) be Y. Now we already know Neural Networks find the underlying function between X and Y.</p><p><em>So X can be seen as High-Entropy distribution of Y. High entropy because X contains the information Y but it also a lot of other information.</em></p><blockquote>Example:</blockquote><blockquote>“This boy is good.” contains enough information to tell us about it’s ‘positive’ sentiment.</blockquote><blockquote>But.</blockquote><blockquote>It also contains things like:</blockquote><blockquote>1. It’s a specific Boy</blockquote><blockquote>2. It’s just one Boy</blockquote><blockquote>3. Tense of sentence is present</blockquote><blockquote>Now no-entropy version of this sentence would be ‘positive’. Yes that’s also the output. We’ll come back to that in a while.</blockquote><p>Now imagine every hidden layer as a single variable H ( so layers will be named as H0, H1 ….. H(n-1) )</p><p>Now every layer becomes a variable and the neural network becomes a <a href="https://brilliant.org/wiki/markov-chains/">Markov Chain</a>. Markov Chain because each variable is only dependent upon the previous layer only.</p><p><em>So essentially each layer forms a partisan of information.</em></p><p>Following is a visualisation of a Neural Network as a Markov Chain.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xomkoSgT4-Icsu8NnqU0mg.jpeg" /></figure><p>The last layer Y_ is supposed to result in least entropy output (with respect to ‘Y’, the original tag/class).</p><p><em>This process of obtaining Y_ is squeezing the information in X layer as it flows through H layers and retaining only the information most relevant to Y. This is the information bottleneck.</em></p><h3>Mutual Information</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GXaA5KmLrl4P-FDSJdtrdA.jpeg" /></figure><p>I(X,Y) = H(X) — H(X|Y)</p><p>H -&gt; Entropy</p><p>H(X) -&gt; Entropy of X</p><p>H(X|Y) -&gt; Conditional Entropy of X given Y</p><p>In other words, H(X|Y) signifies how much uncertainty is removed form X if Y is known.</p><h4>Properties of Mutual Information</h4><ol><li>When you move along the Markov Chain the mutual information only decreases</li><li>Mutual Information is invariant to reparameterization i.e. shuffling values in a layer won’t change the output</li></ol><h3>Revisiting Bottleneck</h3><p>In the Markov Representation of Neural Network, every layer becomes a partition of information.</p><p>In Information Theory these partitions are known as Successive Refinement of Relevant Information. You don’t have to worry about the details.</p><p>Another way of seeing this is the input being encoded and decoded into the output.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fyOYhqxXJNrzN7SxSuKiJA.jpeg" /></figure><p>So, for enough hidden layers:</p><ol><li>the Sample Complexity of deep neural network is determined by encoded Mutual Information at Last Hidden Layer</li><li>the Accuracy is determined by Decoded Mutual Information of Last Hidden Layer</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/558/1*dbVNyK7nacJHhvutuzyEdg.jpeg" /></figure><p><em>Sample Complexity is the number and variety of examples one needs to receive certain accuracy.</em></p><h3>Mutual Information along the Training Phase</h3><p>We calculate Mutual Information between</p><ol><li>Layer and Input</li><li>Layer and Output</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*1rm8So6vBDdN5oWFJeDrHA.jpeg" /><figcaption>Initial Conditions</figcaption></figure><p>Initially,weights are randomly initialised. So barely anything is known about the correct output. With successive layers the mutual information about input decreases and the information in hidden layers about the output is low as well.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*-_YbjhJ9h_6sZR1a6Yj5yQ.jpeg" /><figcaption>Compression Phase</figcaption></figure><p>As we train the Neural Network the plots start moving up, signifying gain of information about the output.</p><p>But.</p><p>Plots also start shifting towards the right side, signifying increase of information in latter layers about the input.</p><p>This his the longest phase. Here the density of the plots is maximum and plots are concentrated at the top right. This signifies compression of information about the input in relation to the output.</p><p>This is called the compression phase.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*JXNRt_1D2zbXK5l6ejVNnQ.jpeg" /><figcaption>Expansion Phase</figcaption></figure><p>After the compression phase, the plots starts shifting towards the top but also to left side.</p><p>This signifies, with successive layers, there is loss of information about the input and what’s retained in the last layer is the <em>lowest entropy information </em>about the output.</p><h3>Virtualisation</h3><p>The Markov Chain version of Neural Network highlights one more point, learning happens from layer to layer. A layer has all the information it needs to predict the output (plus some noise).</p><p>So we use each layer to predict the output. This helps us peep into the layer-wise knowledge of the so called black box.</p><p>This gives us a perspective about how many layers are required to make an accurate enough prediction of output. If saturation is achieved at earlier layers, the layers succeeding this layer can be pruned/dropped.</p><p>These layers are usually very of hundreds or thousands of dimensions. Our evolution doesn’t allow us to visualise anything beyond the 3-dimensions. So we use dimension reduction techniques.</p><p>There are various methods of performing dimension reduction. Cristopher Olah has a brilliant <a href="http://colah.github.io/posts/2014-10-Visualizing-MNIST/">blog</a> explaining those methods. I won’t go into the details of t-SNE, you can check this <a href="https://distill.pub/2016/misread-tsne/">blog</a> for details.</p><p>To keep it short, t-SNE tries to reduce dimension retaining the neighbours from higher dimensions in lower dimensions. So this results in pretty accurate 2D and 3D plots.</p><p>Following are the layer plots of a language model with 2 layers.</p><p>About the plots:</p><ol><li>Selected 16 words</li><li>Used the final language model to find N synonyms of the above 16 words (N=200 for 2D and N=50 for 3D)</li><li>Find the representation vector for each word at each layer</li><li>Find 2D and 3D reduced representation of each of above selected words and their synonyms using t-SNE</li><li>Plot the reduced representations</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/883/1*TkvVA6cxM6rcl0GHGps4ZQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/883/1*7HqPTclRdGWCm2t_49WxZQ.png" /><figcaption>2D plots of Layer 1 and Layer 2</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PbY873h94s7r5qKm7GiSiA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9M3jIfQPi2kLhc3IwVAuFg.png" /><figcaption>3D plots of Layer 1 and Layer 2</figcaption></figure><h3>Summary</h3><ol><li>Most every deep neural network acts like an encoder-decoder</li><li>Most of the training time is spent in compression phase</li><li>Layers are learned bottom-up</li><li>Beyond compression phase, more the neural network forgets about the input, the more accurate it gets (wiping out the irrelevant part of input).</li></ol><p>I have excluded mathematics from this blog. If you are comfortable enough with the mathematics of Information Theory, Game Theory, Learning Theory etc then do watch this <a href="https://www.youtube.com/watch?v=bLqJHjXihK8&amp;t=856s"><strong>video</strong></a> of the Mastero <strong>Naftali Tishby</strong>.</p><blockquote>Thank You \(-_- )/</blockquote><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fupscri.be%2Fdde502%3Fas_embed%3Dtrue&amp;dntp=1&amp;url=https%3A%2F%2Fupscri.be%2Fhackernoon%2F&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=upscri" width="800" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/3c851dac986ab6dbb2d1aaa91205a8eb/href">https://medium.com/media/3c851dac986ab6dbb2d1aaa91205a8eb/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c96a0f0a8d9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/hackernoon/information-theory-of-neural-networks-c96a0f0a8d9">Information Theory of Neural Networks</a> was originally published in <a href="https://medium.com/hackernoon">HackerNoon.com</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Basics of Neural Network]]></title>
            <link>https://becominghuman.ai/basics-of-neural-network-bef2ba97d2cf?source=rss-bd3e39707a79------2</link>
            <guid isPermaLink="false">https://medium.com/p/bef2ba97d2cf</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-intelligence]]></category>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Mukul Malik]]></dc:creator>
            <pubDate>Tue, 03 Apr 2018 07:29:03 GMT</pubDate>
            <atom:updated>2018-04-04T13:36:28.800Z</atom:updated>
            <content:encoded><![CDATA[<p>The aim of this blog is just to get one acquainted with theory of Neural Networks.</p><h3>Introduction</h3><p>Neural Networks started off as an attempt to to replicate the working of the human brain in order to make things more intelligent. Even something like this is not necessarily always complex. At its heart, it is just multiplication and differentiation.</p><blockquote>I am really bad at remembering maths. But. Once I tried learning concepts I realised I am pretty decent at visualising maths. <a href="https://www.youtube.com/channel/UCYO_jab_esuFRV4b17AJtAw">3Blue1Brown</a> has some amazing visualisations of mathematical concepts.</blockquote><blockquote>So there are many ways you can understand a concept, choose whichever suits you, being persistent about the learning part. At the end knowing maths is a useful tool when it comes to optimisations or experimentations.</blockquote><figure><a href="http://bit.ly/BusinessOfBots"><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*S_E42x8_h-YHrGnfJLosoQ.png" /></a></figure><p>Neural Network is, <em>usually</em>, a supervised method of learning. This means there is presence of a training set. Ideally this set contains examples with their absolutely truth values (tags, classes etc). In case of sentiment analysis the training set would be list of sentences and their respective correct sentiment.</p><p><em>Note: Unlabelled datasets can also be used to train neural networks but we are considering the basic case here.</em></p><blockquote>Let’s refer to the texts as X and they labels/tags as Y. There is some function that defines the relationship between X and Y. In case of sentiment analysis that would be what features (words / phrases / sentence structure etc) lead to a sentence being negative or positive. Earlier people used to find these features manually, this was called feature engineering. Neural Network automated this process.</blockquote><h3>Top 3 Most Popular Ai Articles:</h3><blockquote><a href="https://becominghuman.ai/a-noobs-guide-to-implementing-rnn-lstm-using-tensorflow-1907a5bbb1fa">1. A noob’s guide to implementing RNN-LSTM using Tensorflow</a></blockquote><blockquote><a href="https://becominghuman.ai/keras-cheat-sheet-neural-networks-in-python-738c0d170c2e">2. Keras Cheat Sheet: Neural Networks in Python</a></blockquote><blockquote><a href="https://becominghuman.ai/making-a-simple-neural-network-2ea1de81ec20">3. Making a Simple Neural Network</a></blockquote><h3>Working</h3><p>An Artificial Neural Network is made up of 3 components:</p><ul><li>Input Layer</li><li>Hidden (computation) Layers</li><li>Output Layer</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/592/1*_lqiFBTqoskOWZ72FrXr3Q.jpeg" /></figure><p>Furthermore the learning happens in two steps:</p><ul><li>Forward-Propagation</li><li>Back-Propagation</li></ul><p>In simple words</p><ul><li>forward propagation is making a guess about the answer</li><li>back propagation is minimising the error between the actual answer and guessed answer</li></ul><h3>Forward-Propagation</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*6OvCMEC9mptplP4zjd5nbw.png" /></figure><p>Randomly initialize weights</p><ul><li>w1</li><li>w2</li><li>w3</li></ul><p>Data at input layer is multiplied with weights to form hidden layer</p><ul><li>h1 = (x1 * w1) + (x2 * w1)</li><li>h2 = (x1 * w2) + (x2 * w2)</li><li>h3 = (x1 * w3) + (x2 * w3)</li></ul><p>Output of hidden layer is passed through a non-linear function also known as activation function to form guessed output</p><ul><li>y_ = fn( h1 , h2, h3 )</li></ul><h3>Backward Propagation</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_cjsYd9V5dYqBy1yjN5Jwg.jpeg" /></figure><ul><li>total_error is calculated by calculating the difference between the expected value ‘y’ (value in training set) and observed value ‘y_’ (value attained from forward propagation) by passing them through a <em>cost function</em>.</li><li>Partial derivative of error is calculated w.r.t each weight ( these partial differentials are measures of contribution of each weight in total_loss )</li><li>The differentials are then multiplied by a small number called learning rate ( η )</li><li>The resultant is then subtracted from the respective weights</li></ul><p>The result of backprop is the following updated weights:</p><ul><li>w1 = w1 - (η * ∂(err) / ∂(w1))</li><li>w2 = w2 - (η * ∂(err) / ∂(w2))</li><li>w3 = w3 - (η * ∂(err) / ∂(w3))</li></ul><blockquote><em>Basically we initialise random weights and assume they would produce accurate answers, sounds trivial because it is but seems to work pretty well.</em></blockquote><blockquote><em>Those familiar with Taylor Series, backpropogation shares the same end result with it. But instead of an indefinite series we try to optimise the first element only.</em></blockquote><blockquote>Biases are weights added to hidden layers. They too are randomly initialised and updated in similar manner as the hidden layer. While the role of hidden layer is to map the shape of the underlying function in the data, the role of bias is to laterally shift the learned function so it overlaps with the original function.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/585/1*TddvF4PDLfU4veIIf3iMjg.jpeg" /></figure><h3>Partial Derivatives</h3><p>Partial Derivatives are calculated so we know what was the contribution of error by each weight.</p><p>The need of derivatives obvious if you think of it.</p><p>For example think of a neural network trying to find the optimal speed (velocity) of a self driving car. Now if the car finds out the the is either faster or slower than desired speed neural network will c<em>hange its speed by either accelerating or decelerating the car</em>. What is accelerating/decelerating? <em>Derivatives of speed.</em></p><p>Let’s explain the need of ‘Partial Derivatives’ with an example as well:</p><p>Let’s say that a few kids were asked to throw dart at a dart-board, aiming at the center. The initial results were:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*aIAavOk4K-XdY8_ki9AClg.jpeg" /></figure><p>Now if we found total loss and simply subtracted that from all the weights then we generalize the mistakes made by each student. So let’s say a kid aimed too low but we ask all the kids to aim high then it results in:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*OQgwLMHS45MVyyCyGCxziQ.jpeg" /></figure><p>The error of a few students might decrease but overall error still increases.</p><p>By finding partial derivatives we find what was the error by each weight individually. Correcting each weight individually results in following results:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*lbpJZuc21qZt9sCgwk0coQ.jpeg" /></figure><h3>Hyper Parameters</h3><p>While neural network is used to automate feature selection, there are still a few parameters that we have to input manually.</p><h4>Learning Rate</h4><p>Learning Rate is again a very crucial hyper-parameter. If the learning rate is too small then even after training the neural network for long time, it will still be away from the optimal results. Results would look something like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*eX36Ru0zg5NlPapjs09A0Q.jpeg" /></figure><p>Instead, if the learning rate is too high then the learner jumps to conclusions too soon. Producing following results:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*LOOTMiNJ1iU11ZG87w7MRw.jpeg" /></figure><h4>Activation Function</h4><p>Activation Function is one of most powerful arsenal, which is responsible for powers Neural Networks advertised to have. Vaguely, it decides which neurons will be activated, in other words what information would be passed to further layers.</p><p>Without activation functions, deep nets lose a bulk of their representation learning power.</p><p><em>These functions’ non-linearity is responsible for increased degree of freedom of the learners, enabling them to generalize problems of high dimensionality in lower dimensions.</em></p><p>Below are few examples of popular Activation Functions:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/930/1*52pviFr_uDX_JeuBI01e0g.png" /></figure><h4>Cost Function</h4><p>Cost Function is at the centre of Neural Network. It is used to calculate loss given the real and observed results. Our aim throughout is to minimise this loss. So Cost Function effectively drives the learning of neural network towards it’s goal.</p><p>A cost function is a measure of “how good” a neural network did with respect to it’s given training sample and the expected output. It also may depend on variables such as weights and biases.</p><p>A cost function is a single value, not a vector, because it rates how good the neural network did as a whole.</p><p>Some of the most famous cost functions are:</p><ol><li>Quadratic Cost (Root Mean Square)</li><li>Cross Entropy</li><li>Exponential (AdaBoost)</li><li>Kullback–Leibler divergence or Information Gain</li></ol><p>Root Mean Square is the simplest and most used of them all. It is simply defined as:</p><pre>Loss = √(expected_output ** 2) - (real_output ** 2)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/489/1*iMvja0saoB_7F7ej0ZIzGg.jpeg" /></figure><p><em>The Cost function in NN should satisfy two conditions</em></p><ul><li>The cost function must be able to be written as an average</li><li>The cost function must not be dependent on any activation values of a neural network except the output values</li></ul><h3>Deep Nets</h3><p>Deep learning is a class of machine learning algorithms that learn deeper (more abstract) insights from data.</p><p>In more formal terms:</p><ul><li>Uses a cascade (pipeline like flow, successively passed on) of many layers of processing units (nonlinear) for feature extraction and transformation.</li><li>Are based on learning of features (representation of knowledge of data) of the data in unsupervised manner. Higher level features (that are found in latter processing layers layers) are derived from lower level features (that are found in initial processing layers layers).</li><li>Learn multiple levels of representations that correspond to different levels of abstraction; the levels form a hierarchy of concepts.</li></ul><p>Example:</p><p>Let’s consider a single layer Neural Network:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/583/1*tYw39Bv-srfM7Fug6_Omlg.png" /></figure><p>Here whatever the first layer (green neurons) learn, they simply pass it to output.</p><p>Whereas</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/721/1*pskcoikyWf6kWpFGmwVQnQ.png" /></figure><p>In case of two layer Neural Networks, whatever the green hidden layer learns, is then passed on to blue hidden layer where it learns further (about the learning in green layer).</p><p>Hence more the number of hidden layers, more is the learning on previously learned concepts.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/666/1*BW9Tj-sDZkkSNXZE6Qe-eg.png" /></figure><p>This is not to be confused with Wide Neural Networks</p><p>In this case the presence of more neurons in one layer, does not result in learning an insight in depth. Instead it results in learning more number of concepts.</p><p>Example:</p><blockquote><em>Learning English Grammar, it requires understanding of huge number of concepts. In this case a single layer Wide Neural Network works much better than Deep Neural Network which is significantly less wider.</em></blockquote><blockquote><em>But</em></blockquote><blockquote><em>In case of learning the Fourier Transform, the learner (Neural Network) needs to be Deep one because there aren’t many concepts to be learned but each of these concepts is complex enough to require deep learning.</em></blockquote><h3>Balance is Key</h3><p>It’s very tempting to use deep and wide neural networks for every task. That might be a very bad idea because:</p><ul><li>Both require significantly more data to be trained upon to reach a minimum desirable accuracy</li><li>Both have exponentially higher time complexity</li><li>Too Deep NN will try to break a fundamental concept deeper, but at this point it will be making wrong assumptions about the concept and try to find pseudo patterns that do not exist</li><li>Too Wide NN will try to find more number of features (individual measurable property of a pattern being observed) than there exist. So, similar to last point, it will start making wrong assumptions about data.</li></ul><h3>Curse of Dimensionality</h3><p>The curse of dimensionality refers to various phenomena that arise when analyzing and organizing data in high-dimensional spaces (often with hundreds or thousands of dimensions) that do not occur in low-dimensional settings.</p><p>Something like Grammar of English language or Prize of Stocks have Humongous amount of features affecting them. Now with Machine Learning, we have to represent these with features with array/matrix of finite and comparatively much smaller length (than number of features that actually exist). To do this, learners generalize thew concepts. This gives rise to two problems:</p><ul><li>Bias arises due to wrong assumptions made by a learner. High bias can cause an algorithm to miss the relevant relations between features and target outputs. This phenomenon is called underfitting.</li><li>Variance arises from small fluctuations in the training set due insufficient learning about a feature. High variance results overfitting, learning errors as relevant information.</li></ul><h3>The Tradeoff</h3><p>Early in training the Bias is large because the network output is far from desired. The Variance is very small as data had little influence yet.</p><p>Late in training the Bias is small because the network has learned the underlying function. However if trained too long, the network will also learn the noise specific to that dataset. This results in high variance in results when tested on different datasets as noise varies from dataset to dataset.</p><p>Effectively,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*4QViAe58uSklqbUdHBTTGg.jpeg" /></figure><p>Algorithms with high bias typically produce simpler models that don’t tend to overfit, but may underfit their training data, failing to capture important patterns or features’ properties.</p><p>Models with low bias and high variance are usually more complex in terms of their structure, enabling them to represent the training set more accurately. In the process, however, they may also represent a large noise component in the training set, making their predictions less accurate — despite their added complexity.</p><p>Hence, it is typically impossible to have low bias and low variance.</p><blockquote><em>Today, in presence of abundance data and tools to easily create complex ML models with ease, over-fitting takes the centre stage. Because bias effectively occurs when there learner is not provided with enough information. But more examples, mean more variation as number of patterns and variation in those patterns, both increase.</em></blockquote><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fupscri.be%2F8f5f8b%3Fas_embed%3Dtrue&amp;dntp=1&amp;url=https%3A%2F%2Fupscri.be%2F8f5f8b&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=upscri" width="800" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/c43026df6fee7cdb1aab8aaf916125ea/href">https://medium.com/media/c43026df6fee7cdb1aab8aaf916125ea/href</a></iframe><figure><a href="https://becominghuman.ai/artificial-intelligence-communities-c305f28e674c"><img alt="" src="https://cdn-images-1.medium.com/max/255/1*2f7OqE2AJK1KSrhkmD9ZMw.png" /></a></figure><figure><a href="https://upscri.be/8f5f8b"><img alt="" src="https://cdn-images-1.medium.com/max/255/1*v-PpfkSWHbvlWWamSVHHWg.png" /></a></figure><figure><a href="https://becominghuman.ai/write-for-us-48270209de63"><img alt="" src="https://cdn-images-1.medium.com/max/255/1*Wt2auqISiEAOZxJ-I7brDQ.png" /></a></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bef2ba97d2cf" width="1" height="1" alt=""><hr><p><a href="https://becominghuman.ai/basics-of-neural-network-bef2ba97d2cf">Basics of Neural Network</a> was originally published in <a href="https://becominghuman.ai/">Becoming Human: Artificial Intelligence Magazine</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Become Data Scientist in One Day?]]></title>
            <link>https://medium.com/hackernoon/how-to-become-data-scientist-in-one-day-f6c904618190?source=rss-bd3e39707a79------2</link>
            <guid isPermaLink="false">https://medium.com/p/f6c904618190</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Mukul Malik]]></dc:creator>
            <pubDate>Wed, 11 Oct 2017 21:45:51 GMT</pubDate>
            <atom:updated>2019-05-08T08:38:29.015Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>Honestly, I don’t know. It’s probably a decade too early for me to become a Data ‘<strong>Scientist’</strong>.</blockquote><p><strong><em>I am writing this blog because people too often ask “How to become data scientist in a month?”</em></strong></p><p>Here lies the real problem. Two actually:</p><ul><li><em>‘Research’ isn’t fairly understood term</em></li><li><em>People love shortcuts too much</em></li></ul><p>Half of you wouldn’t even have done a single project in Data Science, <em>yet</em>.</p><p>How do I know? Simple, I was one of you.</p><p>Damn, I’ve been in lectures without even making an effort to know what that lecture is about!</p><p>PS : Not the smartest idea.</p><h4>First Issue</h4><p>Let’s clarify the following terms:</p><ul><li><strong>Engineer</strong> : A person who uses existing knowledge and/or <a href="https://hackernoon.com/tagged/tools">tools</a> to build something</li><li><strong>Researcher</strong> : A person who extends the knowledge about understanding of something</li><li><strong>Scientist</strong> : A person who has <strong><em>significantly</em></strong> contributed towards the research</li></ul><p>Data Scientist is ridiculously thrown around. Ask a physicist what it would take him to be called scientist (<em>partial insanity is just the beginning</em>).</p><p>Data Science, currently, is majorly a research oriented field, yes. Doesn’t mean anyone working in it becomes a researcher, let alone a scientist.</p><h4>Second Issue</h4><p>You see, <strong>there are no shortcuts for gaining knowledge</strong>. You can just be <em>smart about the learning techniques.</em></p><p>No tutorials can jump-start your career from novice to pro ( if there is then damn! I wasted life <strong>:/</strong> )</p><p>You become a pro by mastering:</p><ul><li>Execution</li><li>Dedication</li><li>Persistence</li></ul><p>Claims like “Master Neural Networks in One Minute!” will give you an abstract idea about the working of it.</p><p>So you’ll be lost when you are tackling a real life not-so-mainstream problem as default settings won’t work.</p><p>Like me, when I used to read the abstract of a research paper and assume I’ve understood all the concepts.</p><p>PS : Yeah, the old me wasn’t that bright.</p><h4>So how to go about it?</h4><p>Irrespective how experienced you are, these will apply to you:</p><ul><li>Ask the stupidest questions</li><li>Just because nobody thought of it, doesn’t mean it is wrong</li><li>Leave the guessing work to Learner, you code and evaluate</li><li>Keep your mind open to All Fields (Bio, Info Theory, Physics etc.)</li><li>Visualise Everything! It’s the map to the Treasure Chest</li><li>Never be scared to improvise</li></ul><h4>What if you already are a pro?</h4><p>Well if you already have experience in this field then you very well know how a single idea can change the whole landscape.</p><p>This just highlights the fact that Data Science has more potential than one could fathom.</p><blockquote>Take example of Reinforcement <a href="https://hackernoon.com/tagged/learning">Learning</a>. Most of the people I know think of RL as a game-play learning. They barely try to scratch their heads to apply this in text or vision paradigms.</blockquote><blockquote>RL is Goal-Oriented learning. Something like a customer support chatbot is perfect playground for RL is field of text. Where the goal could be anything from answering their queries to persuading them to buy a policy.</blockquote><h4>So where does this leave us?</h4><p>The harder you work the sweeter the fruit and longer you keep licking your fingers (you creep!)</p><p>This also increases one’s chances to ‘beat the odds’ or ‘defy the norms&#39; (with motivational tunes in background).</p><p>As Freddie Mercury once said “do what you want to do …. just don’t make it boring”.</p><p>And</p><blockquote>All the Best! :D</blockquote><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fupscri.be%2Fdde502%3Fas_embed%3Dtrue&amp;dntp=1&amp;url=https%3A%2F%2Fupscri.be%2Fhackernoon%2F&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=upscri" width="800" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/3c851dac986ab6dbb2d1aaa91205a8eb/href">https://medium.com/media/3c851dac986ab6dbb2d1aaa91205a8eb/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f6c904618190" width="1" height="1" alt=""><hr><p><a href="https://medium.com/hackernoon/how-to-become-data-scientist-in-one-day-f6c904618190">How to Become Data Scientist in One Day?</a> was originally published in <a href="https://medium.com/hackernoon">HackerNoon.com</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>