<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Luciano Strika on Medium]]></title>
        <description><![CDATA[Stories by Luciano Strika on Medium]]></description>
        <link>https://medium.com/@StrikingLoo?source=rss-56f7cdc8c678------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*iCpIXyBTQz4WbbFyAR_zgQ.jpeg</url>
            <title>Stories by Luciano Strika on Medium</title>
            <link>https://medium.com/@StrikingLoo?source=rss-56f7cdc8c678------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 04:28:16 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@StrikingLoo/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Feature Visualization on Convolutional Neural Networks (Keras)]]></title>
            <link>https://medium.com/data-science/feature-visualization-on-convolutional-neural-networks-keras-5561a116d1af?source=rss-56f7cdc8c678------2</link>
            <guid isPermaLink="false">https://medium.com/p/5561a116d1af</guid>
            <category><![CDATA[convolutional-network]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Luciano Strika]]></dc:creator>
            <pubDate>Sat, 30 May 2020 01:20:19 GMT</pubDate>
            <atom:updated>2022-07-11T14:10:34.017Z</atom:updated>
            <content:encoded><![CDATA[<h4>What is each individual filter doing in a Convolutional Neural Network? Which kinds of images is it learning to detect? Here’s a way to know</h4><p>According to Wikipedia, <a href="https://en.wikipedia.org/wiki/Apophenia">apophenia</a> is <em>“the tendency to mistakenly perceive connections and meaning between unrelated things”</em>. It is also used as “the human propensity to seek patterns in random information”. Whether it’s a scientist doing research in a lab, or a conspiracy theorist warning us about how “it’s all connected”, I guess people need to feel like we understand what’s going on, even in the face of clearly random information.</p><p>Deep Neural Networks are usually treated like “black boxes” due to their <strong>inscrutability</strong> compared to more transparent models, like XGboost or <a href="https://github.com/interpretml/interpret">Explainable Boosted Machines</a>.</p><p>However, there is a way to interpret what <strong>each individual filter</strong> is doing in a Convolutional Neural Network, and which kinds of images it is learning to detect.</p><p>Convolutional Neural Networks rose to prominence since at least 2012, when <a href="http://image-net.org/challenges/LSVRC/2012/supervision.pdf">AlexNet</a> won the <a href="http://www.image-net.org/challenges/LSVRC/2012/index#workshop">ImageNet computer vision contest</a> with an accuracy of 85%. The second place was at a mere 74%, and <a href="http://www.image-net.org/challenges/LSVRC/2013/results.php#cls">a year later</a> most competitors were switching to this “new” kind of algorithm.</p><p>They are widely used for many different tasks, mostly relating to <strong>image processing</strong>. These include Image Classification, Detection problems, and many others.</p><p>I will not go in depth into how a Convolutional Neural Network works, but if you’re getting started in this subject I recommend you read my <a href="https://www.datastuff.tech/machine-learning/convolutional-neural-networks-an-introduction-tensorflow-eager/">Practical Introduction to Convolutional Neural Networks</a> with working TensorFlow code.</p><p>If you already have a grasp of how a Convolutional Neural Network works, then this article is all you need to know to understand <strong>what Feature Visualization does</strong> and how it works.</p><h3>How does Feature Visualization work?</h3><p>Normally, you would train a CNN feeding it images and labels, and using Gradient Descent or a similar optimization method to <strong>fit the Neural Network’s weights</strong> so that it predicts the right label.</p><p>Throughout this process, one would expect the image to remain untouched, and the same applies to the label.</p><p>However, what do you think would happen if we took any image, picked one convolutional filter in our (already trained) network, and applied Gradient Descent <strong>on the input image</strong> to <strong>maximize that filter’s output</strong>, while <strong>leaving the Network’s weights constant</strong>?</p><p>Suddenly, we have <strong>shifted perspectives</strong>. We’re no longer training a model to predict an image’s label. Rather, we’re now kind of fitting the image to the model, to make it generate whatever output we want.</p><p>In a way, it’s like we’re asking the model “See this filter? What kind of images turn it on?”.</p><p>If our Network has been properly trained, then we expect most filters to carry interesting, valuable information that help the model make accurate predictions for its classification task. We expect a filter’s activation to carry <strong>semantic meaning</strong>.</p><p>It stands to reason then, that an image that “activates” a filter, making it have a large output, should have features that resemble those of one of the objects present in the Dataset (and among the model’s labels).</p><p>However, given that convolutions are a <strong>local transformation</strong>, it is common to see the patterns that trigger that convolutional filter repeatedly “sprout” in many different areas of our image.</p><p>This process generates the kind of picture <a href="https://deepdreamgenerator.com/#gallery">Google’s Deep Dream</a> model made popular.</p><p>In this tutorial, we will use <strong>TensorFlow’s Keras</strong> code to <strong>generate images</strong> that maximize a given filter’s output.</p><p>Since a filter’s output is technically a matrix, the actual function we will be maximizing is the average of that matrix’s components, <strong>averaged over the whole image</strong>. This way our algorithm will be incentivized to generate whatever pattern activates the filter, throughout the whole image.</p><h3>Implementing Filter Visualization</h3><p>As I mentioned before, for this to work we would need to first <strong>train a Neural Network classifier</strong>. Luckily, we don’t need to go through that whole messy and costly process: <strong>Keras</strong> already comes with a whole suite of pre-trained Neural Networks we can just download and use.</p><h3>Using a Pre-trained Neural Network</h3><p>For this article, we will use VGG16, a huge Convolutional Neural Network trained on the same ImageNet competition Dataset. Remember how I mentioned AlexNet won with an 85% accuracy and disrupted the Image Classification field? VGG16 scored 92% on that same task.</p><blockquote><em>VGG16 is a convolutional neural network model proposed by K. Simonyan and A. Zisserman from the University of Oxford in the paper “Very Deep Convolutional Networks for Large-Scale Image Recognition”. The model achieves 92.7% top-5 test accuracy in ImageNet, which is a dataset of over 14 million images belonging to 1000 classes. It was one of the famous model submitted to </em><a href="http://www.image-net.org/challenges/LSVRC/2014/results"><em>ILSVRC-2014</em></a><em>. It makes the improvement over AlexNet by replacing large kernel-sized filters (11 and 5 in the first and second convolutional layer, respectively) with multiple 3×3 kernel-sized filters one after another. VGG16 </em><strong><em>was trained for weeks and was using NVIDIA Titan Black GPU’s</em></strong><em>.</em></blockquote><blockquote><a href="https://neurohive.io/en/popular-networks/vgg16/"><em>https://neurohive.io/en/popular-networks/vgg16/</em></a><em> — VGG16 — Convolutional Network for Classification and Detection (emphasis mine)</em></blockquote><p>For these experiments, I will be using<a href="https://colab.research.google.com/"> Google colab’s </a>GPU machine, and tweaking Keras Library’s example of <a href="https://github.com/keras-team/keras/blob/master/examples/conv_filter_visualization.py">Filter Visualization</a> code.</p><p>For a breakdown of how the original script works, you can see <a href="https://blog.keras.io/how-convolutional-neural-networks-see-the-world.html">Keras Blog</a>. I only made slight changes to it to easily configure file names, and other minor details, so I don’t think it’s worth it to link to my own notebook.</p><p>What the important function does is:</p><ul><li>Define a loss function that’s equal to the chosen <strong>filter’s average output, over the whole image</strong>. In our code, we do this here: loss = K.mean(layer_output[:, filter_index, :, :]).</li><li>Initialize a small <strong>starting picture</strong>, typically with <strong>random uniform noise</strong> centered around RGB(128,128,128) (I actually played around with this a bit, and will expand on it later).</li><li>Compute the <strong>gradient of the input picture</strong> with regards to this loss, and perform gradient descent.<br>Note that we are <strong>feeding the image to the neural network</strong>, but ignoring any layer that comes after the one we are concerned with.</li><li>Repeat N times, then resize the picture to make it slightly bigger (default value was 20%). We start with a small picture and make it increasingly bigger as we generate the filter’s maximizing image, because otherwise the algorithm tends to create a small pattern that repeats many times, instead of making a lower-frequency pattern with bigger (and, subjectively, more aesthetically pleasing) shapes.</li><li>Repeat the last two steps until reaching the desired resolution.</li></ul><p>That’s pretty much it. The code I linked to has a few more things happening (image normalization, and stitching together many filters’ generated images into a cute collage) but that is the most important bit.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/17017399c2a72b09f27c9f4fbc65da9a/href">https://medium.com/media/17017399c2a72b09f27c9f4fbc65da9a/href</a></iframe><p>Here’s the code for that function, not that scary now that you know what’s going on right?</p><p>Now for the fun part, let’s try this out and see which kinds of filters come out.</p><h3>My Results Trying out Feature Visualization</h3><p>I read <a href="https://distill.pub/2017/feature-visualization/">many</a> <a href="https://towardsdatascience.com/how-to-visualize-convolutional-features-in-40-lines-of-code-70b7d87b0030">different</a> <a href="https://arxiv.org/pdf/1311.2901.pdf">examples</a> of Feature Visualization articles before giving it a shot. Here are some of the things I learned.</p><ul><li>The first Convolutional Layers (the ones closer to the inputs) generate simpler visuals. They’re usually just rough textures like parallel wavy lines, or multicolored circles.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xL7HEr7FqNzhkyNHYBszHQ.png" /><figcaption>Visualization of Convolutional Filter on VGG16, second layer.</figcaption></figure><ul><li>The Convolutional layers closer to the outputs generate more intricate textures and patterns. Some even resemble objects that exist, or sorta look like they may exist (in a very uncanny-valley kind of way).</li></ul><p>This is also where I had the most fun, to be honest. I tried out many different “starting images”, from random noise to uniform grey, to a progressive degrade.</p><p>The results for any given filter all came out pretty similar. This makes me think given the number of iterations I used, the starting image itself became pretty irrelevant. At the very least, it did not have a predictable impact on the results.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*I-7-NKpo9008cG7Pj7_jCQ.png" /><figcaption>Feature Visualization for Block 4, filters in first convolutional layer of VGG16. Most of the patterns look regular and granular, but a lot more complicated than the early, rustic textures we saw on the first layers.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_OnIu3CKWARdB6q24niFkw.png" /><figcaption>Filter Visualization for Block 4, filters in second convolutional layer of VGG16. Note how the patterns are very repetitive, but generate textures that look a lot more sophisticated than in the first layers.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CaLf2OWfsLQk4-sw4gleiw.png" /><figcaption>Filter Visualization for Block 4, filters in third convolutional layer of VGG16. Some more porous patterns seem to emerge.</figcaption></figure><p>As we go deeper, and <strong>closer to the fully connected layers</strong>, we reach the <strong>last Convolutional Layer</strong>. The images it generates are the <strong>most intricate</strong> by far, and the patterns they make many times resemble real life items.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DNM6maX9QHItW1JMn_OgzQ.png" /><figcaption>Filter Visualization for Block 5, filters in first convolutional layer of VGG16</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OnqliCU1ldLBYhjYgYzPDA.png" /><figcaption>Filter Visualization for Block 5, filters in third convolutional layer of VGG16</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1NmmKZrU_GmJkeC1PcozBw.png" /><figcaption>Block 5, filters in second Convolutional Layer. Isn’t it just crazy that all of these patterns emerge just from maximizing a “simple” (albeit hyperdimensional) mathematical function?</figcaption></figure><p>Now, looking into these images in search of patterns, it is easy to feel like one is falling into apophenia. However, I think we can all agree some of those images have features that <strong>really look like</strong> … you can zoom in and complete that sentence on your own. Feature Visualization is the new gazing at clouds.</p><p>My own guess is it’s just a new kind of abstract art.</p><p>Let me show you some of the filters I found most visually interesting:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*mBigWTXSW3wu6fhpXa9DuA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*_jNEaXHwzoRhcAG1MVF2oA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*DTt6V6LMZGLMvOMoNWTyXw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*AurcTrzl84qvRk8an_54IQ.png" /><figcaption>Cotton, Orange Peel, eyes? weird moldy spirals.</figcaption></figure><p>I have about 240 more of these and if there’s enough interest, I can make a gallery out of them, but I fear it may turn repetitive after a while.</p><h3>Conclusion</h3><p>To be honest, I had a lot of fun with this project. I hadn’t really heard about Google Colab until a couple weeks ago (thanks to <a href="https://www.reddit.com/r/MediaSynthesis/">r/mediaSynthesis</a>). It feels great to be able to use a good GPU machine for free.</p><p>I’d also read most of the papers on this subject a couple years ago, then never got around to actually testing the code or doing an article like this. I’m glad I finally scratched it out of my list (or Trello, who am I kidding?).</p><p>Finally, in the future, I’d like to try out different network architectures and visualize how the images morph at every iteration, instead of simply looking at the finished product.</p><p>Please let me know in the comments which other experiments or bibliography could be worth checking to keep expanding on this subject!</p><p><em>If you liked this article, please consider tweeting it or sharing anywhere else!</em> <em>Follow me on </em><a href="https://www.twitter.com/strikingloo"><em>Twitter</em></a><em> to discuss any of this further or keep up to date with my latest articles.</em></p><p><em>You can see what I’m working on and my most recent articles and notes in my </em><a href="http://strikingloo.github.io/wiki"><em>personal site</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5561a116d1af" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/feature-visualization-on-convolutional-neural-networks-keras-5561a116d1af">Feature Visualization on Convolutional Neural Networks (Keras)</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Markov Chains: How to Train Text Generation to Write like George R. R. Martin]]></title>
            <link>https://medium.com/data-science/markov-chains-how-to-train-text-generation-to-write-like-george-r-r-martin-cdc42786e4b6?source=rss-56f7cdc8c678------2</link>
            <guid isPermaLink="false">https://medium.com/p/cdc42786e4b6</guid>
            <category><![CDATA[markov-chains]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[fun]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Luciano Strika]]></dc:creator>
            <pubDate>Fri, 25 Oct 2019 05:29:17 GMT</pubDate>
            <atom:updated>2019-10-27T01:41:26.819Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*SOgAK5tqbJ5UqTQ4.jpg" /></figure><p>Markov chains have been around for a while now, and they are here to stay. From predictive keyboards to applications in trading and biology, they’ve proven to be versatile tools.</p><p>Here are some Markov Chains industry applications:</p><ul><li>Text Generation (you’re here for this).</li><li>Financial modelling and forecasting (including trading algorithms).</li><li>Logistics: modelling future deliveries or trips.</li><li>Search Engines: PageRank can seen as modelling a random internet surfer with a Markov Chain.</li></ul><p>So far, we can tell this algorithm is useful, but what exactly are Markov Chains?</p><h3>What are Markov Chains?</h3><p>A Markov Chain is a <strong>stochastic process</strong> that models a finite <strong>set of states</strong>, with fixed <strong>conditional probabilities of jumping</strong> from a given state to another.</p><p>What this means is, we will have an “agent” that randomly jumps around different states, with a certain probability of going from each state to another one.</p><p>To show what a Markov Chain looks like, we can use a <strong>digraph</strong>, where each node is a state (with a label or associated data), and the weight of the edge that goes from node <em>a</em> to node <em>b</em> is the <strong>probability of jumping from state<em> a</em> to state <em>b</em></strong>.</p><p>Here’s an example, modelling the weather as a Markov Chain.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/454/0*Ie2j5RUwEGDuV5dB.png" /><figcaption><a href="http://techeffigytutorials.blogspot.com/2015/01/markov-chains-explained.html">Source</a></figcaption></figure><p>We can express the probability of going from state <em>a </em>to state <em>b</em> as a <strong>matrix component</strong>, where the whole <strong>matrix characterizes our Markov chain</strong> process, corresponding to the <strong>digraph’s adjacency matrix</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/238/0*Zq7gLHPxqYAKQsY6.png" /><figcaption><a href="http://techeffigytutorials.blogspot.com/2015/01/markov-chains-explained.html">Source</a></figcaption></figure><p>Then, if we represent the current state as a one-hot encoding, we can obtain the conditional probabilities for the next state’s values by taking the current state, and looking at its corresponding row.</p><p>After that, if we repeatedly sample the discrete distribution described by the <em>n</em>-th state’s row, we may model a succession of states of arbitrary length.</p><h3>Markov Chains for Text Generation</h3><p>In order to generate text with Markov Chains, we need to define a few things:</p><ul><li>What are our states going to be?</li><li>What probabilities will we assign to jumping from each state to a different one?</li></ul><p>We could do a character-based model for text generation, where we define our state as the last <em>n</em> characters we’ve seen, and try to predict the next one.</p><p>I’ve already gone in-depth on this for my <a href="http://www.datastuff.tech/machine-learning/lstm-how-to-train-neural-networks-to-write-like-lovecraft/">LSTM for Text Generation</a> article, to mixed results.</p><p>In this experiment, I will instead choose to use the previous <em>k</em> words as my current state, and model the probabilities of the next token.</p><p>In order to do this, I will simply create a vector for each distinct sequence of <em>k</em> words, having N components, where N is the total quantity of distinct words in my corpus.</p><p>I will then add 1 to the <em>j</em>-th component of the <em>i</em>-th vector, where i is the index of the <em>i</em>-th k-sequence of words, and <em>j</em> is the index of the next word.</p><p>If I normalize each word vector, I will then have a probability distribution for the next word, given the previous <em>k</em> tokens.</p><p>Confusing? Let’s see an example with a small corpus.</p><h3>Training our chain: toy example.</h3><p>Let’s imagine my corpus is the following sentence.</p><p>This sentence has five words</p><p>We will first choose <em>k</em>: the <strong>quantity of words our chain will consider</strong> before sampling/ <strong>predicting the next</strong> one. For this example, let’s use k=1.</p><p>Now, how many distinct sequences of 1 word does our sentence have? It has 5, one for each word. If it had duplicate words, they wouldn’t add to this number.</p><p>We will first initialize a 5×5 matrix of zeroes.</p><p>After that, we will add 1 to the column corresponding to ‘sentence’ on the row for ‘this’. Then another 1 on the row for ‘sentence’, on the column for ‘has’. We will continue this process until we’ve gone through the whole sentence.</p><p>This would be the resulting matrix:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/122/0*0OQO51wVaqCWI6Vx.png" /><figcaption>The diagonal pattern comes from the ordering of the words.</figcaption></figure><p>Since each word only appears once, this model would simply generate the same sentence over and over, but you can see how adding more words could make this interesting.</p><p>I hope things are clearer now. Let’s jump to some code!</p><h3>Coding our Markov Chain in Python</h3><p>Now for the fun part! We will train a Markov chain on the whole A Song of Ice and Fire corpus (Ha! You thought I was going to reference the show? Too bad, I’m a book guy!).</p><p>We will then generate sentences with varying values for <em>k</em>.</p><p>For this experiment, I decided to treat anything between two spaces as a word or <em>token</em>.</p><p>Conventionally, in NLP we treat punctuation marks (like ‘,’ or ‘.’) as tokens as well. To solve this, I will first add padding in the form of two spaces to every punctuation mark.</p><p>Here’s the code for that small preprocessing, plus loading the corpus:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b2a2e832550735323487183ebe826e59/href">https://medium.com/media/b2a2e832550735323487183ebe826e59/href</a></iframe><p>We will start training our Markov Chain right away, but first let’s look at our dataset:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/763ec3d9324efe8818bd4bdd1fdef573/href">https://medium.com/media/763ec3d9324efe8818bd4bdd1fdef573/href</a></iframe><p>We have over 2 million tokens, representing over 32000 distinct words! That’s a pretty big corpus for a single writer.</p><p><em>If only he could add 800k more…</em></p><h3>Training our chain</h3><p>Moving on, here’s how we initialize our “word after k-sequence” counts matrix for an arbitrary <em>k</em> (in this case, 2).</p><p>There are 2185918 words in the corpus, and 429582 different sequences of 2 words, each followed by one of 32663 words.</p><p>That means only slightly over 0.015% of our matrix’s components will be non-zero.</p><p>Because of that, I used scipy’s <em>dok_matrix</em> ( <em>dok</em> stands for Dictionary of Keys), a sparse matrix implementation, since we know this dataset is going to be extremely sparse.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5ddb622b528d88fe45732ffe77170ede/href">https://medium.com/media/5ddb622b528d88fe45732ffe77170ede/href</a></iframe><p>After initializing our matrix, sampling it is pretty intuitive.</p><p>Here’s the code for that:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/51c98c526d7ad76917588438def3d556/href">https://medium.com/media/51c98c526d7ad76917588438def3d556/href</a></iframe><p>There are two things that may have caught your attention here. The first is the <em>alpha</em> hyperparameter.</p><p>This is our chain’s <em>creativity</em>: a (typically small, or zero) chance that it will pick a totally random word instead of the ones suggested by the corpus.</p><p>If the number is high, then the next word’s distribution will approach uniformity. If zero or closer to it, then the distribution will more closely resemble that seen in the corpus.</p><p>For all the examples I’ll show, I used an <em>alpha</em> value of 0.</p><p>The second thing is the weighted_choice function. I had to implement it since Python’s random package doesn’t support weighted choice over a list with more than 32 elements, let alone 32000.</p><h3>Results: Generated Sentences</h3><p>First of all, as a baseline, I tried a deterministic approach: what happens if we pick a word, use k=1, and always jump to the most likely word after the current one?</p><p>The results are underwhelming, to say the least.</p><pre><strong>I</strong> am not have been a man , and the Wall . &quot; &quot; &quot; &quot;<br><strong>he</strong> was a man , and the Wall . &quot; &quot; &quot; &quot; &quot; &quot; &quot;<br><strong>she</strong> had been a man , and the Wall . &quot; &quot; &quot; &quot; &quot; &quot;</pre><p>Since we’re being deterministic, ‘a’ is always followed by ‘man’, ‘the’ is always followed by ‘Wall’ (hehe) and so on.</p><p>This means our sentences will be boring, predictable and kind of nonsensical.</p><p>Now for some actual generation, I tried using a stochastic Markov Chain of 1 word, and a value of 0 for alpha.</p><h3>1-word Markov Chain results</h3><p>Here are some of the resulting 15-word sentences, with the seed word in bold letters.</p><pre>&#39;<strong>the</strong> Seven in front of whitefish in a huge blazes burning flesh . I had been&#39; <br>&#39;<strong>a </strong>squire , slain , they thought . &quot; He bathed in his head . The&#39; <br>&#39;<strong>Bran</strong> said Melisandre had been in fear I&#39;ve done . &quot; It must needs you will&#39; <br>&#39;<strong>Melisandre</strong> would have feared he&#39;d squired for something else I put his place of Ser Meryn&#39; <br>&#39;<strong>Daenerys</strong> is dead cat - TOOTH , AT THE GREAT , Asha , which fills our&#39; <br>&#39;<strong>Daenerys</strong> Targaryen after Melara had worn rich grey sheep to encircle Stannis . &quot; The deep&#39;</pre><p>As you can see, the resulting sentences are quite nonsensical, though a lot more interesting than the previous ones.</p><p>Each individual pair of words makes some sense, but the whole sequence is pure non-sequitur.</p><p>The model did learn some interesting things, like how Daenerys is usually followed by Targaryen, and ‘would have feared’ is a pretty good construction for only knowing the previous word.</p><p>However, in general, I’d say this is nowhere near as good as it could be.</p><p>When increasing the value of alpha for the single-word chain, the sentences I got started turning even more random.</p><h3>Results with 2-word Markov chains</h3><p>The 2-word chain produced some more interesting sentences.</p><p>Even though it too usually ends sounding completely random, most of its output may actually fool you for a bit at the beginning <em>(emphasis mine)</em>.</p><pre>&#39;<strong>the world</strong> . <em>And Ramsay loved the feel of grass</em> <em>welcomed them warmly</em> , the axehead flew&#39; <br>&#39;<strong>Jon Snow</strong> . <em>You are to strike at him</em> . <em>The bold ones have had no sense</em>&#39; <br>&#39;<strong>Eddard Stark</strong> had done his best to give her <em>the promise was broken</em> . By tradition the&#39; <br>&#39;<strong>The game</strong> of thrones , so you must tell her the next buyer who comes running ,&#39; <br>&#39;<strong>The game</strong> trail brought her messages , strange spices . <em>The Frey stronghold was not large enough</em>&#39; <br>&#39;<strong>heard the</strong> scream of fear . I want to undress properly . Shae was there , fettered&#39;</pre><p>The sentences maintain local coherence ( <em>You are to strike at him</em>, or <em>Ramsay loved the feel of grass</em>), but then join very coherent word sequences into a total mess.</p><p>Any sense of syntax, grammar or semantics is clearly absent.</p><p>By the way, I didn’t cherry-pick those sentences at all, those are the first outputs I sampled.</p><p>Feel free to <a href="https://github.com/StrikingLoo/ASOIAF-Markov">play with the code yourself</a>, and you can share the weirdest sentences you get in the comments!</p><p>As a last experiment, let’s see what we get with a 3-word Markov Chain.</p><h3>3-Word Chain Results</h3><p>Here are some of the sentences the model generated when trained with sequences of 3 words.</p><pre>&#39;<strong>I am a</strong> master armorer , lords of Westeros , sawing out each bay and peninsula until the&#39; <br>&#39;<strong>Jon Snow is</strong> with the Night&#39;s Watch . I did not survive a broken hip , a leathern&#39; <br>&#39;<strong>Jon Snow is</strong> with the Hound in the woods . He won&#39;t do it . &quot; Please don&#39;t&#39; <br>&#39;<strong>Where are the</strong> chains , and the Knight of Flowers to treat with you , Imp . &quot;&#39; <br>&#39;<strong>Those were the</strong> same . Arianne demurred . &quot; So the fishwives say , &quot; It was Tyrion&#39;s&#39; <br>&#39;<strong>He thought that</strong> would be good or bad for their escape . If they can truly give us&#39; <br>&#39;<strong>I thought that</strong> she was like to remember a young crow he&#39;d met briefly years before . &quot;&#39;</pre><p>Alright, I really liked some of those, especially the last one. It kinda sounds like a real sentence you could find in the books.</p><h3>Conclusion</h3><p>Implementing a Markov Chain is a lot easier than it may sound, and training it on a real corpus was fun.</p><p>The results were frankly better than I expected, though I may have set the bar too low after my little LSTM fiasco.</p><p>In the future, I may try training this model with even longer chains, or a completely different corpus.</p><p>In this case, trying a 5-word chain had basically deterministic results again, since each 5-word sequence was almost always unique, so I did not consider 5-words and upwards chains to be of interest.</p><p>Which corpus do you think would generate more interesting results, Especially for a longer chain? Let me know in the comments!</p><p><em>If you wish to learn even more about Markov Chains, consider checking </em><a href="https://amzn.to/31IDAHp"><em>this in-depth book</em></a><em>. That’s an affiliate link, which means I get a small commission from it</em>.</p><p><em>Originally published at </em><a href="http://www.datastuff.tech/machine-learning/markov-chains-teaching-ai-to-write-game-of-thrones/"><em>http://www.datastuff.tech</em></a><em> on October 25, 2019.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cdc42786e4b6" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/markov-chains-how-to-train-text-generation-to-write-like-george-r-r-martin-cdc42786e4b6">Markov Chains: How to Train Text Generation to Write like George R. R. Martin</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[MapReduce in C from Scratch Using Threads: Map]]></title>
            <link>https://medium.com/data-science/mapreduce-in-c-from-scratch-using-threads-map-c6b1b01f020c?source=rss-56f7cdc8c678------2</link>
            <guid isPermaLink="false">https://medium.com/p/c6b1b01f020c</guid>
            <category><![CDATA[programming-languages]]></category>
            <category><![CDATA[functional-programming]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Luciano Strika]]></dc:creator>
            <pubDate>Sat, 19 Oct 2019 05:45:55 GMT</pubDate>
            <atom:updated>2019-10-21T04:08:08.791Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*LUwfTibMtoVwtKHl.jpg" /><figcaption>Source: <a href="https://pixabay.com/photos/knowledge-book-library-glasses-1052011/">Pixabay</a></figcaption></figure><p>Hadoop’s MapReduce is not just a Framework, it’s also a problem-solving philosophy.</p><p>Borrowing from functional programming, the MapReduce team realized a lot of different problems could be divided into two common operations: <strong>map</strong>, and <strong>reduce</strong>.</p><p>Both <strong>mapping</strong> and <strong>reducing</strong> steps can be done <strong>in parallel</strong>.</p><p>This meant as long as you could <strong>frame your problem</strong> in that specific way, there would be a solution to it that could easily be run in parallel. This will usually result in a big <strong>performance boost</strong>.</p><p>That all sounds good, and running things on parallel is usually a good thing, especially when working at scale. But, some of you on the back may be wondering, what are <strong>Map</strong> and <strong>Reduce</strong>?</p><h3>What is MapReduce?</h3><p>In order to understand the MapReduce framework, we need to understand its two basic operations: <strong>Map</strong> and <strong>Reduce</strong>.</p><p>They’re both high order functions: Meaning they are functions that can take other functions as their argument.</p><p>Specifically, when you need to convert a certain sequence of elements of type A into a result, or series of results of type B, you will:</p><ul><li><strong>Map</strong> all your inputs to a different domain: that means you will <strong>transform each of them</strong> with a chosen function, applying it to each element.</li><li><strong>Group</strong> the mapped elements by some criterion, usually a grouping <strong>key</strong>.</li><li><strong>Reduce</strong> the mapped elements on each group with some other function. This function needs to take two arguments and return a single one of the same type, successively running an operation between an <strong>accumulator</strong> and each value in our collection. It should be <strong>commutative and associative</strong>, as parallel execution <strong>won’t guarantee any order</strong> for the operations.</li></ul><p>To make this clearer, let’s see an example.</p><h3>Example of a MapReduce solution</h3><p>Suppose you’re working for an e-commerce company, and they give you a log file of this form:</p><pre>John Surname bought 2 apples <br>Alice Challice bought 3 bananas <br>John Surname bought 5 pineapples</pre><p>Then they ask you to tell them how many fruits each customer bought.</p><p>In this case, after parsing this file to turn it into an actual format, like CSV, you could easily go through each line, and add the number of bought fruits on a dictionary under each name.</p><p>You could even solve it with a bit of <a href="http://www.datastuff.tech/programming/files-strings-shell-tutorial/">Bash scripting</a>, or load the CSV on a <a href="http://www.datastuff.tech/data-science/exploratory-data-analysis-with-pandas-and-jupyter-notebooks/">Pandas DataFrame</a> and get some statistics.</p><p>However, if the log file was a trillion lines long, bash scripting wouldn’t really cut it. Especially not if you’re not immortal.</p><p>You would need to run this in parallel. Let me propose a MapReduce-y way of doing it:</p><ul><li><strong>Map</strong> each line to a Pair of the form &lt;Name, Quantity&gt; by parsing each string.</li><li>Group by Name.</li><li><strong>Reduce</strong> by summing the quantities.</li></ul><p>If you’re familiar with SQL and relational databases, you may have thought of a similar solution. The query would look something like</p><p>select user, sum(bought_fruits) <br>from fruit_transactions group by user;</p><h3>Why MapReduce scales</h3><p>Notice how <strong>the mapper doesn’t need to see the whole file</strong>, just some of the lines. The <strong>reducer</strong>, on the other hand, <strong>only needs to have the lines that have the same Name</strong> (the ones that belong to the same group).</p><p>You could do this with many different threads on the same computer, and then just join the results.</p><p>Or, you could have many different processes running the map jobs, and feeding their output to another set running the reducing job.</p><p>If the log was big enough, you could even be running Mapper and Reducer processes on many different computers (say, on a cluster), and then joining their results on some lake in the end.</p><p>This kind of solution is very common in ETL jobs and other data-intensive applications, but I won’t delve any further into applications.</p><p>If you wish to learn more about this kind of scalable solutions, I recommend you check <a href="https://amzn.to/33Dwh56">this O’Reilly book on designing applications at scale</a>.</p><h3>Programming MapReduce in C</h3><p>Now that you have an understanding of what MapReduce is, and why MapReduce scales, let’s cut to the chase.</p><p>For this first article, we will program two different implementations of the <em>Map</em> function.</p><p>One of them will be <strong>single-threaded</strong>, to introduce a few concepts and show a <strong>simple solution</strong>. The other one will use the <em>pthread</em> library to make an actually <strong>multi-threaded</strong>, and <strong>much faster</strong> version of <em>Map</em>. Finally, we will compare the two and run some benchmarks.</p><p>As usual, all the code is available on <a href="https://github.com/StrikingLoo/mapReduCe">this C GitHub project</a>.</p><h3>Single threaded implementation of Map in C</h3><p>First of all, let’s remember what <em>Map</em> does.</p><blockquote><em>The Map function receives a </em><strong><em>sequence</em></strong><em> and a </em><strong><em>function</em></strong><em>, and returns the result of </em><strong><em>applying that function to each element</em></strong><em> in the sequence.</em></blockquote><p>Since this is C, representing a sequence can be very straight forward: we can just use a pointer to whatever type we’re mapping over!</p><p>However, there’s a catch. <strong>C is statically typed</strong>, and we would like our Map function to be <strong>as generic as possible</strong>. We want it to be able to map over a sequence of elements of any type (provided they all share a type. Let’s not get carried away here, boys).</p><p>How do we solve this? There are probably a few different solutions to this problem. I chose the one that looked like the most simple one, but feel free to pitch in with other ideas.</p><p>We will use sequences of void*, and cast everything to this type. This means every element will be represented as a pointer to a random memory address, without specifying a type (or size).</p><p>We will trust whatever function we are calling over all these sequence elements knows how to cast them to the right type before using them. We’re effectively delegating that problem away.</p><p>A smaller problem we need to solve is sequence length. A pointer to void doesn’t carry the information of how many elements the sequence has. It only knows where it starts, not where it ends.</p><p>We will solve this other problem by passing sequence length as a second argument. Knowing that, our <em>Map</em> function becomes pretty straightfoward.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/348712f0d9156fae47f2cf4c1d57c55f/href">https://medium.com/media/348712f0d9156fae47f2cf4c1d57c55f/href</a></iframe><p>You see, the function receives a void** to represent the sequence it will map over, and a void* (*f)(void*) function that transforms elements of some generic type to another (or the same) one.</p><p>After that, we can use our <em>Map</em> function on any sequence. We only need to do some awkward wrapping and pointer arithmetic beforehand.</p><p>Here’s an example, using a function that returns 1 for prime numbers and 0 for the others.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7abd12dc4005d46a78c8ce7cb56c88c5/href">https://medium.com/media/7abd12dc4005d46a78c8ce7cb56c88c5/href</a></iframe><p>As expected, the resulting pointer points to a sequence of integers: 1 corresponds to prime numbers, 0 to composite ones.</p><p>Now we’ve gone through the single-threaded <em>Map</em> function, let’s see how to make this run on parallel in C.</p><h3>Multi-threaded Map function in C</h3><p><em>(If you want to add a benchmark using processes and forking, feel free to make a pull request!)</em></p><p>In order to use parallel execution in C, we can either turn to processes, or threads.</p><p>For this project, we will be using threads, as they’re more lightweight and, in my opinion, their API is a bit more intuitive for this kind of tutorial.</p><h3>How to use threads in C</h3><p>Threads’ API in C is quite intuitive, if only a bit obscure at first.</p><ul><li>A pointer to a pthread_t: the actual thread.</li><li>A configuration struct. In this case, we will use NULL for default config.</li><li>The function we want the thread to run. Unlike a process, a thread will only run a function until it returns, rather than continuing the execution of arbitrary code. This function must take a single void* argument and return another void* value.</li><li>The input of the aforementioned function. It must be cast to void*.</li></ul><p>To use them, we will have to #include &lt;pthread.h&gt;. Pthreads&#39; man page explains their interface quite nicely. However, for this tutorial, all we will use is the pthread_create function.</p><p>pthread_create takes four arguments:</p><p>After calling on pthread_create, a parallel thread of execution will begin running the given function.</p><p>Once we call pthread_create for each of the chunks we wish to map, we will have to call pthread_join on each of them, which makes the parent (original) thread <strong>wait</strong> until all the threads it spun <strong>finish running</strong>.</p><p>Otherwise, the program would end before the mapping was done.</p><p>Now, let’s feast our eyes on some code.</p><h3>Using pthread for Parallel MapReduce in C</h3><p>To code MapReduce’s <em>Map</em> function in C, the first thing we are going to do is define a struct that can store the generic inputs and outputs for it, as well as the function we will be mapping with.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3961b63ec80e53489cd47ae89659477d/href">https://medium.com/media/3961b63ec80e53489cd47ae89659477d/href</a></iframe><p>Since parallel execution requires some manner of slicing and <strong>partitioning</strong>, we will store that logic inside this structure as well, using two different <strong>indices</strong> for the start and end of our slice.</p><p>Next, we will code the function that actually does the mapping: it will cycle the inputs from start to end, storing the result of applying the mapped function to each input in the outputs&#39; pointer.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/cbf6240359f496651ebf4015f3bd319a/href">https://medium.com/media/cbf6240359f496651ebf4015f3bd319a/href</a></iframe><p>Finally the star of the show, the function that starts the threads, assigns a map_argument to each of them, and waits for all the map jobs to run, finally returning the results.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5bf0f77c6a64edde661c44d1d62dfc2e/href">https://medium.com/media/5bf0f77c6a64edde661c44d1d62dfc2e/href</a></iframe><p>Notice how this function allows us to choose how many threads we want, and partitions the data accordingly. It also handles <em>pthreads</em> ‘ creation and joining.</p><p>Finally, the way we would call this function in main looks something like this:</p><pre>concurrent_map( (void**) numbers, twice, N, NTHREADS)</pre><p>Where NTHREADS is the number of threads we want, and N is how many elements numbers has.</p><p>Now the code is done, let’s run some benchmarks! Is this really going to be faster? Will all this wrapper code make things a lot slower? Let’s find out!</p><h3>Map in C, Benchmarks: Single-threaded vs Multi-threaded</h3><p>In order to measure performance improvements from using parallel <em>Map</em>, I tested some single-threaded algorithms against their multi-threaded counterparts.</p><h4>First benchmark: slow_twice</h4><p>For my first test, I used the <em>slow_twice</em> function, which simply multiplies each number by 2.</p><p>You may be wondering, ‘why is it called slow?’. The answer is simple: we will double each number 1000 times.</p><p>This makes the operation slower, so we can measure time differences without having to use so many numbers that initialization takes too long. It also lets us benchmark the case of many memory writes.</p><p>Since execution time for each number is constant, the non-parallel algorithm’s time grows pretty much linearly on input size.</p><p>I then ran it with 2, 4 and 8 threads. My laptop has 4 cores, and I found that to be the optimum number of threads to use as well. For some other algorithms, I’ve found using a multiple of my quantity of cores to be optimum, but this hasn’t been the case.</p><h4>Benchmark Results</h4><p>I ran each benchmark 10 times and took the average, just in case.</p><p>Here are the results:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/511/1*57R9MFiE3cFnGBikwtWVDg.png" /></figure><p>For both test cases, using <strong>4 threads</strong> was about <strong>three times faster</strong> than the single-threaded implementation. This proves using Parallel <em>Map</em> is a lot faster than using a common single-threaded version.</p><p>There was also a cost to adding more than 4 threads, probably due to the overhead of initialization and context switching.</p><h4>Second benchmark: is_prime</h4><p>For this benchmark I coded a naive prime testing function: it simply iterates through all the numbers smaller than the input, and returns 1 if any divides it, 0 otherwise.</p><p>Notice how this function takes O(n) instead of O(1) for each element, so a few partitions of our data (which is ordered) will be a lot slower than the others. I wonder how this will affect running times?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/695/1*qAPJ7Zm8P5L8u-2JtZLmjg.png" /></figure><p>In this case, again the parallel algorithm beats the single-threaded one. No big surprises there. However, this time there’s an <strong>improvement when using over 4 threads</strong>!</p><p>I think this is because when partitioning our inputs, dividing it into smaller chunks makes the <strong>slowest partition take less time</strong>, thus making our bottleneck smaller.</p><h3>Conclusion</h3><p>I had a lot of fun running this experiment.</p><p>Picking <strong>how many threads to use</strong> turns out to be a lot harder than just “use the same amount as cores”, and <strong>depends a lot on our input</strong> even for very dumb algorithms.</p><p>This may help us understand why optimizing a cluster’s configuration can be such a daunting task for a big application.</p><p>In the future, I may add a parallel <em>reduce</em> implementation to complete this little framework.</p><p>A few other benchmarks that might’ve been fun and I may run in the future are <em>Map</em> in C vs <a href="http://www.datastuff.tech/programming/pythons-list-comprehensions-uses-and-advantages/">Python List Comprehensions</a>, and C vs SIMD-Assembly.</p><p><em>If you want to level up as a Data scientist, check out my </em><a href="http://www.datastuff.tech/data-science/3-machine-learning-books-that-helped-me-level-up-as-a-data-scientist/"><em>best Machine Learning books</em></a><em> list and my </em><a href="http://www.datastuff.tech/programming/terminal-tutorial-more-productive/"><em>Bash tutorial</em></a><em>.</em></p><p>Remember you can use this code any way you like, or run your own experiments, and if you do <em>please</em> don’t forget to let me know your results in the comments!</p><p>Feel free to contact me on <a href="http://www.twitter.com/strikingloo">Twitter</a>, <a href="http://www.medium.com/@strikingloo">Medium</a> or <a href="http://www.dev.to/strikingloo">dev.to</a> for anything you want to say or ask to me!</p><p><em>Originally published at </em><a href="http://www.datastuff.tech/programming/how-to-code-mapreduce-in-c-from-scratch-using-threads-pt-1-map/"><em>http://www.datastuff.tech</em></a><em> on October 19, 2019.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c6b1b01f020c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/mapreduce-in-c-from-scratch-using-threads-map-c6b1b01f020c">MapReduce in C from Scratch Using Threads: Map</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why do Neural Networks Need an Activation Function?]]></title>
            <link>https://medium.com/data-science/why-do-neural-networks-need-an-activation-function-3a5f6a5f00a?source=rss-56f7cdc8c678------2</link>
            <guid isPermaLink="false">https://medium.com/p/3a5f6a5f00a</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[mathematics]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[neural-networks]]></category>
            <dc:creator><![CDATA[Luciano Strika]]></dc:creator>
            <pubDate>Mon, 01 Jul 2019 00:21:12 GMT</pubDate>
            <atom:updated>2019-07-02T13:15:02.942Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*BpLK0UQUQGjqctt5.jpg" /></figure><p>Why do Neural Networks Need an Activation Function? Whenever you see a Neural Network’s architecture for the first time, one of the first things you’ll notice is they have a lot of interconnected layers.</p><p><strong>Each layer in a Neural Network has an activation function, but why are they necessary? And why are they so important? Learn the answer here.</strong></p><h3>What are activation functions?</h3><p>To answer the question of what Activation Functions are, let’s first take a step back and answer a bigger one: What is a Neural Network?</p><h3>What are Neural Networks?</h3><p>A Neural Network is a Machine Learning model that, given certain input and output vectors, will try to “fit” the outputs to the inputs.</p><p>What this means is, given a set of observed instances with certain values we wish to predict, and some data we have on each instance, it will try to generalize those data so that it can predict the values correctly for new instances of the problem.</p><p>As an example, we may be designing an image classifier (typically with a <a href="http://www.datastuff.tech/machine-learning/convolutional-neural-networks-an-introduction-tensorflow-eager/">Convolutional Neural Network</a>). Here, the inputs are a vector of pixels. The output could be a numerical class label (for instance, 1 for dogs, 0 for cats).</p><p>This would train a Neural Network to predict whether an image contains a cat or a dog.</p><p>But what is a mathematical function that, given a set of pixels, returns 1 if they correspond to the image of a dog, and 0 to the image of a cat?</p><p>Coming up with a mathematical function that did that by hand would be impossible. <strong>For a human</strong>.</p><p>So what we did is invent a Machine that finds that function for us.</p><p>It looks something like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/395/0*jRmjehyeDenFkVOT.png" /><figcaption>Single hidden layer Neural Network. <a href="https://docs.opencv.org/2.4/modules/ml/doc/neural_networks.html">Source</a>.</figcaption></figure><p>But you may have seen this picture many times, recognize it for a Neural Network, and still not know exactly what it represents.</p><p>Here, each circle represents a neuron in our Neural Network, and the vertically aligned neurons represent each layer.</p><h3>How do Neural Networks work?</h3><p>A neuron is just a <strong>mathematical function</strong>, that takes inputs (the outputs of the neurons pointing to it) and <strong>returns outputs</strong>.</p><p>These outputs serve as inputs for the next layer, and so on until we get to the final, output layer, which is the actual value we return.</p><p>There is an input layer, where each neuron will simply return the corresponding value in the inputs vector.</p><p>For each set of inputs, the Neural Network’s goal is to make each of its outputs <strong>as close as possible to the actual expected values</strong>.</p><p>Again, think back at the example of the image classifier.</p><p>If we take 100x100px pictures of animals as inputs, then our input layer will have 30000 neurons. That’s 10000 for all the pixels, times three since a pixel is already a triple vector (RGB values).</p><p>We will then run the inputs through each layer. We get a <strong>new vector as each layer’s output, feed it to the next layer as inputs</strong>, and so on.</p><p>Each neuron in a layer will return a single value, so a layer’s output vector will have as many dimensions as the layer has neurons.</p><p>So, which value will a neuron return, given some inputs?</p><h3>What does a Neuron do?</h3><p>A neuron will take an input vector, and do three things to it:</p><ul><li>Multiply it by a weights vector.</li><li>Add a bias value to that product.</li><li>Apply an <strong>activation function</strong> to that value.</li></ul><p>And we finally got to the core of our business: that’s what activation functions do.</p><p>We’ll typically use <strong>non-linear functions as activation functions</strong>. This is because the linear part is already handled by the previously applied product and addition.</p><h3>What are the most commonly used activation functions?</h3><p>I’m saying non-linear functions and it sounds logic enough, but what are the typical, commonly used activation functions?</p><p>Let’s see some examples.</p><h3>ReLU</h3><p>ReLU stands for “Rectified Linear Unit”.</p><p>Of all the activation functions, this is the one that’s <strong>most similar to a linear one</strong>:</p><ul><li>For non-negative values, it just applies the identity.</li><li>For negative values, it returns 0.</li></ul><p>In mathematical words,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/133/0*_HZ9xRVv9WeUP7HA.png" /></figure><p>This means all negative values will become 0, while the rest of the values just stay as they are.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*n9edLWjCUcRb-a3v.png" /></figure><p>This is a biologically inspired function, since neurons in a brain will either “fire” (return a positive value) or not (return 0).</p><p>Notice how combined with a bias, this actually <strong>filters out any value beneath a certain threshold</strong>.</p><p>Suppose our bias had a value of -b. Any input value lower than b, after adding the bias will become negative. This turns to a 0 after applying ReLU to it.</p><h3>Sigmoid</h3><p>The sigmoid function takes any real number as input, and <strong>returns a value between 0 and 1</strong>. Since it is continuous, it effectively “ <strong>smushes</strong> “ values:</p><p>If you apply the sigmoid to 3, you get 0.95. Apply it to 10, you get 0.999… And it will keep approaching 1 without ever reaching it.</p><p>The same happens in the negative direction, except there it converges to 0.</p><p>Here’s the mathematical formula for the sigmoid function.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/119/0*cFjqP1ztAkAz8B4A.png" /></figure><p>As you see, it approaches 1 as x approaches infinity, and approaches 0 if x approaches minus infinity.</p><p>It is also symmetrical, and has a value of 1/2 when its input is 0.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/320/0*xW3Mc8rvAvWFrYB_.png" /></figure><p>Since it takes values between 0 and 1, this function is extremely useful as an output if you want to model a probability.</p><p>It’s also helpful if you wish to apply a “filter” to partially keep a certain value (like in an <a href="http://www.datastuff.tech/machine-learning/lstm-how-to-train-neural-networks-to-write-like-lovecraft/">LSTM’s forget gate</a>).</p><h3>Why do Neural Networks Need an Activation Function?</h3><p>We’ve already talked about the applications some different activation functions have, in different cases.</p><p>Some let a signal through or obstruct it, others filter its intensity. There’s even the <a href="https://en.wikipedia.org/wiki/Hyperbolic_function">tanh</a> activation function: instead of filtering, it turns its input into either a negative or positive value.</p><p>But what why do our Neural Networks need Activation Functions? <strong>What would happen if we didn’t use them</strong>?</p><p>I found the explanation for this question in Yoshua Bengio’s awesome <a href="https://amzn.to/305g2MF">Deep Learning book</a>, and I think it’s perfectly explained there.</p><p>We could, instead of composing our linear transformations with non-linear functions, make each neuron simply return their result (effectively composing them with the identity instead).</p><p>But then all of our layers would simply stack one affine (product plus addition) transformation after another. Each layer would simply add a vector product, and vector addition, to the previous one.</p><p>It can be shown (and you can even convince yourself if you try the math with a small vector on a whiteboard) that this composition of affine transformations, is equivalent to a single affine transformation.</p><p>Effectively, this whole “Neural Network” where all activation functions have been replaced by the identity would be <strong>nothing more than a vector product and a bias addition</strong>.</p><p>There are many problems a linear transformation can’t solve, so we would effectively be <strong>shrinking the quantity of functions</strong> our model could estimate.</p><p>As a very simple but earthshaking example, consider the XOR operator.</p><figure><img alt="XOR values table" src="https://cdn-images-1.medium.com/proxy/1*FDAAQeaaE8s_Il8K9llr8w.png" /></figure><p>Try to find a two-element vector, plus a bias that can take x1 and x2, and turn them into x1 XOR x2. Go ahead, I’ll wait.</p><p>…</p><p>Exactly, you can’t. nobody can. However, consider</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/307/0*_Vh988l6o4Jto5Vr.png" /></figure><p>If you work the math, you’ll see this has the desired output for each possible combination of 1 and 0.</p><p>Congratulations! You’ve just trained your first Neural Network!</p><p>And it’s learned a problem a linear model could never have learned.</p><h3>Conclusions</h3><p>I hope after this explanation, you now have a better understanding of why Neural Networks need an Activation Function.</p><p>In future articles, I may cover other Activation Functions and their uses, like SoftMax and the controversial Cos.</p><p>So what do you think? Did you learn anything from this article? Did you find it interesting? Was the math off?</p><p>Feel free to contact me on <a href="http://www.twitter.com/strikingloo">Twitter</a>, <a href="http://www.medium.com/@strikingloo">Medium</a> or <a href="http://www.dev.to/strikingloo">dev.to</a> for anything you want to say or ask to me!</p><p><em>If you want to level up as a Data scientist, check out my </em><a href="http://www.datastuff.tech/data-science/3-machine-learning-books-that-helped-me-level-up-as-a-data-scientist/"><em>best Machine Learning books</em></a><em> list and my </em><a href="http://www.datastuff.tech/programming/terminal-tutorial-more-productive/"><em>Bash tutorial</em></a><em>.</em></p><p><em>Originally published at </em><a href="http://www.datastuff.tech/machine-learning/why-do-neural-networks-need-an-activation-function/"><em>http://www.datastuff.tech</em></a><em> on July 1, 2019.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3a5f6a5f00a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/why-do-neural-networks-need-an-activation-function-3a5f6a5f00a">Why do Neural Networks Need an Activation Function?</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[LSTM: How To Train Neural Networks to Write like Lovecraft]]></title>
            <link>https://medium.com/data-science/lstm-how-to-train-neural-networks-to-write-like-lovecraft-e56e1165f514?source=rss-56f7cdc8c678------2</link>
            <guid isPermaLink="false">https://medium.com/p/e56e1165f514</guid>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[tensorflow]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Luciano Strika]]></dc:creator>
            <pubDate>Mon, 24 Jun 2019 00:37:59 GMT</pubDate>
            <atom:updated>2019-06-24T13:58:27.829Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*FW21WQHMWSmLJ6L5.jpg" /></figure><p>LSTM Neural Networks have seen a lot of use in the recent years, both for text and music generation, and for Time Series Forecasting.</p><p>Today, I’ll teach you how to train a LSTM Neural Network for text generation, so that it can write with H. P. Lovecraft’s style.</p><p>In order to train this LSTM, we’ll be using TensorFlow’s Keras API for Python.</p><p>I learned about this subject from this awesome <a href="http://colah.github.io/posts/2015-08-Understanding-LSTMs/">LSTM Neural Networks tutorial</a>. My code follows this <a href="https://chunml.github.io/ChunML.github.io/project/Creating-Text-Generator-Using-Recurrent-Neural-Network/">Text Generation tutorial</a> ‘s closely.</p><p>I’ll show you my Python examples and results as usual, but first, let’s do some explaining.</p><h3>What are LSTM Neural Networks?</h3><p>The most vanilla, run-of-the-mill Neural Network, called a Multi-Layer-Perceptron, is just a composition of fully connected layers.</p><p>In these models, the input is a vector of features, and each subsequent layer is a set of “neurons”.</p><p>Each neuron performs an affine (linear) transformation to the previous layer’s output, and then applies some non-linear function to that result.</p><p>The output of a layer’s neurons, a new vector, is fed to the next layer, and so on.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/535/0*rv7S_vTlogprTuG0.jpg" /><figcaption><a href="https://www.researchgate.net/figure/A-hypothetical-example-of-Multilayer-Perceptron-Network_fig4_303875065">Source</a></figcaption></figure><p>A LSTM (Long Short-term Memory) Neural Network is just another kind of <a href="http://www.datastuff.tech/machine-learning/autoencoder-deep-learning-tensorflow-eager-api-keras/">Artificial Neural Network</a>, containing LSTM cells as neurons in some of its layers.</p><p>Much like <a href="http://www.datastuff.tech/machine-learning/convolutional-neural-networks-an-introduction-tensorflow-eager/">Convolutional Layers</a> help a Neural Network learn about image features, LSTM cells help the Network learn about temporal data, something which other Machine Learning models traditionally struggled with.</p><p>How do LSTM cells work? I’ll explain it now, though I highly recommend you give those tutorials a chance too.</p><h3>How do LSTM cells work?</h3><p>An LSTM layer will contain many LSTM cells.</p><p>Each LSTM cell in our Neural Network will only look at a single column of its inputs, and also at the previous column’s LSTM cell’s output.</p><p>Normally, we feed our LSTM Neural Network a whole matrix as its input, where each column corresponds to something that “comes before” the next column.</p><p>This way, each LSTM cell will have <strong>two different input vectors</strong>: the previous LSTM cell’s output (which gives it some information about the previous input column) and its own input column.</p><h3>LSTM Cells in action: an intuitive example.</h3><p>For instance, if we were training an LSTM Neural Network to predict stock exchange values, we could feed it a vector with a stock’s closing price in the last three days.</p><p>The first LSTM cell, in that case, would use the first day as input, and send some extracted features to the next cell.</p><p>That second cell would look at the second day’s price, and also at whatever the previous cell learned from yesterday, before generating new inputs for the next cell.</p><p>After doing this for each cell, the last one will actually have a lot of temporal information. It will receive, from the previous one, what it learned from yesterday’s closing price, and from the previous two (through the other cells’ extracted information).</p><p>You can experiment with different time windows, and also change how many units (neurons) will look at each day’s data, but this is the general idea.</p><h3>How LSTM Cells work: the Math.</h3><p>The actual math behind what each cell extracts from the previous one is a bit more involved.</p><h4>Forget Gate</h4><p>The “forget gate” is a sigmoid layer, that regulates how much the previous cell’s outputs will influence this one’s.</p><p>It takes as input both the previous cell’s “hidden state” (another output vector), and the actual inputs from the previous layer.</p><p>Since it is a sigmoid, it will return a vector of “probabilities”: values between 0 and 1.</p><p>They will <strong>multiply the previous cell’s outputs</strong> to regulate how much influence they hold, creating this cell’s state.</p><p>For instance, in a drastic case, the sigmoid may return a vector of zeroes, and the whole state would be multiplied by 0 and thus discarded.</p><p>This may happen if this layer sees a very big change in the inputs distribution, for example.</p><h4>Input Gate</h4><p>Unlike the forget gate, the input gate’s output is added to the previous cell’s outputs (after they’ve been multiplied by the forget gate’s output).</p><p>The input gate is the dot product of two different layers’ outputs, though they both take the same input as the forget gate (previous cell’s hidden state, and previous layer’s outputs):</p><ul><li>A <strong>sigmoid unit</strong>, regulating how much the new information will impact this cell’s output.</li><li>A <strong>tanh unit</strong>, which actually extracts the new information. Notice tanh takes values between -1 and 1.</li></ul><p>The <strong>product of these two units</strong> (which could, again, be 0, or be exactly equal to the tanh output, or anything in between) is added to this neuron’s cell state.</p><h4>The LSTM cell’s outputs</h4><p>The cell’s state is what the next LSTM cell will receive as input, along with this cell’s hidden state.</p><p>The hidden state will be <strong>another tanh unit</strong> applied to this neuron’s state, multiplied by another <strong>sigmoid unit</strong> that takes the previous layer’s and cell’s outputs (just like the forget gate).</p><p>Here’s a visualization of what each LSTM cell looks like, borrowed from the tutorial I just linked:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/782/0*CaDdr4CJjAz9qBey.png" /><figcaption>Source: <a href="https://chunml.github.io/ChunML.github.io/project/Creating-Text-Generator-Using-Recurrent-Neural-Network/">Text Generating LSTMs</a></figcaption></figure><p>Now that we’ve covered the theory, let’s move on to some practical uses!</p><p>As usual, all of the code is <a href="https://github.com/StrikingLoo/LoveCraftLSTM">available on GitHub</a> if you want to try it out, or you can just follow along and see the gists.</p><h3>Training LSTM Neural Networks with TensorFlow Keras</h3><p>For this task, I used this <a href="https://github.com/vilmibm/lovecraftcorpus">dataset containing 60 Lovecraft tales</a>.</p><p>Since he wrote most of his work in the 20s, and he died in 1937, it’s now mostly in the public domain, so it wasn’t that hard to get.</p><p>I thought training a Neural Network to write like him would be an interesting challenge.</p><p>This is because, on the one hand, he had a very distinct style (with abundant purple prose: using weird words and elaborate language), but on the other he used a very complex vocabulary, and a Network may have trouble understanding it.</p><p>For instance, here’s a random sentence from the first tale in the dataset:</p><blockquote><em>At night the subtle stirring of the black city outside, the sinister scurrying of rats in the wormy partitions, and the creaking of hidden timbers in the centuried house, were enough to give him a sense of strident pandemonium</em></blockquote><p>If I can get a Neural Network to write “pandemonium”, then I’ll be impressed.</p><h3>Preprocessing our data</h3><p>In order to train an LSTM Neural Network to generate text, we must first preprocess our text data so that it can be consumed by the network.</p><p>In this case, since a Neural Network takes vectors as input, we need a way to convert the text into vectors.</p><p>For these examples, I decided to train my LSTM Neural Networks to predict the next M characters in a string, taking as input the previous N ones.</p><p>To be able to feed it the N characters, I did a one-hot encoding of each one of them, so that the network’s input is a matrix of CxN elements, where C is the total number of different characters on my dataset.</p><p>First, we read the text files and concatenate all of their contents.</p><p>We limit our characters to be alphanumerical, plus a few punctuation marks.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ac4219a44577c03cece80c0af08f0500/href">https://medium.com/media/ac4219a44577c03cece80c0af08f0500/href</a></iframe><p>We can then proceed to one-hot encode the strings into matrices, where every element of the <em>j</em>-th column is a 0 except for the one corresponding to the <em>j</em>-th character in the corpus.</p><p>In order to do this, we first define a dictionary that assigns an index to each character.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d7c8a07569fe8c6cd8a746e6e2ef0c02/href">https://medium.com/media/d7c8a07569fe8c6cd8a746e6e2ef0c02/href</a></iframe><p>Notice how, if we wished to sample our data, we could just make the variable <em>slices</em> smaller.</p><p>I also chose a value for <em>SEQ_LENGTH</em> of 50, making the network receive 50 characters and try to predict the next 50.</p><h3>Training our LSTM Neural Network in Keras</h3><p>In order to train the Neural Network, we must first define it.</p><p>This Python code creates an LSTM Neural Network with two LSTM layers, each with 100 units.</p><p>Remember each unit has one cell for each character in the input sequence, thus 50.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c6800309c758fa437f5e839610b7a6df/href">https://medium.com/media/c6800309c758fa437f5e839610b7a6df/href</a></iframe><p>Here <em>VOCAB_SIZE</em> is just the amount of characters we’ll use, and <em>TimeDistributed</em> is a way of applying a given layer to each different cell, maintaining temporal ordering.</p><p>For this model, I actually tried many different learning rates to test convergence speed vs overfitting.</p><p>Here’s the code for training:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5cefa8229b96a781efe7282a6acc9542/href">https://medium.com/media/5cefa8229b96a781efe7282a6acc9542/href</a></iframe><p>What you are seeing is what had the best performance in terms of loss minimization.</p><p>However, with a binary_cross_entropy of 0.0244 in the final epoch (after 500 epochs), here’s what the model’s output looked like.</p><pre>Tolman hast toemtnsteaetl nh otmn tf titer aut tot tust tot ahen h l the srrers ohre trrl tf thes snneenpecg tettng s olt oait ted beally tad ened ths tan en ng y afstrte and trr t sare t teohetilman hnd tdwasd hxpeinte thicpered the reed af the satl r tnnd Tev hilman hnteut iout y techesd d ty ter thet te wnow tn tis strdend af ttece and tn aise ecn</pre><p>There are many <strong>good things</strong> about this output, and <strong>many bad ones</strong> as well.</p><p>The way the spacing is set up, with words mostly between 2 and 5 characters long with some longer outliers, is pretty similar to the actual word length distribution in the corpus.</p><p>I also noticed the <strong>letters</strong> ‘T’, ‘E’ and ‘I’ were <strong>appearing very commonly</strong>, whereas ‘y’ or ‘x’ were <strong>less frequent</strong>.</p><p>When I looked at <strong>letter relative frequencies</strong> in the sampled output versus the corpus, they were pretty similar. It’s the <strong>ordering</strong> that’s <strong>completely off</strong>.</p><p>There is also something to be said about how <strong>capital letters only appear after spaces</strong>, as is usually the case in English.</p><p>To generate these outputs, I simply asked the model to predict the next 50 characters for different 50 character subsets in the corpus. If it’s this bad with training data, I figured testing or random data wouldn’t be worth checking.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d49e3fec4b157793388c236a5106cbb7/href">https://medium.com/media/d49e3fec4b157793388c236a5106cbb7/href</a></iframe><p>The nonsense actually reminded me of one of H. P. Lovecraft’s most famous tales, “Call of Cthulhu”, where people start having hallucinations about this cosmic, eldritch being, and they say things like:</p><blockquote><em>Ph’nglui mglw’nafh </em>Cthulhu R’lyeh<em> wgah’nagl fhtagn.</em></blockquote><p>Sadly the model wasn’t overfitting that either, it was clearly <strong>underfitting</strong>.</p><p>So I tried to make its task smaller, and the model bigger: 125 units, predicting only 30 characters.</p><h3>Bigger model, smaller problem. Any results?</h3><p>With this smaller model, after another 500 epochs, some patterns began to emerge.</p><p>Even though the loss function wasn’t that much smaller (at 210), the character’s frequency remained similar to the corpus’.</p><p>The ordering of characters improved a lot though: here’s a random sample from its output, see if you can spot some words.</p><pre>the sreun troor Tvwood sas an ahet eae rin and t paared th te aoolling onout The e was thme trr t sovtle tousersation oefore tifdeng tor teiak uth tnd tone gen ao tolman aarreed y arsred tor h tndarcount tf tis feaont oieams wnd toar Tes heut oas nery tositreenic and t aeed aoet thme hing tftht to te tene Te was noewked ay tis prass s deegn aedgireean ect and tot ced the sueer anoormal -iuking torsarn oaich hnher tad beaerked toring the sars tark he e was tot tech</pre><p>Tech, the, and, was… <strong>small words</strong> are where it’s at! It also realized many words ended with <strong>common suffixes</strong> like -ing, -ed, and -tion.</p><p>Out of 10000 words, 740 were “ <em>the</em>”, 37 ended in “ <em>tion</em>” (whereas only 3 contained without ending in it), and 115 ended in — <em>ing</em>.</p><p>Other common words were “than” and “that”, though the model was clearly still unable to produce English sentences.</p><h3>Even bigger model</h3><p>This gave me hopes. The Neural Network was clearly learning <em>something</em>, just not enough.</p><p>So I did what you do when your model underfits: I tried an even bigger Neural Network.</p><p>Take into account, I’m running this on my laptop.</p><p>With a modest 16GB of RAM and an i7 processor, these models take hours to learn.</p><p>So I set the amount of units to 150, and tried my hand again at 50 characters.</p><p>I figured maybe giving it a smaller time window was making things harder for the Network.</p><p>Here’s what the model’s output was like, after a few hours of training.</p><pre>andeonlenl oou torl u aote targore -trnnt d tft thit tewk d tene tosenof the stown ooaued aetane ng thet thes teutd nn aostenered tn t9t aad tndeutler y aean the stun h tf trrns anpne thin te saithdotaer totre aene Tahe sasen ahet teae es y aeweeaherr aore ereus oorsedt aern totl s a dthe snlanete toase af the srrls-thet treud tn the tewdetern tarsd totl s a dthe searle of the sere t trrd eneor tes ansreat tear d af teseleedtaner nl and tad thre n tnsrnn tearltf trrn T has tn oredt d to e e te hlte tf the sndirehio aeartdtf trrns afey aoug ath e -ahe sigtereeng tnd tnenheneo l arther ardseu troa Tnethe setded toaue and tfethe sawt ontnaeteenn an the setk eeusd ao enl af treu r ue oartenng otueried tnd toottes the r arlet ahicl tend orn teer ohre teleole tf the sastr ahete ng tf toeeteyng tnteut ooseh aore of theu y aeagteng tntn rtng aoanleterrh ahrhnterted tnsastenely aisg ng tf toueea en toaue y anter aaneonht tf the sane ng tf the</pre><p>Pure nonsense, except a lot of “the” and “and”s.</p><p>It was actually saying “the” more often than the previous one, but it hadn’t learned about gerunds yet (no -ing).</p><p>Interestingly, many words here ended with “-ed” which means it was kinda grasping the idea of the <strong>past tense</strong>.</p><p>I let it go at it a few hundred more epochs (to a total of 750).</p><p>The output didn’t change too much, still a lot of “the”, “a” and “an”, and still no bigger structure. Here’s another sample:</p><pre>Tn t srtriueth ao tnsect on tias ng the sasteten c wntnerseoa onplsineon was ahe ey thet tf teerreag tispsliaer atecoeent of teok ond ttundtrom tirious arrte of the sncirthio sousangst tnr r te the seaol enle tiedleoisened ty trococtinetrongsoa Trrlricswf tnr txeenesd ng tispreeent T wad botmithoth te tnsrtusds tn t y afher worsl ahet then</pre><p>An interesting thing that emerged here though, was the use of prepositions and pronouns.</p><p>The network wrote “I”, “you”, “she”, “we”, “of” and other similar words a few times. All in all, <strong>prepositions and pronouns</strong> amounted to about <strong>10% of the total sampled words</strong>.</p><p>This was an improvement, as the Network was clearly learning low-entropy words.</p><p>However, it was still far from generating coherent English texts.</p><p>I let it train 100 more epochs, and then killed it.</p><p>Here’s its last output.</p><pre>thes was aooceett than engd and te trognd tarnereohs aot teiweth tncen etf thet torei The t hhod nem tait t had nornd tn t yand tesle onet te heen t960 tnd t960 wndardhe tnong toresy aarers oot tnsoglnorom thine tarhare toneeng ahet and the sontain teadlny of the ttrrteof ty tndirtanss aoane ond terk thich hhe senr aesteeeld Tthhod nem ah tf the saar hof tnhe e on thet teauons and teu the ware taiceered t rn trr trnerileon and</pre><p>I knew it was doing its best, but it wasn’t really going anywhere, at least not quickly enough.</p><p>I thought of accelerating convergence speed with <strong>Batch Normalization</strong>.</p><p>However, I read on StackOverflow that BatchNorm is not supposed to be used with LSTM Neural Networks.</p><p>If any of you is more experienced with LSTM nets, please let me know if that’s right in the comments!</p><p>At last, I tried this same task with 10 characters as input and 10 as output.</p><p>I guess the model wasn’t getting enough context to predict things well enough though: the results were awful.</p><p>I considered the experiment finished for now.</p><h3>Conclusions</h3><p>While it is clear, looking at other people’s work, that an LSTM Neural Network <em>could</em> learn to write like Lovecraft, I don’t think my PC is powerful enough to train a big enough model in a reasonable time.</p><p>Or maybe it just needs more data than I had.</p><p>In the future, I’d like to repeat this experiment with a word-based approach instead of a character-based one.</p><p>I checked, and about 10% of the words in the corpus appear only once.</p><p>Is there any good practice I should follow if I removed them before training? Like replacing all nouns with the same one, sampling from <a href="http://www.datastuff.tech/machine-learning/k-means-clustering-unsupervised-learning-for-recommender-systems/">clusters</a>, or something? Please let me know! I’m sure many of you are more experienced with LSTM neural networks than I.</p><p>Do you think this would have worked better with a different architecture? Something I should have handled differently? Please also let me know, I want to learn more about this.</p><p>Did you find any rookie mistakes on my code? Do you think I’m an idiot for not trying XYZ? Or did you actually find my experiment enjoyable, or maybe you even learned something from this article?</p><p>Contact me on <a href="http://www.twitter.com/strikingloo">Twitter</a>, <a href="http://linkedin.com/in/luciano-strika">LinkedIn</a>, <a href="http://medium.com/@strikingloo">Medium</a> or <a href="http://www.dev.to/strikingloo">Dev.to</a> if you want to discuss that, or any related topic.</p><p><em>If you want to become a Data scientist, or learn something new, check out my </em><a href="http://www.datastuff.tech/data-science/3-machine-learning-books-that-helped-me-level-up-as-a-data-scientist/"><em>Machine Learning Reading List</em></a><em>!</em></p><p><em>Originally published at </em><a href="http://www.datastuff.tech/machine-learning/lstm-how-to-train-neural-networks-to-write-like-lovecraft/"><em>http://www.datastuff.tech</em></a><em> on June 24, 2019.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e56e1165f514" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/lstm-how-to-train-neural-networks-to-write-like-lovecraft-e56e1165f514">LSTM: How To Train Neural Networks to Write like Lovecraft</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Probability Distributions Every Data Scientist Should Know]]></title>
            <link>https://medium.com/data-science/probability-distributions-every-data-scientist-should-know-e0a850c19157?source=rss-56f7cdc8c678------2</link>
            <guid isPermaLink="false">https://medium.com/p/e0a850c19157</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[mathematics]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Luciano Strika]]></dc:creator>
            <pubDate>Mon, 17 Jun 2019 04:34:11 GMT</pubDate>
            <atom:updated>2019-06-18T02:16:15.557Z</atom:updated>
            <content:encoded><![CDATA[<h3>5 Probability Distributions Every Data Scientist Should Know</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*9p8A14DlVSVwAFYv.jpg" /><figcaption><strong>Probably</strong> a nice dashboard. Source: <a href="https://pixabay.com/photos/digital-marketing-technology-1433427/">Pixabay</a>.</figcaption></figure><p>Probability Distributions are like 3D glasses. They allow a skilled Data Scientist to recognize patterns in otherwise completely random variables.</p><p>In a way, most of the other Data Science or Machine Learning skills are based on certain assumptions about the probability distributions of your data.</p><p>This makes probability knowledge part of the basis on which you can build your toolkit as a statistician. The first steps if you are figuring out <a href="http://www.datastuff.tech/data-science/3-machine-learning-books-that-helped-me-level-up-as-a-data-scientist/">how to become a Data Scientist</a>.</p><p>Without further ado, let us cut to the chase.</p><h3>What are Probability Distributions?</h3><p>In Probability and Statistics, a <strong>random variable</strong> is a thing that <strong>takes random values</strong>, like “the height of the next person I see” or “the amount of cook’s hairs in my next ramen bowl”.</p><p>Given a random variable <em>X</em>, we’d like to have a way of describing which values it takes. Even more than that, we’d like to characterize <strong>how likely</strong> it is for that variable to <strong>take a certain value</strong> <em>x</em>.</p><p>For instance, if <em>X</em> is “how many cats my girlfriend has”, then there’s a non-zero chance that number could be 1. One could argue there’s a non-zero probability that value could even be 5 or 10.</p><p>However, there’s no way (and therefore no probability) that a person will have negative cats.</p><p>We therefore would like an unambiguous, mathematical way of expressing every possible value <em>x</em> a variable <em>X</em> can take, and how likely the event <em>(X= x)</em> is.</p><p>In order to do this, we define a function <em>P</em>, such that <em>P(X = x)</em> is the probability of the variable <em>X</em> having a value of <em>x</em>.</p><p>We could also ask for P(X &lt; x), or P(X &gt; x), for intervals instead of discrete values. This will become even more relevant soon.</p><p><em>P</em> is the variable’s <strong>density function</strong>, and it characterizes that variable’s <strong>distribution</strong>.</p><p>Over time, scientists have come to realize many things in nature and real life tend to <strong>behave similarly</strong>, with variables sharing a distribution, or having the same density functions (or a similar function changing a few constants in it).</p><p>Interestingly, for <em>P</em> to be an actual density function, some things have to apply.</p><ul><li><strong><em>P(X =x)</em> &lt;= 1</strong> for any value <em>x</em>. Nothing’s more certain than certain.</li><li><strong><em>P(X =x)</em> &gt;= 0</strong> for any value <em>x</em>. A thing can be impossible, but not less likely than that.</li><li>And the last one: the <strong>sum</strong> of <em>P(X=x)</em> for all possible values <em>x </em><strong>is 1</strong>.</li></ul><p>This last one means something like “the probability of X taking <em>any</em> value in the universe, <em>has</em> to add up to 1, since we know it will take <em>some</em> value”.</p><h3>Discrete vs Continuous Random Variable Distributions</h3><p>Lastly, random variables can be thought of as belonging to two groups: <strong>discrete</strong> and <strong>continuous</strong> random variables.</p><h4>Discrete Random Variables</h4><p><strong>Discrete variables</strong> have a discrete set of possible values, each of them with a non-zero probability.</p><p>For instance, when flipping a coin, if we say</p><blockquote><em>X = “1 if the coin is heads, 0 if tails”</em></blockquote><p>Then <em>P(X = 1) = P(X = 0) = 0.5</em>.</p><p>Note however, that a discrete set need not be finite.</p><p>A <strong>geometric distribution</strong>, is used for modelling the chance of some event with probability <em>p</em> <strong>happening after <em>k</em> retries</strong>.</p><p>It has the following density formula.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/172/0*B0Mg0_Vi4y7L_-4-.png" /></figure><p>Where <strong><em>k</em> can take any non-negative value with a positive probability</strong>.</p><p>Notice how the sum of all possible values’ probabilities still <strong>adds up to 1</strong>.</p><h4>Continuous Random Variables</h4><p>If you said</p><blockquote><em>X = “the length in millimeters (without rounding) of a randomly plucked hair from my head”</em></blockquote><p>Which possible values can <em>X</em> take? We can all probably agree a negative value doesn’t make any sense here.</p><p>However if you said it is exactly 1 millimeter, and not 1.1853759… or something like that, I would either doubt your measuring skills, or your measuring error reporting.</p><p>A continuous random variable can take <strong>any value</strong> in a given (continuous) interval.</p><p>Therefore, if we assigned a <strong>non-zero probability to all of its possible values</strong>, their sum would <strong>not add up to 1</strong>.</p><p>To solve this, if <em>X</em> is continuous, we set<em> P(X=x) = 0</em> for all <em>k</em>, and instead assign a non-zero chance to <em>X</em> taking a value <strong>in a certain interval.</strong></p><p>To express the probability of X laying between values<em> a</em> and <em>b</em>, we say <br><em>P(a &lt; X &lt; b)</em>.</p><p>Instead of just replacing values in a density function, to get <em>P(a &lt; X &lt; b)</em> for <em>X</em> a continuous variable, you’ll integrate <em>X</em>’s density function from <em>a</em> to <em>b</em>.</p><p>Whoah, you’ve made it through the whole theory section! Here’s your reward.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*ugmZEp0ffv5igMMG.jpg" /><figcaption>Source: <a href="https://pixabay.com/photos/pug-puppy-dog-animal-cute-690566/">pixabay</a>.</figcaption></figure><p>Now that you know what a probability distribution is, let’s learn about some of the most common ones!</p><h3>Bernoulli Probability Distribution</h3><p>A Random Variable with a Bernoulli Distribution is among the simplest ones.</p><p>It represents a <strong>binary event</strong>: “this happened” vs “this didn’t happen”, and takes a value <em>p</em> as its <strong>only parameter</strong>, which represents the <strong>probability</strong> that <strong>the event will occur</strong>.</p><p>A random variable <em>B</em> with a Bernoulli distribution with parameter <em>p</em> will have the following density function:</p><blockquote><em>P(B = 1) = p, P(B =0)= (1-p)</em></blockquote><p>Here <em>B=1</em> means the event happened, and <em>B=0</em> means it didn’t.</p><p>Notice how both probabilities add up to 1, and therefore no other value for <em>B</em> will be possible.</p><h3>Uniform Probability Distribution</h3><p>There are two kinds of uniform random variables: discrete and continuous ones.</p><p>A <strong>discrete uniform distribution</strong> will take a <strong>(finite)</strong> set of values <em>S</em>, and assign a probability of <em>1/n</em> to each of them, where <em>n</em> is the amount of elements in<em> S</em>.</p><p>This way, if for instance my variable<em> Y</em> was uniform in {1,2,3}, then there’d be a 33% chance each of those values came out.</p><p>A very typical case of a discrete uniform random variable is found in <strong>dice</strong>, where your typical dice has the set of values {1,2,3,4,5,6}.</p><p>A <strong>continuous uniform distribution</strong>, instead, only takes as parameters, and assigns the same density to each value in the <strong>interval between them</strong>.</p><p>That means the probability of Y taking a <strong>value in an interval</strong> (from <em>c</em> to <em>d</em>) is <strong>proportional to its size</strong> versus the size of the whole interval ( <em>b-a</em>).</p><p>Therefore if<em> Y</em> is uniformly distributed between <em>a</em> and <em>b</em>, then</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/355/0*PiiV2650PVJu0Ehb.png" /></figure><p>This way, if <em>Y</em> is a uniform random variable between 1 and 2,</p><p>Python’s random package&#39;s random method samples a uniformly distributed continuous variable between 0 and 1.</p><p>Interestingly, it can be shown that <strong>any other distribution</strong> can be sampled given a <a href="https://www.mathworks.com/help/stats/generate-random-numbers-using-the-uniform-distribution-inversion-method.html">uniform random values generator and some calculus</a>.</p><h3>Normal Probability Distribution</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/340/0*_Qc_D5Ru9C5x8Oe_.png" /><figcaption>Normal Distributions. source: <a href="https://en.wikipedia.org/wiki/Normal_distribution">Wikipedia</a></figcaption></figure><p><strong>Normally distributed variables</strong> are so commonly found in nature, they’re actually . That’s actually where the name comes from.</p><p>If you round up all your workmates and measure their heights, or weigh them all and plot a histogram with the results, odds are it’s gonna approach a normal distribution.</p><p>I actually saw this effect when I showed you <a href="http://www.datastuff.tech/data-analysis/data-analysis-pandas-seaborn-kaggle-dataset/">Exploratory Data Analysis</a> examples.</p><p>It can also be shown that if you <strong>take a sample</strong> of any random variable and <strong>average those measures</strong>, and repeat that process many times, that average will also have a <strong>normal distribution</strong>. That fact’s so important, it’s called the <a href="https://math.tutorvista.com/statistics/fundamental-theorem-of-statistics.html">fundamental theorem of statistics</a>.</p><p>Normally distributed variables:</p><ul><li>Are <strong>symmetrical</strong>, centered around a mean (usually called <strong>μ</strong>).</li><li>Can take <strong>all values on the real space</strong>, but only deviate two sigmas from the norm 5% of the time.</li><li>Are <strong>literally everywhere</strong>.</li></ul><p>Most often if you measure any empirical data and it’s symmetrical, assuming it’s normal will kinda work.</p><p>For example, rolling <em>K</em> dice and adding up the results will distribute pretty much normally.</p><h3>Log-Normal Probability Distribution</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/0*m99NvAtL_cu-5MHH.png" /><figcaption>Lognormal distribution. source: <a href="https://en.wikipedia.org/wiki/Log-normal_distribution">Wikipedia</a></figcaption></figure><p>Log-normal probability distribution is Normal Probability Distribution’s smaller, less frequently seen sister.</p><p>A variable <em>X</em> is said to be <strong>log-normally distributed</strong> if the variable <em>Y = log(X)</em> follows a normal distribution.</p><p>When plotted in a histogram, log-normal probability distributions are <strong>asymmetrical</strong>, and become even more so if their standard deviation is bigger.</p><p>I believe <strong>lognormal</strong> distributions to be worth mentioning, because <strong>most money-based variables</strong> behave this way.</p><p>If you look at the probability distributions of any variable that relates to money, like</p><ul><li>Amount sent on the latest transfer of a certain bank.</li><li>Volume of the latest transaction in Wall Street.</li><li>A set of companies’ quarterly earnings for a given quarter.</li></ul><p>They will usually not have a normal probability distribution, but will behave much closer to a lognormal random variable.</p><p><em>(For other Data Scientists: chime in in the comments if you can think of any other empirical lognormal variables you’ve come across in your work! Especially anything outside of finances)</em>.</p><h3>Exponential Probability Distribution</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/325/1*SkmRHZpApUuQM8pYkj2cTg.png" /><figcaption>Source: <a href="https://en.wikipedia.org/wiki/Exponential_distribution">Wikipedia</a></figcaption></figure><p><strong>Exponential probability distributions</strong> appear everywhere, too.</p><p>They are heavily linked to a Probability concept called a <strong>Poisson Process</strong>.</p><p>Stealing straight from Wikipedia, a <a href="https://en.wikipedia.org/wiki/Poisson_point_process">Poisson Process</a> is “ <em>a process in which events occur continuously and independently at a constant average rate</em> “.</p><p>All that means is, if:</p><ul><li>You have a lot of events going.</li><li>They happen at a certain rate (which <strong>does not change</strong> over time).</li><li>Just because one happened the chances of another one happening don’t change.</li></ul><p>Then you have a Poisson process.</p><p>Some examples could be requests coming to a server, transactions happening in a supermarket, or birds fishing in a certain lake.</p><p>Imagine a Poisson Process with a frequency rate of λ (say, events happen once every second).</p><p>Exponential random variables model the time it takes, after an event, for the next event to occur.</p><p>Interestingly, in a Poisson Process <strong>an event can happen anywhere between 0 and infinity times</strong> ( <em>with decreasing probability</em>), in any interval of time.</p><p>This means there’s a <strong>non-zero chance that the event won’t happen, no matter how long you wait</strong>. It also means it could happen a lot of times in a very short interval.</p><p>In class we used to joke bus arrivals are Poisson Processes. I think the response time when you send a WhatsApp message to <em>some people</em> also fits the criteria.</p><p>However, the λ parameter <strong>regulates the frequency</strong> of the events.</p><p>It will make the <strong>expected time</strong> it actually takes for an event to happen <strong>center around a certain value</strong>.</p><p>This means if we know a taxi passes our block every 15 minutes, even though theoretically we <em>could</em> wait for it forever, it’s extremely likely we won’t wait longer than, say, 30 minutes.</p><h3>Exponential Probability Distribution: In Practice</h3><p>Here’s the density function for an exponential distribution random variable:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/227/0*R7xrYVso497WLl0E.png" /></figure><p>Suppose you have a sample from a variable and want to see if it can be modelled with an Exponential distribution Variable.</p><p>The optimum <strong>λ parameter can be easily estimated</strong> as the inverse of the average of your sampled values.</p><p>Exponential variables are very good for modelling any probability distributions with very infrequent, but huge (and mean-breaking) <strong>outliers</strong>.</p><p>This is because they can <strong>take any non-negative value</strong> but center in smaller ones, with decreased frequency as the value grows.</p><p>In a particularly <strong>outlier-heavy sample</strong>, you may want to estimate λ as the <strong>median instead of the average</strong>, since the median is more <strong>robust to outliers</strong>. Your mileage may vary on this one, so take it with a grain of salt.</p><h3>Conclusions</h3><p>To sum up, as Data Scientists, I think it’s important for us to learn the basics.</p><p>Probability and Statistics may not be as flashy as <a href="http://www.datastuff.tech/machine-learning/autoencoder-deep-learning-tensorflow-eager-api-keras/">Deep Learning</a> or <a href="http://www.datastuff.tech/machine-learning/k-means-clustering-unsupervised-learning-for-recommender-systems/">Unsupervised Machine Learning</a>, but they are the <strong>bedrock of Data Science</strong>. Especially Machine Learning.</p><p>Feeding a Machine Learning model with features without knowing which distribution they follow is, in my experience, a poor choice.</p><p>It’s also good to remember the <strong>ubiquity of Exponential and Normal Probability Distributions</strong>, and their smaller counterpart, the lognormal distribution.</p><p>Knowing their properties, uses and appearance is <strong>game-changing when training a Machine Learning model</strong>. It’s also generally good to keep them in mind while doing any kind of Data Analysis.</p><p>Did you find any part of this article useful? Was it all stuff you already knew? did you learn anything new? Let me know in the comments!</p><p><em>Contact me on </em><a href="http://www.twitter.com/strikingloo"><em>Twitter</em></a><em>, </em><a href="http://www.medium.com/@strikingloo"><em>Medium</em></a><em> of </em><a href="http://www.dev.to/strikingloo"><em>dev.to</em></a><em> if there’s anything you don’t think was clear enough, anything that you disagree with, or just anything that’s plain wrong. Don’t worry, I don’t bite.</em></p><p><em>Originally published at </em><a href="http://www.datastuff.tech/data-science/5-probability-distributions-every-data-scientist-should-know/"><em>http://www.datastuff.tech</em></a><em> on June 17, 2019.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e0a850c19157" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/probability-distributions-every-data-scientist-should-know-e0a850c19157">Probability Distributions Every Data Scientist Should Know</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Convolutional Neural Networks: an Introduction (TensorFlow Eager API)]]></title>
            <link>https://medium.com/data-science/convolutional-neural-networks-an-introduction-tensorflow-eager-api-7e99614a2879?source=rss-56f7cdc8c678------2</link>
            <guid isPermaLink="false">https://medium.com/p/7e99614a2879</guid>
            <category><![CDATA[image-classification]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[tensorflow]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Luciano Strika]]></dc:creator>
            <pubDate>Wed, 12 Jun 2019 18:05:35 GMT</pubDate>
            <atom:updated>2019-06-14T01:48:44.171Z</atom:updated>
            <content:encoded><![CDATA[<h3>How to Train Convolutional Neural Networks in Python (TensorFlow Eager API)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tBjPkUtIyepvIbl1ZRgyOQ.jpeg" /><figcaption>Keep an eye out for Deep Learning. Source: <a href="https://pixabay.com/photos/eye-iris-look-focus-green-1132531/">Pixabay</a>.</figcaption></figure><p>Convolutional Neural Networks are a part of what made Deep Learning reach the headlines so often in the last decade. Today we’ll train an <strong>image classifier</strong> to tell us whether an image contains a dog or a cat, using TensorFlow’s eager API.</p><p>Artificial Neural Networks have disrupted several industries lately, due to their unprecedented capabilities in many areas. However, Different Deep Learning architectures excel on each one:</p><ul><li>Image Classification (Convolutional Neural Networks).</li><li>Image, audio and text generation (GANs, RNNs).</li><li>Time Series Forecasting (RNNs, LSTM).</li><li>Recommendations Systems.</li><li>A huge et cetera (e.g., regression).</li></ul><p>Today we’ll focus on the first item of the list, though each of those deserves an article of its own<em>.</em></p><h3>What are Convolutional Neural Networks?</h3><p>In MultiLayer Perceptrons (MLP), the <em>vanilla</em> Neural Networks, each layer’s neurons connect to <strong>all</strong> the neurons in the next layer. We call this type of layers <strong>fully connected</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*YpFy4XNcRdH0Dhow.png" /><figcaption>A MLP. Source: <a href="http://www.astroml.org/book_figures/appendix/fig_neural_network.html">astroml</a></figcaption></figure><p>A Convolutional Neural Network is different: they have Convolutional Layers.</p><p>On a fully connected layer, each neuron’s output will be a linear transformation of the previous layer, composed with a non-linear activation function (e.g., <em>ReLu</em> or <em>Sigmoid</em>).</p><p>Conversely, the output of each neuron in a <strong>Convolutional Layer</strong> is only a function of a (typically small) <strong>subset</strong> of the previous layer’s neurons.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1022/0*ulpGWgOsrF74H5KU" /><figcaption>Source: <a href="https://brilliant.org/wiki/convolutional-neural-network/">Brilliant</a></figcaption></figure><p>Outputs on a Convolutional Layer will be the result of applying a <strong>convolution</strong> to a subset of the previous layer’s neurons, and then an activation function.</p><h3>What is a convolution?</h3><p>The convolution operation, given an input matrix <em>A</em> (usually the previous layer’s values) and a (typically much smaller) weight matrix called a <strong>kernel</strong> or <strong>filter</strong> <em>K</em>, will output a new matrix <em>B</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*4yv0yIH0nVhSOv3AkLUIiw.png" /><figcaption>by @<a href="https://medium.com/@RaghavPrabhu/understanding-of-convolutional-neural-network-cnn-deep-learning-99760835f148">RaghavPrabhu</a></figcaption></figure><p>If <em>K</em> is a <em>CxC</em> matrix, the first element in <em>B</em> will be the result of:</p><ul><li>Taking the first <em>CxC</em> submatrix of <em>A</em>.</li><li>Multiplying each of its elements by its corresponding weight in <em>K</em>.</li><li>Adding all the products.</li></ul><p>These two last steps are equivalent to flattening both <em>A</em>’s submatrix and <em>K</em>, and computing the dot product of the resulting vectors.</p><p>We then slide K to the right to get the next element, and so on, repeating this process for each of <em>A’</em>s rows.</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*MrGSULUtkXc0Ou07QouV8A.gif" /><figcaption>Convolution visualization by @<a href="https://medium.com/@RaghavPrabhu/understanding-of-convolutional-neural-network-cnn-deep-learning-99760835f148">RaghavPrabhu</a></figcaption></figure><p>Depending on what we want, we could only start with our kernel centered at the <em>Cth</em> row and column, to avoid “going out of bounds”, or assume all elements “outside A” have a certain default value (typically 0) -This will define whether <em>B</em>’s size is smaller than <em>A</em> ‘s or the same.</p><p>As you can see, if <em>A</em> was an <em>NxM</em> matrix, now each neuron’s value in<em> B</em> won’t depend on <em>N*M</em> weights, but only on <em>C*C</em> (much less) of them. This makes a convolutional layer much lighter than a fully connected one, helping convolutional models learn a lot faster.</p><p>Granted, we will end up using many kernels on each layer (getting a stack of matrices as each layer’s output). However, that will still require a lot less weights than our good old MLP.</p><h3>Why does this work?</h3><p>Why can we <strong>ignore</strong> how each neuron affects most of the others? Well, this whole system holds up on the premise that each neuron is <strong>strongly affected by its “neighbors”</strong>. Faraway neurons, however, have only a small bearing on it.</p><p>This assumption is <strong>intuitively true in images</strong> -if we think of the input layer, each neuron will be a pixel or a pixel’s RGB value. And that’s part of the reason why this approach works so well for image classification.</p><p>For example, if I take a region of a picture where there’s a blue sky, it’s likely that nearby regions will show the sky as well, using similar tones.</p><p>A pixel’s neighbors will usually have similar RGB values to it. If they don’t, then that probably means we are on the edge of a figure or object.</p><p>If you do some convolutions with pen and paper (or a calculator), you’ll realize certain kernels will increase an input’s intensity if it’s on a certain kind of edge. In other edges, they could decrease it.</p><p>As an example, let’s consider the following kernels <em>V</em> and <em>H</em>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/297/0*pyY6Yqi5xyzpkmYC.png" /></figure><p><em>V</em> filters vertical edges (where colors above are very different from colors below), whereas <em>H</em> filters horizontal edges. Notice how one is the transposed version of the other.</p><h3>Convolutions by example</h3><p>Here’s an unfiltered picture of a litter of kittens:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/0*8dZsX54Gpku0XtJn.jpg" /></figure><p>Here’s what happens if we apply the horizontal and vertical edge filters, respectively:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/530/0*2u3YVp5qkTaBwtPJ.png" /></figure><p>We can see how some features become a lot more noticeable, whereas others fade away. Interestingly, each filter showcases different features.</p><p>This is how Convolutional Neural Networks learn to identify features in an image. Letting it fit its own kernel weights is a lot easier than any manual approach. Imagine trying to figure out how you should express the relationship between pixels… <em>by hand</em>!</p><p>To really grasp what each convolution does to a picture, I strongly recommend you play around on <a href="http://setosa.io/ev/image-kernels/">this website</a>. It helped me more than any book or tutorial could. Go ahead, bookmark it. It’s fun.</p><p>Alright, you’ve learned some theory already. Now let’s move on to the practical part.</p><h3>How do you train a Convolutional Neural Network in TensorFlow?</h3><p>TensorFlow is Python’s most popular Deep Learning framework. I’ve heard good things about PyTorch too, though I’ve never had the chance to try it.</p><p>I’ve already written one tutorial on <a href="http://www.datastuff.tech/machine-learning/autoencoder-deep-learning-tensorflow-eager-api-keras/">how to train a Neural Network with TensorFlow’s Eager API</a>, focusing on AutoEncoders.</p><p>Today will be different: we will try three different architectures, and see which one does better. As usual, all the code is available on <a href="https://github.com/StrikingLoo/Cats-and-dogs-classifier-tensorflow-CNN">GitHub</a>, so you can try everything out for yourself or follow along. Of course I’ll also be showing you Python snippets.</p><h3>The Dataset</h3><p>We will be training a neural network to predict whether an image contains a dog or a cat. To do this we’ll use Kaggle’s <a href="https://www.kaggle.com/c/dogs-vs-cats">cats and dogs Dataset</a>. It contains 12500 pictures of cats and 12500 of dogs, with different resolutions.</p><h3>Loading and Preprocessing our Image Data with NumPy</h3><p>A neural network receives a features vector or matrix as an input, typically with <strong>fixed dimensions</strong>. How do we generate that from our pictures?</p><p>Lucky for us, Python’s Image library provides us an easy way to load an image as a NumPy array. A HeightxWidth matrix of RGB values.<br>We already did that on <a href="http://www.datastuff.tech/machine-learning/k-means-clustering-with-dask-editing-pictures-of-kittens/">this article</a>, so I’ll just reuse that code.</p><p>However we still have to fix the fixed dimensions part: which dimensions do we choose for our input layer? This is important, since we will have to resize every picture to the chosen resolution. We do not want to distort aspect ratios too much in case it brings too much noise for the network.</p><p>Here’s how we can see what the most common shape is in our dataset.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/bb9647a1887dbdcabcad05df05a55578/href">https://medium.com/media/bb9647a1887dbdcabcad05df05a55578/href</a></iframe><p>I sampled the first 1000 pictures for this, though the result did not change when I looked at 5000. The most common shape was 375×500, though I decided to divide that by 4 for our network’s input.</p><p>This is what our image loading code looks like now.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/05b3f398feb5645199b0e93fcd5346c3/href">https://medium.com/media/05b3f398feb5645199b0e93fcd5346c3/href</a></iframe><p>Finally, you can load the data with this snippet. I chose to use a sample of 4096 pictures for the training set and 1024 for validation. However, that’s just because my PC couldn’t handle much more due to RAM size.</p><p>Feel free to increase these numbers to the max (like 10K for training and 2500 for validation) if you try this at home!</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/07a221c185e516a15ab9e4668778f143/href">https://medium.com/media/07a221c185e516a15ab9e4668778f143/href</a></iframe><h3>Training our Neural Networks</h3><p>First of all, as a sort of baseline, let’s see how good a normal <strong>MLP</strong> does on this task. If Convolutional Neural Networks are so revolutionary, I’d expect the results to be <strong>terrible</strong> for this experiment.</p><p>So here’s a single hidden layer fully connected neural network.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5e0526d87ac6c908d3652137ca2c2a4c/href">https://medium.com/media/5e0526d87ac6c908d3652137ca2c2a4c/href</a></iframe><p>All the trainings for this article were made using AdamOptimizer, since it’s the fastest one. I only tuned the learning rate per model (here it was 1e-5).</p><p>I trained this model for 10 epochs, and it basically converged to <strong>random guessing</strong>. I made sure to <strong>shuffle the training data</strong>, since I loaded it in order and that could’ve biased the model.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6f4f9e6718efdff04aeb1f491a112581/href">https://medium.com/media/6f4f9e6718efdff04aeb1f491a112581/href</a></iframe><p>I used <strong>MSE</strong> as loss function, since it’s usually <strong>more intuitive to interpret</strong>. If your MSE is 0.5 in binary classification, you’re as good as <strong>always predicting 0</strong>. However, MLPs with more layers, or different loss functions <strong>did not perform better</strong>.</p><h3>Training a Convolutional Neural Network</h3><p>How much good can a single convolutional layer do? Let’s add one to our model and see.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9bb8831a2c5269b79a0886c1eac81506/href">https://medium.com/media/9bb8831a2c5269b79a0886c1eac81506/href</a></iframe><p>For this network, I decided to add a single convolutional layer (with 24 kernels), followed by 2 fully connected layers.</p><p>All Max Pooling does is reduce every four neurons to a single one, with the highest value between the four.</p><p>After only 5 epochs, it was already <strong>performing much better</strong> than the previous networks. With a validation MSE of 0.36, it was a lot better than random guessing already. Notice however that I had to use a <strong>much smaller learning rate</strong>. Also, even though it learned in less epochs, <strong>each epoch</strong> took <strong>much longer</strong>. The model is also quite a lot heavier (200+ MB).</p><p>I decided to also start measuring the Pearson correlation between predictions and validation labels. This model scored a 15.2%.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d5338c56d528d6edcb28ae680e351d48/href">https://medium.com/media/d5338c56d528d6edcb28ae680e351d48/href</a></iframe><h3>Neural Network with two Convolutional Layers</h3><p>Since that model had done so much better, I decided I would try out a bigger one. I added <strong>another convolutional layer</strong>, and made both a lot bigger (48 kernels each). This means the model gets to learn <strong>more complex features</strong> from the images. However it also predictably meant my RAM almost exploded. Also, training took <strong>a lot longer</strong> (half an hour for 15 epochs).</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f368fb0e89978f7906b00aca8495f1d1/href">https://medium.com/media/f368fb0e89978f7906b00aca8495f1d1/href</a></iframe><p>Results were superb. The Pearson correlation coefficient between predictions and labels reached 0.21, with validation MSE reaching as low as 0.33.</p><p>Let’s measure the network’s accuracy. Since 1 is a cat and 0 is a dog, I could say “If the model predicts a value higher than some threshold <em>t</em>, then predict <em>cat</em>. Else predict <em>dog</em>.” After trying 10 straightforward thresholds, this network had a <strong>maximum accuracy of 61%</strong>.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/436c059701032983a11e79200017d64a/href">https://medium.com/media/436c059701032983a11e79200017d64a/href</a></iframe><h3>Even bigger Convolutional Neural Network</h3><p>Since clearly adding size to the model made it learn better, I tried making both convolutional layers a lot bigger, with 128 filters each. I left the rest of the model untouched, and didn’t change the learning rate.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/818be901efea4a06dca9656a6ab46b09/href">https://medium.com/media/818be901efea4a06dca9656a6ab46b09/href</a></iframe><p>This model finally reached a correlation of 30%! Its best <strong>accuracy was 67%</strong>, which means it was right two thirds of the time. I assume an even bigger model could’ve fit the data even better. However, this one was taking 7 minutes per epoch already, and I didn’t want to leave the next one training all morning.</p><p>Usually, there’s a <strong>tradeoff</strong> to be made between a model’s <strong>size</strong>, and <strong>time constraints</strong>. Size limits how well the network can fit the data (a <strong>small model</strong> will <strong>underfit</strong>), however I won’t wait 3 hours for my model to learn.</p><p>The same concerns may apply if you have a business deadline.</p><h3>Conclusions</h3><p>We’ve seen Convolutional Neural Networks are <strong>significantly better</strong> than vanilla architectures at <strong>image classification</strong> tasks. We also tried different <strong>metrics</strong> to measure <strong>model performance</strong> (correlation, accuracy).</p><p>We learned about the <strong>tradeoff</strong> between a <strong>model’s size</strong> (which prevents underfitting) and its <strong>convergence speed</strong>.</p><p>Lastly, we used TensorFlow’s eager API to easily train a Deep Neural Network, and numpy for (albeit simple) image preprocessing.</p><p>For future articles, I believe we could experiment a lot more with different pooling layers, filter sizes, striding and a different preprocessing for this same task.</p><p>Did you find this article useful? Would you have preferred to learn more about anything else? Is anything not clear enough? Let me know in the comments!</p><p><em>Find me on </em><a href="http://www.twitter.com/strikingloo"><em>Twitter</em></a><em>, </em><a href="http://www.medium.com/@strikingloo"><em>Medium</em></a><em> or </em><a href="http://www.dev.to/strikingloo"><em>Dev.to</em></a><em> if you have any questions, or want to contact me for anything. If you want to start a career in Machine Learning, here’s my </em><a href="http://www.datastuff.tech/data-science/3-machine-learning-books-that-helped-me-level-up-as-a-data-scientist/"><em>recommended reading list</em></a><em>.</em></p><p><em>Originally published at </em><a href="http://www.datastuff.tech/machine-learning/convolutional-neural-networks-an-introduction-tensorflow-eager/"><em>http://www.datastuff.tech</em></a><em> on June 12, 2019.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7e99614a2879" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/convolutional-neural-networks-an-introduction-tensorflow-eager-api-7e99614a2879">Convolutional Neural Networks: an Introduction (TensorFlow Eager API)</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[K Means Clustering with Dask (Image Filters for Cat Pictures)]]></title>
            <link>https://medium.com/data-science/k-means-clustering-with-dask-image-filters-for-cat-pictures-78284a3e83f6?source=rss-56f7cdc8c678------2</link>
            <guid isPermaLink="false">https://medium.com/p/78284a3e83f6</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[unsupervised-learning]]></category>
            <category><![CDATA[dask]]></category>
            <dc:creator><![CDATA[Luciano Strika]]></dc:creator>
            <pubDate>Sun, 02 Jun 2019 19:14:19 GMT</pubDate>
            <atom:updated>2019-06-02T23:41:15.202Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/808/0*8K3b78cbeINWTEE7.jpg" /><figcaption>300+ colors to 24 colors.</figcaption></figure><p>Applying filters to images is not a new concept to anyone. We take a picture, make a few changes to it, and now it looks cooler. But where does Artificial Intelligence come in? Let’s try out a fun use for <strong>Unsupervised Machine Learning</strong> with K Means Clustering in Python.</p><p>I’ve written before about <strong>K Means Clustering</strong>, so I will assume you’re <strong>familiar</strong> with the algorithm this time. If you’re not, this is the <a href="http://www.datastuff.tech/machine-learning/k-means-clustering-unsupervised-learning-for-recommender-systems/">in-depth introduction I wrote</a>.</p><p>And I also tried my hand at <strong>image compression</strong> (well, reconstruction) with <a href="http://www.datastuff.tech/machine-learning/autoencoder-deep-learning-tensorflow-eager-api-keras/">autoencoders</a>, to varying degrees of success.</p><p>However this time, my goal is <strong>not to reconstruct</strong> the best possible image, but just to see the effects of <strong>recreating a picture</strong> with the least possible colors.</p><p>Instead of making the picture look as similar to the original as possible, I just want us to look at it and say “neat!”.</p><p>So how do we do this? I’m glad you asked.</p><h3>How to do image filters with K Means Clustering</h3><p>First of all, it’s always good to remember an <strong>image</strong> is just a <strong>vector of pixels</strong>. Each pixel is a tuple of three integer values between 0 and 255 (an unsigned byte), which represent that pixel’s color’s RGB values.</p><p>We want to use K Means clustering to find the<em> k </em><strong>colors</strong> that best <strong>characterize an image</strong>. That just means we could treat each <strong>pixel</strong> as a <strong>single data point</strong> (in 3-dimensional space), and cluster them.</p><p>So first, we’ll want to turn an image into a<strong> vector of pixels</strong> in Python. Here’s how we do it.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/94393f3fa90f07291734a1f8901600d3/href">https://medium.com/media/94393f3fa90f07291734a1f8901600d3/href</a></iframe><p>As an aside, I don’t think the <em>vector_of_pixels</em> function needs to use a Python list. I’m sure there has to be some way to <strong>flatten a numpy array</strong>, I just couldn’t find any (at least not one that did it in the order I wanted). <br>If you can think of any way, let me know in the comments!</p><p>The next step is <strong>fitting</strong> the model to the image, so that it <strong>clusters the pixels</strong> into <em>k </em>colors. Then, it’s just a matter of assigning the corresponding cluster color to each position in the image.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/189da3cd533394d99e3d5c3773410ccb/href">https://medium.com/media/189da3cd533394d99e3d5c3773410ccb/href</a></iframe><p>For instance, maybe our pic has only three colors: two reddish ones and a greenish one. If we fit that to 2 clusters, all the reddish pixels would turn some different shade of red (getting clustered together), and the other ones would turn into some greenish one.</p><p>But enough with the explanations, let’s see the program in action! <br>As usual, you are free to run it yourself with any pic you want, here’s the <a href="https://github.com/StrikingLoo/K-means-image-compression">GitHub repository</a> with the code.</p><h3>The results</h3><p>We will apply the filter to pictures of kittens, taken from the awesome “Cats vs Dogs” <a href="https://www.kaggle.com/c/dogs-vs-cats">kaggle dataset</a>.</p><p>We’ll start with a picture of a cat, and apply the filter with different values for <em>k</em>. Here’s the original picture:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/499/0*G0bJLlaIqeUJbjNJ.jpg" /></figure><p>First, let’s check how many colors this picture originally had.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/94ae7aedcf26cab18a7e438ef8d25403/href">https://medium.com/media/94ae7aedcf26cab18a7e438ef8d25403/href</a></iframe><p>With just one line of <em>numpy</em>, we count the unique values a pixel takes on this picture. This image in particular has <strong>243 different colors</strong>, even though it has a total of <strong>166167 pixels</strong>.</p><p>Now, let’s see the result of clustering it to 2, 5 and 10 different colors only.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/499/0*w0-OFLttEBVoy1HP.jpg" /></figure><p>With only two colors, all it’s doing is marking the darkest and lightest areas. This could however be useful if you’re an artist, drawing something in black and white (with ink, for instance) and would like to see what your reference’s outlines look like.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/499/0*mWZz6bBSrwRFOaxM.jpg" /><figcaption>With only 5 different colors, the cat is already recognizable!</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/499/0*WcUtubwQV7DIEYnk.jpg" /><figcaption>The 10 colors picture may look a bit trippy, but it represents the original’s content pretty clearly.</figcaption></figure><p>Did you notice a trend? Each color we add has diminishing returns. The difference between having 2 colors and having 5, is a lot more than the difference between 5 and 10. However with 10 colors, the flat areas are smaller, and we have more granularity. Moving on to 15 and 24 colors!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/499/0*W5CdmoudtEFO4zb6.jpg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/499/0*pnzi9i0Oo0u92KEV.jpg" /></figure><p>Though it’s clear this is a filtered picture, with 24 colors (10% of the original amount) we are representing the cat well enough and to a certain level of detail.</p><p>Moving on to a different picture: Here’s the original (256 different colors) and here’s a compressed one (24 colors again).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/760/0*adxO8bwRiFxTIsX0.jpg" /></figure><p>As an interesting note, the “compressed” image weighs 18KB and the uncompressed one 16KB. I don’t really know why this is, since compressors are pretty complicated beasts, but would love to read your theories in the comments.</p><h3>Conclusions</h3><p>We were able to make new images with only 10% of the original’s colors, which looked very similar to them. We also got some cool looking filters thanks to K means clustering. Can you think of any other fun application for clustering? Do you think other clustering techniques could have yielded more interesting results?</p><p>If you want to answer any of these questions, feel free to contact me on <a href="http://twitter.com/strikingloo">Twitter</a>, <a href="http://medium.com/@strikingloo">Medium</a> or <a href="http://www.dev.to/strikingloo">Dev.to</a>.</p><p>Are you interested in starting a career in Data Science? Do you want to be an awesome Machine Learning professional? Check out my recommended reading list: <a href="http://www.datastuff.tech/data-science/3-machine-learning-books-that-helped-me-level-up-as-a-data-scientist/">“3 Machine Learning Books that will Help You Level Up as a Data Scientist”</a>. One of them actually taught me what I know about K Means Clustering.</p><p><em>Originally published at </em><a href="http://www.datastuff.tech/machine-learning/k-means-clustering-with-dask-editing-pictures-of-kittens/"><em>http://www.datastuff.tech</em></a><em> on June 2, 2019.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=78284a3e83f6" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/k-means-clustering-with-dask-image-filters-for-cat-pictures-78284a3e83f6">K Means Clustering with Dask (Image Filters for Cat Pictures)</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[3 Machine Learning Books that Helped me Level Up]]></title>
            <link>https://medium.com/data-science/3-machine-learning-books-that-helped-me-level-up-a95764c458e3?source=rss-56f7cdc8c678------2</link>
            <guid isPermaLink="false">https://medium.com/p/a95764c458e3</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[book-review]]></category>
            <dc:creator><![CDATA[Luciano Strika]]></dc:creator>
            <pubDate>Sun, 28 Apr 2019 06:57:44 GMT</pubDate>
            <atom:updated>2022-08-12T16:39:37.660Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Hl0uiyswFpgE9sgt.jpg" /><figcaption>Source: <a href="https://pixabay.com/photos/book-read-old-literature-pages-1659717/">Pixabay</a></figcaption></figure><p>There is a Japanese word, <em>tsundoku </em>(積ん読), which means buying and keeping a growing collection of books, even though you don’t really read them all.</p><p>I think we Developers and Data Scientists are particularly prone to falling into this trap. Personally, I even hoard bookmarks: my phone’s Chrome browser has so many open tabs, the counter was replaced with a “:D” emoji.</p><p>In that zeal for <strong>reading</strong> and <strong>learning</strong> most of us experience, we usually end up <strong>lost</strong>, not sure of <strong>which book to pick up next</strong>. That’s why today I’ll give you a very <strong>short list</strong>: just 3 Machine Learning books, so that you won’t just bookmark it and forget it.</p><p>Each of these books has <strong>helped me immensely</strong> in different stages of my <strong>career</strong> as a Data Scientist, particularly in my role as a Machine Learning Engineer.</p><p>Here come the books!</p><h3>O’Reilly: Data Science from Scratch with Python</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/381/1*k5BWrgTDU9PBcEzm4iayxw.jpeg" /></figure><p>I have a very personal attachment to this <a href="https://amzn.to/2DDhIUH">book</a>, since it’s the one that <strong>got me my job</strong>. That’s right! I knew next to nothing about Data Science, even what Data Science was, before picking up this book.</p><p>I did have a pretty strong Probability and Statistics background, and knew enough Python to defend myself. However, I was missing the <strong>practical side</strong> of it.</p><p>This book did many things for me. It:</p><ul><li>Showed me how to <strong>process data in Python</strong> efficiently and elegantly (following Python’s<strong> good practices</strong>).</li><li>Taught me how to <strong>implement</strong> most simple <strong>Machine Learning algorithms</strong> from scratch.</li><li>Showed me what the <strong>day-to-day job</strong> of a Data Scientist may look like.</li><li>Taught me how to <strong>communicate my results</strong> to others clearly.</li></ul><p>I wholeheartedly recommend it if you’re <strong>new to the Data Science community</strong>. It will give you a clear overview of most topics you’ll need in order to start being a productive Data Scientist.</p><p>It will also showcase Python’s most commonly used libraries and expose you to a lot of <strong>idiomatic code</strong>, which is always a plus.</p><p>Here’s a link to <a href="https://amzn.to/2DDhIUH"><em>Data Science from Scratch</em></a> on Amazon.</p><h3>Springer: Introduction to Statistical Learning</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/334/1*qpExo1SV-PG5cildoOk9Dw.jpeg" /></figure><p><a href="https://amzn.to/2ZP93YS">This book</a> is the most comprehensive <strong>Machine Learning book</strong> I’ve found so far. I learned a lot from it, from <strong>Unsupervised Learning</strong> algorithms like <a href="http://www.datastuff.tech/machine-learning/k-means-clustering-unsupervised-learning-for-recommender-systems/">K-Means Clustering</a>, to <strong>Supervised Learning</strong> ones like <a href="http://www.datastuff.tech/machine-learning/xgboost-predicting-life-expectancy-with-supervised-learning/">Boosted Trees</a>.</p><p>The first chapters may feel a bit too introductory if you’re already working in this field (at least that was my experience). However, they also sum up many things you may not have learned in such an organized way before.</p><p>The later chapters are, however, where I think this book really shines. Its explanation of random forests, boosted trees and support vector machines are spot on.</p><p>Here are some of the topics you can learn from <em>Introduction to Statistical Learning</em>:</p><ul><li><strong>Regression </strong>and<strong> Supervised Learning </strong>Algorithms: from <strong>Linear Regression</strong> and <strong>SVM</strong>s to <strong>tree-based</strong> methods.</li><li><strong>Unsupervised Learning</strong> techniques: especially Clustering, including the K-Means algorithm.</li><li>Sampling methods, and other general Machine Learning <strong>core concepts</strong>.</li><li>The meaning, advantages and disadvantages of <strong>metrics</strong> such as <strong>accuracy, recall, precision</strong>, etc.</li></ul><p>I think this book has been my best read so far this year, and it’s made me into a more round up Data Scientist. I recommend it if you have a bit more experience, but want to <strong>polish your edges.</strong> It is also a very good <strong>reference</strong> book to keep on your shelf.</p><p>It also shows everything’s implementation in R, which I didn’t find particularly useful, but it didn’t hurt. You’ll probably import most of this code from SciKit learn anyway.</p><p>As before, here’s a link to <a href="https://amzn.to/2ZP93YS">Springer’s Introduction to Statistical Learning</a> on Amazon.</p><h3>Deep Learning by Goodfellow, Bengio et al.</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/375/1*EqVW5k_ZSDlAxvkZgRDYlw.jpeg" /></figure><p><a href="https://amzn.to/2J0WjZe">This book</a> blows my mind every time I open it. I’ll be the first to admit I haven’t really read it from start to finish. Yet. <br>The only reason it’s the last one in the list is because of its very <strong>specific scope</strong>: Artificial <strong>Neural Networks</strong> or <strong>Deep Learning</strong>.</p><p>However its first chapters, with an overview of Deep Learning’s <strong>precursors</strong> and what makes it different, and then the explanation of <strong>how Deep Learning works</strong>, are marvelous.</p><p>It even starts off by explaining everything you need to know <strong>before studying deep learning</strong>, with whole chapters dedicated to <strong>linear algebra</strong>, <strong>probability</strong> and <strong>information theory</strong>, and <strong>numerical computation</strong> methods.</p><p>The next chapters, which I’ve only partially read, serve as an awesome <strong>reference</strong> whenever you need to dive deeper into a particular Neural Network <strong>architecture</strong>.</p><p>They include in-depth explanations of <strong>Convolutional</strong> Neural Networks and <strong>Recurrent</strong> Neural Networks, along with many <strong>regularization</strong> or optimization methods.</p><p>The third and last section, which revolves around <strong>cutting-edge technology</strong>, explains <strong>Generative models</strong>, <a href="http://www.datastuff.tech/machine-learning/autoencoder-deep-learning-tensorflow-eager-api-keras/">Autoencoders</a> and many other interesting algorithms. Adding them to your own toolkit will probably give you a great boost!</p><p>The authors of this book are the <strong>rock stars</strong> of Machine Learning right now. One of them even <a href="https://venturebeat.com/2019/03/27/geoffrey-hinton-yann-lecun-and-yoshua-bengio-honored-with-the-turing-award/">won a Turing award</a> recently, so I can’t think of better people to teach this subject.</p><p>Here’s an <a href="https://amzn.to/2J0WjZe">Amazon link</a> if you’re interested in the <em>Deep Learning</em> book.</p><h3>Conclusion</h3><p>I went from a broad, introductory book to an advanced, specific one.</p><p>Each of these Machine Learning books has had a profound impact in my career and, to some degree, the way I see the world.</p><p>I really hope at least some of them will have the same positive impact on your life!</p><p>And if you’ve already read, or are reading, any of them, tell me what you think of them in the comments! I’d love to discuss any of them further, especially the Deep Learning book.</p><p>We can also discuss them on <a href="http://twitter.com/strikingloo">Twitter</a>, <a href="http://medium.com/@strikingloo">Medium</a> of <a href="http://dev.to/strikingloo">dev.to</a> if you’re interested.<br>I want to hear your opinions!</p><p><em>(small disclaimer: all of these links are Amazon affiliate links, which means I get a small commission if you buy the books. However, I’ll only review books I’ve actually read, and have genuinely recommended to people in real life)</em></p><p><em>Originally published at </em><a href="https://strikingloo.github.io/3-ml-books"><em>https://strikingloo.github.io</em></a><em> on April 28, 2019.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a95764c458e3" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/3-machine-learning-books-that-helped-me-level-up-a95764c458e3">3 Machine Learning Books that Helped me Level Up</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Autoencoders: Deep Learning with TensorFlow’s Eager API]]></title>
            <link>https://medium.com/data-science/autoencoders-deep-learning-with-tensorflows-eager-api-data-stuff-378318784ae?source=rss-56f7cdc8c678------2</link>
            <guid isPermaLink="false">https://medium.com/p/378318784ae</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[tensorflow]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Luciano Strika]]></dc:creator>
            <pubDate>Sat, 20 Apr 2019 06:02:21 GMT</pubDate>
            <atom:updated>2019-05-30T18:54:50.444Z</atom:updated>
            <content:encoded><![CDATA[<h3>Autoencoders: Deep Learning with TensorFlow’s Eager Execution</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*8j2rHzY4oiP9ZSxd.jpg" /><figcaption>We are so deep. Source: <a href="https://pixabay.com/photos/coral-cay-aquarium-sea-reef-water-1053834/">Pixabay</a>.</figcaption></figure><p>Deep Learning has revolutionized the Machine Learning scene in the last years. Can we apply it to image compression? How well can a Deep Learning algorithm reconstruct pictures of kittens? What’s an autoencoder?</p><p>Today we’ll find the answers to all of those questions.</p><h3>Image Compression: all about the patterns</h3><p>I’ve talked about <a href="http://www.datastuff.tech/machine-learning/k-means-clustering-unsupervised-learning-for-recommender-systems/">Unsupervised Learning</a> before: applying Machine Learning to discover <strong>patterns</strong> in <strong>unlabelled data</strong>.</p><p>In the case of Image Compression, it makes a lot of sense to assume most images are <strong>not completely random</strong>.</p><p>In more proper words, it is safe to assume most images are not completely made of <strong>noise</strong> (like the static when you turn on an old TV), but rather follow <em>some</em> <strong>underlying structure</strong>.</p><p>It would be even better if we knew all the images will have things in common. One of the most rudimentary ways of doing image compression is finding repeating streams of pixels, assigning them a (lighter) label, and replacing them with that label.</p><p>Ideally, if two images are almost the same, a compressor could store both taking barely more space than either of them. That should apply to parts of the image as well. Finally, I would guess a pattern-finding machine like Unsupervised Learning should be good at this.</p><p>For the purposes of this work, we’ll be using a dataset of pictures of kittens I downloaded from <a href="https://www.kaggle.com/c/dogs-vs-cats">Kaggle</a>. That will guarantee there is some common structure to most pictures.</p><p>Here’s the <a href="https://github.com/StrikingLoo/AutoEncatter">GitHub project</a> with the code and data for today, in case you want to follow along. As usual however, I’ll be adding the most important snippets here.</p><p>But before we get our hands dirty with code, it’s time to learn some Deep Learning!</p><h3>Autoencoders: Unsupervised-ish Deep Learning</h3><p>Even though we call Autoencoders “Unsupervised Learning”, they’re actually a Supervised Learning Algorithm in disguise. I know, I was shocked too!</p><p>They do however have a very peculiar property, which makes them stand out from normal classifiers: <strong>their input and output are the same</strong>.</p><h3>Training an Autoencoder</h3><p>When we train an Autoencoder, we’ll actually be training an Artificial Neural Network that</p><ul><li>Takes an input vector <em>X</em>.</li><li>Applies some math to it (I won’t get into the specifics of Deep Learning right now, but <a href="https://amzn.to/2LU5uNR">this is the book</a> I used to learn these subjects).</li><li>Returns another vector. In this particular case, we want it to be X again, or as close as possible.</li></ul><p>However, there’s a trick: in the feed-forward process (going from input to output) the Neural Network will reduce the input’s dimensionality, and increase it again before returning the last layer.</p><h3>The curse of dimensionality</h3><p>The Autoencoder takes a vector <em>X</em> as input, with potentially a lot of components. For instance, for a 3 channels — <em>RGB</em>- picture with a 48×48 resolution, <em>X</em> would have 6912 components. Even if each of them is just a float, that’s 27Kb of data for each (very small!) image. That adds up fast. The time it takes to do all the mathematical operations on so many elements is also very relevant.</p><p>We will thus have to map that vector to a smaller space. In this particular case, I tried reducing it to half of its original dimensions. <br>We intend, however, to keep most of the data.</p><h3>Architecture of an Autoencoder</h3><p>We call the layer of the Network that reduces the input vector’s dimensions the <em>encoder</em>, because it <em>encodes</em> the data into a smaller vector.</p><p>We then operate on that smaller vector, in our case reducing its dimensions by a factor of 2 again. This would be the version of the input that we would store if we were designing an actual image compressor, saving 75% space. However during training, that is just an intermediate step before <em>decoding</em> it.</p><p>Finally, using that reduced vector, the Autoencoder will have to reconstruct the original image as well as it can.</p><p>In order to do this, it will first make it go through a <em>decoder</em> layer, which again maps the reduced vector to one with half of the input vector’s dimensions. And then a final layer, where we obtain a vector of the same dimensions as the original image, hopefully very similar.</p><p>Here’s what an Autoencoder’s layer architecture looks like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/361/1*bm6EaErSPQDETh7gGyTA5A.png" /><figcaption>Source: Medium</figcaption></figure><h3>Results of using an Autoencoder</h3><p>Training an Autoencoder will have many effects:</p><ul><li>The reconstructed images won’t be exactly similar to the original ones.</li><li>We’ll be able to save the much lighter reduced vector, and then use the reconstruction, provided it’s good enough (however we define that), instead of the original image.</li><li>We’ll now have a Neural Network that has learned some of the underlying patterns in pictures of kittens.</li></ul><p>This last one is the most valuable result.</p><p>Some of the industry applications for Autoencoders are:</p><ul><li>Feature Engineering for Supervised Deep Learning models: instead of feeding them with data directly, we’ll feed them with the Autoencoder’s hidden layer.</li><li>Anomaly Detection: The Autoencoder will be very bad at reconstructing pictures of dogs, landscapes or bugs. This gives us a way to check if a picture is effectively a kitten automatically.</li></ul><p>Now that you know why we’re doing what we’re doing, let’s get our hands dirty with some actual code!</p><h3>Training an Autoencoder with Tensorflow</h3><p>For this tutorial we’ll be using Tensorflow’s eager execution API. I have to say, it is a lot more intuitive than that old <em>Session</em> thing, so much so that I wouldn’t mind if there had been a drop in performance (which I didn’t perceive).</p><p>Our dataset contains over 8000 images of kittens, of various sizes and colors, and in different positions. It is available for you to look at at the <a href="https://github.com/StrikingLoo/AutoEncatter">GitHub project</a>, along with the code that loads it.</p><p>Due to PyPlot’s way of handling numpy float arrays, and to accelerate convergence for the network, the images are loaded as an array of floats ranging from 0 to 1, instead of 0 to 255.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/880b3c98c32c86fea7d0901f62428ebb/href">https://medium.com/media/880b3c98c32c86fea7d0901f62428ebb/href</a></iframe><h3>Designing the Autoencoder</h3><p>Before Tensorflow swallowed Keras and became eager, writing a Neural Network with it was quite cumbersome. Now, its API has become intuitive.</p><p>Here’s the first Autoencoder I designed using Tensorflow’s Keras API.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0a589fea2afa9b51570154059187be73/href">https://medium.com/media/0a589fea2afa9b51570154059187be73/href</a></iframe><p>However, this model had some major flaws! First of all, it was converging too slowly, even with a very high learning rate (we’ll get there). Secondly, it was returning a vector with the right size… except some of the elements were higher than 1! That didn’t mesh well with PyPlot’s graphs, nor with our inputs’ priors -each pixel has three values between 0 and 1-.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/222/0*VpvRTFAgBhd_ozBD.png" /></figure><p>I fixed that problem by replacing the ReLu on the last layer with a sigmoid activation function, so it always returned numbers between 0 and 1.</p><p>However, there was still the factor of convergence speed. What could I do to solve that? Luckily, I’ve read a few Deep Learning papers here and there, and there’s this thing I’ve used a few times called <strong>Batch Normalization</strong>. It deserves its own article, so I won’t be delving too deep on it.</p><h4>Batch Normalization</h4><p>The short version is, a batch normalization layer calculates the mean and standard deviation for the previous layer, for the current batch of training instances. It then subtracts the mean and divides by the standard deviation, thus normalizing the layer’s output (for the batch). In feed-forwards after the training, the layer will start normalizing using parameters estimated from the whole training dataset.</p><p>This makes the model converge a lot faster, since it becomes less sensitive to changes in the distribution of the inputs, or the hidden layers. It will also become more robust to natural distribution changes in the inputs, being more sustainable in the long term on a productive environment.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/35b434753cdda215dc747298a7d0abda/href">https://medium.com/media/35b434753cdda215dc747298a7d0abda/href</a></iframe><p>This way, the decoder just needs to learn how to decode centered variables with a variance of 1, and not just any random thing. The same thing applies to the encoder. This change alone made the model start converging in the first 5 or 6 epochs.</p><h3>Training the Autoencoder</h3><p>It’s the moment we were all waiting for! Time to train our Autoencoder!</p><p>Tensorflow’s eager execution requires we first compile the model. <br>This means we define an optimizer (I’m using Adam, it’s fast), a loss (in this case, mean squared error, which is a pretty standard way to measure reconstruction error), and monitoring metrics. Most importantly, here is where we will choose the model’s learning rate.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ba9e31470129349956c03ba5c401cf71/href">https://medium.com/media/ba9e31470129349956c03ba5c401cf71/href</a></iframe><p>And finally, we start training the model.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2732cd92137f9724ae6108f3f7086663/href">https://medium.com/media/2732cd92137f9724ae6108f3f7086663/href</a></iframe><p>This will begin the training (and make my laptop sound like a Victorian furnace) and print messages like this:</p><pre>Epoch 2/2 7373/7373 [==============================] - 59s 8ms/sample - loss: 0.0183 - mean_squared_error: 0.0183 - val_loss: 0.0350 - val_mean_squared_error: 0.0350</pre><p>We will know the model is learning if <em>val_mean_squared_error</em>, the MSE on the validation dataset, keeps converging towards 0. If the train loss keeps dropping but the validation loss is stable or, worse, going up, then that means that the model is overfitting. What do we do then? We can</p><ul><li>add more data,</li><li>make the compression smaller, or</li><li>increase the learning rate.</li></ul><p>This wasn’t a problem I had too often.</p><p>On the opposite side of the spectrum, the training loss may be bad. <br>How bad? Well, the value we’d get if we sampled random numbers between 0 and 1 would be around 25%, since that’s the square of 1/2, which is the average random error.</p><p>For most configurations I tried, that was the training loss for a long while (for instance, without using batch normalization, or when I sampled less than half of the data).</p><p>If that happens, we should decrease the learning rate (to avoid converging to a local minimum), increase the training dataset size, or be less ambitious and compress the data less.</p><p>I first tried compressing to a ninth of the original weight instead of a fourth, and there was no way the network learned that.</p><h3>The results</h3><p>And finally, here are some of the results!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/222/0*VPsNd1mMO-8rFcPR.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/222/0*dlINFpijWfCN7PLk.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/222/0*VBKj-uy0NbH1yvV3.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/222/0*H4YN-UTk4NYHpXdc.png" /></figure><p>The model was particularly good with black or white cats, probably because they had the most contrast, and most extreme colors.</p><p>This is what happened when I tried to compress the input to a ninth of its original size:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/222/0*nKp3UltN3fbLJT6_.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/222/0*2fxJ_PetXHp_mxQj.png" /></figure><p>It basically only got the outlines right, and it only worked on black or dark-grey cats.</p><h3>Conclusions</h3><p>I was surprised with the results: compressing the image to a fourth of its size with the cat still being recognizable, means a classifier could probably tell there was a cat in the picture.</p><p>Tensorflow’s current API is a lot more comfortable and intuitive than the old one, and I’m glad I can finally do deep learning without thinking of sessions and graphs.</p><p>I think batch-normalization proved to be quite effective at accelerating the training, and it’s a tool I should use more often.</p><p>Lastly, in the future I’d like to try different image compression algorithms. K-Means Clustering based ones, and maybe a Convolutional Autoencoder. However, that’s an idea for a different article.</p><p>What about you? Think you can train a better autoencoder for this dataset? What are some of the 1000 things I should’ve done better, or hyperparameters I forgot to tweak and could’ve improved the model’s performance greatly? Let me know in the comments, I’m still very much learning.</p><p>I hope you’ve found this article entertaining or helpful, and maybe that you’ve learned something along the way.</p><p>If you wish to learn more about Deep Learning and become a professional in it, you can’t go wrong with Goodfellow and Bengio’s <a href="https://amzn.to/2LU5uNR">Deep Learning book</a>.<br>Its first chapters made Deep Learning finally click for me, and the one on Autoencoders is pure gold.</p><p><em>Follow me on </em><a href="http://medium.com/@strikingloo"><em>Medium</em></a><em>, </em><a href="http://www.dev.to/strikingloo"><em>dev.to</em></a><em> or </em><a href="http://twitter.com/strikingloo"><em>Twitter</em></a><em> for more Machine Learning tutorials.</em></p><p><em>Originally published at </em><a href="http://www.datastuff.tech/machine-learning/autoencoder-deep-learning-tensorflow-eager-api-keras/"><em>http://www.datastuff.tech</em></a><em> on April 20, 2019.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=378318784ae" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/autoencoders-deep-learning-with-tensorflows-eager-api-data-stuff-378318784ae">Autoencoders: Deep Learning with TensorFlow’s Eager API</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>