<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Brian Williams on Medium]]></title>
        <description><![CDATA[Stories by Brian Williams on Medium]]></description>
        <link>https://medium.com/@rantlab?source=rss-96380c147b55------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*Q-DkSPCobhH7rubt.jpeg</url>
            <title>Stories by Brian Williams on Medium</title>
            <link>https://medium.com/@rantlab?source=rss-96380c147b55------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 22:09:40 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@rantlab/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Contrastive Loss Explained]]></title>
            <link>https://medium.com/data-science/contrastive-loss-explaned-159f2d4a87ec?source=rss-96380c147b55------2</link>
            <guid isPermaLink="false">https://medium.com/p/159f2d4a87ec</guid>
            <category><![CDATA[contrastive-loss]]></category>
            <category><![CDATA[loss-function]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[softmax]]></category>
            <category><![CDATA[siamese-networks]]></category>
            <dc:creator><![CDATA[Brian Williams]]></dc:creator>
            <pubDate>Tue, 03 Mar 2020 05:49:40 GMT</pubDate>
            <atom:updated>2023-03-16T00:28:31.288Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/926/1*61b62NNTAqmYSORV-QRo9Q.png" /></figure><p>Contrastive loss has been used recently in a number of papers showing state of the art results with unsupervised learning. <a href="https://arxiv.org/abs/1911.05722">MoCo</a>, <a href="https://arxiv.org/abs/1912.01991">PIRL</a>,<em> </em>and <a href="https://arxiv.org/abs/2002.05709">SimCLR</a> all follow very similar patterns of using a siamese network with contrastive loss. When reading these papers I found that the general idea was very straight forward but the translation from the math to the implementation wasn’t well explained. As is often the case in machine learning papers, the math isn’t difficult once you’ve got the main idea. I will attempt to clearly explain how contrastive loss works and provide a complete implementation using Python and Numpy.</p><h4>Siamese Networks</h4><p>Before digging into the details, it will be helpful to talk about siamese networks (also called <strong>twin networks</strong> but it’s not a widely used term). When training a siamese network, 2 or more inputs are encoded and the output features are compared. This comparison can be done in <a href="https://gombru.github.io/2019/04/03/ranking_loss/">a number of ways</a>. Some of the comparisons are triplet loss, <a href="https://arxiv.org/abs/2001.07685">pseudo labeling with cross-entropy loss</a>, and contrastive loss.</p><p>A siamese network is often shown as two different encoding networks that share weights, but in reality the same network is just used twice before doing backpropagation.</p><figure><img alt="A generic siamese model" src="https://cdn-images-1.medium.com/max/998/1*ChM99rEEcqL0K9ZUoB_UrA.png" /></figure><h4>Example</h4><p>Let’s look at an example where we want to make features out of MNIST numbers. Each image of an MNIST number should encode into a vector that is close to vectors from images of the same class. Conversely different numbers should encode into vectors that are far from each other.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/846/1*E6UUEmxKp5ZTRgCRNbIP-g.png" /><figcaption>Positive (top) and negative samples embedded into a vector space</figcaption></figure><p>Since we have the class labels for MNIST inputs we could use a regular network and Categorical Cross-Entropy loss. The problem is when we don’t have nicely labeled data, as is usually the case. There is much more <a href="https://syncedreview.com/2019/02/22/yann-lecun-cake-analogy-2-0/">unlabeled data than labeled data available</a> in the world. This is where contrastive loss comes in.</p><p>Contrastive loss takes the output of the network for a positive example and calculates its distance to an example of the same class and contrasts that with the distance to negative examples. Said another way, the loss is low if positive samples are encoded to similar (closer) representations and negative examples are encoded to different (farther) representations.</p><p>This is accomplished by taking the cosine distances of the vectors and treating the resulting distances as prediction probabilities from a typical categorization network. The big idea is that you can treat the distance of the positive example and the distances of the negative examples as output probabilities and use cross entropy loss.</p><p>When performing supervised categorization, the network outputs are typically run through a softmax function then the negative log-likelihood loss.</p><p>Let’s make this more concrete. This example will have two vectors that are similar and an array of dissimilar ones. P1 and p2 are the positive vectors, with p2 being a slightly modified version of p1. Neg is the array of dissimilar vectors.</p><h4>Example Setup</h4><pre>import numpy as np</pre><pre>p1 = np.array([-0.83483301, -0.16904167, 0.52390721])<br>p2 = np.array([-0.83455951, -0.16862266, 0.52447767])<br>neg = np.array([<br> [ 0.70374682, -0.18682394, -0.68544673],<br> [ 0.15465702,  0.32303224,  0.93366556],<br> [ 0.53043332, -0.83523217, -0.14500935],<br> [ 0.68285685, -0.73054075,  0.00409143],<br> [ 0.76652431,  0.61500886,  0.18494479]])</pre><p>Please note that I have used pre-normalized vectors (aka unit vectors ) here.</p><h4>Calculating Distance</h4><p>In order to measure how similar two vectors are to each other, we need a way of measuring distance. In 2 or 3 dimensions the euclidian distance (“ordinary” or straight-line distance) is a great choice for measuring the distance between two points. However, in a large dimensional space, all points tend to be far apart by the euclidian measure. In higher dimensions, the angle between vectors is a more effective measure. The cosine distance measures the cosine of the angle between the vectors. The cosine of identical vectors is 1 while orthogonal and opposite vectors are 0 and -1 respectively. More similar vectors will result in a larger number. Calculating the cosine distance is done by taking the dot product of the vectors. <em>When not using unit vectors, you would have either normalize the vectors or divide but the product to the normed vectors.</em></p><pre># P1 and p2 are nearly identically, thus close to 1.0<br>pos_dot = p1.dot(p2)<br>pos_dot -&gt; 0.999999716600668</pre><pre># Most of the negatives are pretty far away, so small or negative<br>num_neg = len(neg)<br>neg_dot = np.zeros(num_neg)<br>for i in range(num_neg):<br>    neg_dot[i] = p1.dot(neg[i])</pre><pre>neg_dot -&gt; [-0.91504053,  0.30543542, -0.37760565, -0.44443608, -0.64698801]</pre><h4>Softmax</h4><p>The softmax function takes a vector of real numbers and forces them into a range of 0 to 1 with the sum of all the numbers equaling 1. One other nice property of softmax is that one of the values is usually much bigger than the others. When calculating the loss for categorical cross-entropy, the first step is to take the softmax of the values, then the negative log of the labeled category.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JI8EbX6CmkphXa5rnWmLcg.png" /><figcaption>Softmax, the mathy version</figcaption></figure><p>Let’s take the softmax of pos_dot appended with neg_dot vector.</p><pre># make a vector from the positive and negative vectors comparisons<br>v = np.concatenate(([pos_dot], neg_dot))</pre><pre># take e to the power of each value in the vector<br>exp = np.exp(v)</pre><pre># divide each value by the sum of the exponentiated values<br>softmax_out = exp/np.sum(exp)</pre><pre>softmax_out -&gt; [0.4296791, 0.0633071, 0.2145353, 0.1083572, 0.1013523, 0.0827687]</pre><p>Our positive example (0.4296791) is now much bigger than the random ones, and all bigger than zero and less than one.</p><h4>Contrastive Loss</h4><p>Finally, we get to focus of this article, contrastive loss.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/908/1*4dQo-60fwy1Cki4nKjeGuQ.png" /><figcaption>The contrastive loss function</figcaption></figure><p>Contrastive loss looks suspiciously like the softmax function. That’s because it is, with the addition of the vector similarity and a temperature normalization factor. The similarity function is just the cosine distance that we talked about before. The other difference is that values in the denominator are the cosign distance from the positive example to the negative samples. Not very different from CrossEntropyLoss. The intuition here is that we want our similar vectors to be as close to 1 as possible, since -log(1) = 0, that’s the optimal loss. We want the negative examples to be close to 0 , since any non-zero values will reduce the value of similar vectors.</p><pre># Contrastive loss of the example values<br># temp parameter<br>t = 0.07</pre><pre># concatenated vector divided by the temp parameter<br>logits = np.concatenate(([pos_dot], neg_dot))/t</pre><pre>#e^x of the values<br>exp = np.exp(logits)</pre><pre># we only need to take the log of the positive value over the sum of exp. <br>loss = - np.log(exp[0]/np.sum(exp))<br>loss -&gt; 4.9068650660314756e-05</pre><p>That’s all there is to it. Contrastive loss can be implemented as a modified version of cross-entropy loss. Contrastive loss, like triplet and magnet loss, is used to map vectors that model the similarity of input items. These mappings can support many tasks, like unsupervised learning, one-shot learning, and other distance metric learning tasks. I hope that this article has helped you better understand contrastive loss.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=159f2d4a87ec" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/contrastive-loss-explaned-159f2d4a87ec">Contrastive Loss Explained</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Do androids see electric blue? — visualizing the latent space of color perception]]></title>
            <link>https://medium.com/data-science/do-androids-see-electric-blue-visualizing-the-latent-space-of-color-perception-fcd46fb8848e?source=rss-96380c147b55------2</link>
            <guid isPermaLink="false">https://medium.com/p/fcd46fb8848e</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[perception]]></category>
            <category><![CDATA[pytorch]]></category>
            <category><![CDATA[color-theory]]></category>
            <dc:creator><![CDATA[Brian Williams]]></dc:creator>
            <pubDate>Wed, 05 Sep 2018 18:45:18 GMT</pubDate>
            <atom:updated>2018-09-07T16:31:03.767Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/320/1*OjqbgHH3q0roMA5korWeKw.gif" /><figcaption>Visualization of the learned color space</figcaption></figure><h4>TL;DR</h4><p>I wanted to better understand the process of using a neural network to map data into a latent vector space. I remembered this <a href="https://blog.xkcd.com/2010/03/01/color-name-survey/">survey that asked people to name randomly presented colors</a>. Since colors make compelling visualizations and since I hadn’t seen this done before, I thought that it would make a good training example. The task of the neural network was to group colors with the same name into similar regions of a new color space. The results were visualized by plotting each color’s new location in the learned color space.</p><h4>Introduction</h4><p>In machine learning, vector spaces can be used to group semantically similar items. Examples include face recognition, sentence similarity and content creation. The idea is to take input data that is variable, such as pictures of the same face in different lighting conditions, and map it to the same or similar region in a vector space. I’ll be using data from a color perception survey to illustrate this concept. I haven’t seen this done before, so I’m really excited to share it with you.</p><p>So, what is a latent vector space?</p><p>Essentially, a latent vector space is a representation of an input that’s encoded to be semantically meaningful. Semantically similar items should be mapped to similar areas of the space, even if the raw input data are quite different.</p><p>Let’s break the definition down: a vector is just a collection of numbers. In a vector space, each number represents a position along an independent axis. You can think of a vector pointing to a position, or a dot, in a multi-dimensional space. Latent is defined as “(of a quality or state) existing but not yet developed or manifest; hidden or concealed”. In a latent vector space, the meaning of the axes are discovered through the training process, rather than being determined a priori.</p><p>A common example is <a href="https://blog.acolyer.org/2016/04/21/the-amazing-power-of-word-vectors/">word2vec</a>, where words are encoded into a vector. The meaning of a word is determined solely from the words that are adjacent to it in a sentence. The axes of this new vector space encode interesting semantic properties. The illustration below shows a gender axis.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/596/1*p0cReUgY5WCTUhtImLk2Qg.png" /></figure><p>With a latent vector space, we are hoping to discover some shape or order to our data that is hidden, but present.</p><h4>Mapping Colors</h4><p>One way of understanding the process of discovering these hidden qualities is to start with a vector space that we already know. This is where colors come in. Colors are usually represented with a vector of 3 numbers, representing the values of red, green and blue (RGB). Other common color spaces are <a href="https://en.wikipedia.org/wiki/List_of_color_spaces_and_their_uses">HSV, YUV, CIE etc.</a></p><p>Training a network requires a task to be performed, like a labeling a photo, or predicting the next word in a sentence. Word2Vec and sentence vectors are structured this way. In the <a href="https://iksinc.online/tag/continuous-bag-of-words-cbow/">bag-of-words</a> model, a sentence fragment is presented with one of the words removed. The network tries to guess the missing word based on the words that are adjacent.</p><h4>Data cleanup and dataset creation</h4><p>The webcomic XKCD ran a <a href="https://blog.xkcd.com/2010/03/01/color-name-survey/">survey</a> in which participants were asked to name colors that were presented. The raw survey data consists of each color that was presented and the name that the participants typed in response. Before this data could be used, some cleanup was required. I took the liberty of combining entries that I thought were obviously the same color name. For instance I combined entries for bluish and blueish, brick red and brick, fuscia and fuchsia. Interested parties can have a look at this <a href="https://github.com/briandw/ColorEmbeddings">notebook</a> to see exactly what I did.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/320/1*yWNmFhzP-ZtYUv5tq098kQ.png" /><figcaption>Sample of the colors labeled “red”</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/579/1*IdMMXm9FfSOnqSCghSJYbQ.png" /><figcaption>Plot of the input colors from the survey</figcaption></figure><h4>Training the neural network</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/379/1*Zn_ttYvkPt4TZlJ9qzGMzQ.png" /></figure><p>The network I used was a fully connected network with an input, 2 hidden layers, and an output. The input layer takes 3 floats, corresponding to our original RGB color values and outputs 50 values. The two hidden layers are also 50 values wide. The output layer will take the 50 hidden values from the 3rd layer and output 3 floats. The output will be our new color space.</p><p>When neural networks are trained, the output is scored for how well it predicted the correct outcome based on the input. This score is called the loss function. The simplest loss function would be the L1 loss, which is just the absolute value of the difference between the output and the expected value. You can read more about <a href="https://medium.com/data-science-group-iitr/loss-functions-and-optimization-algorithms-demystified-bb92daff331c">loss functions here</a>. In this case, the loss function is a bit more interesting. We can’t use the usual guessing game since we are asking the network to create a new vector, not categorize data that has already been labeled. The task is to create vectors that are close to each other for inputs that are similar and far apart for inputs that are dissimilar. One way of accomplishing this is to use the Triplet Margin Loss function.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/716/1*BwPl_DoSDsAoIkQE5QlXwg.png" /><figcaption>Sample of the triplet dataset colors. The top, middle and bottom are the anchor, positive and negative colors respectively</figcaption></figure><p>Triplet loss takes three values as an input and outputs a score. The three values are an anchor value, a positive value, and a negative value. The anchor is the color in question, the positive is a color that was given the same name as the anchor color and the negative is a color that was given a different name. When training the network, these three values are run through the network, one at a time and the outputs are saved. These three values are then given to the loss function, which produces a score based on how close the anchor and positive colors are and how far away the negative color is. This is then used as the error value in backpropagation. After each batch of colors is run through the network, similar colors will have vectors that are closer to each other, while dissimilar colors will be further apart.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/271/1*Z3jFlL_8nU6MF77TTeYN7w.png" /></figure><p>Once this process is complete, we should have a new color map.</p><p>But a map of what?</p><p>Before training, I saw the following possibilities:</p><p>1) The network doesn’t train, that is to say, that the loss value never goes down and the network doesn’t learn anything useful.</p><p>2) The network trains and the loss function decreases but the results are nonsensical. This would be the result of over-fitting.</p><p>3) The network trains and results in a perceptual color space that is understandable.</p><p>4) The network trains and results in a color space that groups colors but is otherwise idiosyncratic.</p><p>5) The network trains and produces RGB values similar to the original color values.</p><p>Before training started, my guess was that it would be #4.</p><h4>Training Results</h4><p>After training the network and plotting the colors, I’d say that the results are a mix of #3 and #4. The greens and reds are pretty far apart, as you’d expect with what is known about human <a href="https://en.wikipedia.org/wiki/Opponent_process">color perception</a>. There also seems to be a warm/cold axis as well as a light/dark one. The greens seem to occupy a larger volume than the other colors. A bit of green bias also makes sense. I have no idea what’s going on with the green points that ended up in the corner.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*XB_VGol7mFFb8PBKOERK6g.jpeg" /><figcaption>One view of the 3D color space</figcaption></figure><h4>Conclusion</h4><p>So there it is, a machine learned color space from survey data. I think that this illustrates the power of a machine learning system to extract meaning from a vast amount of data. Here the signal is small, just individual color / name pairs, but with such a large amount of data, a larger structure can be learned. It would be interesting to run the training again from scratch and see how much the results differ. If the results were very similar then there is something about the final structure that’s inevitable given the input and the network topology.</p><p>Please clap 👏 if you enjoyed this article and would like to see more like it.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fcd46fb8848e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/do-androids-see-electric-blue-visualizing-the-latent-space-of-color-perception-fcd46fb8848e">Do androids see electric blue? — visualizing the latent space of color perception</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[25 Lights]]></title>
            <link>https://medium.com/data-science/25-lights-part-ii-e021b66e449b?source=rss-96380c147b55------2</link>
            <guid isPermaLink="false">https://medium.com/p/e021b66e449b</guid>
            <category><![CDATA[coreml]]></category>
            <category><![CDATA[tensorflow]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[ios]]></category>
            <dc:creator><![CDATA[Brian Williams]]></dc:creator>
            <pubDate>Fri, 08 Dec 2017 22:32:35 GMT</pubDate>
            <atom:updated>2017-12-18T00:21:42.904Z</atom:updated>
            <content:encoded><![CDATA[<h4>Part II: Working with real world data</h4><h3>Brian on Twitter</h3><p>Final results from my ML experiment. Trained a deep CNN to recognize the 25 bit pattern displayed on the screen. Medium post to follow. #CoreML #MachineLearning #TensorFlow https://t.co/PyposM5iw9</p><p>In <a href="https://towardsdatascience.com/25-lights-c4cc2c3f1832">part </a>I, I showed that a simple, fully connected autoencoder, can reproduce an image with a random bit pattern. An autoencoder was a good choice to illustrate what’s happening as the network is trained and to show that it works in theory. However, a simple fully connected network won’t work in real life. This is because flattening out the image into a single dimension loses much of the spatial information. Additionally, a single hidden layer isn’t able to process all the variations in size, position, rotation, blur etc… that a real photo will have.</p><h4><strong>Working with photographs</strong></h4><p>My goal was to build a demo iPhone app that takes photos of the pattern and shows the detected bit pattern. This means that the training set will have to use photos and not purely computer-generated images. Working with real-world photographs requires a more sophisticated approach than was used in the first post. Fortunately, there is a large body of research in image recognition to draw from. The <a href="http://www.image-net.org/challenges/LSVRC/">ImageNet</a> contest winners post their results and copying the winning network is a good place to start. One popular network to use for transfer learning is<a href="https://arxiv.org/abs/1409.1556v6"> VGG-16</a>. We can borrow heavily from that network in our design here. VGG-16 is a deep convolutional network (DCNN). If you are unfamiliar with CNNs, I found this <a href="https://www.youtube.com/watch?v=FmpDIaiMIeA">video</a> to be an excellent tutorial. The short explanation is that CNNs use a stack of convolutional filters that are good at extracting spatial information and spatial relationships.</p><p>Below is the network that I used. It’s different from VGG in that it uses average pooling in place of max pooling, has fewer layers and a different loss function.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3589102a54efe177c2407db33942ea77/href">https://medium.com/media/3589102a54efe177c2407db33942ea77/href</a></iframe><p>For the rest of the notebook check out the <a href="https://github.com/briandw/25Lights">full jupyter notebook and supporting tools</a>. I’ve included the macOS image server, iOS image recorder and final iOS recognition application.</p><h4>Creating a training and validation set</h4><p>Now that we have our network set up, we need better data to train on. In our first example, we used simple images drawn on demand. These are too clean and don’t have any of the irregularities that a real photo will have. Using real photos is a challenge due to the quantity needed. My initial attempts to use purely synthetic images failed to validate with real photographs. I then created a training set of about 15k photographs. These were taken by an iPhone pointed at a computer screen. The phone connected over the local network to retrieve the bit pattern and then save a picture to disk. I wasn’t able to get good results until around 90k images were included in the training set. In order to improve the quality of the training, the image had a random affine transform applied. Applying this transform moves, scales and skews the image.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cNjULs1OYlHaDsmnVXe2VQ.jpeg" /><figcaption>iOS app capturing images from the screen</figcaption></figure><p>After collecting the images, I reserved 10% of them for a validation set. A good validation set will enable you to know how training is progressing and what additional steps are needed to reach your goals. Rachel Thomas at fast.ai has an<a href="http://www.fast.ai/2017/11/13/validation-sets/"> excellent article on the topic</a>.</p><h4>Overfitting</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*m1TocH-TeB4TsbUgpSzKiQ.png" /><figcaption>This is what overfitting looks like</figcaption></figure><p>My first attempts at using a VGG like network didn’t include any regularization. This was clearly a mistake. The above graph is a classic case of overfitting. The <strong>loss</strong> graph on the left shows how the network is doing on the training data. The <strong>val_loss</strong> on the right is showing how the network is doing with the validation set. I like to think of neural networks as lazy cheaters, they will take any advantage to get the right answer. A big network and a small training set is bound to result in overfitting. The initial training set was 10k images. At that size, it’s easier for the network to “remember” the values of each image, rather than learn to see the dots in a generalizable way.</p><h4>How to avoid overfitting</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3vvxmEcupDQlnX9VZmiKWg.png" /><figcaption>The same data as above, but with batch normalization and dropout</figcaption></figure><p>Some techniques to avoid overfitting are:</p><ul><li>Larger training sets (bigger training set, data argumentation, bigger variety of input sources)</li><li>Regularization (batch normalization, dropout)</li><li>Changing the network topology (making the network smaller)</li></ul><p>Adding dropout and batch normalization was very effective in preventing the runaway overfitting that’s seen in the first graph. However, the data set was still too small to get good results on the validation set. Early training only resulted in a validation loss of 0.4 and a training loss of under 0.1. This discrepancy is called <a href="https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff">variance</a>. A validation loss of under 0.02 is needed for good results. When a bigger, more varied, training set was used, better results were obtained.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/904/1*-2bhIe7l1tmQo7jDNlIs-w.png" /><figcaption>Final training run</figcaption></figure><h4>Debugging the network</h4><p>Once training has reached a reasonable level of performance, it’s a good idea to check the cases where the network is getting it wrong. A simple visual inspection of those cases is often sufficient.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/77d0163d1cb5b286cc738fe2550da8bd/href">https://medium.com/media/77d0163d1cb5b286cc738fe2550da8bd/href</a></iframe><p>The first thing was to calculate how many images had one or more incorrect bit. It was getting about 92% of the images completely correct. So whats up with that other 8%? Here are some of the missed images:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MApHH-vmZ3dMraBR8PIwKA.png" /></figure><p>There are a bunch of double images and blurry photos, reasonable things to miss. I went back and removed all the photos that had double images or other major problems, then re-ran the training from the beginning. Here are the results:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/870/1*s8YvFbYR8Juv66HPQvSlmw.png" /><figcaption>Final loss of 0.04 and validation loss of 0.02</figcaption></figure><p>The final error rate is 97%. If it was necessary to get closer to 100% a wider variety of images would be needed, concentrating on those cases that the current network isn’t currently doing well on.</p><h4>iPhone implementation and CoreML</h4><p>I was going to write a 3rd post about making an iOS app with the trained model. However, it’s really simple and doesn’t warrant its own post.</p><p>Once you have your model trained, install the <a href="https://pypi.python.org/pypi/coremltools">CoreML Tools</a>. You will need to have a Python 2.7 environment since CoreML doesn’t support Python 3. Then create a new cell in your notebook and run the following.</p><pre>import coremltools</pre><pre>coreml_model = coremltools.converters.keras.convert(model)<br>coreml_model.save(“Lights.mlmodel”)</pre><p>Once you have exported your model, create a new project and import it. The only tricky part is converting the image into an input for the CoreML model.</p><p>To convert a image into CoreML:</p><ol><li>Make a MLMultiArray in the shape of our models input. In this case 1 x 128 X 128, 32 bit float.</li><li>Render the image into an 8 bit gray scale drawing context.</li><li>Extract the bytes and convert them to floats and write them to the array.</li><li>Create an input object for the model and get a prediction from the MLModels predictionFromFeatures method.</li></ol><p>For more detail have a look at the <a href="https://gist.github.com/briandw/3662437e975ace767caa9ef27d9d6712">iOS CoreML Code</a>.</p><p>For a full tutorial on how to use a CoreML model, check out <a href="https://www.raywenderlich.com/164213/coreml-and-vision-machine-learning-in-ios-11-tutorial">CoreML and Vision Machine Learning</a>.</p><h4>Conclusion</h4><p>That was an awful lot of trouble to solve this problem the wrong way! There are plenty of examples of image recognition systems that could do this kind of task without resorting to using a deep convolutional network. Traditional CV techniques could solve this faster and with a much smaller binary. However this was a really fun learning exercise and helped me to figure out how to solve the problems that come with a new ML task.</p><p>When I thought about this problem as an ML training exercise, I wasn’t sure that it would work. All the examples that I had encountered were classification tasks, like ImageNet. Those classification tasks have a final one-hot vector output and used loss functions like categorical crossentropy. After doing more reading about other networks and understanding loss functions a little better I realized that something like binary crossentropy should work. In hindsight I should have <a href="http://yeephycho.github.io/2017/09/16/Loss-Functions-In-Deep-Learning/">started reading in detail about loss functions sooner</a>.</p><p>The size and breath of the training data needed is also a bit of a revlation. Over 85k examples isn’t that big compared to other training sets, but it’s still really big. It would be fun to try this again with a <a href="https://medium.com/ai%C2%B3-theory-practice-business/understanding-hintons-capsule-networks-part-i-intuition-b4b559d1159b">capsule network</a>. Capsule networks should have a better sense of relative position and require fewer examples.</p><p>Putting all the techniques that I’ve learned to use and seeing them work was very gratifying. <a href="http://www.andrewng.org/">Andrew Ng’s</a> and <a href="http://course.fast.ai/">Jeremy Howard’s</a> video courses have been excellent guides. In most ML course material there’s a good amount of time spent on regularization and managing your training data and validation sets. I hope I’ve shown that there is good reason for that. Thanks for reading about my project. I’d love to hear any comments or suggestions about this article in the comments below.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e021b66e449b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/25-lights-part-ii-e021b66e449b">25 Lights</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[25 Lights]]></title>
            <link>https://medium.com/data-science/25-lights-c4cc2c3f1832?source=rss-96380c147b55------2</link>
            <guid isPermaLink="false">https://medium.com/p/c4cc2c3f1832</guid>
            <category><![CDATA[tensorflow]]></category>
            <category><![CDATA[keras]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[internet-of-things]]></category>
            <category><![CDATA[computer-vision]]></category>
            <dc:creator><![CDATA[Brian Williams]]></dc:creator>
            <pubDate>Mon, 02 Oct 2017 04:42:49 GMT</pubDate>
            <atom:updated>2017-12-18T00:07:04.592Z</atom:updated>
            <content:encoded><![CDATA[<h4>Part I: Problem Introduction and Simple Solution</h4><p>Part II : <a href="https://towardsdatascience.com/25-lights-part-ii-e021b66e449b">Working with real world data</a></p><p>It’s an amazing time to get into Machine Learning. There are tools and resources available to help anyone with some coding skills and a problem to solve, to do interesting work. I’ve been following along with <a href="http://course.fast.ai/"><strong>Practical Deep Learning For Coders</strong></a><strong> </strong>and the<strong> </strong><a href="https://www.youtube.com/watch?v=2pWv7GOvuf0"><strong>Reinforcement Learning Course by David Silve</strong></a><strong>r. </strong><a href="https://www.youtube.com/watch?v=vq2nnJ4g6N0&amp;t=2s"><strong>Machine Learning without a PhD</strong></a><strong> </strong>is an exellent introduction to deep learning techinques. These, along with all the papers linked from Hacker News and <a href="https://www.youtube.com/user/keeroyz"><strong>Two Minute Papers</strong></a><strong> </strong>have inspired me to give some ideas a try.</p><p>I’ve worked in the past on the setup process for an IoT device, connecting to it from an iPhone over BTLE. One of the vulnerabilities of relying on a single channel for setup is the <a href="https://en.wikipedia.org/wiki/Man-in-the-middle_attack">Man-in-the-middle attack</a>. One way of beating MitM is to use out-of-band authentication of the public keys that are exchanged. This can be done by encoding a random bit pattern on the devices LEDs and reading them from the camera on the phone. This would be a less sophisticated version of the Apple Watch setup protocol, but still very effective.</p><p>The device I’m thinking about has a 5x5 array of lights on the front. 2²⁵ can encode 33,554,432 possibilities, perfect for key exchange validation. The challenge here is now to find a way to automatically extract the bit pattern from an image.</p><p>First, I tried to adapt the VGG16 net, used in the early part of the Fast.ai MOOC. I tried several rounds of training and applying the standard advice for fine-tuning a network. This failed to produce meaningful results. VGG16 and deep nets like it are great at identifying a single category but I wasn’t sure if it could track 25 lights. In <a href="https://towardsdatascience.com/25-lights-part-ii-e021b66e449b">Part II</a> I show how a VGG16 like network can work. After taking a break, thinking and reading about other projects and <a href="https://arxiv.org/pdf/1412.6622.pdf">techniques</a>, I came across <a href="https://en.wikipedia.org/wiki/Autoencoder">autoencoders</a>. An autoencoder is a neural network that encodes the input into a smaller vector, then decodes that vector back to the original size. A loss function is used to compare the difference between the input and the output. The autoencoder can then be trained to minimize the loss.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*P7aFcjaMGLwzTvjW3sD-5Q.jpeg" /><figcaption>Autoencoder diagram from the Keras blog</figcaption></figure><p>Autoencoders aren’t the most sophisticated technique, but it’s better to start with a simple solution. <a href="https://blog.keras.io/building-autoencoders-in-keras.html"><strong>Building Autoencoderss in Keras</strong></a><strong> </strong>is an excellent introduction and my explorations here are following that post closely.</p><h3>Image Generation</h3><p>In order to train the autoencoder, we need some images. Below is the Python code to draw the image that we want to recognize. It’s fortunate that we can easily generate the images that are needed to train against. Real world images will be harder to acquire in the volumes that are needed here.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b7447de0e0efa462dff84fb2a411a57e/href">https://medium.com/media/b7447de0e0efa462dff84fb2a411a57e/href</a></iframe><h3>Simple Autoencoder</h3><p>Now that we have our image generator, we can make a very simple autoencoder. This is nearly identical to the <a href="https://blog.keras.io/building-autoencoders-in-keras.html">first autoencoder</a> from the Keras blog.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f9f93a0d3047f55366bf1651a59b33be/href">https://medium.com/media/f9f93a0d3047f55366bf1651a59b33be/href</a></iframe><p>After 500 iterations with a batch size of 128 we get to a loss of 0.1012. This is about the level of loss in the of the first try in the autoencoder blog. Here’s what we see. Input image on top, decoded on the bottom.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9Wi4gXUmDqntMbFl9M6Mkg.png" /></figure><p>This isn’t looking as good as I had hoped, but notice the slight variation in dot color. Let’s give it a few thousand iterations more.</p><p>5000 iterations get to a loss of 0.0031.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ACbssc-FYiW6dWiNs2n4FA.png" /></figure><p>That’s more like it. Now 5,000 more and it should be good.</p><p>After 10,000 iterations, the loss is 0.000288.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PZHWHO5o23y_zbsxaiqBhw.png" /></figure><p>It’s looking good, but this simple of a network won’t hold up to real world input. Have a look at my next post where I show what it takes to recognize the dot pattern in photographs. Then how to run the network on the iPhone with CoreML.</p><h4>Part II : <a href="https://towardsdatascience.com/25-lights-part-ii-e021b66e449b">Working with real world data</a></h4><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c4cc2c3f1832" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/25-lights-c4cc2c3f1832">25 Lights</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>