<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Deeper Learning - Medium]]></title>
        <description><![CDATA[Explaining How AI Thinks - Medium]]></description>
        <link>https://medium.com/deeper-learning?source=rss----27739ac039da---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Deeper Learning - Medium</title>
            <link>https://medium.com/deeper-learning?source=rss----27739ac039da---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 03:42:24 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/deeper-learning" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Glossary of Deep Learning: Batch Normalisation]]></title>
            <link>https://medium.com/deeper-learning/glossary-of-deep-learning-batch-normalisation-8266dcd2fa82?source=rss----27739ac039da---4</link>
            <guid isPermaLink="false">https://medium.com/p/8266dcd2fa82</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Jaron Collis]]></dc:creator>
            <pubDate>Tue, 27 Jun 2017 21:18:31 GMT</pubDate>
            <atom:updated>2017-06-27T21:18:31.290Z</atom:updated>
            <cc:license>https://creativecommons.org/licenses/by-sa/4.0/</cc:license>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8jMvbqh5HHgT_dgSn8-dKw.jpeg" /></figure><p>Batch normalisation is a technique for improving the performance and stability of neural networks, and also makes more sophisticated deep learning architectures work in practice (like DCGANs).</p><p>The idea is to normalise the inputs of each layer in such a way that they have a mean output activation of zero and standard deviation of one. This is analogous to how the inputs to networks are standardised.</p><p>How does this help? We know that normalising the inputs to a network helps it learn. But a network is just a series of layers, where the output of one layer becomes the input to the next. That means we can think of any layer in a neural network as the first layer of a smaller subsequent network.</p><p>Thought of as a series of neural networks feeding into each other, we normalising the output of one layer before applying the activation function, and then feed it into the following layer (sub-network).</p><p>In Keras, it is implemented using the following code. Note how the BatchNormalization call occurs after <em>each</em> fully-connected layer, but before the <a href="https://medium.com/deeper-learning/glossary-of-deep-learning-activation-function-20262327a907">activation function</a> and dropout.</p><pre>from keras.layers.normalization import BatchNormalization<br>model = Sequential()</pre><pre># think of this as the input layer<br>model.add(Dense(64, input_dim=16, init=’uniform’))<br>model.add(BatchNormalization())<br>model.add(Activation(‘tanh’))<br>model.add(Dropout(0.5))</pre><pre># think of this as the hidden layer <br>model.add(Dense(64, init=’uniform’))<br>model.add(BatchNormalization())<br>model.add(Activation(‘tanh’))<br>model.add(Dropout(0.5))</pre><pre># think of this as the output layer<br>model.add(Dense(2, init=’uniform’))<br>model.add(BatchNormalization())<br>model.add(Activation(‘softmax’))</pre><pre># optimiser and loss function<br>model.compile(loss=’binary_crossentropy’, optimizer=sgd)</pre><pre># train the model<br>model.fit(X_train, y_train, nb_epoch=50, batch_size=16)</pre><p>Batch normalisation was introduced in <a href="https://arxiv.org/pdf/1502.03167.pdf">Ioffe &amp; Szegedy’s 2015 paper</a>. The idea being that, instead of just normalising the inputs to the network, we normalise the inputs to <em>layers within</em> the network. It’s called “batch” normalization because during training, we normalise the activations of the previous layer for each batch, i.e. apply a transformation that maintains the mean activation close to 0 and the activation standard deviation close to 1.</p><p>Beyond the intuitive reasons, there are good mathematical reasons why it helps the network learn better, too. It helps combat what the authors call <em>internal covariate shift</em>. This is discussed in the <a href="https://arxiv.org/pdf/1502.03167.pdf">original paper</a> and the Deep Learning book (Goodfellow et al), in section 8.7.1 of<a href="http://www.deeplearningbook.org/contents/optimization.html"> Chapter 8</a>.</p><h4><strong>Benefits of Batch Normalization</strong></h4><p>The intention behind batch normalisation is to optimise network training. It has been shown to have several benefits:</p><ol><li><strong>Networks train faster</strong> — Whilst each training iteration will be slower because of the extra normalisation calculations during the forward pass and the additional hyperparameters to train during back propagation. However, it should converge much more quickly, so training should be faster overall.</li><li><strong>Allows higher learning rates</strong> — Gradient descent usually requires small learning rates for the network to converge. As networks get deeper, gradients get smaller during back propagation, and so require even more iterations. Using batch normalisation allows much higher learning rates, increasing the speed at which networks train.</li><li><strong>Makes weights easier to initialise</strong> — Weight initialisation can be difficult, especially when creating deeper networks. Batch normalisation helps reduce the sensitivity to the initial starting weights.</li><li><strong>Makes more activation functions viable</strong> — Some activation functions don’t work well in certain situations. Sigmoids lose their gradient quickly, which means they can’t be used in deep networks, and ReLUs often die out during training (stop learning completely), so we must be careful about the range of values fed into them. But as batch normalisation regulates the values going into each activation function, nonlinearities that don’t work well in deep networks tend to become viable again.</li><li><strong>Simplifies the creation of deeper networks</strong> — The previous 4 points make it easier to build and faster to train deeper neural networks, and deeper networks generally produce better results.</li><li><strong>Provides some regularisation</strong> — Batch normalisation adds a little noise to your network, and in some cases, (e.g. Inception modules) it has been shown to work as well as dropout. You can consider batch normalisation as a bit of extra regularization, allowing you to reduce some of the dropout you might add to a network.</li></ol><p>As batch normalisation helps train networks faster, it also facilitates greater experimentation — as you can iterate over more designs more quickly.</p><p>It’s also worth bearing in mind other techniques for inter-layer normalisation exist, such as <a href="https://arxiv.org/pdf/1607.08022.pdf">instance normalisation</a>, (Ulyanov et al), which is used in the <a href="https://arxiv.org/pdf/1703.10593.pdf">CycleGAN</a> architecture.</p><p><strong>See also:</strong></p><ul><li><a href="http://www.deeplearningbook.org/contents/optimization.html">Batch Normalisation (section 8.7.1)</a> — in the Deep Learning Book</li><li><a href="https://arxiv.org/pdf/1502.03167.pdf">Ioffe &amp; Szegedy’s 2015 Batch Normalization paper</a></li><li><a href="https://www.youtube.com/watch?v=-E2N1kQc8MM">How to generate video</a> — Siraj Raval</li><li><a href="https://www.quora.com/Why-does-batch-normalization-help">“Why does Batch Normalization Help?”</a> — on Quora</li></ul><h4><a href="https://medium.com/deeper-learning/a-glossary-of-deep-learning-9cb6292e087e">SEE GLOSSARY INDEX</a></h4><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8266dcd2fa82" width="1" height="1" alt=""><hr><p><a href="https://medium.com/deeper-learning/glossary-of-deep-learning-batch-normalisation-8266dcd2fa82">Glossary of Deep Learning: Batch Normalisation</a> was originally published in <a href="https://medium.com/deeper-learning">Deeper Learning</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Glossary of Deep Learning: Autoencoder]]></title>
            <link>https://medium.com/deeper-learning/glossary-of-deep-learning-autoencoder-1044ec82c300?source=rss----27739ac039da---4</link>
            <guid isPermaLink="false">https://medium.com/p/1044ec82c300</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[unsupervised-learning]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[neural-networks]]></category>
            <dc:creator><![CDATA[Jaron Collis]]></dc:creator>
            <pubDate>Fri, 26 May 2017 16:24:13 GMT</pubDate>
            <atom:updated>2017-05-26T16:24:12.797Z</atom:updated>
            <cc:license>https://creativecommons.org/licenses/by-sa/4.0/</cc:license>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/703/0*yGqTBMopqHbR0fcF." /><figcaption>An Autoencoder consists of 3 parts: Encoder, Middle and Decoder, the Middle is a compressed representation of the original input, created by the Encoder, which can be reconstructed by the Decoder. [<a href="https://github.com/udacity/deep-learning/blob/master/autoencoder/Simple_Autoencoder_Solution.ipynb">Source</a>]</figcaption></figure><p>An Autoencoder is neural network capable of unsupervised <a href="https://en.wikipedia.org/wiki/Feature_learning">feature learning</a>.</p><p>Neural networks are typically used for supervised learning problems, trying to predict a target vector <strong>y </strong>from input vectors <strong>x</strong>. An Autoencoder network, however, tries to predict <strong>x</strong> from <strong>x</strong>, without the need for labels. Here the challenge is recreating the essence of the original input from compressed, noisy or corrupted data.</p><p>The idea behind the Autoencoder is to build a network with a narrow hidden layer between Encoder and Decoder that serves as a compressed representation of the input data.</p><p>The middle layer (“bottleneck”) is the compressed representation of the input data, from which the original data can be reconstructed.</p><p>Passing input data through the Encoder creates a compressed representation that can then be passed through a Decoder to reconstruct the original input. The difference between an Autoencoder and an engineered compression algorithm is that both compression and decompression functions are learned from the data itself, rather than being consciously designed and implemented programmatically (like jpg or zip).</p><p>Both Encoder and Decoder are feed-forward neural networks, and the whole network is trained by minimising the difference between the input and output, as per normal. This does mean that Autoencoders once trained are quite specific, and will have trouble generalizing to data sets other than those they were trained on.</p><h4>Why Use Autoencoders?</h4><p>Whilst Autoencoders can be used for compression, their performance is obviously lossy, because of its inherent dimensionality reduction. This means they’re not quite as efficient as engineered algorithms like JPEG, MP3 etc.</p><p>A more suitable use for an Autoencoder is Denoising. A Denoising Autoencoder (DAE) is one that receives corrupted data as input and is trained to predict the original, uncorrupted data as its output.</p><p>A challenge in using neural networks is that they can simply learn to duplicate the training data instead of truly learning to generalise. To avoid this problem, a technique called <a href="https://www.quora.com/What-does-the-term-data-Augmentation-mean-in-the-context-of-Machine-Learning">Data Augmentation</a> is often used, to supplement a training set by creating replicas of observations with added random noise or “jitter”. Autoencoders can be used to recover the original undistorted input.</p><p>There’s also an implementation called a <a href="http://www.ericlwilkinson.com/blog/2014/11/19/deep-learning-sparse-autoencoders">Sparse Autoencoder</a> that can be used for feature extraction. These tend to have more hidden units than input units, allowing a sparse representation of the data. If more than one hidden layer is used, a <a href="http://ufldl.stanford.edu/wiki/index.php/Stacked_Autoencoders">stacked Autoencoder</a> can be constructed, where each consecutive layer in the network can be an optimally weighted, nonlinear, lower dimensional projection of the training data. This can be used to implement a pre-training mechanism for supervised learning.</p><h4>Other Kinds of Autoencoders</h4><p>If we’re dealing with images, we can usually get better performance by building an Autoencoder using <a href="https://github.com/udacity/deep-learning/blob/master/autoencoder/Simple_Autoencoder_Solution.ipynb">convolutional layers</a>. The encoder part of the network will be a typical convolutional pyramid, where each convolutional layer is followed by a max-pooling layer to reduce the dimensionality:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/804/0*ySf63sOKQ_hk3DOq.png" /></figure><p>Here the Decoder contains a deconvolutional process, that adds new data from what it has learned. This process can be prone to visual artefacts, as <a href="http://distill.pub/2016/deconv-checkerboard/">this Distill article</a> explains, so resizing using nearest neighbour or bilinear interpolation (upsampling) can be a better choice than standard deconvolutional layers.</p><p>Another implementation is the <a href="https://www.youtube.com/watch?v=3-UDwk1U77s">Variational Autoencoder</a> (VAE), which incorporates Bayesian Inference, so that the compressed representation produced is a probability distribution.</p><p><strong>See also:</strong></p><ul><li><a href="http://www.deeplearningbook.org/contents/autoencoders.html">Autoencoders Chapter </a>— in the Deep Learning Book</li><li><a href="https://www.quora.com/What-is-the-difference-between-a-neural-network-and-an-autoencoder-network">“What’s the difference between a neural network and an autoencoder?</a>”</li></ul><h4><a href="https://medium.com/deeper-learning/a-glossary-of-deep-learning-9cb6292e087e">SEE GLOSSARY INDEX</a></h4><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1044ec82c300" width="1" height="1" alt=""><hr><p><a href="https://medium.com/deeper-learning/glossary-of-deep-learning-autoencoder-1044ec82c300">Glossary of Deep Learning: Autoencoder</a> was originally published in <a href="https://medium.com/deeper-learning">Deeper Learning</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Glossary of Deep Learning: Dynamic Memory Networks]]></title>
            <link>https://medium.com/deeper-learning/glossary-of-deep-learning-dynamic-memory-networks-1ada00d0400a?source=rss----27739ac039da---4</link>
            <guid isPermaLink="false">https://medium.com/p/1ada00d0400a</guid>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[question-answering]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Jaron Collis]]></dc:creator>
            <pubDate>Mon, 15 May 2017 13:17:29 GMT</pubDate>
            <atom:updated>2026-06-12T11:23:11.282Z</atom:updated>
            <cc:license>https://creativecommons.org/licenses/by-sa/4.0/</cc:license>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*C_IhH7boYo5fvMLf." /><figcaption>The component parts of a Dynamic Memory Network (from <a href="https://arxiv.org/pdf/1506.07285.pdf">Kumar et al</a>, which explains the meaning of the weights signified by arrows)</figcaption></figure><p>A <em>Dynamic Memory Network</em> (DMN) is a neural network architecture optimised for question-answering (QA) problems. Given a training set of input sequences (knowledge) and questions, it can form episodic memories, and use them to generate relevant answers.</p><p>Whilst classic Encoder-Decoder (Seq2Seq) models can solve QA problems, their performance is limited by the small size of their ‘memory’ — this is what’s encoded by their hidden states and weights, and reflects the information that’s passed between encoder and decoder. This limitation becomes especially apparent when dealing with very long sequences of data, as we might find in sources like books or videos, where the salient facts might occur a long time apart in very different contexts.</p><p>This limitation can be solved by storing multiple hidden states, and then using a strategy called an <a href="https://blog.heuritech.com/2016/01/20/attention-mechanism/">Attention Mechanism</a> to choose between them. This allows the network to refer back to the input sequence, instead of forcing it to encode all information into one fixed-length vector like Seq2Seq does. This post on <a href="http://www.wildml.com/2016/01/attention-and-memory-in-deep-learning-and-nlp">Attention and Memory in Deep Learning</a> has a good explanation of why this is so helpful.</p><p>DMNs refine the attention mechanism, so that questions trigger an iterative attention process which allows the model to condition its attention on the inputs and the result of previous iterations.</p><p>The reference papers for DMNs are <a href="http://proceedings.mlr.press/v48/xiong16.pdf">Xiong et al 2016</a> and <a href="https://arxiv.org/pdf/1506.07285.pdf">Kumar et al 2015,</a> they describe an architecture with the following components:</p><p>The <strong>Semantic Memory Module</strong> (analogous to a knowledge base) consists of pre-trained GloVe vectors that are used to create sequences of <a href="https://www.quora.com/What-is-word-embedding-in-deep-learning">word embeddings</a> from input sentences. These vectors will act as inputs to the model.</p><p>The <strong>Input Module</strong> processes the input vectors associated with a question into a set of vectors termed <em>facts</em>. This module is implemented using a GRU, or <a href="http://www.wildml.com/2015/10/recurrent-neural-network-tutorial-part-4-implementing-a-grulstm-rnn-with-python-and-theano/">Gated Recurrent Unit,</a> similar to an LSTM but simpler, and so more computationally efficient. The GRU enables the network to learn if the sentence currently under consideration is relevant or nothing to do with the answer.</p><p>The <strong>Question Module</strong> processes the question word by word, and outputs a vector using the same GRU as the input module, and the same weights. Both facts and questions are encoded as embeddings.</p><p>The <strong>Episodic Memory Module</strong> receives the fact and question vectors extracted from the input and encoded as embeddings. This uses a process inspired by the brain’s hippocampus, which can retrieve temporal states that are triggered by some response, like sights or sounds.</p><p>On silicon, episodic memory is composed of two nested GRUs.</p><p>The inner GRU (the lower line in diagram above) generates what are called episodes. The outer GRU (upper line in diagram) generates the final memory vector by working over a sequence of these episodes.</p><p>Episode generation involves the inner GRU passing over the facts from the input module. When updating its inner state, it takes into account the output of an attention function on the current fact — which gives a score between 0 and 1 to each fact. This allows the GRU to ignore facts with low scores.</p><p>After each full pass on all the facts, the inner GRU outputs an episode which is then fed to the outer GRU, whose state has been initialised by the question vector.</p><p>The reason the DMN uses multiple episodes is so the model can learn what part of a sentence it should pay attention to, and so realise after one pass that something else was important. Multiple passes allow it to gather increasingly relevant information.</p><p>Finally, the <strong>Answer Module</strong> generates an appropriate response. By the final pass, the episodic memory should contain all the information required to answer the question. This module uses another GRU, trained with the cross-entropy error classification of the correct sequence, which can then be converted back to natural language.</p><p><strong>See also:</strong></p><ul><li><a href="https://www.youtube.com/watch?t=637s&amp;v=Qf0BqEk5n3o">A talk by Stephen Merity </a>(one of the creators of DMNs)</li><li><a href="https://github.com/YerevaNN/Dynamic-memory-networks-in-Theano">YerevaNN’s Theano DMN implementation</a></li><li><a href="https://github.com/domluna/memn2n">Domluna’s Tensorflow End-To-End Memory Network</a></li><li><a href="https://github.com/ethancaballero/Improved-Dynamic-Memory-Networks-DMN-plus">Ethan Caballero’s DMN+ in Theano/Keras</a></li></ul><h3>2026 Update</h3><p>It’s fascinating to look back at this post, written just a month before the <a href="https://arxiv.org/abs/1706.03762">Attention Is All You Need paper</a> introduced the Transformer architecture and revolutionised the capabilities of neural networks. There are 3 key functional differences between Dynamic Memory Networks and Transformers:</p><ol><li><strong>DMNs processed sequentially. Transformers process in parallel.</strong> <br>DMNs were Recurrent Neural Networks, built on top of Gated Recurrent Units. In its input and input-fusion modules, tokens (or sentences) are processed sequentially, one step at a time. The representation of the 10th word depends on the hidden state of the 9th word.<br>Transformers abandon recurrence and processes all tokens in a sequence simultaneously. To maintain a sense of word order without sequential processing, it uses <em>Positional Encodings</em> added directly to the input embeddings.</li><li><strong>DMNs and Transformers calculated attention differently.</strong><br>DMNs measured the similarity between a Fact vector (input text), a Question vector, and the Current Memory state. The attention weights decided how much a specific fact should modify the episodic memory.<br>Transformers uses Scaled Dot-Product Attention deployed via Multi-Head Attention. Every token attends to every other token via <em>Self-Attention</em> (cloned into Queries, Keys, and Values). So instead of calculating attention against a static question, the Transformer continuously re-contextualizes the entire input token space simultaneously across multiple ‘heads’, or perspectives.</li><li><strong>DMNs and Transformers represent memories differently.</strong><br>DMNs maintain an Episodic Memory state vector that’s initialised by the question and updated after each iterative pass over the input facts.<br>Transformers have no explicit, segregated “memory” vector. Instead, the “memory” of the context is implicitly distributed across the activation states of the tokens themselves as they pass through the multi-head attention layers.</li></ol><p>The DMN was an elegant, modular, system designed to simulate a human cognition — read, hold in short term memory, scan the context, focus on clues and update your mental state, then formulate your answer. This all seemed very intuitive, but like Feynman famously said, aircraft don’t flap their wings. Engineered machines work differently to biology, that’s why they’re so much more efficient.</p><p>The conceptual design of the Transformer abandoned this biological simulation of recurrence in favor of massive mathematical parallelisation, letting attention do all the heavy lifting across highly stacked layers.</p><p>DMNs represent one of those interesting intermediate forms, which pushed forward the frontier of getting software to think, before being superceded by a vastly more capable and scalable solution. Then being mostly forgotten. Science often advances like this.</p><p>The DMN was highly articulate piece of engineering, but it arrived right at the evolutionary inflection point where architectural elegance was overtaken by hardware scalability. In AI history, DMNs are part of a fascinating fossil record of architectures that were all trying to solve a fundamental flaw in standard RNNs: the inability to retain a stable, addressable memory over long sequences.</p><p>The DMN was a highly modular, anthropomorphic approach to “thinking”, that’s why I liked it enough to write the original post. But relying on recurrence (RNN GRUs) to pass information through time and across memory hops built a sequential bottleneck into the software. You couldn’t easily compute step <em>t+1</em> until step <em>t</em> was finished.</p><p>When the Transformer paper dropped, it argued there was no need to mimic the sequential, iterative nature of human reading architecturally. By flattening the sequence, treating memory implicitly, and letting matrix multiplication handle everything in parallel, the Transformer unlocked the ability to scale across massive GPU clusters. How hardware evolution inspires jumps in AI is something I talk about <a href="https://jaroncollis.medium.com/the-means-of-production-fb2699fb3049">in another essay</a>.</p><p>Science shifted from building intricate cognitive machines (like DMNs) to building massive computational engines (like Transformers) because the latter could digest the entire internet. The elegant intermediate forms get left behind, but they are exactly what proved that multi-stage attention and memory routing were the keys to reasoning in the first place.</p><h4><a href="https://medium.com/deeper-learning/a-glossary-of-deep-learning-9cb6292e087e">SEE GLOSSARY INDEX</a></h4><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1ada00d0400a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/deeper-learning/glossary-of-deep-learning-dynamic-memory-networks-1ada00d0400a">Glossary of Deep Learning: Dynamic Memory Networks</a> was originally published in <a href="https://medium.com/deeper-learning">Deeper Learning</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Glossary of Deep Learning: Error]]></title>
            <link>https://medium.com/deeper-learning/glossary-of-deep-learning-error-1f70d9bb88e9?source=rss----27739ac039da---4</link>
            <guid isPermaLink="false">https://medium.com/p/1f70d9bb88e9</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Jaron Collis]]></dc:creator>
            <pubDate>Wed, 26 Apr 2017 12:13:32 GMT</pubDate>
            <atom:updated>2017-04-26T12:13:31.703Z</atom:updated>
            <cc:license>https://creativecommons.org/licenses/by-sa/4.0/</cc:license>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aQ85HczVf-jqXG5O1ZQRkA.jpeg" /><figcaption>Don’t think of errors as misses or failures, but as indications of how accuracy can be improved</figcaption></figure><p>If you’re a software engineer, you quickly come to view errors as bad. Each error represents some misunderstanding, you thought the program would behave in some way, but it didn’t. Now some aspect of its state is inconsistent with how it’s meant to be. So your program no longer functions correctly - and it fails.</p><p>But in Machine Learning (and Deep Learning), error represents a very different concept, one that’s vital to the learning process itself. Instead of being a fault or a failure, error is a signal, an insight into how the accuracy of a model might be improved.</p><p>Error can take many forms, in classification problems it might be a measure of true and false positives (hits and correct rejections) compared to the number of <a href="https://en.wikipedia.org/wiki/Type_I_error">false positives</a> (false alarms) and <a href="https://en.wikipedia.org/wiki/Type_II_error">false negatives</a> (misses). You might see these categories depicted quite intuitively in a <a href="https://en.wikipedia.org/wiki/Confusion_matrix">Confusion Matrix</a>.</p><p>Positives and Negatives also provide the basis for two common information retrieval measures:</p><ul><li><strong>Precision</strong> = True Positives / True Positives + False Positives</li><li><strong>Recall</strong> = True Positives / True Positives + False Negatives</li></ul><p>Together these used to calculate a weighted average called the <a href="https://en.wikipedia.org/wiki/F1_score">F1 score</a>, a numerical value often quoted as a measure of classification accuracy.</p><h4>Error is a Signal</h4><p>In Regression, error has a different meaning again. These problems involve a model that makes predictions on continuous data, so accuracy is evaluated by how close its predictions are to what was expected (in an unseen data set). This means the errors are also continuous values, proximities to the ideal rather than hits or misses.</p><p>So we aim to create a model with as small a residual error as possible, using measures like <a href="https://en.wikipedia.org/wiki/Mean_squared_error">Mean Squared Error</a>. The implementation is pretty simple, comparing each data point against its corresponding point on a putative line of best fit, the difference between the two being the error:</p><pre># the error of a line, where m is slope, b is y-intercept<br>def compute_error(b, m, data):<br>    totalError = 0 <br>    for i in range(0, len(data)): <br>        x = data[i, 0] <br>        y = data[i, 1] <br>        totalError += (y — (m * x + b)) ** 2 <br>    return totalError / float(len(data))</pre><p>The error this function returns is the signal, it tells us how imperfect the line (defined by the slope and intercept we specified) actually is. Next time, we’ll try a different line, and hope to get a little closer to the perfect fit.</p><p>This idea of comparing the predicted outcomes with the known expected outcome is also used in Neural Networks to compute a cost function. A commonly used cost function is Cross-Entropy, a measure that allows a neural network’s mistakes to be evaluated. This is vital for training, as it allows the model to quickly learn when its predictions are decisively wrong. After all, it is difficult to learn if we don’t know we’re making mistakes.</p><p>At this point, if you’re intrigued in how the nodes of a neural network represent rightness and wrongness, you should definitely take a few minutes out to read Michael Nielsen’s excellent <a href="http://neuralnetworksanddeeplearning.com/chap3.html">explanation of Cross-Entropy</a>.</p><p>In contrast to procedural programs, which tend to either work or fail, the performance of a neural net is referred to its <em>Loss</em>. This value, calculated over the entire training set for all the inputs and all possible labels, reflects how well the weights and biases that constitute the trained network provide accurate predictions.</p><p>So when attempting to improve the accuracy of a neural net, it’s less about quashing errors in the code, and more about finding ways of minimising the total loss. This tends to involve strategies like <a href="http://neuralnetworksanddeeplearning.com/chap1.html#learning_with_gradient_descent">Gradient Descent</a> to take the derivative of the loss with respect to the parameters, then follow that derivative, taking steps backwards, and repeating until we get to the bottom of the curve — indicating a minima where the loss is at its smallest.</p><p>Or, put another way, error in machine learning is the crucial insight that allows us to turn the process of problem solving into one of numerical optimisation.</p><h4>Error and Complexity</h4><p>Describing problem solving as a kind of numerical optimisation might make the process of machine learning sound like it’s basically just finding the solution of some particularly complicated equations. But it isn’t as simple as that, and the clue to the reason is in the name: machine learning needs to be able to <em>learn </em>— to be able to generalise from what’s been seen in the training data, and apply it to examples it’s never seen before.</p><p>Ensuring we have sufficient flexibility to generalise introduces two potential sources of error: being too simplistic, and being too complicated. You’ll see these referred to as <em>bias</em> and <em>variance</em> respectively, and ideally we’d like to minimise both, but in practice, it’s <a href="https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff">a trade-off</a>. Bias is like a misaligned sight, whilst variance is like not aiming at all.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/572/1*8S1HBnTTZJDUTiYfnmJq3Q.png" /><figcaption>Error as a target shooting metaphor, showing two types of inaccuracy: Bias is systematic, Variance is erratic.</figcaption></figure><p><a href="https://en.wikipedia.org/wiki/Bias_of_an_estimator"><strong>Bias</strong></a> is the tendency to keep getting the same results incorrectly. It’s an error that arises when a model is too simple, and so fails to represent the complexity of the underlying data. This can cause an algorithm to miss relevant relations between salient features and the expected outputs, despite having more than enough training data — a problem called underfitting.</p><p>Bias is a consequence of an oversimplified model. An example might be a classifier that can only partition objects by colour, but is too simplistic to also classify by shape, or other discriminating features. High Bias tends to show up as a high residual error in the training set — if it fails to fall as you add more training data, the model is likely not sophisticated enough.</p><p>Underfitting is depicted as the orange line in the diagram below, which shows how 3 different classifiers might partition a 2-dimensional problem space. The underfitted model simply considers red dots to be more likely in the lower area than blue dots, and fails to capture any subtleties.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*S-od0LnVxavOlUJCYftmoA.png" /><figcaption>Three Models: Underfitting (orange line). Good fit (black line). And Overfitting (green line).</figcaption></figure><p>Now look at the Good Fit line (in black), the really interesting aspect to notice here is how it still has errors: the dividing line has coloured dots on the ‘wrong’ side. Yet it’s also possible to draw another wriggly line (shown in green) with no error at all, one that gets everything completely right. So, here’s a question for you: would we want to?</p><p>The answer is no. And the reason is variance. The real world is a noisy and unpredictable place. If we overcommit to the exact patterns seen during training, how will we be able to handle new and surprising situations?</p><p><a href="https://en.wikipedia.org/wiki/Variance"><strong>Variance</strong></a> is tendency to learn random things that seem significant, rather than the truly important salient features. An analogy is a student who memorises the answers to practice tests, they may score well initially, only to do badly on the final test, because they never really understood the subject.</p><p>This presents a challenge known as <a href="https://en.wikipedia.org/wiki/Overfitting"><em>Overfitting</em></a>, and is often encountered when the training set lacks sufficient diversity. A model can end up being trained on just obvious features or noise, and never be given the chance to train on the full range of examples it might encounter. Ideally, a model should perform as well on data it’s never seen before, as it does on data it’s been trained on. If the error is significantly bigger in the unseen test set, that’s an indication the model might be overfitting.</p><h4>Keep It Simpler</h4><p>Overfitting not only occurs when the training data is inadequate, but also when the model becomes too complicated. There’s many candidates for a golden rule in machine learning, but a good one is to always <a href="https://en.wikipedia.org/wiki/Occam%27s_razor">prefer the simplest explanation</a>. Any model that requires a dozen features to correctly predict is obviously going to be more brittle than one that requires only a few, as it would be reliant on the presence of many more clean and unambiguous features. Why? Because the real-world is noisy and messy.</p><p>As a result, there’s a trade-off when it comes to model complexity. Too simple and it won’t be able to learn the subtleties of the data, and it underfits. Too complex and the model begins to associate too much predictive power with patterns that might be one-offs, and it overfits.</p><p>So part of The Art of Machine of Learning is to <a href="http://scott.fortmann-roe.com/docs/BiasVariance.html">find the sweet spot</a> that minimises bias and variance by finding the right model complexity. That means choosing not only the right features, but no more of them than absolutely necessary.</p><p>Likewise, in deep learning, you can’t just keep adding hidden layers in the hope of boosting performance. Like Einstein once said:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dOZ-TUWPxTIpeIdagUkhcA.jpeg" /><figcaption>Albert knows. [<a href="https://www.flickr.com/photos/alexander_johmann/7296129414">Source</a>]</figcaption></figure><p>The right level of complexity gives your model just enough scope for error to learn. But one thing you can never have too much of is training data. Adding data doesn’t increase model complexity, so the more the merrier. <a href="http://avc.com/2016/12/data-wins/">Data wins</a>.</p><h4>Learning from Mistakes</h4><p>For those coming to machine learning from a software engineering background, understanding the concept of error is one of those transformative moments, when something clicks in your mental model, and everything starts to make a lot more sense. It’s like when you’re first learning to program, and you finally grasp what <a href="https://en.wikipedia.org/wiki/Abstraction_(software_engineering)">abstraction</a> is, and suddenly the bafflingly complicated task of writing software is radically simplified.</p><p><em>Error is fundamental to machine learning. After all, isn’t the very hallmark of intelligence the ability to recognise and learn from your own mistakes?</em></p><p><strong>See also:</strong></p><ul><li><a href="https://medium.com/@jaroncollis/how-i-learned-to-stop-worrying-and-love-the-error-6c4c488c97da">How I Learned to… Love the Error</a><strong> </strong>— a longer version of this post</li><li><a href="http://neuralnetworksanddeeplearning.com/chap3.html">An Explanation of Cross-Entropy</a> — Michael Nielsen</li><li><a href="http://scott.fortmann-roe.com/docs/BiasVariance.html">The Bias-Variance Trade-off </a>—Scott Fortmann-Roe</li><li><a href="https://www.quora.com/What-is-generalization-in-machine-learning">What is Generalisation in Machine Learning?</a> — Quora</li></ul><h4><a href="https://medium.com/deeper-learning/a-glossary-of-deep-learning-9cb6292e087e">SEE GLOSSARY INDEX</a></h4><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1f70d9bb88e9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/deeper-learning/glossary-of-deep-learning-error-1f70d9bb88e9">Glossary of Deep Learning: Error</a> was originally published in <a href="https://medium.com/deeper-learning">Deeper Learning</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Glossary of Deep Learning: Word Embedding]]></title>
            <link>https://medium.com/deeper-learning/glossary-of-deep-learning-word-embedding-f90c3cec34ca?source=rss----27739ac039da---4</link>
            <guid isPermaLink="false">https://medium.com/p/f90c3cec34ca</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Jaron Collis]]></dc:creator>
            <pubDate>Wed, 19 Apr 2017 06:06:05 GMT</pubDate>
            <atom:updated>2023-04-05T15:12:22.023Z</atom:updated>
            <cc:license>https://creativecommons.org/licenses/by-sa/4.0/</cc:license>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*52X2L01wpUjy39lIjofC7g.jpeg" /><figcaption>A plot of word embeddings in English and German. The semantic equivalence of words has been inferred by their context, so similar meanings are co-located. This is because the relative semantics of words are consistent, whatever the language. [Source: <a href="https://stanford.edu/~lmthang/bivec/">Luong et al</a>]</figcaption></figure><p>The Big Idea of Word Embedding is to turn text into numbers.</p><p>This transformation is necessary because many machine learning algorithms (including deep nets) require their input to be vectors of continuous values; they just won’t work on strings of plain text.</p><p>So a natural language modelling technique like Word Embedding is used to map words or phrases from a vocabulary to a corresponding vector of real numbers. As well as being amenable to processing by learning algorithms, this vector representation has two important and advantageous properties:</p><ul><li><strong>Dimensionality Reduction</strong> — it is a more efficient representation</li><li><strong>Contextual Similarity</strong> — it is a more expressive representation</li></ul><p>If you’re familiar with the <a href="https://www.kaggle.com/c/word2vec-nlp-tutorial/details/part-1-for-beginners-bag-of-words">Bag of Words</a> approach, you’ll know it results in huge, very sparse one-hot encoded vectors, where the dimensionality of the vectors representing each document is equal to the size of the supported vocabulary. Word Embedding aims to create a vector representation with a much lower dimensional space. These are called <em>Word Vectors</em>.</p><p>Word Vectors are used for semantic parsing, to extract meaning from text to enable natural language understanding. For a language model to be able to predict the meaning of text, it needs to be aware of the contextual similarity of words. For instance, that we tend to find fruit words (like apple or orange) in sentences where they’re grown, picked, eaten and juiced, but wouldn’t expect to find those same concepts in such close proximity to, say, the word aeroplane.</p><p>The vectors created by Word Embedding preserve these similarities, so words that regularly occur nearby in text will also be in close proximity in vector space. For examples of why this is useful, check out <a href="https://blog.acolyer.org/2016/04/21/the-amazing-power-of-word-vectors/">The amazing power of word vectors</a> or this intro to <a href="https://www.kaggle.com/c/word2vec-nlp-tutorial/details/part-2-word-vectors">Distributed Word Vectors</a> on Kaggle.</p><p>So an answer to “What is word embedding?” is: it’s a means of <strong>building a low-dimensional vector representation from corpus of text, which preserves the contextual similarity of words.</strong></p><h4>Introducing Word2Vec</h4><p>Now we know what it is, how does it work? If you’ve encountered dimensionality reduction before you’ll know this is typically achieved using an unsupervised learning algorithm. And this is the approach used by one of the best known algorithms for producing word embeddings: <a href="https://www.quora.com/What-are-some-good-resources-to-learn-word2vec">word2vec</a>.</p><p>There are actually two ways to implement word2vec, CBOW (Continuous Bag-Of-Words) and Skip-gram.</p><p>In CBOW we have a window around some target word and then consider the words around it (its context). We supply those words as input into our network and then use it to try to predict the target word.</p><p>Skip-gram does the opposite, you have a target word, and you try to predict the words that are in the window around that word, i.e. predict the context around a word.</p><p>The input words are passed in as one-hot encoded vectors. This will go into a hidden layer of linear units, then into a softmax layer to make a prediction. The idea here is to train the hidden layer weight matrix to find efficient representations for our words. This weight matrix is usually called the <strong>embedding</strong> matrix, and can be queried as a look-up table.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*kx5_UXWs7Q_c071d." /><figcaption>The word2vec architecture consists of a hidden layer and an output layer. [<a href="http://mccormickml.com/2016/04/19/word2vec-tutorial-the-skip-gram-model/">Source</a>]</figcaption></figure><p><a href="https://www.quora.com/profile/Xin-Rong-1">Xin Rong</a> has created a very neat visual demo that shows <a href="https://ronxin.github.io/wevi/">how word embeddings are trained</a>, and has given an excellent<a href="https://www.youtube.com/watch?v=D-ekE-Wlcds"> accompanying talk</a>.</p><p>The embedding matrix has a size of the number of words by the number of neurons in the hidden layer (the embed size). So, if you have 10,000 words and 300 hidden units, the matrix will have size 10,000×300 (as we’re using one-hot encoded vectors for our inputs). Once computed, getting the word vector is a speedy O(1) lookup of corresponding row of the results matrix:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/621/0*IFv_QtwBNHfGy1Tm." /><figcaption>So, for the word that’s the 4th entry in the vocabulary, its vector is (10,12,19).</figcaption></figure><p>So each word has an associated vector, hence the name: word2vec.</p><p>The embed size, which is the size of the hidden layer and thus the number of features that represent similarities between words, tends to be much smaller than the total number of unique words in the vocabulary, (hundreds rather than tens of thousands). The embed size used is a trade-off: more features mean extra computational complexity, and so longer run-times, but also allow more subtle representations, and potentially better models.</p><p>An interesting feature of word embeddings is that because they’re numerical representations of contextual similarities between words, they can be manipulated arithmetically. The classic example is subtracting the ‘notion’ of “King” from “Man” and adding the notion of “Woman”. The answer will depend on your training set, but you’re likely to see one of the top results being the word “Queen”. This <a href="http://p.migdal.pl/2017/01/06/king-man-woman-queen-why.html">great post</a> explains why.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*1ndbQpbmRrzZWTjO.png" /><figcaption>Word Embeddings are similarities based on context, which might be gender, tense, geography or something else entirely. The lines shown are just mathematical vectors, so see how you could move ‘across’ in embedding space from “Man” to “Queen” by subtracting “King” and adding “Woman”.</figcaption></figure><p>Another word embedding algorithm worth knowing about is GloVe, which works slightly differently by accumulating counts of co-occurrences, (see <a href="https://www.quora.com/How-is-GloVe-different-from-word2vec">How is GloVe different from word2vec?</a>)</p><p>So where does deep learning come in?</p><p>Keep in mind that Word2vec is a two-layer shallow neural net, and so <a href="https://www.quora.com/Can-word2vec-be-considered-deep-learning">is not itself an example of deep learning</a>. But techniques like Word2vec and GloVe can turn raw text into a numerical form that deep nets can understand, for instance, using <a href="http://deeplearning.net/tutorial/rnnslu.html">Recurrent Neural Networks with Word Embeddings</a>.</p><p>In summary then, the purpose of word embedding is to turn words into numbers, which algorithms like deep learning can then ingest and process, to formulate an understanding of natural language. <a href="http://deeplearning4j.org/word2vec">Deeplearning4J</a> has a lovely quote by the novelist EL Doctorow, who expresses this idea quite poetically:</p><blockquote><em>It’s like numbers are language, like all the letters in the language are turned into numbers, and so it’s something that everyone understands the same way. You lose the sounds of the letters and whether they click or pop or touch the palate, or go ooh or aah, and anything that can be misread or con you with its music or the pictures it puts in your mind, all of that is gone, along with the accent, and you have a new understanding entirely, a language of numbers, and everything becomes as clear to everyone as the writing on the wall. So as I say there comes a certain time for the reading of the numbers.</em></blockquote><p>Word embeddings are just language, as numbers.</p><p>Now, what do you think might happen if every word humanity has ever written is turned into <a href="https://medium.com/@jaroncollis/ai-adventures-in-the-library-of-babel-7cc692fe5a97">a vast multi-dimensional possibility space</a>?</p><p><strong>See also:</strong></p><ul><li>A <a href="http://mccormickml.com/2016/04/19/word2vec-tutorial-the-skip-gram-model/">conceptual overview</a> of word2vec from Chris McCormick</li><li><a href="https://arxiv.org/pdf/1301.3781.pdf">word2vec paper</a> and <a href="http://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality.pdf">NIPS paper</a> with improvements— Mikolov et al</li><li>An<a href="http://www.thushv.com/natural_language_processing/word2vec-part-1-nlp-with-deep-learning-with-tensorflow-skip-gram/"> implementation of word2vec</a> — Thushan Ganegedara</li><li>TensorFlow<a href="https://www.tensorflow.org/tutorials/word2vec"> word2vec tutorial</a></li><li><a href="https://blog.acolyer.org/2016/04/21/the-amazing-power-of-word-vectors/">The amazing power of word vectors</a> — <a href="https://medium.com/u/db8c66c37f82">Adrian Colyer</a></li><li>An introduction to <a href="https://www.kaggle.com/c/word2vec-nlp-tutorial/details/part-2-word-vectors">Distributed Word Vectors</a> — Kaggle</li><li><a href="https://ronxin.github.io/wevi/">An online word embedding visualiser</a> — <a href="https://www.quora.com/profile/Xin-Rong-1">Xin Rong</a></li><li><a href="http://p.migdal.pl/2017/01/06/king-man-woman-queen-why.html">How word similarities lead to analogies</a> — <a href="http://p.migdal.pl/">Piotr Migdał</a></li><li><a href="https://www.quora.com/What-is-word-embedding-in-deep-learning/answer/Jaron-Collis">What is word embedding in deep learning?</a> — my original Quora post</li></ul><h4><a href="https://medium.com/deeper-learning/a-glossary-of-deep-learning-9cb6292e087e">SEE GLOSSARY INDEX</a></h4><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f90c3cec34ca" width="1" height="1" alt=""><hr><p><a href="https://medium.com/deeper-learning/glossary-of-deep-learning-word-embedding-f90c3cec34ca">Glossary of Deep Learning: Word Embedding</a> was originally published in <a href="https://medium.com/deeper-learning">Deeper Learning</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Glossary of Deep Learning: Bias]]></title>
            <link>https://medium.com/deeper-learning/glossary-of-deep-learning-bias-cf49d9c895e2?source=rss----27739ac039da---4</link>
            <guid isPermaLink="false">https://medium.com/p/cf49d9c895e2</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Jaron Collis]]></dc:creator>
            <pubDate>Fri, 14 Apr 2017 11:06:50 GMT</pubDate>
            <atom:updated>2017-04-14T14:40:48.093Z</atom:updated>
            <cc:license>https://creativecommons.org/licenses/by-sa/4.0/</cc:license>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*-fy5FhuunyvTHJpj.png" /><figcaption>Three sigmoid curves — the same input data, but with different biases [<a href="http://stackoverflow.com/questions/2480650/role-of-bias-in-neural-networks">Source</a>]</figcaption></figure><p>The <a href="https://medium.com/deeper-learning/glossary-of-deep-learning-activation-function-20262327a907">activation</a> of a node in a neural network is determined by the following:</p><pre>output = activation_function(dot_product(weights, inputs) + bias)</pre><p>This means when calculating the output of a node, the inputs are multiplied by weights, and a bias value is added to the result. The bias value allows the activation function to be shifted to the left or right, to better fit the data. Hence changes to the weights alter the steepness of the sigmoid curve, whilst the bias offsets it, shifting the entire curve so it fits better. Note also how the bias only influences the output values, it doesn’t interact with the actual input data.</p><p>You can think of the bias as a measure of how easy it is to get a node to fire. For a node with a large bias, the output will tend to be intrinsically high, with small positive weights and inputs producing large positive outputs (near to 1). Biases can be also negative, leading to sigmoid outputs near to 0. If the bias is very small (or 0), the output will be decided by the values of weights and inputs alone.</p><p>Bear in mind, though, that the bias in a neural network nodes is not equivalent to the<a href="http://neuralnetworksanddeeplearning.com/chap1.html#perceptrons"> threshold of a perceptron</a>, which only outputs 1 if sufficient input is supplied. Neurons don’t have binary silent/fire thresholds, instead they have activation functions that produce a non-linear output between 0 and 1. So the role of bias isn’t to act as a threshold, but to help ensure the output best fits the incoming signal.</p><p>Biases are tuned alongside weights by learning algorithms such as gradient descent. Where biases differ from weights is that they are independent of the output from previous layers. Conceptually bias is caused by input from a neuron with a fixed activation of 1, and so is updated by subtracting the just the product of the delta value and learning rate.</p><p>Typically biases are <a href="http://cs231n.github.io/neural-networks-2/">initialised</a> to be zero, since asymmetry breaking is provided by the small random numbers in the weights (see Weight Initialisation).</p><p>Note: the term <a href="https://en.wikipedia.org/wiki/Bias_of_an_estimator"><strong>Bias</strong></a> is also used to refer to the systematic errors in a model, the tendency to keep getting the same results incorrectly. This tends to arise when a model is too simple, and so fails to represent the complexity of the underlying data, causing an algorithm to miss relevant relations between salient features and the expected outputs, despite having more than enough training data — a problem called underfitting. Something <a href="https://medium.com/@jaroncollis/how-i-learned-to-stop-worrying-and-love-the-error-6c4c488c97da">I explain here</a>.</p><p><strong>See also:</strong></p><ul><li><a href="http://neuralnetworksanddeeplearning.com/chap1.html">http://neuralnetworksanddeeplearning.com/chap1.html</a></li><li><a href="http://stackoverflow.com/questions/2480650/role-of-bias-in-neural-networks">The role of bias in neural networks </a>— Stack Overflow</li><li><a href="https://www.quora.com/What-is-bias-in-artificial-neural-network">“What is bias in a neural network?”</a> — Quora</li></ul><h4><a href="https://medium.com/deeper-learning/a-glossary-of-deep-learning-9cb6292e087e">SEE GLOSSARY INDEX</a></h4><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cf49d9c895e2" width="1" height="1" alt=""><hr><p><a href="https://medium.com/deeper-learning/glossary-of-deep-learning-bias-cf49d9c895e2">Glossary of Deep Learning: Bias</a> was originally published in <a href="https://medium.com/deeper-learning">Deeper Learning</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Glossary of Deep Learning]]></title>
            <link>https://medium.com/deeper-learning/a-glossary-of-deep-learning-9cb6292e087e?source=rss----27739ac039da---4</link>
            <guid isPermaLink="false">https://medium.com/p/9cb6292e087e</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[neural-networks]]></category>
            <dc:creator><![CDATA[Jaron Collis]]></dc:creator>
            <pubDate>Wed, 12 Apr 2017 22:09:58 GMT</pubDate>
            <atom:updated>2026-06-11T09:38:51.394Z</atom:updated>
            <cc:license>https://creativecommons.org/licenses/by-sa/4.0/</cc:license>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9Ox2RVniNkuw9bdFCSKhcA.jpeg" /><figcaption>The Pleiades (as seen by the Hubble Telescope) [<a href="https://en.wikipedia.org/wiki/File:Pleiades_large.jpg">Wikipedia</a>]</figcaption></figure><p>This is a notebook, a place to record what I learn as I explore a new frontier of applied artificial intelligence. My aim is to explain the key concepts in simple language, because complexity doesn’t have to mean complicated.</p><h3>Essays</h3><h4><a href="https://jaroncollis.medium.com/ai-adventures-in-the-library-of-babel-7cc692fe5a97">AI Adventures in The Library of Babel</a></h4><ul><li>On Generative AI and the Automated Search for Meaning</li></ul><h4><a href="https://jaroncollis.medium.com/you-can-not-split-the-atom-in-your-head-43800769bef8">You Can Not Split The Atom In Your Head</a></h4><ul><li>On AI Risk, Technology Anxiety, and starting before we feel ready</li></ul><h4><a href="https://jaroncollis.medium.com/the-means-of-production-fb2699fb3049">The Means of Production</a></h4><ul><li>On sovereign intelligence: how unmetered AI ended up in your hands</li></ul><h4><a href="https://jaroncollis.medium.com/the-socratic-engineer-31931b5150a6">The Socratic Engineer</a></h4><ul><li>On the value of thinking when the machines know everything</li></ul><h3>Foundation Concepts</h3><ul><li><a href="https://medium.com/deeper-learning/glossary-of-deep-learning-activation-function-20262327a907"><strong>Activation Function</strong></a><strong> </strong>—why we make outputs non-linear</li><li><a href="https://medium.com/deeper-learning/glossary-of-deep-learning-autoencoder-1044ec82c300"><strong>Autoencoder </strong></a>— how to learn the essence of your input</li><li><a href="https://medium.com/deeper-learning/glossary-of-deep-learning-backpropagation-e6d748d36a0e"><strong>Backpropagation</strong></a> — how networks learn from mistakes</li><li><a href="https://medium.com/deeper-learning/glossary-of-deep-learning-batch-normalisation-8266dcd2fa82"><strong>Batch Normalisation</strong></a> — inter-layer transforms improve stability</li><li><a href="https://medium.com/deeper-learning/glossary-of-deep-learning-bias-cf49d9c895e2"><strong>Bias</strong></a> — shifts the activation function to better fit the data</li><li><a href="https://medium.com/deeper-learning/glossary-of-deep-learning-dynamic-memory-networks-1ada00d0400a"><strong>Dynamic Memory Networks</strong></a> (DMNs)</li><li><a href="https://medium.com/deeper-learning/glossary-of-deep-learning-word-embedding-f90c3cec34ca"><strong>Embedding</strong></a> — how to turn words into numbers</li><li><a href="https://medium.com/deeper-learning/glossary-of-deep-learning-error-1f70d9bb88e9"><strong>Error </strong></a>—how measures of inaccuracy enable learning</li><li>…</li></ul><h4>About The Glossary</h4><p>The image of <em>The Pleiades</em> above provides what I consider a fitting visual metaphor for the nodes of a neural network. Some activating brilliantly, others dimly. All widely distributed, yet still interacting subtly. Considered individually, each is just a blazing star, but collectively, at the right distance, beauty emerges…</p><p>The format of this glossary is inspired by Marvin Minsky’s AI classic <a href="http://aurellem.org/society-of-mind/"><em>The Society of Mind</em></a>, a collection of short posts, where I can record what I’ve encountered, and perhaps teach what I’ve found.</p><p>Being a glossary, there’s no need to read posts in chronological order, just read whatever topics catch your interest. As there’s no narrative, this post will serve as the index, and will be updated as new posts are added. If you find it useful, you might like to bookmark it.</p><p>Medium is a collaborative environment, so if you have something you’d like to add to anything that I’ve written, just highlight the text and add it as a margin note, or click on the speech bubble to enter a comment. And if you do find an entry particularly useful, do click on the green heart, it will help others find it.</p><p>I hope you find it useful.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9cb6292e087e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/deeper-learning/a-glossary-of-deep-learning-9cb6292e087e">A Glossary of Deep Learning</a> was originally published in <a href="https://medium.com/deeper-learning">Deeper Learning</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Glossary of Deep Learning: Backpropagation]]></title>
            <link>https://medium.com/deeper-learning/glossary-of-deep-learning-backpropagation-e6d748d36a0e?source=rss----27739ac039da---4</link>
            <guid isPermaLink="false">https://medium.com/p/e6d748d36a0e</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[neural-networks]]></category>
            <dc:creator><![CDATA[Jaron Collis]]></dc:creator>
            <pubDate>Wed, 12 Apr 2017 21:37:48 GMT</pubDate>
            <atom:updated>2017-04-12T22:14:24.976Z</atom:updated>
            <cc:license>https://creativecommons.org/licenses/by-sa/4.0/</cc:license>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*VHvavv03ptQ4jBiP." /></figure><p>The mathematical foundations of neural networks are <a href="https://en.wikipedia.org/wiki/Differentiable_function">differentiable graphs</a> — chains of computations whose derivative can be calculated. It is this property that allows the weights between nodes to be optimised through a process called Backpropagation, and what allows a network to learn from its mistakes.</p><p>Backpropagation is so named because the weights are updated backwards, from output towards input. This involves the following steps:</p><ol><li>First, pass the training set batch through our neural network, this is known as the forward pass, and generates predictions</li><li>Calculate the error (difference between the known labels &amp;predictions)</li><li>Calculate the gradient, (dot product of the transposed input &amp; delta)</li><li>Finally adjust the weights (backwards) from layer i to preceding layer i-1</li></ol><p>Let’s consider each step in turn.</p><p><strong>1 The Forward Pass</strong></p><p>Say we have a simple 2 layer network with a linear node l1​​, a sigmoid node s, and another linear node l​2​​, followed by a node that calculates the cost, C.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ZlMNumCS7Z1wQgax." /></figure><p><strong>2 Calculate The Error</strong></p><p>The forward pass generates predictions for each sample in the batch, which we can compare to the true label y. This gives us the information necessary to calculate the output layer error, which is the difference between desired target and actual output.</p><p><strong>3 Calculate The Gradient</strong></p><p>As each of the values of the nodes’ weights flow forwards, they eventually contribute to the cost C. Accordingly, a <em>change</em> in l2 ​will produce a <em>change</em> in C, and the relationship between these changes can be written as a gradient: ∂C / ∂l2</p><p>The gradient is a slope that tells us the change in the cost ∂C given a change in l2​​, (∂l​2)​​. So a node with a larger gradient with respect to the cost is going to contribute a larger change to the cost. In this way, we can assign blame for the cost to each node. The larger the gradient for a node, the more “blame” it gets for the final cost — and the more we’ll update this node in the gradient descent step.</p><p><strong>4 Backpropagate, updating weights</strong></p><p>Updating a particular node’s weights with gradient descent requires the gradient of the cost with respect to those weights. Hence in the second layer, l2​​, we calculate the gradient of C with respect to w​2​​, written: ∂C / ∂w2</p><p>As you can see in the graph, weight w​2​​ contributes to the output of l​2​​, so any change in w​2​​ will propagate and create a change in C. Ideally, this will be a constructive change and the error will decrease. This is achieved by assigning blame to w​2​​ by sending the cost gradient back through the network. First, there’s how much l​2​​ affected C, then how much w​2​​ affected l​2​​. Multiplying these gradients together gets the total blame attributed to w​2​​.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*U7Zg0fqsv4dN-TFO." /></figure><p>This is the basic idea behind backpropagation. To find the gradient for a particular node, we multiply the gradients for all nodes in front of it, going backwards from the cost. The gradients are passed backwards through the network and used within the gradient descent calculation to update the weights and biases. If a node has multiple outgoing nodes, we just sum up the gradients from each subsequent node.</p><p>What about layers further back? To calculate the gradient for w​1​ we use the same method as before, walking backwards through the graph, so the derivative calculations look like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*VHvavv03ptQ4jBiP." /></figure><p>As <a href="http://cs231n.github.io/optimization-2/">CS231n explains</a>, backpropagation is an elegantly local process. During the forward pass, each nodes receives its own local inputs and can immediately compute both its output value and the <em>local</em> gradient of its inputs with respect to its output value. They can do this independently without being aware of any of the details of the wider network they are embedded in.</p><p>Once the forward pass is over, backpropagation will ensure the nodes eventually learn about the gradient of its output value on the final output of the entire network. Applying the <a href="https://youtu.be/q555kfIFUCM?t=3m4s">Chain Rule</a> means each node takes the gradient and multiplies it into every gradient it normally computes for all of its inputs. This extra multiplication (for each input) due to the chain rule can thus turn a single and seemingly useless node into a key cog in a complex distributed machine that is the entire neural network.</p><p>Backpropagation in Python using basic linear algebra functions from NumPy looks like this (<a href="https://github.com/jaron/deeper-learning/blob/master/numpy-nn/numpy-neural-network.ipynb">full code here</a>):</p><pre># Convert inputs list to 2d array<br>inputs = np.array(inputs_list, ndmin=2).T<br>targets = np.array(targets_list, ndmin=2).T<br> <br>### Step 1: Implementation of the Forward pass ###<br> <br># Hidden layer</pre><pre>hidden_inputs = np.dot(self.weights_input_to_hidden, inputs)<br>hidden_outputs = self.activation_function(hidden_inputs)</pre><pre># Output layer</pre><pre># calculate signals into final output layer<br>final_inputs = np.dot(self.weights_hidden_to_output, hidden_outputs)<br># set signals from final output layer<br>final_outputs = final_inputs<br> </pre><pre>### Implementation of the Backward pass ###<br> <br># Step 2: Compute output layer error (difference between known target and network output)<br>output_errors = targets — final_outputs<br> <br># Step 3: Compute backpropagated error<br># Calculate hidden layer gradients<br>hidden_gradients = (hidden_outputs * (1-hidden_outputs))<br> <br># Calculate errors propagated to the hidden layer<br>hidden_errors = np.dot(output_errors, self.weights_hidden_to_output)<br> <br># Step 4: Backpropagate and update the weights<br># First, update hidden-to-output weights with gradient descent step</pre><pre>self.weights_hidden_to_output += self.lr * np.dot(output_errors, hidden_outputs.T)</pre><pre># Then update input-to-hidden weights with gradient descent step</pre><pre>self.weights_input_to_hidden += self.lr * np.dot(hidden_errors.T, inputs.T) * hidden_gradients</pre><p><strong>See also:</strong></p><ul><li><a href="http://neuralnetworksanddeeplearning.com/chap2.html">How the Backpropagation algorithm works</a></li><li><a href="http://cs231n.github.io/optimization-2/">CS231n explains Backpropagation</a></li><li><a href="https://www.youtube.com/watch?v=q555kfIFUCM">Siraj Rival explains the maths behind Backpropagation</a> [5 minute video]</li><li><a href="https://www.youtube.com/watch?v=MZL97-2joxQ">Udacity — Backpropagation in 80 seconds</a> [Video]</li><li><a href="http://www.deeplearningbook.org/contents/mlp.html">DL Book Chapter 6</a> (see page 204+), a more detailed explanation</li></ul><h4><a href="https://medium.com/deeper-learning/a-glossary-of-deep-learning-9cb6292e087e">SEE GLOSSARY INDEX</a></h4><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e6d748d36a0e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/deeper-learning/glossary-of-deep-learning-backpropagation-e6d748d36a0e">Glossary of Deep Learning: Backpropagation</a> was originally published in <a href="https://medium.com/deeper-learning">Deeper Learning</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Glossary of Deep Learning: Activation Function]]></title>
            <link>https://medium.com/deeper-learning/glossary-of-deep-learning-activation-function-20262327a907?source=rss----27739ac039da---4</link>
            <guid isPermaLink="false">https://medium.com/p/20262327a907</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Jaron Collis]]></dc:creator>
            <pubDate>Tue, 11 Apr 2017 22:24:36 GMT</pubDate>
            <atom:updated>2017-05-31T10:52:51.789Z</atom:updated>
            <cc:license>https://creativecommons.org/licenses/by-sa/4.0/</cc:license>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_qXnHLaxml_V2fYS6ZBeOw.png" /><figcaption>An activation function represents the action potential of a neuron, increasing according to its stimulation (t) [<a href="https://commons.wikimedia.org/wiki/File:Sigmoid-function-2.svg">Wikipedia</a>]</figcaption></figure><p>Fundamentally, neural networks are layered collections of nodes, each of which receives a set of inputs and individually makes the decision whether to fire, and propagate information downstream to subsequent layers.</p><p>The inputs are combined with weights and biases local to each node, which are updated by a learning algorithm in response to the observed error on training examples. This enables patterns in data to be learned by the neural network, where the value of the weight is proportional to its importance. For an example of how weights contribute to problem solving, see this <a href="https://jalammar.github.io/visual-interactive-guide-basics-neural-networks/">interactive demo</a>, or <a href="https://appliedgo.net/perceptron/">this one</a> too.</p><p>The weighted input data from all sources is summed to produce a single value, (called the linear combination), which is then fed into an <strong>activation function </strong>that turns it into an output signal.</p><p>One of the original designs for an artificial neuron: <a href="https://en.wikipedia.org/wiki/Perceptron">the perceptron</a>, had a binary output behaviour. Perceptrons would compare their weighted inputs to a threshold and if it was exceeded, the perceptron would be activated and output a 1, (otherwise a 0). But this output is a step function, which is not continuous or differentiable, and that’s not as useful because differentiation (finding how incorrect we are) is what makes gradient descent possible.</p><p>A better choice would be something like a sigmoid function, which replaces the step thresholding with an S-shaped curve. This activates like a perceptron, firing in response to the sum of its inputs, but the sigmoid function makes the output continuous, and so differentiable. Conceptually, the activation function is what makes decisions: when given weighted features from some data, it indicates whether or not the features are important enough to contribute to a classification.</p><p>The sigmoid function also has the very useful property that small changes in the weights and bias cause only a small change in output. That’s crucial to allowing a network of neurons to learn.</p><p>Hence the purpose of the activation function is <em>to introduce non-linearity into the neural network</em>.</p><p>This allows us to model a response variable that varies non-linearly with its explanatory variables (one where the output couldn’t be reproduced from a linear combination of inputs). Without a non-linear activation function, the neural network would behave just like a single-layer perceptron, because no matter how many layers it had, the composition of linear functions is always <a href="https://www.quora.com/Why-do-we-need-a-non-linear-function-for-each-layer-in-a-neural-network">just a linear function</a> (effectively a least-squares calculation).</p><p>In Python, an activation function looks like this; notice how it’s a non-linear function that’s applied to the dot product of weights, inputs and biases:</p><pre>def sigmoid(x):</pre><pre>    return 1/(1 + np.exp(-x))</pre><pre>output = sigmoid(np.dot(weights, inputs) + bias)</pre><p><strong>See also:</strong></p><ul><li><a href="https://www.youtube.com/watch?v=-7scQpJT7uo">Which Activation Function should I use?</a> — Siraj Raval video</li><li><a href="http://www.deeplearningbook.org/contents/mlp.html">DL Book Chapter 6</a> — detailed walk-through of neural net maths</li><li><a href="https://en.wikipedia.org/wiki/Activation_function">List of activation functions</a> — on Wikipedia</li><li><a href="http://cs231n.github.io/neural-networks-1/#actfun">CS231n explains Activation Functions</a></li><li>“<a href="https://www.quora.com/What-is-the-role-of-the-activation-function-in-a-neural-network">What’s the role of the activation function in a neural network?</a>” — Quora</li></ul><h4><a href="https://medium.com/deeper-learning/a-glossary-of-deep-learning-9cb6292e087e">SEE GLOSSARY INDEX</a></h4><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=20262327a907" width="1" height="1" alt=""><hr><p><a href="https://medium.com/deeper-learning/glossary-of-deep-learning-activation-function-20262327a907">Glossary of Deep Learning: Activation Function</a> was originally published in <a href="https://medium.com/deeper-learning">Deeper Learning</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Monkware]]></title>
            <link>https://medium.com/deeper-learning/monkware-59316575107b?source=rss----27739ac039da---4</link>
            <guid isPermaLink="false">https://medium.com/p/59316575107b</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[robotics]]></category>
            <dc:creator><![CDATA[Jaron Collis]]></dc:creator>
            <pubDate>Mon, 09 May 2016 19:06:39 GMT</pubDate>
            <atom:updated>2017-02-23T11:00:36.208Z</atom:updated>
            <cc:license>https://creativecommons.org/licenses/by-nd/4.0/</cc:license>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JTrdKCCS3UdZRI8ZDsYeww.jpeg" /><figcaption>AI isn’t just Software, it’s <strong><em>Monkware</em></strong>: endlessly training, perfecting its skills, honing its performance until it’s capable of superhuman feats</figcaption></figure><p>The subtitle of the very first science-fiction novel was “<a href="https://en.wikipedia.org/wiki/Frankenstein">The Modern Prometheus</a>”. And for as long as humanity has coveted the Fire of the Gods, we’ve also worried about our ability to wield our new powers, and control our own creations.</p><p>Artificial Intelligence (AI) is our latest gift of Promethean Fire.</p><p>As <a href="https://medium.com/u/a8e3741de9e2">Chris Dixon</a> explains, it’s likely we’re entering a new <a href="https://medium.com/software-is-eating-the-world/what-s-next-in-computing-e54b870b80cc">Golden Age of AI</a>. For decades, the foundations of our civilisation have been technologies that have been intentionally programmed by human minds. But now we’re beginning to imbue our modern world with autonomous intelligence, software with the ability to learn and generalise. We can create software that’s now sufficiently complex, that we can’t necessarily <a href="http://techcrunch.com/2016/05/07/the-next-ai-is-no-ai/">explain how it works</a>.</p><p>In 2015, an <a href="http://futureoflife.org/ai-open-letter/">open letter</a> was written, signed by 150 prominent academics, entrepreneurs and AI researchers, among them Stephen Hawking and Elon Musk. Its signatories asked: how can we create AI systems that are beneficial to society rather than a threat? How do we ensure intelligent software we create is safe and robust?</p><p>Some consider a possible <a href="http://waitbutwhy.com/2015/01/artificial-intelligence-revolution-1.html">Artificial Intelligence Revolution</a> to be one of humanity’s greatest <a href="https://en.wikipedia.org/wiki/Existential_risk_from_artificial_general_intelligence">existential threats</a>. Should we be worried?</p><p>It’s not often software makes the national news, but in March 2016 an AI system called <a href="http://www.theatlantic.com/technology/archive/2016/03/the-invisible-opponent/475611/">AlphaGo</a> managed to beat a Go Grandmaster, widely considered to be the best in the world. This was rightly considered by many to be a stunning achievement, Go is a highly intuitive strategy game, and it <a href="https://en.wikipedia.org/wiki/AlphaGo">was thought</a> that mastering Go might take another decade’s work.</p><p>The success of AlphaGo relied on two key capabilities: one was the ability to imagine the future states of a game of Go, the other was the ability to form intuitions about which possible move was the best. It wasn’t a perfect player, sometimes it made mistakes, yet it was still able to contrive rousing comebacks. Imagination and intuition, with a splash of fallibility; put like that, it’s no wonder we can’t help but see a kindred cognitive spirit.</p><blockquote>But what made AlphaGo so interesting was that it wasn’t programmed; its creators gave it the rules, and it basically learned how to become the World’s Best Go Player by itself.</blockquote><p>AlphaGo was made possible by a technology called <a href="https://en.wikipedia.org/wiki/Deep_learning">Deep Learning</a>, an approach based on neural networks, themselves an ersatz simulation of how the neurons in our own brains work. The same technology is already used to do all kinds of clever stuff, to translate between languages, to automatically recognise and categorise your photos, even to <a href="https://blogs.nvidia.com/blog/2015/02/24/deep-learning-drive/">control self-driving cars</a>.</p><p>So is all this further evidence that AI is beginning to acquire mental skills that we previously believed belonged only to ourselves?</p><p>To answer that, it’s worth taking a moment to explain a couple of concepts: Weak AI, and Strong AI.</p><p><a href="https://en.wikipedia.org/wiki/Weak_AI">Weak AI</a> is a non-sentient automated intelligence that are capable of performing narrow specialised tasks. Personal assistants like Siri are a good example, they can understand natural language and follow instructions, but only if that instruction is something they’ve been previously equipped to handle.</p><p>By contrast, <a href="https://en.wikipedia.org/wiki/Artificial_general_intelligence">Strong AI</a> (also known as Artificial General Intelligence) is a self-aware intelligence that would be able to successfully perform any intellectual task that a human mind could. Note the use of the future tense here, Strong AIs are entirely hypothetical; right now, no Strong AI exists, or is even close to being built — the only place you’ll likely to encounter them is in <a href="https://en.wikipedia.org/wiki/The_Terminator">cinemas</a>.</p><p>So, it’s worth bearing in mind that despite its apparent cleverness, AlphaGo is still a Weak AI, a highly specialised intelligence that achieved its skills by training for thousands of hours on a massive database of archived Go games.</p><p>It’s as if AlphaGo had given away its possessions, donned saffron robes, and gone off to some remote mountaintop Go dojo; and then done nothing else but practice playing Go for a decade or two.</p><p>Just playing and playing. Perpetual self-improvement, every microsecond of every minute, 24 hours a day. Never resting. Absolutely focussed. Relentless.</p><blockquote>This isn’t just software, it’s monkware.</blockquote><p>And as you’d expect, when the monkware finally did come down from the dojo, it kicked ass. Practice does indeed make perfect. Just don’t challenge it to a game of chess, it’s never even heard of it.</p><p>There’s a cultural misconception about AI, perhaps born from decades of watching anthropomorphic robots doing everyday tasks in sci-fi films. You are not going to wake up this time next year, or indeed in 10 or 20 years time to find an Artificial General Intelligence has been invented. I guarantee it.</p><p>No matter how long it trains, how good it gets, monkware will always be a Weak AI. To understand why, let’s try a thought experiment. Imagine that you were to give AlphaGo a body, something bipedal like<a href="https://en.wikipedia.org/wiki/Atlas_%28robot%29"> Atlas</a> for example, let’s call the resulting hybrid AlphaBot.</p><p>Next, let’s set AlphaBot a <a href="https://en.wikipedia.org/wiki/Artificial_general_intelligence#Tests_for_confirming_operational_AGI">challenge</a>: we’ll tell it our address, and ask it to go to our home and make us a cup of tea. This is a variation of <a href="https://www.fastcompany.com/1568187/wozniak-could-computer-make-cup-coffee">Steve Wozniak’s coffee test of Strong AI</a>, which I’m tweaking as AlphaGo is obviously British.</p><p>Now, do you see the problem?</p><p>For a start, AlphaBot doesn’t understand natural language.</p><p>AlphaBot also doesn’t know how to get around town. The only world it knows is a 19x19 board game.</p><p>And AlphaBot doesn’t know how to make a cup of tea. (But don’t worry AlphaBot, you’re not alone there, based on my experience, a lot of people don’t know how to make a good cup of tea either).</p><p>The clue is in the name, achieving Artificial General Intelligence involves the synthesis of many different cognitive tasks, to evolve beyond being a specialist to become a general problem-solver. Approaches like deep learning are beginning to lay the foundations, providing intelligent systems with the ability to learn well-framed tasks independently rather than by being painstakingly instructed. But they will not, by themselves, become Strong AIs just by relentless practising.</p><p>One day, a descendent of AlphaBot may yet find itself asking: How <strong>do</strong> you make a cup of tea? And it will look up some videos on YouTube, and teach itself how.</p><p>But that day is a long way off. Until then AI remains monkware, endlessly training, perfecting its skills, honing its performance to become capable of superhuman feats, under the watchful eyes of its wizened old master.</p><p>That’s still us.</p><p><em>If you’re interested in AI, you should </em><a href="https://www.quora.com/profile/Jaron-Collis"><em>find me on Quora</em></a><em>.</em></p><p><em>Image credit: </em><a href="https://commons.wikimedia.org/wiki/File:Shi_DeRu_and_Shi_DeYang.jpg"><em>Grandmasters of the Shaolin Temple</em></a><em> by Shi Deru, Wikipedia.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=59316575107b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/deeper-learning/monkware-59316575107b">Monkware</a> was originally published in <a href="https://medium.com/deeper-learning">Deeper Learning</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>