<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Tristan on Medium]]></title>
        <description><![CDATA[Stories by Tristan on Medium]]></description>
        <link>https://medium.com/@tianyu.tristan?source=rss-c0336b6e3f59------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*zf7_WtpsCH_QqjoGsd50-Q.png</url>
            <title>Stories by Tristan on Medium</title>
            <link>https://medium.com/@tianyu.tristan?source=rss-c0336b6e3f59------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 05:42:24 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@tianyu.tristan/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Visual Attention Model in Deep Learning]]></title>
            <link>https://medium.com/data-science/visual-attention-model-in-deep-learning-708813c2912c?source=rss-c0336b6e3f59------2</link>
            <guid isPermaLink="false">https://medium.com/p/708813c2912c</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[towards-data-science]]></category>
            <dc:creator><![CDATA[Tristan]]></dc:creator>
            <pubDate>Mon, 17 Jul 2017 08:23:21 GMT</pubDate>
            <atom:updated>2017-07-17T18:11:39.008Z</atom:updated>
            <content:encoded><![CDATA[<h3>Abstract</h3><p>The main idea of this exercise is to study the evolvement of the state of the art and main work along topic of visual attention model. There are two datasets that are studied: augmented MNIST and SVHN. The former dataset focused on canonical problem — handwritten digits recognition, but with cluttering and translation, the latter focus on real world problem — street view house number (SVHN) transcription. In this exercise, the following papers are studied in the way of developing a good intuition to choose a proper model to tackle each of the above challenges.</p><p>For augmented MNIST dataset:</p><ul><li>The baseline model is based on classical 2 layer CNN</li><li>The target model is recurrent attention model (RAM) with LSTM, refer to paper [2]</li></ul><p>For SVHN dataset:</p><ul><li>The baseline model is based on 11 layer CNN: with convolutional network to extract image feature, then use multiple independent dense layer to predict ordered sequence, refer to paper [1]</li><li>The target model is deep recurrent attention model (DRAM) with LSTM and convolutional network, refer to paper [3]</li></ul><p>Additionally:</p><ul><li>Spatial Transformer Network is also studied as latest development in the visual attention regime, refer to paper [5]</li></ul><p>Both of the above dataset challenges focuses on digit recognition. In this exercise, MNIST is used to demonstrate the solution for single digit recognition, whereas SVHN is used to show the result of multiple digit sequence recognition.</p><h3>Augmented MNIST Dataset</h3><h4>Data Augmentation</h4><p>Since the original MNIST dataset is a canonical dataset used for illustration of deep learning, where a simple multi-layer perceptron could get a very high accuracy, therefore the original MNIST is augmented with additional noise and distortion in order to make the problem more challenging and leads towards real-world problem such as SVHN.</p><p>An example of MNIST dataset after augmentation is shown below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CRR5WONVcTn8OtQyLnAuuA.png" /></figure><p>The above augmented MNIST dataset is cluttered non-centred. The augmentation process is as following:<br>(1) The translation is generated by randomly select a position to translate the original 28x28 image in a 100x100 canvas.<br>(2) The distortion noise is generated by randomly select a training/validation 28x28 image, then randomly crop a 9x9 patch for 8 times, then stitch all the 8 cropped images to the augmented 100x100 image</p><p>The augmented MNIST dataset is seemingly difficult for classical method, let’s try it first.</p><h4>Data Preprocessing</h4><p>In addition to data augmentation, each image pixel value is normalized in the range of [0, 1].</p><p>The MNIST dataset is perfectly balanced. There is no additional data preprocessing is applied.</p><h4>Baseline Approach: CNN Only</h4><p>The baseline approach is to use a classical convolutional model architecture that works well with original MNIST dataset. MLP is not used as baseline model as it does not present the best effort with classical model. The convolutional model architecture is taken from <a href="https://github.com/fchollet/keras/blob/master/examples/mnist_cnn.py">keras example</a> which claims to reach 99.25% accuracy on original MNIST dataset.</p><p>The model architecture is shown as below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/570/1*Wa6YoI28PpuRqt9IbUwqjg.png" /></figure><p>Since MNIST hand written digits are relatively simple, therefore only 2 convolution layers followed by max pooling is used for feature reduction and spatial invariance. Dropout is used for regularization. Flatten/Dense layers are used for feature interaction.</p><p>There’re 60k MNIST image data, where 54k are used for training, 6k used for validation and 10k used for testing. Testing data is only used once in the last step.</p><p>The validation dynamics is shown below. We can see that the validation accuracy plateaus at around 50% accuracy. And the final one-time test accuracy is <strong>50.87%</strong> at 100 epochs (since each training image is 100x100, the total training time in a Tesla K80 GPU server takes few hours).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rCxAhtGiLSy55Mq0jznYoA.png" /></figure><h4>Target Approach: Recurrent Attention Model (RAM)</h4><p>To better tackle the problem above, there’s a very neat and cool idea published by Google in 2014, refer to paper [2].</p><p><strong><em>Inspiration<br></em></strong>The general idea is to take inspiration from how human eye works, i.e. retina, as illustrated below. Our eye can capture a very wide view of the world, however, we tend to “glance” the overview and only pay attention to a particular region of the view, while the other part is kind of “blurred out”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/945/0*vfFr0dSwDMzAb5xj." /><figcaption>source: <a href="https://en.wikipedia.org/wiki/Visual_perception">https://en.wikipedia.org/wiki/Visual_perception</a></figcaption></figure><p>The visual attention model is trying to leverage on this idea, to let the neural network be able to “focus” its “attention” on the interesting part of the image where it can get most of the information, while paying less “attention” elsewhere.</p><p>The advantage of this approach is:<br>(1) the model can pay more attention to the relevant region of the image<br>(2) and therefore the image processing power required is reduced<br>(3) “off-focus” blurry area still captures a general concept of what’s happening</p><p>This intuition may help in our augmented MNIST dataset challenge where only the region of image around the digit need more attention.</p><p><strong><em>Architecture<br></em></strong>There’re few components in the overall architecture.</p><p><strong><em>Glimpse Sensor<br></em></strong>Glimpse Sensor is the implementation of Retina, as illustrated below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/396/1*4Ip1oL5zJqoEAzqFgnDgGQ.png" /></figure><p>The idea is to allow our network to “take a glance” at the image around a given location, called a glimpse, then extract and resize this glimpse into various scales of image crops, but each scale is using the same resolution. For example, the glimpse in the above example contains 3 different scales, each scale has the same resolution (a.k.a. sensor bandwidth), e.g. 12x12. Therefore, the smallest scale of crop in the centre is most detailed, whereas the largest crop in the outer ring is most blurred. In summary, Glimpse Sensor takes a full-sized image and a location, outputs the “Retina-like” representation of the image around the given location.</p><p><strong><em>Glimpse Network</em></strong><br>Once we have defined glimpse sensor, Glimpse Network is simply a wrapped around Glimpse Sensor, to take a full-sized image and a location, extract a retina representation of the image via Glimpse Sensor, flatten, then combine the extracted retina representation with the glimpse location using hidden layers and ReLU, emitting a single vector g. This vector contains the information of both “what” (our retina representation) and “where” (the focused location within the image).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/448/1*n8V7ObNtU85oJupqfTD_sA.png" /></figure><p><strong><em>Recurrent Network</em></strong><br>Recurrent Network takes feature vector input from Glimpse Network, remembers the useful information via it’s hidden states (and memory cell).</p><p><strong><em>Location Network</em></strong><br>Location Network takes hidden states from Recurrent Network as input, and tries to predict the next location to look at. This location prediction will become input to the Glimpse Network in the next time step in the unrolled recurrent network. The Location Network is the key component in this whole idea since it directly determines where to pay attention to in the next time step. In order to maximize the performance of this Location Network, the paper introduce a stochastic process (i.e. gaussian distribution) to generate next location, and use reinforcement learning techniques to learn. It is also known as “hard” attention, since this stochastic process is non-differentiable (compared to “soft” attention). The intuition behind stochasticity is to balance between exploitation (to predict future using the history) and exploration (to try unprecedented stuff). Note that, this stochasticity makes the component non-differentiable, which will incur problem during back-propagation. And REINFORCE gradient policy algorithm is used to solve this problem, the detail is discussed in reinforcement learning section.</p><p><strong><em>Activation Network</em></strong><br>Activation Network takes hidden states from Recurrent Network as input, and tries to predict the digit. In addition, the prediction result is used to generate the reward point, which is used to train the Location Network (since the stochasticity makes it non-differentiable).</p><p><strong><em>Architecture Combined</em></strong><br>Combining all the element illustrated above, we have our network architecture below. f_g is Glimpse Network, f_h is Recurrent Network, f_l is Location Network and f_a is Activation Network. The diagram below is the unrolled recurrent for 2 time steps for illustration purpose. In particular, the # of time steps = # of glimpse per image</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/953/1*71KTu5p-USzQPSTMPFTZzA.png" /></figure><p>Reinforcement Learning<br>Since Location Network is non-differentiable, reinforcement learning, specifically, policy gradient method, is used to train this special part of network. First let’s look at an overview of reinforcement learning.</p><p><strong><em>Overview<br></em></strong>Reinforcement learning is all about trial and error. An overview of reinforcement learning is shown in diagram below. We have the following components:</p><ul><li>Agent: the brain, which is the neural network we’re modelling.</li><li>Environment: the globe, which is the unknown world that we can interact with. By default we don’t know anything about the environment.</li><li>Interactions: the red arrows, which is the interaction between agent and environment. It can be action (the agent output that will change the environment), observation (the outcome of environment change as consequence of the agent’s action) and reward (some scalar value that represents “how good” the agent’s action is). The goal of the agent is to maximize the reward it collects.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/447/0*71d7JALfaEwIWmUn." /><figcaption>source: <a href="http://www0.cs.ucl.ac.uk/staff/d.silver/web/Teaching.html">http://www0.cs.ucl.ac.uk/staff/d.silver/web/Teaching.html</a></figcaption></figure><p>The various method of reinforcement learning depends on the objective we tries to optimize, as following taxonomy:</p><ul><li>Policy: mapping from states to actions. Which action to execute given where we are (i.e. state)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/342/0*SjB7gStAB99cDl1d." /></figure><ul><li>Value: evaluate how good a state is. How good it is to be at a certain state</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/677/0*IHfiiyzToYl6OOKZ." /></figure><ul><li>Model: predict what the environment will do. Knowing the enemy helps to win the war.</li></ul><p>More detailed and through explanation about RL can be found <a href="/service/source: http://www0.cs.ucl.ac.uk/staff/d.silver/web/Teaching.html">here</a>. In this paper, policy gradient is chosen as RL method, which falls under the first category above.</p><p><strong><em>Application to our problem<br></em></strong>Back to our problem, since we’re using policy gradient to solve our problem, let’s talk about “policy” itself. In our problem, the “policy” determines how do we choose our next glimpse location. Our answer (i.e. policy) is, by sampling from a gaussian distribution with parameter μ and σ, where μ is mean and σ is standard deviation. This gaussian distribution is our stochastic process. In this implementation, we fixed σ as a network hyperparameter, and only tries to learn μ.</p><p>To illustrate the training process of stochastic network, the following diagram shows a high level concept. The blue arrow represents forward pass, red arrow represents backward pass. Our process starts from passing an input image into our network. The differentiable network will work as normal (e.g. XW + b), represented as a robot head. Then the stochastic network (in our case the Location Network), will apply a random stochastic process (in our case gaussian distribution) to generate a result (in our case the prediction of next glimpse location). It is represented as flipping a coin, to denote a random process. Then the location prediction (together with classification result using the glimpse) is evaluated by the environment, and now we have a reward to start backward propagation. Conceptually, the backward propagation through stochastic network can be think of as if we make our coin more biased towards the direction that will get us more reward in the future, in other words, we learn the gaussian distribution parameter. This step is represented as a hammer on a coin, to denote that, we want to “forge” the coin in our favour.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BuhOGBJTjjz8lxzaiYBzzQ.png" /></figure><p>So how to hammer our coin? Let’s take a look at derivation of policy gradient. Assume we have a function f(x) that can tells us how much reward we’re going to have if we have a sample point x, i.e. a score/reward function</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/625/0*i1Z-RhKYePCQzlH6." /><figcaption>Source: <a href="http://karpathy.github.io/2016/05/31/rl/">http://karpathy.github.io/2016/05/31/rl/</a></figcaption></figure><p>The above tells us, if we have a sample point x, in order to have a higher score next round, the way we should “forge” our coin is the direction of gradient of score function (the left hand side of equation), which is the same as the direction of maximizing loglikelihood of sample point x, with magitude of f(x) itself (the right hand side of equation).</p><p>To put the above idea in a formal way, the way we “hammer” our coin is the REINFORCE algorithm, which stands for <strong>RE</strong>ward <strong>I</strong>ncrement = <strong>N</strong>onnegative <strong>F</strong>actor times <strong>O</strong>ffset <strong>R</strong>einforcement times <strong>C</strong>haracteristic <strong>E</strong>ligibility (someone tried really hard to spell “reinforce”…). The original paper see [6]</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/785/0*UlT5LDDxAilaHkNO." /><figcaption>Source: <a href="http://www0.cs.ucl.ac.uk/staff/d.silver/web/Teaching.html">http://www0.cs.ucl.ac.uk/staff/d.silver/web/Teaching.html</a></figcaption></figure><p>There’s a much better explanation from <a href="http://karpathy.github.io/2016/05/31/rl/">Andrej Karpathy’s blog</a>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*BRXBhq5ZznYaJE8j." /><figcaption>A visualization of the score function gradient estimator. <strong>Left</strong>: A gaussian distribution and a few samples from it (blue dots). On each blue dot we also plot the gradient of the log probability with respect to the gaussian’s mean parameter. The arrow indicates the direction in which the mean of the distribution should be nudged to increase the probability of that sample. <strong>Middle</strong>: Overlay of some score function giving -1 everywhere except +1 in some small regions (note this can be an arbitrary and not necessarily differentiable scalar-valued function). The arrows are now color coded because due to the multiplication in the update we are going to average up all the green arrows, and the <em>negative</em> of the red arrows. <strong>Right</strong>: after parameter update, the green arrows and the reversed red arrows nudge us to left and towards the bottom. Samples from this distribution will now have a higher expected score, as desired.</figcaption></figure><p>Now we know how to compute gradient for stochastic network. However, in practise, tensorflow computation graph will calculate gradients automatically by default. Therefore, in order to achieve a customised gradient update as described by REINFORCE, there’re two way to do it: <br>(1) set tf.stop_gradient to pass through gradient directly to differentiable network, then use a hybrid loss function to train only the differentiable network as an approximation. The argument here is, since we fixed std of gaussian distribution and only try to “learn” mean as distribution parameter, hammering the coin itself will roughly be equivalent to hammering our differentiable network parameters in order to have our desired mean emitted from differentiable network. Refer to <a href="https://github.com/zhongwen/RAM">this repo</a> and <a href="https://github.com/jlindsey15/RAM">this repo</a><br>(2) build a separate computation graph with optimizer and usetf.gradients to compute specifically the desired gradients w.r.t relevant parameters, then use optimizer.apply_gradients to update only those parameters. Refer to <a href="https://github.com/yanji84/deep-recurrent-attention-model">this repo</a></p><p><strong><em>Training Methodology</em></strong><br>As the implementation details discussed above, in this exercise, a hybrid loss is used in the implementation, the approach (1) as above.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1016/1*pZWai5XO0qinKbhkfVV2QQ.png" /><figcaption>R = actual reward, b = score function, b_freeze = tf.stop_gradients(b)</figcaption></figure><p>Since plus gate works as gradient distributor in backward propagation process, we’re effectively optimizing only our differentiable network with following 3 targets with equal weights: (1) Cross Entropy for prediction correctness (2) MLE for regressing the optimal μ as parameter input to stochastic network (3) Score Function to regress score function onto actual reward (so the score function is more accurate).</p><p>Another trick done is Monte Carlo sampling. Since we have stochastic process in the network, we can duplicate the same image 10 times (which will yield 10 slightly different result due to stochasticity), then average them as our prediction for both training and testing.</p><p>The training parameters is shown as below. Only 2 scenarios are compared.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VlyM88b3xyMaBTSVANXM4A.png" /></figure><p>The training dynamic is shown below. Training time in the above table is benchmarked using a Tesla K80 GPU server. The training accuracy (averaged per batch) plateaus at 80%, and the one-time testing accuracy in the end is <strong>79.68%</strong>, a significant improvement compared to the baseline approach of <strong>50%</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*V759ECxy3mNRh7Zc." /></figure><p><strong><em>Limitation<br></em></strong>The limitation of the RAM method as discussed in this section includes:</p><p>(1) Glimpse Sensor is using Flatten + Dense layer. It may be sufficient for MNIST features, however, to fit for real-world image, convolution layers may be used</p><p>(2) Recurrent Network is using single layer of RNN Cell. This RNN Cell is supposed to learn both image feature (in order to predict digits label) and location feature (in order to predict next glimpse location). This mixture of responsibility may cause some issue internally.</p><p>(3) At the initial step of unrolled RNN, the Location Network randomly choose a position to focus its glimpse as a starting point. This may be improved to have a “less random” start.</p><p>All the above limitations are addressed in the follow up paper published by Google in reference [3]. This paper is discussed in the section below as target architecture to handle SVHN dataset.</p><h3>SVHN Dataset</h3><p>SVHN dataset is a real-world street view house number dataset. As shown below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/567/0*h7T9SwepLu7chv0G." /><figcaption>source: reference [1] paper</figcaption></figure><p>In the SVHN dataset, the following information is provided:<br>(1) length of digits<br>(2) each digit number<br>(3) maximum length of digits is 5</p><p>SVHN dataset is the extension to our augmented MNIST dataset challenge, in a sense that:<br>(1) there’s noise and blurry effect in the image<br>(2) there’s translation of digits<br>(3) it is an ordered sequence of digits instead of a single digit in our augmented MNIST dataset</p><h4><strong><em>Data Preprocessing</em></strong></h4><p>For the image, the following preprocessing is done:</p><ul><li>Take bounding box position from training (and testing) data</li><li>Crop union of bounding boxes, in order to shrink image to maximum</li><li>Resize to 32x32</li><li>Convert to grayscale</li><li>Normalize between [0,1]</li></ul><p>An example image is shown below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/200/1*e_3NNG9IVk8Dgy8T7m6ftw.png" /></figure><p>For the labels, the following preprocessing is done:</p><ul><li>0–9 represent digits, 10 represents end token (or null token, if multiple)</li><li>Set max sequence length to 5</li><li>Padding using end/null token 10</li><li>Throw away data that is more than 5 digits long</li></ul><p>E.g. the above image label would be: [1, 0, 5, 10, 10]</p><h4><strong><em>Introduction</em></strong></h4><p>The following two approaches are studied.</p><p><strong>(1) A “CNN only” approach</strong>. The main idea is to look at multiple digits at the same time, and aim to “classify” the digit sequences in a fixed length digits order with blank padding if that digit is missing. This idea is borrowed from paper published in reference [1]. However the implementation varies to some extend from the paper, for the purpose of quickly building a baseline for comparison.</p><p><strong>(2) A “RNN + CNN” deep recurrent attention model approach</strong>. The main idea is to introduce deep visual attention model (DRAM) refer to [3] in extension to recurrent attention model (RAM) their previous work [2]. Where two layers of LSTM is proposed to separate glimpse location prediction and image feature handling.</p><h4>Baseline Approach: CNN Only</h4><p><strong><em>Problem Description<br></em></strong>The problem of house number transcription has the following factors that we need to pay attention to:</p><ol><li>Multiple digits are present (we need to recognize all of them)</li><li>All digits are ordered (we need to recognize the sequence)</li><li>There’s a specific length of digit sequence (the length of sequence varies)</li></ol><p>Let P denote probability, S represents the output digit sequence, L represents the length of digit sequence and X represent input image data. Our goal can be interpreted to maximize the following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/385/1*dXbdFIJWnx384UcPiCO3wg.png" /></figure><p>At test time, our prediction is the argmax of the above model:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/401/1*c7zQoUco7izcjmKLVWfv4Q.png" /></figure><p>This model captures all the 3 factors mentioned above. In this exercise, to quickly form a baseline, a small variation of the above model is implemented, although the 3 factors are still kept, as following:</p><ol><li>house number of length greater than 5 is discarded</li><li>the model classifies 5 digits in order, where each digits ranges from 0–10. 0–9 represents the digits, 10 represents blank padding. E.g. house number “258” will result in a correct label sequence of [2,5,8,10,10]</li><li>did not use the length since blank padding implies the length information</li></ol><p><strong><em>Approach<br></em></strong>The general idea adopted in this exercise is to have two networks in the architecture: convolutional network and classification network.</p><p>The convolution network is used to extract features of house number digits from the feed image, followed by classification network that use 5 independent dense layers to collectively classify an ordered sequence of 5 digits, where 0–9 representing digits and 10 represent blank padding.</p><p>We expect the classification network to learn the spatial information of digits layout through backward propagation, so that each dense layer is picking up only one digit in the sequence respectively.</p><p><strong><em>Network Architecture<br></em></strong>The original paper uses a network architecture with 8 convolutional layers. In this exercise, to quickly form a baseline, a simpler network is used (while still preserving the type of layers), as following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6AqltShKFXQz9KBnFAe5eA.png" /></figure><p>Note that in the classification network, 5 independent dense layers are in charged to classify existing features into 0–10 class label, where 0–9 represents digit label and 10 represents blank padding.</p><p><strong><em>Training Methodology<br></em></strong>The paper achieves 96.03% transcription accuracy with its best model performance on SVHN dataset. In this exercise, with the variation of models, image preprocessing and network architecture, an accuracy of 87.64% has been achieved, refer to <a href="https://github.com/ritchieng/NumNum/blob/master/NumNum/report/report.pdf">this paper</a></p><p><strong><em>Limitation<br></em></strong>Ideally, the image preprocessing step should only apply to training data (e.g. cropping off union of bounding-boxes), where leaving validation/testing data untouched. However, in the implementation refer to <a href="https://github.com/ritchieng/NumNum/">this repo</a>, validation/testing data is also cropped around bounding-boxes. The original paper mentioned that bounding-box rectangles around digits are localized first in image preprocessing stage. However it did not mention the exact method to perform such localization.</p><p><strong>TODO</strong>: Alternatively, another benchmark shall be performed with only cropping training data</p><h4>Target Approach: Deep Recurrent Attention Model (DRAM)</h4><p>In addition to previous work in Recurrent Attention Model (RAM), let’s extend to fix the limitations and apply to SVHN dataset. This idea is called Deep Recurrent Attention Model (DRAM).</p><p><strong><em>Architecture<br></em></strong>The architecture of DRAM is similar to RAM, except that:</p><ul><li>Location Network → Emission Network</li><li>Two layer of RNN Cell are proposed: Location RNN which is dedicated to predict next glimpse location (denoted as r superscript 2), and Classification RNN which is dedicated to predict digit label (denoted as r superscript 1).</li><li>Context Network is introduced to use a down sampled coarse image to set hidden state of Location RNN. The idea is to feed a general context of image to Location RNN in order for it to generate a good guess of the first position to focus glimpse (compared to random position to start in RAM approach).</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/611/0*L52vgshbTP6f7G6S." /></figure><p><strong><em>Sequential Classification<br></em></strong>To extend the idea of RAM and apply for sequential classification, the following work is proposed.</p><p><strong><em>Monte Carlo Sampling<br></em></strong>It is the same idea as RAM, by duplicating same image multiple times then take the average as prediction (the paper mentioned taking log average gives best performance)</p><p><strong><em>Curriculum-like learning<br></em></strong>The idea is to continue training until the first mistake made by the network to classify a digit in sequence, and then stop training (i.e. stop parameter update) after that point. In this way, the network will be more focused on classifying an ordered sequence with imminent error.</p><p><strong><em>Accumulated Reward<br></em></strong>In order to simplify multiple objects classification, an accumulated reward is used, that is, the reward proportional to the correctly predicted labels.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/157/1*C2Gj_U-2-T5oozflU3gr7w.png" /></figure><p><strong><em>Sequence Log-Likelihood<br></em></strong>The sequence log-likelihood is presented in the following objective function:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/649/1*MsOt9BVdqZMi1Yk8v8uxuQ.png" /></figure><p>This objective function is MLE that merges location probability from Emission Network (same as Location Network in RAM) and classification probability from Classification Network (same as Action Network in RAM).</p><p><strong><em>Forward-Backward<br></em></strong>It is mentioned in the paper that, a single DRAM with Monte Carlo sampling tends to have problem predicting the end token (i.e. when to stop). The solution here is to take the same training data, but let the network predict the reverse sequence of digits, then take the average as final prediction. This method improves accuracy.</p><p><strong><em>Result<br></em></strong>DRAM is not implemented in this exercise (at least not successfully).</p><p><strong><em>Accuracy<br></em></strong>Referring to the paper, the accuracy of DRAM on SVHN dataset is reported in the table below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/506/1*3SGHs2U6LLydAaxnCAJeHg.png" /></figure><p>This is not a significant improvement (3.9% error rate)against our baseline method of 11 layer CNN (3.96% error rate). However, the improvement is more on computation cost.</p><p><strong><em>Computation Cost<br></em></strong>The computation cost is shown in the paper as below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/921/1*swguGZTl6_gTspj1mMYRQQ.png" /></figure><p>As observed, in terms of both number of parameters used and number of floating-point operation, DRAM is significantly better due to the fact that it adopts the idea of glimpse for visual attention.</p><h3>One More Thing: Spatial Transformer Network</h3><p>I feel like an obligation to mention this paper (too popular not to mention). It tackles the problem in a whole new different way. The idea is not to use “hard” attention and stay away from Reinforcement Learning, but to build something fully differentiable as a “soft” attention model. Refer to paper [5] for detail.</p><p>The intuition is that, since the attention is all about selectively cropping a portion of image as “attention”, why not use an affine transformation for that, which handles cropping, translation, rotation, (isotropic) scaling and skewing in a fully differentiable way. The only thing the neural network need to do is to learn these affine transformation parameters. Hence the cool idea illustrated in the diagram below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/566/0*0bm4l5o-O27uwRrO." /></figure><p>U is our original input image. It is first passed to Localisation Net, which is an affine transformation with learnable parameter θ, e.g. as shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/0*sYwc4cY56WMbfv4J." /></figure><p>The output of transformed image will be then sampled by the sampler to provide a “rectified and resized” version of original input image. Typically the sampling process is bilinear interpolation, which is also a differentiable process, as shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/488/0*oyYtqbp2pXLVSz8E." /></figure><p>In this way, the original image will be affine transformed (or attended) to rectified output image, as shown below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/573/0*AjM5Sfzs8nKkIPY-." /></figure><p><a href="https://goo.gl/qdEhUu">This video</a> shows the result of spatial transformer which is very impressive.</p><h3>Conclusion</h3><p>To summarize, we have discussed the following approaches in order to solve augmented MNIST and SVHN problem:</p><p>For MNIST:<br>(1) Use a classical 2 layer CNN, refer to keras example<br>(2) Use “hard” attention model RAM, refer to paper [2]</p><p>For SVHN:<br>(1) Use 11 layer CNN, refer to paper [1]<br>(2) Use DRAM as extension to RAM, refer to paper [3]</p><p>Last but not the least, spatial transformer network, as “soft” attention solution</p><h3>About Dataset</h3><h4>MNIST</h4><p>The original MNIST dataset is available from here: <a href="http://yann.lecun.com/exdb/mnist/">http://yann.lecun.com/exdb/mnist/</a></p><h4>SVHN</h4><p>Google Street View House Number images (SVHN), available from here: <a href="http://ufldl.stanford.edu/housenumbers/">http://ufldl.stanford.edu/housenumbers/</a></p><ul><li>10 classes, 1 for each digit. Digit ‘1’ has label 1, ‘9’ has label 9 and ‘0’ has label 10.</li><li>73257 digits for training, 26032 digits for testing, and 531131 additional, somewhat less difficult samples, to use as extra training data</li></ul><h3>Reference</h3><p>[1] Multi-digit Number Recognition from Street View Imagery using Deep Convolutional Neural Networks. Ian J. Goodfellow, Yaroslav Bulatov, Julian Ibarz, Sacha Arnoud, and Vinay Shet (2013). <a href="https://arxiv.org/abs/1312.6082">https://arxiv.org/abs/1312.6082</a></p><p>[2] Recurrent Models of Visual Attention, Volodymyr Mnih, Nicolas Heess, Alex Graves, Koray Kavukcuoglu, <a href="https://arxiv.org/abs/1406.6247">https://arxiv.org/abs/1406.6247</a></p><p>[3] Multiple Object Recognition with Visual Attention, Jimmy Ba, Volodymyr Mnih, Koray Kavukcuoglu, <a href="https://arxiv.org/abs/1412.7755">https://arxiv.org/abs/1412.7755</a></p><p>[4] Reading Digits in Natural Images with Unsupervised Feature Learning , Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, Andrew Y. Ng on NIPS Workshop on Deep Learning and Unsupervised Feature Learning 2011. <a href="http://ufldl.stanford.edu/housenumbers/nips2011_housenumbers.pdf">http://ufldl.stanford.edu/housenumbers/nips2011_housenumbers.pdf</a></p><p>[5] Spatial Transformer Networks, <a href="https://arxiv.org/find/cs/1/au:+Jaderberg_M/0/1/0/all/0/1">Max Jaderberg</a>, <a href="https://arxiv.org/find/cs/1/au:+Simonyan_K/0/1/0/all/0/1">Karen Simonyan</a>, <a href="https://arxiv.org/find/cs/1/au:+Zisserman_A/0/1/0/all/0/1">Andrew Zisserman</a>, <a href="https://arxiv.org/find/cs/1/au:+Kavukcuoglu_K/0/1/0/all/0/1">Koray Kavukcuoglu</a>, <a href="https://arxiv.org/abs/1506.02025">https://arxiv.org/abs/1506.02025</a></p><p>[6] Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning, Williams et al, <a href="http://www-anw.cs.umass.edu/~barto/courses/cs687/williams92simple.pdf">http://www-anw.cs.umass.edu/~barto/courses/cs687/williams92simple.pdf</a></p><p>[7] <a href="https://github.com/tianyu-tristan/Visual-Attention-Model">Github Repo</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=708813c2912c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/visual-attention-model-in-deep-learning-708813c2912c">Visual Attention Model in Deep Learning</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>