<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Md Ashiqur Rahman on Medium]]></title>
        <description><![CDATA[Stories by Md Ashiqur Rahman on Medium]]></description>
        <link>https://medium.com/@ashiqbuet14?source=rss-fc1a468ef277------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*PbjTAmptKltYF60-vw_n2A.jpeg</url>
            <title>Stories by Md Ashiqur Rahman on Medium</title>
            <link>https://medium.com/@ashiqbuet14?source=rss-fc1a468ef277------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 23:11:31 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@ashiqbuet14/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Understanding Conditional Variational Autoencoders]]></title>
            <link>https://medium.com/data-science/understanding-conditional-variational-autoencoders-cd62b4f57bf8?source=rss-fc1a468ef277------2</link>
            <guid isPermaLink="false">https://medium.com/p/cd62b4f57bf8</guid>
            <category><![CDATA[generative-model]]></category>
            <category><![CDATA[unsupervised-learning]]></category>
            <category><![CDATA[variational-autoencoder]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Md Ashiqur Rahman]]></dc:creator>
            <pubDate>Sat, 16 May 2020 16:20:16 GMT</pubDate>
            <atom:updated>2025-03-13T20:31:08.020Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/921/1*gA2JdLZJ12gQtnqakU7KAQ.png" /><figcaption>modified from<a href="https://www.researchgate.net/profile/Abu_Kamruzzaman/publication/332540618_Developing_Deep_Learning_Models_to_Simulate_Human_Declarative_Episodic_Memory_Storage/links/5d2f8b33458515c11c392f78/Developing-Deep-Learning-Models-to-Simulate-Human-Declarative-Episodic-Memory-Storage.pdf">*</a></figcaption></figure><p>The variational autoencoder or VAE is a directed graphical generative model which has obtained excellent results and is among the state of the art approaches to generative modeling. It assumes that the data is generated by some random process, involving an unobserved continuous random variable <strong><em>z. </em></strong>it is assumed that the <strong><em>z</em></strong> is generated from some prior distribution <strong><em>P_θ(z) </em></strong>and the data is generated from some condition distribution <strong><em>P_θ(X|Z)</em></strong>, where <strong><em>X</em></strong> represents that data. The <strong><em>z</em></strong> is sometimes called the hidden representation of data <strong><em>X</em></strong>.</p><p>Like any other autoencoder architecture, it has an encoder and a decoder. The encoder part tries to learn<strong><em> q_φ(z|x)</em></strong>, which is equivalent to learning hidden representation of data <strong><em>X </em></strong>or encoding the <strong><em>X </em></strong>into the hidden representation (probabilistic encoder). The decoder part tries to learn<strong><em> P_θ(X|z)</em></strong> which decoding the hidden representation to input space. The graphical model can be expressed as the following figure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/939/1*iO1uxR--NmFCgLkmG_gnXA.png" /><figcaption>(<a href="https://arxiv.org/pdf/1312.6114.pdf">source</a>)</figcaption></figure><p>The model is trained to minimize the objective function</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/930/1*qpeycNFiCNfhBKDb7PH1jQ.png" /></figure><p>The first term in this loss is the reconstruction error or expected negative log-likelihood of the datapoint. The expectation is taken with respect to the encoder’s distribution over the representations by taking a few samples. This term encourages the decoder to learn to reconstruct the data when using samples from the latent distribution. A large error indicates the decoder is unable to reconstruct the data.</p><p>The second term is the Kullback-Leibler divergence between the encoder’s distribution <strong><em>q_φ(z|x)</em></strong> and <strong><em>p(z)</em></strong>. This divergence measures how much information is lost when using <strong><em>q</em></strong> to represent a prior over<strong><em> z</em></strong> and encourages its values to be Gaussian.</p><p>During generation, samples from <strong><em>N(0,1)</em></strong> is simply fed into the decoder. The training and the generation process can be expressed as the following</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/491/1*6MxYa1CLA4Kuy0i65Wc4sw.png" /><figcaption>A training-time variational autoencoder implemented as a feedforward neural network, where P(X|z) is Gaussian. Red shows sampling operations that are non-differentiable. Blue shows the loss calculation. (<a href="https://arxiv.org/pdf/1606.05908.pdf">source</a>)</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/338/1*3eiBobCP6g1mqscka7MVag.png" /><figcaption>The testing-time variational “autoencoder,” which allows us to generate new samples. The “encoder” pathway is simply discarded. (<a href="https://arxiv.org/pdf/1606.05908.pdf">source</a>)</figcaption></figure><p>The reason for such a brief description of VAE is, it is not the main focus but very much related to the main topic.</p><p>The one problem for generating data with VAE is we do not have any control over what kind of data it generates. For example, if we train a VAE with the MNIST data set and try to generate images by feeding<strong><em> Z ~ N(0,1)</em></strong> into the decoder, it will also produce different random digits. If we train it well, the images will be good but we will have no control over what digit it will produce. For example, you can not tell the VAE to produce an image of digit ‘2’.</p><p>For this, we need to have a little change to our VAE architecture. Let’s say, given an input <strong><em>Y</em></strong>(label of the image) we want our generative model to produce output <strong><em>X</em></strong>(image). So, the process of VAE will be modified as the following: given observation <strong><em>y, z</em></strong> is drawn from the prior distribution <strong><em>P_θ(z|y</em></strong>), and the output <strong>x </strong>is generated from the distribution <strong><em>P_θ(x|y, z)</em></strong>. Please note that, for simple VAE, the prior is<strong><em> P_θ(z)</em></strong> and the output is generated by<strong><em> P_θ(x|z)</em></strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/513/1*QjUR_IRCfq7sNV6yb6W6cQ.png" /><figcaption>Visual representation task in conditional VAE (<a href="https://ijdykeman.github.io/ml/2016/12/21/cvae.html">source</a>)</figcaption></figure><p>So, here the encoder part tries to learn<strong><em> q_φ(z|x,y)</em></strong>, which is equivalent to learning hidden representation of data <strong><em>X</em></strong> or encoding the <strong><em>X</em></strong> into the hidden representation conditioned <strong><em>y</em></strong>. The decoder part tries to learn<strong><em> P_θ(X|z,y)</em></strong> which decoding the hidden representation to input space conditioned by <strong><em>y</em></strong>. The graphical model can be expressed as the following figure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/803/1*zcqQjh9NsvJD72PU8xv9Rg.png" /><figcaption>(<a href="https://arxiv.org/pdf/1606.05908.pdf">Source</a>)</figcaption></figure><p>The neural network architecture of Conditional VAE (CVAE) can be represented as the following figure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/921/1*gA2JdLZJ12gQtnqakU7KAQ.png" /><figcaption>X is the image. Y is the label of the image which can be in 1 hot-vector representation.</figcaption></figure><p>The implementation of CVAE in Keras is available <a href="https://github.com/nnormandin/Conditional_VAE/blob/master/Conditional_VAE.ipynb">here</a>.</p><h3>References:</h3><ol><li><a href="https://papers.nips.cc/paper/5775-learning-structured-output-representation-using-deep-conditional-generative-models.pdf">Learning Structured Output Representation using Deep Conditional Generative Models</a></li><li><a href="https://arxiv.org/pdf/1606.05908.pdf">Tutorial on Variational Autoencoders</a></li><li><a href="https://arxiv.org/pdf/1312.6114.pdf">Auto-Encoding Variational Bayes</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cd62b4f57bf8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/understanding-conditional-variational-autoencoders-cd62b4f57bf8">Understanding Conditional Variational Autoencoders</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Comprehensive Guide to Correlational Neural Network with Keras]]></title>
            <link>https://medium.com/data-science/a-comprehensive-guide-to-correlational-neural-network-with-keras-3f7886028e4a?source=rss-fc1a468ef277------2</link>
            <guid isPermaLink="false">https://medium.com/p/3f7886028e4a</guid>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[keras]]></category>
            <dc:creator><![CDATA[Md Ashiqur Rahman]]></dc:creator>
            <pubDate>Sun, 04 Aug 2019 10:02:23 GMT</pubDate>
            <atom:updated>2025-03-13T20:33:05.385Z</atom:updated>
            <content:encoded><![CDATA[<h3>A Comprehensive Guide to the Correlational Neural Network with Keras</h3><h4>From Theory to Implementation</h4><p>Human beings, along with many other animals, have 5 basic senses:<em> Sight, Hearing, Taste, Smell, and Touch</em>. We also have additional senses like a <em>sense of balance and acceleration</em>, a <em>sense of time</em>, etc. Every single moment, the human brain processes information from all these sources, and each of these senses affects our decision-making process. During any conversation, the movement of lips, facial expressions, and the sound produced by the vocal cord help us to fully understand the meaning of words pronounced by the speaker. We can even understand words only by seeing the lips movement without any sound. This visual information is not just supplementary but necessary. This was first exemplified in the <strong>McGurk effect</strong> (McGurk &amp; MacDonald, 1976), where a visual <strong>/ga/</strong> with a voiced <strong>/ba/</strong> is perceived as <strong>/da/ </strong>by most subjects. As we want our machine learning models to achieve human-level performance, it is also necessary to enable them to use data from various sources.</p><p>In machine learning, these types of data that come from different heterogeneous sources are known as multimodal data. For example, audio and visual information is used for speech recognition. It is difficult to directly use these multimodal data in the training, as they may be of different dimensionality and data types. So much of the attention is paid to learning a common representation of these multimodal data. Learning common representations for such multiple views of data will help in several downstream applications. For example, learning a common representation for videos and their audios could help in generating more accurate subtitles for that video than if it generated by only using audio. But how do you learn this common representation?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*AJcD5AFwJGSek1i03MJZHw.png" /><figcaption><strong>An abstract view of CorrNet. It tires to learn a common representation of both views of data and tries to reconstruct both the view from that encoded representation.</strong></figcaption></figure><p>Correlational neural network (CorrNet) is one of the methods for learning common representations. Its architecture is almost the same as a conventional single-view deep autoencoder. But it contains one encoder-decoder pair for each modality of data.</p><p>Let’s consider a two-view input,<strong> Z = [Ia, Iv]</strong>, where <strong>Ia </strong>and <strong>Iv </strong>are two different views of data such as audio and video. In the figure below, one simple architecture of CorrNet with this data is shown.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/788/1*C3xccrXaVLAsaPf86s9nLQ.png" /><figcaption>Example of CorrNet with bimodal data<strong> Z = [Ia, Iv]</strong> where Ia and Iv are two different views of data like (audio and video )</figcaption></figure><p>Where both encoder and decoder are single-layered. <strong><em>H </em></strong>is the encoded representation. <strong><em>H</em></strong>a<strong><em> = f( W</em></strong>a<strong><em>.I</em></strong>a<strong><em>+b) </em></strong>is encoded representation Ia.<strong><em> f() </em></strong>is any non linearity ( sigmoid, tanh , etc.). Same is the case of <strong><em>Hv </em></strong>= <strong><em>f( W</em></strong>a<strong><em>.I</em></strong>a<strong><em>+b). </em></strong>And<strong><em> </em></strong>the common representation of bimodal data <strong>Z </strong>is given as :</p><p><strong>H = <em>f(</em> <em>W</em>a<em>.I</em>a + <em>Wv.Iv + b).</em></strong></p><p>In the decoder part, the model tries to reconstruct the input from the common representation <strong>H </strong>by<strong> <em>I’a</em></strong> = <strong><em>g(</em>W’a.H+b’<em>) </em></strong>and<strong><em> I’v = g(</em>W’vH+b’<em>)</em></strong>, where <strong><em>g()</em></strong> is any activation and <strong>I’a </strong>and <strong>I’v </strong>is the reconstructed Inputs.</p><p>During training, the gradient is calculated based on three losses:</p><p>i) Minimize the self-reconstruction error, that is, minimize the error in reconstructing <strong><em>Ia </em></strong>from <strong><em>Ia </em></strong>and <strong><em>Iv </em></strong>from <strong><em>Iv</em></strong>.</p><p>ii) Minimize the cross-reconstruction error, that is, minimize the error in reconstructing <strong>Ia </strong>from <strong>Iv </strong>and <strong>Iv </strong>from <strong>Ia</strong>.</p><p>iii) Maximize the correlation between the hidden representations of both views, that is, maximize the correlation between <strong>Ha </strong>and <strong>Hv</strong>.</p><p>The total loss can be written as</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Z1dcFbJUggZcKPHnC_Gonw.png" /></figure><p>Here, <strong>Lr()</strong> represents reconstruction loss which may be ‘<strong>mean squared error’</strong> or ‘<strong>mean absolute error’.</strong> Our target is to minimize this loss. And as we want to increase the correlation, it is subtracted from the loss, i.e., the higher is the correlation the lower is the loss.</p><h4>Implementation</h4><p>Total implementation can be divided into three parts: Model Building, Setting loss functions, and Training.</p><p>In the Model Building phase, we have to create that auto encodes architecture. First, we have to include all the necessary packages.</p><pre>from <strong>keras </strong>import  <strong>Model</strong><br>from <strong>keras.layers import  Input,Dense,concatenate,Add</strong><br>from <strong>keras </strong>import backend as <strong>K,activationsfrom tensorflow</strong> <br>import <strong>Tensor </strong>as Tfrom<strong> keras.engine.topology</strong> <br>import <strong>Layer</strong><br>import numpy as <strong>np </strong></pre><p>Then we have to create CorrNet architecture. For simplicity, it has a single-layered encoder and decoder.</p><pre>class <strong>ZeroPadding</strong>(Layer):<br>     def <strong>__init__</strong>(self, **kwargs):<br>          super(ZeroPadding, self).__init__(**kwargs)</pre><pre>     def <strong>call</strong>(self, x, mask=None):<br>          return K.zeros_like(x) <br>     <br>     def <strong>get_output_shape_for</strong>(self, input_shape):<br>          return input_shape</pre><pre>#inputDimx,inputDimy are the dimentions two input modalities. And #hdim_deep is the dimentions of shared representation( <em>kept as #global veriable</em>) </pre><pre>inpx = <strong>Input</strong>(<strong>shape</strong>=(inputDimx,))               <br>inpy = <strong>Input</strong>(<strong>shape</strong>=(inputDimx,)) <br><strong><em>#Encoder</em></strong>                <br>hl = <strong>Dense</strong>(hdim_deep,<strong>activation</strong>=&#39;relu&#39;)(inpx)                              hr = <strong>Dense</strong>(hdim_deep,<strong>activation</strong>=&#39;relu&#39;)(inpy)                             h =  Add()([hl,hr]) <strong><em>#Common representation/Encoded representation</em></strong></pre><pre><strong><em>#decoder</em></strong></pre><pre>recx = <strong>Dense</strong>(inputDimx,activation=&#39;relu&#39;)(h)                                         recy = <strong>Dense</strong>(inputDimy,activation=&#39;relu&#39;)(h)</pre><pre><strong>CorrNet</strong> = <strong>Model</strong>( [inpx,inpy],[recx,recy,h])<br> <br><strong>CorrNet</strong>.summary()</pre><pre>&#39;&#39;&#39;we have to create a separate model for training this <strong>CorrNet</strong><br>As during training we have to take gradient from 3 different loss function and which can not be obtained from signle input.If you look closely to the loss function, we will see it has different input parameter&#39;&#39;&#39;</pre><pre>[recx0,recy0,h1] = <strong>CorrNet</strong>( [inpx, inpy])<br>[recx1,recy1,h1] = <strong>CorrNet</strong>( [inpx, ZeroPadding()(inpy)])[recx2,recy2,h2] = <strong>CorrNet</strong>( [ZeroPadding()(inpx), inpy ])                <br>H= <strong>concatenate</strong>([h1,h2]) <br><strong>model</strong> = Model( [inpx,inpy],[recx0,recx1,recx2,recy0,recy1,recy2,H])</pre><p>Now we have to write the correlational loss function for our model.</p><pre><strong>Lambda </strong>= 0.01<br>#by convention error function need to have two argument, but in #correlation loss we do not need any argument. Loss is computed #competely based on correlation of two hidden representation. For #this a &#39;<strong>fake</strong>&#39; argument is used.</pre><pre>def <strong><em>correlationLoss</em></strong>(<em>fake</em>,H):<br>  y1 = H[:,:hdim_deep]<br>  y2 = H[:,hdim_deep:]<br>  y1_mean = K.mean(y1, axis=0)<br>  y1_centered = y1 - y1_mean<br>  y2_mean = K.mean(y2, axis=0)<br>  y2_centered = y2 - y2_mean<br>  corr_nr = K.sum(y1_centered * y2_centered, axis=0) <br>  corr_dr1 = K.sqrt(K.sum(y1_centered * y1_centered, axis=0) + 1e-8)<br>  corr_dr2 = K.sqrt(K.sum(y2_centered * y2_centered, axis=0) + 1e-8)<br>  corr_dr = corr_dr1 * corr_dr2<br>  corr = corr_nr / corr_dr <br>  <strong>return </strong>K.sum(corr) * Lambda<br>def <strong><em>square_loss</em></strong>(y_true, y_pred):<br>  error = ls.mean_squared_error(y_true,y_pred)<br>  <strong>return </strong>error</pre><p>Now we have to compile the model and train</p><pre>model.<strong>compile</strong>(<strong>loss</strong>=[<em>square_loss,square_loss,square_loss,  square_loss,square_loss,square_loss,correlationLoss</em>],<strong>optimizer</strong>=&quot;adam&quot;)<br>model.<strong>summary</strong>()<br>&#39;&#39;&#39;<br>Suppose you have already prepared your data and kept one moadlity data in Ia(e.g. Audio) and another in Iv( e.g. Video).To be used by this model Audios and videos must be converted into 1D tensor.<br>&#39;&#39;&#39;</pre><pre>model.<strong>fit</strong>([Ia,Iv],[Ia,Ia,Ia,Iv,Iv,Iv,<em>np.ones((Ia.shape[0],Ia.shape[1]))</em>],nb_epoch=100)</pre><pre>&#39;&#39;&#39;<br><strong><em>np.ones((Ia.shape[0],Ia.shape[1]))</em></strong> is fake tensor that will be   passed to <strong><em>correlationLoss </em></strong>function but will have no use<br>&#39;&#39;&#39;<br>using this model we can generate Ia to Iv.For example, from video Iv we can generate corresponding audio Ia<br>np.zeros(Ia.shape) gives tensors of 0 of dimestions same as output tensor <strong><em>audio</em></strong>  &#39;&#39;&#39; <br><strong>audio</strong>,_,_ = <strong>CorrNet.</strong>predict([np.zeros(Ia.shape),Iv])</pre><p>After training, the common representation learned by this model can be used in different prediction tasks. For example, common representations learned using CorrNet can be used for: cross-language document classification or transliteration equivalence detection. Various studies have found that using common representation increases performance. It can also be used in data generation. For example, in a certain dataset, you have 10000 audio clips with corresponding videos, 5000 audio clips with their videos missing and 5000 videos with their audios missing. In such cases, we can train a CorrNet with 10000 audios and videos and can be used to generate missing videos for audio and vice versa.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3f7886028e4a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/a-comprehensive-guide-to-correlational-neural-network-with-keras-3f7886028e4a">A Comprehensive Guide to Correlational Neural Network with Keras</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Neural Architecture Search (NAS)- The Future of Deep Learning]]></title>
            <link>https://medium.com/data-science/neural-architecture-search-nas-the-future-of-deep-learning-c99356351136?source=rss-fc1a468ef277------2</link>
            <guid isPermaLink="false">https://medium.com/p/c99356351136</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[automl]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[autodl]]></category>
            <dc:creator><![CDATA[Md Ashiqur Rahman]]></dc:creator>
            <pubDate>Sat, 08 Jun 2019 22:06:21 GMT</pubDate>
            <atom:updated>2025-03-13T20:33:42.199Z</atom:updated>
            <content:encoded><![CDATA[<h4>All you need to know as a beginner</h4><p>Most of us have probably heard about the success of ResNet, winner of ILSVRC 2015 in image classification, detection, and localization and Winner of MS COCO 2015 detection, and segmentation<strong>.</strong> It is an enormous architecture with skip connections all over. While using this ResNet as a pre-trained network for my machine learning project, I thought “ How can someone come out with such an architecture? ’’</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*6hF97Upuqg_LdsqWY6n_wg.png" /><figcaption>Large Human Engineered Images Classification Architectures(<a href="https://arxiv.org/pdf/1512.03385.pdf">image source</a>)</figcaption></figure><p>Soon after I learned that many engineers and scientists with their years of experience build this architecture. And there is more of a hunch than complete math that will tell you “ we need to have a 5x5 filter now to achieve the best accuracy ’’. We have wonderful architectures for the image classification task, but for other tasks, we have to spend much of our energy to find an architecture with reasonable performance for those tasks. It would certainly be better if we can automate this architecture modeling task just as we learn the parameters of machine learning models.</p><p>Neural Architecture Search (NAS), the process of automating architecture engineering i.e. finding the design of our machine learning model. Where we need to provide a NAS system with a dataset and a task (classification, regression, etc), and it will give us the architecture. And this architecture will perform best among all other architecture for that given task when trained by the dataset provided. NAS can be seen as a subfield of AutoML and has a significant overlap with hyperparameter optimization. To understand NAS we need to look deeply into what it is doing. It finds an architecture from <strong>all possible architectures</strong> by following a <strong>search strategy</strong> that will <strong>maximize the performance</strong>. The following figure summarizes the NAS algorithm.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/941/1*cr1OHTq9lE4GDyOCYeknxQ.png" /><figcaption>Dimensions of NAS method(<a href="https://arxiv.org/pdf/1808.05377.pdf">reference</a>)</figcaption></figure><p>It has 3 separate dimensions: Search Space, Search Strategy and Performace Estimation.</p><p><strong>Search space </strong>defines<strong> </strong>what neural architecture a NAS approach might discover in principle. It can be chain-like architecture where the output of layer (n-1) is fed as an input of layer (n). Or it can be a modern complex architecture with skip connection (multi-branch network).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/592/1*rxBO_zugD4EGXTzwjPXoOQ.png" /><figcaption>A chain-like and multi-branch network (<a href="https://arxiv.org/pdf/1808.05377.pdf">image source</a>)</figcaption></figure><p>Sometimes people do want to use handcrafted outer architecture( macro-architectures) with repeated motifs or cells. In such cases, the outer structure is fixed and NAS only searches for the cell architectures. This type of search is known as micro-search or cell search.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/659/1*gLod-YUqTRInKm90QvI3fA.png" /><figcaption>left: The cell architecture right: cells are put in the handcrafted outer structure(<a href="https://arxiv.org/pdf/1808.05377.pdf">source</a>).</figcaption></figure><p>In many NAS methods, both micro and macro structures are searched in a hierarchical fashion; which consists of several levels of motifs. The first level consists of the set of primitive operations, the second level of different motifs that connect primitive operations via a directed acyclic graph, the third level of motifs that encode how to connect second-level motifs, and so on.</p><p>To explain the <strong>search strategy</strong> and <strong>performance estimation, </strong>three different NAS methods will be discussed in the following part.</p><p><strong>Reinforcement Learning</strong></p><p>Reinforcement learning is the problem faced by an agent( a program) that must learn behavior through trial and error interactions with a dynamic environment to maximize some reward. An agent( performs some action according to some policy parametrized by θ. Then that agent updates the policy θ from the reward for that action taken. In the case of NAS, the agent produces the model architecture, <em>child network</em>(<em> the action).</em> Then the model is trained on the dataset and the performance of the model on the validation data is taken as a reward.</p><figure><img alt="neural architecture search à¦à¦° à¦à¦¬à¦¿à¦° à¦«à¦²à¦¾à¦«à¦²" src="https://cdn-images-1.medium.com/proxy/1*hIif88uJ7Te8MJEhm40rbw.png" /><figcaption>The controller plays the role of an agent and accuracy is taken as the reward(<a href="https://arxiv.org/pdf/1611.01578.pdf">reference</a>)</figcaption></figure><p>In general, a Recurrent Neural Network (RNN) is taken as a controller or agent. It produces string and the model is built form that string stochastically.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/756/1*jsdG_98K-kCLyqCY4Hxp7w.png" /><figcaption>RNN predicts filter height, filter width, stride height, stride width, and the number of filters for one layer and repeats in a form of string. Each prediction is carried out by a softmax classifier and then fed into the next time step as input. The RNN stops after generating a certain number of outputs. (<a href="https://arxiv.org/pdf/1611.01578.pdf">image source</a>)</figcaption></figure><p>A sample output for such RNN is shown in the above figure. The hyperparameters for each layer of a convolutional neural net are produced repeatedly up to a certain number of times. And then a model is then built following the output of RNN, which is then trained and validation accuracy is calculated.</p><p>Here, the parameters of RNN acts as the policy θ of the agent. Production string (height, filter width, stride height, etc) is the action of the agent. The performance of the model on the validation set is the reward of the agent in this reinforcement method. The RNN updates its parameters θ in a way such that, the expected validation accuracy of the proposed architectures is maximized.</p><p>The RNN is trained by a policy gradient method to iteratively update the policy θ. The main target is to maximize the expected accuracy i.e.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/393/1*LQm7juMhHjdWpY0AHRsyMA.png" /></figure><p>Here <strong>R </strong>is the accuracy of the model, <strong>a_1: T</strong> is the string of length <strong>T </strong>produced by RNN( the action), <strong>θ </strong>is the parameter of RNN.</p><p>As J(θ) is not differentiable, a policy gradient method is used to iteratively update θ. The update function looks like</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/427/1*z7vnBUhIOkGkbO-R1FacgQ.png" /></figure><p>This is an approximation of the original equation. The expectation term of J(θ) is replaced by sampling. The outer summation of this updated equation is for that sampling i.e. <strong><em>m </em></strong>numbers of models sampled from RNN and their average is taken. <strong>P(a_t|a_(t−1):1; θc), </strong>is the probability of taking an action <strong>a_t </strong>at time<strong> t </strong>given all actions from<strong> 1 to (t-1) </strong>according to policy<strong> θ or </strong>predicted by RNN with parameter<strong> θ </strong>(<a href="https://arxiv.org/pdf/1611.01578.pdf">reference</a>)<strong>.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*hHxhu85PiOjVphcFIDpO5Q.jpeg" /><figcaption>Example of strings produced by RNN to create a model(<a href="https://arxiv.org/pdf/1611.01578.pdf">reference</a>)</figcaption></figure><p>Encoding skip connections is a bit tricky. For this RNN for each layer generates an output called <strong>Anchor point</strong>. The anchor<strong> point</strong> is used to indicate skip connections. At layer <em>N, </em>the anchor point<em> </em>will contain <em>N − 1</em> content-based sigmoids to indicate the previous layers that need to be connected.</p><p><strong>Progressive Neural Architecture Search (PNAS)</strong></p><p>PNAS does a cell search as discussed in the search space part of this tutorial. They construct cells from blocks and construct the Complete network by adding the cells in a predefined manner.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/781/1*uJzNgTITofNC1P5ure2UdA.png" /><figcaption>Hierarchy of model architecture(<a href="https://cs.jhu.edu/~cxliu/slides/pnas-talk-eccv.pdf">reference</a>)</figcaption></figure><p>Cells are connected in a series in a predefined number to form the network. And each cell is formed by several blocks( 5 used in the original paper).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/450/1*p69zLeq5W3nY4GrxHVBoBA.png" /><figcaption>(<a href="https://cs.jhu.edu/~cxliu/slides/pnas-talk-eccv.pdf">image source</a>)</figcaption></figure><p>The blocks consist of predefined operations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/984/1*dDkL1l47D6HwVcjRYqHosw.png" /><figcaption>structure of a block. The <strong>Combination </strong>function is just an element-wise addition (reference)</figcaption></figure><p>Operations that have shown in the figure are used in the original paper and can be extended.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/780/1*Pq1bgTkcWfDOfE86NWbz5Q.png" /><figcaption>the complete structure of a Cell(<a href="https://cs.jhu.edu/~cxliu/slides/pnas-talk-eccv.pdf">reference</a>)</figcaption></figure><p>On the left, an example of a complete is shown. Even in this cell or micro search, there are in total 10 ¹⁴ valid combinations to check to find the best cell structure.</p><p>So, to reduce complexity first only cells with only 1 block are constructed. It is easy because with the mentioned operation there are only 256 of different cells that are possible. Then top <strong>K </strong>best-performing cells are chosen to expand for 2 block cells and it is repeated up to 5 blocks.</p><p>But for a reasonable <strong>K</strong>, too many 2-block candidates to train. As a solution to this problem, a “cheap” surrogate model that predicts the final performance simply by reading the string(<em>cell is encoded into a string</em>) is trained. The data for this training is collected when the cells are constructed, trained and validated.</p><p>For example, we can construct all 256 of single block cells and measure their performance. And train the surrogate model with this data. Then use this model to predict the performance of 2 block cells without training or testing them. And of course, the surrogate model should be capable of handling variable size input. And then top K best performing 2 block cells as predicted by the model is chosen. These 2 block cells are then trained, the “surrogate’’ model is finetuned and these cells are expanded to 3 blocks and it is iterated until we get 5 block cells.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/758/1*R3jY5TK6qmuzqOO0au6BVA.png" /><figcaption>Steps in PNAS (reference)</figcaption></figure><p><strong>Differentiable Architecture Search (DARTS)</strong></p><p>The search space for neural architectures is discrete i.e one architecture is different from the other by at least a layer or some parameter in the layer, for example, <em>5x5</em> filter vs <em>7x7</em> filter. In this method, continuous relaxation is applied to this discrete search which enables direct gradient-based optimization.</p><p>The cell we search can be though as a directed acyclic graph where each node <strong><em>x </em></strong>is a latent representation (e.g. a feature map in convolutional networks) and each directed <strong><em>edge (i, j)</em> </strong>is associated with some operation <strong>o(i,j)</strong> (convolution, max-pooling, etc) that transforms <strong><em>x(i) </em></strong>and stored a latent representation at node <strong><em>x(j)</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/696/1*-RyYy4caGkegCZbH5qgjAg.png" /></figure><p>the output of each node can be calculated by the equation on the left. The nodes are enumerated in such a way, that there is an <strong><em>edge(i,j)</em></strong> from node <strong><em>x(i)</em></strong> to <strong><em>x(j),</em></strong> then <strong>i&lt;j</strong>.</p><p>In continuous relaxation, instead of having a single operation between two nodes. a convex combination of each possible operation is used. To model this in the graph, multiple edges between two nodes are kept, each corresponding to a particular operation. And each edge also has a weight <strong><em>α</em></strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/429/1*R84oGRqPaM-64K4BH1KznA.png" /><figcaption>continuous relaxation of the discrete problem(<a href="https://arxiv.org/pdf/1806.09055">reference</a>)</figcaption></figure><p>Now <strong>O(i,j) </strong>the operation between node x(i) and x(j) is a convex combination of a set of operations <strong>o(i,j)</strong> where<strong> o(.)</strong>ϵ <strong>S</strong>, where S is the set of all possible operations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/526/1*ZDixS-R-GDw8gtO5R9A9QA.png" /></figure><p>The output of <strong>O(i,j) </strong>is calculated by the left equation<strong>.</strong></p><p>The training and the validation loss are denoted by <strong><em>L_train </em></strong>and <strong><em>L_val</em></strong> respectively. Both losses are determined not only by the architecture parameters <strong>α</strong> but also by the weights <strong>‘w’ </strong>in the network. The goal for architecture search is to find<strong> α∗</strong> that minimizes the validation loss <strong><em>L_val</em>(w ∗ , α∗ )</strong>, where the weights <strong>‘w∗’</strong> associated with the architecture is obtained by minimizing the training loss.</p><p><strong>w∗</strong> = argmin <strong><em>L_train</em>(w, α∗ )</strong>.</p><p>This implies a bilevel optimization problem with <strong><em>α</em></strong> as the upper-level variable and <strong><em>w </em></strong>as the lower-level variable:</p><p><strong><em>α *</em></strong> = argmin<strong> <em>L_val</em>(w ∗ (α), α)</strong></p><p>s.t. <strong>w ∗ (α) </strong>= argmin <strong><em>L_train</em></strong>(w, α)</p><p>After training some α’s of some edges become much larger than the others. To derive the discrete architecture for this continuous model, in between two nodes the only edge with maximum weight is kept.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/843/1*0hN1wLBbc4gxTllRAVV-NA.png" /><figcaption>a) Operations on the edges are initially unknown. b) Continuous relaxation of the search space by placing a mixture of candidate operations on each edge c) some weights increases and some falls during bilevel optimization d) final architecture is constructed only by taking edges with the maximum weight between two nodes. (<a href="https://arxiv.org/pdf/1806.09055">reference</a>)</figcaption></figure><p>When the cells are found, these are then used to construct larger networks.</p><p>There are many other methods that are also used in neural architecture search. And they achieved an impressive performance. But still, we have very little knowledge for answering the question ‘why does a particular architecture work better?’. I think we are on our way to answering that question.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c99356351136" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/neural-architecture-search-nas-the-future-of-deep-learning-c99356351136">Neural Architecture Search (NAS)- The Future of Deep Learning</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>