<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Kiran Sudhir on Medium]]></title>
        <description><![CDATA[Stories by Kiran Sudhir on Medium]]></description>
        <link>https://medium.com/@kiransudhir95?source=rss-facae13d21a------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*8GjhLsBFx8qkbYa9.</url>
            <title>Stories by Kiran Sudhir on Medium</title>
            <link>https://medium.com/@kiransudhir95?source=rss-facae13d21a------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 04:34:19 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@kiransudhir95/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Generative Adversarial Networks- History and Overview]]></title>
            <link>https://medium.com/data-science/generative-adversarial-networks-history-and-overview-7effbb713545?source=rss-facae13d21a------2</link>
            <guid isPermaLink="false">https://medium.com/p/7effbb713545</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[generative-model]]></category>
            <category><![CDATA[towards-data-science]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Kiran Sudhir]]></dc:creator>
            <pubDate>Wed, 21 Jun 2017 04:00:44 GMT</pubDate>
            <atom:updated>2017-06-22T17:12:18.056Z</atom:updated>
            <content:encoded><![CDATA[<p>Of late, generative modeling has seen a rise in popularity. In particular, a relatively recent model called Generative Adversarial Networks or GANs introduced by Ian Goodfellow et al. shows promise in producing realistic samples. This blog post has been divided into two parts. Part-1 consists of an introduction to GANs, the history behind it, and its various applications. Part-2 consists of an implementation of GANs (with code) to produce image samples.</p><h3>Part 1- Understanding GANs</h3><h4>Generative Modeling</h4><p>Let us first take a look at what generative models are, and how the differ from discriminative models. Say you have input data <em>x</em>, and corresponding output labels <em>y</em>. A discriminative model tries to directly learn the conditional probability distribution <em>P(y|x)</em>. On the other hand, a generative model tries to learn the joint probability distribution <em>P(x,y)</em>. This can be transformed into <em>P(y|x)</em> using Bayes rule. However, additionally, as opposed to discriminative models, generative models can use the joint distribution <em>P(x,y)</em> to generate likely <em>(x,y)</em> samples.</p><h4>What’s the hype all about?</h4><p>So why would one want to study generative models? One might wonder what the big deal is in simply generating more data, especially since there’s such an abundance of data already available. But in reality, this could be put to several uses. For instance, one can feed some text written in a particular handwriting as input to a generative model to get generate more text in the same handwriting. Generative models, and GANs in particular, can also be used in exploration in reinforcement learning where they can be used to generate artificial environments. Other applications include conversion of sketches to images, image denoising, conversion of low resolution images to high resolution, generation of art, and conversion of satellite images to maps, to name a few. Apart from the wide range of applications, generative models are particularly useful when a majority of the labels are missing, because of their ability to perform semi-supervised learning.</p><h4>Generative models over the years</h4><p>Now that we have laid the ground on what generative modeling is, and why they are useful, let’s take a look at the various approaches in generative modeling. For the purpose of comparing the models, they can all be described to perform maximum likelihood.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/479/1*otb47mG5KQWNh91prmyxzg.png" /><figcaption>Taxonomy of Generative Models (from Ian Goodfellow’s NIPS tutorial, 2016)</figcaption></figure><p>The above graph depicts the various families of generative models as described by Ian Goodfellow in his NIPS tutorial. Let’s now take a look at the pros and cons of some of the popular approaches in the families of models stated above.</p><h4>Fully Visible Belief Networks</h4><p>They use the chain rule of probability to decompose the probability<br>distribution over a vector, into a product over each of the members of the vector.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*EEOWOQSAOams58A4gMfjkQ.png" /></figure><p>The most popular model in this family is PixelCNN.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vioNjVcI6vpplIuvTLWyQg.jpeg" /><figcaption>Images generated by PixelCNN (van den Ord et al 2016)</figcaption></figure><p>The biggest drawback with FVBNs is that the rate of generating samples is very slow. Every time you want to generate a new sample, you will have to run the model again. This cannot be done in parallel.</p><h4>Models based on change of variables (Non-linear ICA)</h4><p>Such models begin with a simple distribution like a Gaussian and use a non-linear function to transform the distribution to another space. The main disadvantage with this is that the transformation needs to be designed to be invertible, and the latent variables must have the same dimensionality as the data. So if you want to generate 5000 pixels, you need to have 5000 latent variables.</p><h4>Variational Autoencoder</h4><p>The way Variational autoencoders work is by marginalizing out the random variable <em>z</em> from the density function <em>log p(x)</em>. Since this is intractable, it uses a variational approximation. The model wishes to maximize the lower bound on the log likelihood of the data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/360/1*Eg_zDuOlOvpS5yuOtFcC4Q.jpeg" /><figcaption>Images of celebrity-like faces generated by a VAE (By Alec Radford)</figcaption></figure><p>The main drawback here is that the model is asymptotically consistent if the distribution <em>q</em> is perfect. Otherwise, there would be a gap between the lower bound and actual density of the data. Another disadvantage is that the samples generated are of relatively low quality.</p><h4>Boltzmann Machines</h4><p>A Boltzmann machine can be defined by an energy function, and the probability of a particular state, is proportional to each of the value of the energy. To convert this to an actual probability distribution, renormalization is done by dividing the sum over the different states. This sum is intractable, which calls for approximation using Monte Carlo methods. The drawback is that these methods, especially the Markov Chain Monte Carlo methods, don’t perform well in high dimensional spaces. So although they may perform well on images like <a href="http://yann.lecun.com/exdb/mnist/">MNIST</a>, you won’t get a similar performance on images from <a href="http://image-net.org/">ImageNet</a>.</p><h4>Generative Adversarial Networks</h4><p>Now that we have touched upon other popular generative models, we can take a look at GANs, and how they compare against the rest.</p><p>GANs were designed to overcome many of the drawbacks stated in the above models. As opposed to Fully Visible Belief Networks, GANs use a latent code, and can generate samples in parallel. Unlike Variational Autoencoders, GANs are asymptotically consistent. Also, GANs do not require Markov chains, which is an advantage over Boltzmann Machines. Finally, GANs are often highly regarded as producing the best samples, although this is very subjective, and currently a topic of debate, with models like PixelCNN competing with them.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/506/1*_d22i5gAB8UWL68EnjXQwA.png" /><figcaption>Images generated by GAN using the Toronto Face Database (Ian Goodfellow et al 2014)</figcaption></figure><h4>How GANs work</h4><p>Now that we’ve established why GANs are worth studying about, let’s delve deeper into how they really work.</p><p>The main idea behind GANs can be interpreted as a game between two players- the generator and the discriminator. The generator tries to generate samples that follow the same underlying distribution as the train data. The discriminator tries to distinguish between the samples generated by the generator (fake data), and the actual data from the train set. The goal of the generator is to fool the discriminator by closely approximating the underlying distribution in order to generate samples that are indistinguishable from the actual data. On the other hand, the goal of the discriminator is to identify the fake data from the real data. The discriminator simply has the task of a binary classification problem, where it determines if the data is real or fake. A common analogy of this game is that of a forger and the police. The forger is like the generator, trying to counterfeit money, and making it look as legitimate as possible, in order to fool the police. The police is like the discriminator whose goal is to be able to identify money that has been counterfeited.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KF-XzsW2F44sCxlgdDy_9w.png" /><figcaption>GAN overview</figcaption></figure><p>The generator is a differentiable function <em>G</em>, that has parameters that can be learned by gradient descent. The input to <em>G</em> is obtained by sampling the latent vector <em>z</em> from some prior distribution over latent variables. So essentially, <em>z</em> is a vector of unstructured noise. <em>G</em> is applied to z to obtain a sample <em>x</em> from the model, which should ideally be similar to the actual data from the train set. Like the generator, the discriminator is also a differentiable function <em>D</em> that has parameters that can be learned by gradient descent. The function <em>D</em>, when applied to the sample <em>x</em> obtained from <em>G(z)</em> should ideally output a value close to zero, indicating that the sample is fake. When an actual sample from the data is fed as input to <em>D</em>, it should output a value close to one.</p><p>Say 𝞱<em>(D)</em> and 𝞱<em>(G)</em> are the parameters of <em>D</em> and <em>G</em> respectively. The discriminator would want to minimize it’s cost <em>J(D)(</em>𝞱<em>(D)</em>,𝞱<em>(G))</em>, but has no control over 𝞱<em>(G)</em>, whereas the generator would want to minimize <em>J(G)(</em>𝞱<em>(D),</em>𝞱<em>(G))</em> without control over 𝞱<em>(D)</em>. So we would want to find the Nash equilibrium values for <em>(</em>𝞱<em>(D)</em>,𝞱<em>(G))</em> such that <em>J(D)</em> is minimum with respect to 𝞱<em>(D)</em> and <em>J(G)</em> is minimum with respect to 𝞱<em>(G)</em>.</p><p>So how is training actually performed? The training procedure is to<br>choose an optimization algorithm like Adam, for instance, and apply it simultaneously to two minibatches of data, one from the actual training data, and the other from the samples generated by <em>G</em>. Additionally, you could update one player more often than the other player.</p><h4>Generator and Discriminator costs</h4><p>The way the algorithm will proceed depends on the cost of each player. The simplest way to specify the cost is to use the Minimax game where the generator cost is the negative of the discriminator cost. So what exactly is this cost that the discriminator would want to maximize, and the generator minimize? It is simply a standard cross-entropy function between the discriminator’s output and the actual labels (real/fake).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Fqbu2D0jvv5es9lx98iF7A.png" /></figure><p>The first term in J(D) represents feeding the actual data to the discriminator, and the discriminator would want to maximize the log probability of predicting one, indicating that the data is real. The second term represents the samples generated by G. Here, the discriminator would want to maximize the log probability of predicting zero, indicating the the data is fake. The generator, on the other hand tries to minimize the log probability of the discriminator being correct. The solution to this problem is an equilibrium point of the game, which is a saddle point of the discriminator loss.</p><p>The main problem with this Minimax game is that when the discriminator becomes increasingly smart, the gradient for the generator vanishes. One way to fix this is by flipping the order of the arguments in the cross entropy function instead of simply flipping the sign of the discriminator’s cost.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/320/1*BmABwFGxU70CITe909PmjA.png" /></figure><p>The generator would now want to maximize the log probability of the discriminator being wrong. Now, the equilibrium cannot be described by a single value function, and the motivations for this particular cost are far more heuristic.</p><h4>Scaling GANs</h4><p>The initial GANs themselves don’t really scale well to large applications. To overcome this, the Deep Convolution GAN architecture was introduced by Radfort at al. Although originally, GANs were already deep and convolutional, DCGANs stress on having more convolutional layers, and additionally use techniques such as batch normalization. Batch normalization is applied to every layer except the last layer of the generator in order to make the learning process more stable.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/815/1*wm4IDdMOigfTEXNXjFKuBw.png" /><figcaption>DCGAN Architecture (Radford et al 2015)</figcaption></figure><h4>Some more applications</h4><p>It’s no myth that GANs have taken the AI world by a storm, and they are indeed here to stay. Before concluding this section, let’s take a look at some of the fascinating applications of GANs today.</p><p>One popular application is in the generation of a high resolution image from a low resolution image using super resolution techniques.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rkSvwXsWcJ2Aqu6gAMskTA.png" /><figcaption>SRGAN used to obtain high resolution images. The original HR image is first downsampled to make a low-resolution image, and different methods are used to recover the original HR image (Christian Ledig et all 2016)</figcaption></figure><p>Another recent advancement was made by Scott Reed et al in 2016, where GANs were used to synthesize realistic images from text.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*h9ziqVloZE2nLmOQH-LDug.png" /><figcaption>Generative Adversarial Text to Image Synthesis (Scott Reed et al 2016)</figcaption></figure><p>As described earlier, GANs also have many applications in reinforcement learning. They have also been applied in image denoising, and generation of art, to name a few.</p><h3>Part 2— Implementing GANs</h3><p>Now that you’ve been introduced to what GANs are and how exactly they work, you’ve officially reached the exciting part of the blog. It’s time to get started and write some code!</p><p>Since the most prominent application of GANs is in the generation of images, it makes sense to get yourself familiar with coding an image generator in GANs. The code should take a set of images as input and generate a set of similar images as output. Most people often start out with images like MNIST and try to generate more hand-drawn digits using GANs. Here, we will be using images from ImageNet as our input. Since these images are pretty high dimensional, we will be using DCGANs for image generation.</p><p>For the input, the images from the ‘plant’ dataset from ImageNet is used. Each image resized to obtain 128x128 images.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/256/1*a-g7_k_bUO2WG90pAiMjLw.png" /><figcaption>Plant data from ImageNet</figcaption></figure><p>The following DCGAN implementation is in TensorFlow. The crux of this implementation is contained in a Python class called DCGAN, where we will be defining methods for defining the model and training it.</p><p>The generator is defined below. It consists of a linear layer followed by four convolutional layers. The methods linear and conv2dTranspose are helper functions used to obtain the layers. Batch normalization is applied to every layer except the last one.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*E9AXCJIID9BjrFWImgrfBg.png" /></figure><p>Next, we define the discriminator. We define it to have four convolutional layers, followed by a linear layer. We use helper function lrelu to define a custom activation function (Leaky ReLU).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LnipgrZJIiRooKKIKZySJg.png" /></figure><p>Now we can create our generator and discriminator models. We define two discriminators which share the same parameters. One is fed a minibatch of actual images from the train data, whereas the other is fed a minibatch of the images generated by the generator.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rB7mdryw6NAuhhk4PNfAGg.png" /></figure><p>Next, we have to define the loss functions for the generator and discriminator. As described in Part-1 of the blog, we use the cross-entropy function between the discriminator’s output and the actual labels (real/fake). Here, the label ‘real’, corresponds to 1. And the label ‘fake’ corresponds to 0. So the discriminator should aim to output values close to 1 for real images, and output values close to 0 for images produced by the generator. The generator loss is computed using the heuristic specified in Part-1. It is calculated based on the probability of the discriminator being wrong, i.e. the generator would want the discriminator to output values close to 1 for the images it generates.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*k6uqvQTU1MzDp3k7toq0TA.png" /></figure><p>Before we begin training, we need to first define an optimization function to minimize the above losses. Here we make use of the Adam optimizer, with learning rate = 0.0002, for both the generator and the discriminator. Here, d_theta represents all the parameters of the discriminator, and g_theta represents all the parameters of the generator.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QuP9i-AegqxfPkxjym5t0w.png" /></figure><p>The final step is to actually begin training the data and generating images. For each epoch, pairs of two minibatches are generated, one from the train images, and the other from sampling Z. Also, in order to prevent the discriminator loss from going to zero, the generator is updated twice for each update of the discriminator.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_5f2iX0_0k9pEArHKg-v7w.png" /></figure><h4>Results</h4><p>The following is the output that was obtained on running the DCGAN with the plant data from ImageNet as input (resized to 128x128), with number of epochs set to 250.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/256/1*5kIiFpkG_lRhCg7bZ8g44Q.png" /><figcaption>Output images generated by DCGAN</figcaption></figure><p>Each of the four losses are plotted below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VfN3VnmmgJrX5m6i4bWyWw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ELTnivYLwXMxq87RukjasA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qDYvcNaUhwH6srw_DljbwQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4MNlXBYoZZsad-YBksuW0Q.png" /></figure><h4>Conclusions and future work</h4><p>For most epochs, the generator loss turned out to be lower than the discriminator loss. This could be attributed to the fact that the generator was updated twice for every update of the discriminator.</p><p>Although not perfect, many of the generated images do resemble plants/flowers, rather than just a distribution of some random pixels. Hence, it can be concluded that the DCGAN is able to intelligently construct images of its own. However, there is plenty of scope for improvement. One of the major restrictions in this implementation is the lack of computational resources. Image generation itself is a computationally intensive task. In this particular example, the images are of relatively higher resolution as compared to other more primitive image datasets like MNIST. Additionally, the images are colored. Hence, using a GPU will make a huge difference in run-time, enabling the possibility of running more epochs.</p><p>However, simply increasing the epochs might not guarantee the generation of better images, and if the learning rate is too low, the discriminator will start winning, and the quality of the images will start deteriorating. One obvious way to improve the model is to add more layers to the generator. I started out with just one layer, and the results improved significantly on adding more layers. One can also experiment with different types of activation functions as well as optimization functions. I have used the AdamOptimizer here because it has been recommended by Ian Goodfellow himself.</p><p>Although the above example of generating plant images might not be particularly useful, the same underlying model could be used to generate other images like that of kittens and puppies, for instance. This also lays the ground for the development of more complicated applications using GANs such as the generation of art.</p><h4><em>References:</em></h4><iframe src="https://drive.google.com/viewerng/viewer?url=https%3A//arxiv.org/pdf/1701.00160.pdf&amp;embedded=true" width="600" height="780" frameborder="0" scrolling="no"><a href="https://medium.com/media/5caf26d3cd9533ba2a862974431f33c5/href">https://medium.com/media/5caf26d3cd9533ba2a862974431f33c5/href</a></iframe><p><a href="https://github.com/carpedm20/DCGAN-tensorflow"><em>https://github.com/carpedm20/DCGAN-tensorflow</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7effbb713545" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/generative-adversarial-networks-history-and-overview-7effbb713545">Generative Adversarial Networks- History and Overview</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>