<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by mrgrhn on Medium]]></title>
        <description><![CDATA[Stories by mrgrhn on Medium]]></description>
        <link>https://medium.com/@mrgrhn?source=rss-87eead11556c------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*5qacM_Afcu_pHRyHCg635w.jpeg</url>
            <title>Stories by mrgrhn on Medium</title>
            <link>https://medium.com/@mrgrhn?source=rss-87eead11556c------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 21:04:20 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@mrgrhn/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Convolutional AutoEncoders (CAE) with Tensorflow]]></title>
            <link>https://ai.plainenglish.io/convolutional-autoencoders-cae-with-tensorflow-97e8d8859cbe?source=rss-87eead11556c------2</link>
            <guid isPermaLink="false">https://medium.com/p/97e8d8859cbe</guid>
            <category><![CDATA[convolutional-network]]></category>
            <category><![CDATA[tensorflow]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[convolutional-autoencoder]]></category>
            <dc:creator><![CDATA[mrgrhn]]></dc:creator>
            <pubDate>Mon, 25 Jan 2021 16:57:30 GMT</pubDate>
            <atom:updated>2021-01-26T21:46:29.302Z</atom:updated>
            <content:encoded><![CDATA[<h3>Convolutional Autoencoders (CAE) with Tensorflow</h3><p>Autoencoders has been in the deep learning literature for a long time now, most popular for data compression tasks. With their easy structure and not so complicated underlying mathematics, they became one of the first choices when it comes to dimensionality reduction in simple data. However, using basic fully connected layers fail to capture the patterns in pixel-data since they do not hold the neighboring information. For a good capturing of the image data in <em>latent variables</em>, <em>convolutional</em> <em>layers </em>are usually used in autoencoders.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*AzonYpYa-Ft-FonV.png" /></figure><p>For the previous post on Autoncoders, please visit:</p><p><a href="https://mrgrhn.medium.com/autoencoders-with-tensorflow-2f0a7315d161">AutoEncoders with TensorFlow</a></p><h4>Introduction</h4><p>Autoencoders are unsupervised neural network models that summarize the general properties of data in fewer parameters while learning how to reconstruct it after compression[1]. In order to extract the textural features of images, convolutional neural networks provide a better architecture. Moreover, CAEs can be stacked in such a way that each CAE takes the latent representation of the previous CAE for higher-level representations[2]. Nevertheless, in this article, a simple CAE will be implemented having 3 convolutional layers and 3 subsampling layers in between.</p><p>The tricky part of CAEs is at the decoder side of the model. During encoding, the image sizes get shrunk by subsampling with either average pooling or max-pooling. Both operations result in information loss which is hard to re-obtain while decoding.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/850/0*_SmPQ7ClKW-yvjau.jpg" /><figcaption>Max-pooling and average pooling comparison (Figure is taken from [3])</figcaption></figure><p>There are both straightforward and smarter ways to perform this upsampling. The most basic method is populating the same image pixel while increasing the image size by the upsampling rate.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/594/1*fLZ-7T6qHXl0mrEOL0_cuQ.png" /><figcaption>A simple upsampling example (Figure is taken from [4])</figcaption></figure><p>Some models hold the location of the maximum value during encoding and then only give activation to that location while surpassing the others.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/658/1*RIXHJh5Ze5ToKx-TkZoH_w.png" /><figcaption>A max-pooling example (Figure is taken from [5])</figcaption></figure><p>It is also possible to do the mapping from smaller size to larger size with learned weights. However, in this implementation, we will continue with the basic upsampling procedure. The overall architecture mostly resembles the autoencoder that is implemented in the previous post, except 2 fully connected layers are replaced by 3 convolutional layers.</p><h4>A Simple Convolutional Autoencoder with TensorFlow</h4><p>A CAE will be implemented including convolutions and pooling in the encoder, and deconvolution in the decoder. The model is tested on the Stanford Dogs Dataset[6]. For a simple implementation, <a href="https://www.tensorflow.org/api_docs/python/tf/keras">Keras API on TensorFlow</a> backend is preferred with <a href="https://colab.research.google.com/">Google Colab</a> GPU services.</p><p>First, needed libraries are imported.</p><pre>import tensorflow as tf<br>import matplotlib.pyplot as plt<br>from tensorflow.keras import datasets, layers, models, losses, Model<br>from random import randint<br>import numpy as np</pre><p><strong><em>The Data</em></strong></p><p>The data is directly loaded into the Google Colab session from Kaggle. You can follow the steps nicely explained <a href="https://www.kaggle.com/general/74235">here</a> to load the data. Please first go to <a href="https://www.kaggle.com/">Kaggle</a> and sign in to your account. From the top-right of the page click on the icon and go to <em>Account</em>. Then click <em>“Expire API Tokens”</em> and <em>“Create New API Token”</em>, respectively. You will download a file named “<em>kaggle.json”</em>.</p><pre>! pip install -q kaggle</pre><pre>from google.colab import files<br>files.upload()</pre><pre>! mkdir ~/.kaggle</pre><pre>! cp kaggle.json ~/.kaggle/</pre><pre>! chmod 600 ~/.kaggle/kaggle.json</pre><pre>! kaggle datasets download -d miljan/stanford-dogs-dataset-traintest</pre><p>As you run the code above you will</p><ol><li>Install the Kaggle API</li><li>Upload the “<em>.json”</em> file</li><li>Create a directory called “<em>.kaggle”</em> and copy the downloaded file there</li><li>Make the file executable</li><li>Finally, download the dataset.</li></ol><p>The data is unzipped in the path, <em>“/content/”.</em></p><pre>import os<br>os.chdir(&#39;/content&#39;)</pre><pre>!unzip -q &#39;stanford-dogs-dataset-traintest.zip&#39;</pre><p>Then, the files are stored in variables for ease of use. The file names for the train and test images are stored in a list using <em>os.walk</em>. Then the images are read into a tensor.</p><pre>import numpy as np<br>import matplotlib.pyplot as plt</pre><pre>files=[]</pre><pre>for root, _, filenames in os.walk(&#39;/content/cropped/train&#39;):<br>  for filename in filenames:<br>    files.append(os.path.join(root,filename))</pre><pre>x_train = []<br>for f in files:<br>  x_train.append(plt.imread(f))</pre><pre>files=[]</pre><pre>for root, _, filenames in os.walk(&#39;/content/cropped/test&#39;):<br>  for filename in filenames:<br>    files.append(os.path.join(root,filename))</pre><pre>x_test = []<br>for f in files:<br>  x_test.append(plt.imread(f))</pre><pre>x_train = np.asarray(x_train)<br>x_test = np.asarray(x_test)</pre><p>A sample from the dataset is as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/888/0*CKd-7yHzzcTQU17Q.png" /><figcaption>The whole Stanford Dogs Dataset is available at [7]</figcaption></figure><p><strong><em>The Encoder</em></strong></p><p>The encoder model consists of three convolution layers having features maps of sizes 32, 64, and 128 each followed by a max-pooling subsampling. The 3x3 kernels and ReLU activations are also used.</p><pre>encoder = models.Sequential()</pre><pre>encoder.add(layers.Conv2D(32, 3, strides=1, padding=&#39;same&#39;, activation=&#39;relu&#39;, input_shape=x_train.shape[1:]))<br>encoder.add(layers.MaxPooling2D(2, strides=2))</pre><pre>encoder.add(layers.Conv2D(64, 3, strides=1, padding=&#39;same&#39;, activation=&#39;relu&#39;))<br>encoder.add(layers.MaxPooling2D(2, strides=2))</pre><pre>encoder.add(layers.Conv2D(128, 3, strides=1, padding=&#39;same&#39;, activation=&#39;relu&#39;))<br>encoder.add(layers.MaxPooling2D(2, strides=2))</pre><pre>encoder.summary()</pre><pre>Model: &quot;sequential_9&quot; _________________________________________________________________ Layer (type)                 Output Shape              Param #    ================================================================= conv2d_27 (Conv2D)           (None, 224, 224, 32)      896        _________________________________________________________________ max_pooling2d_24 (MaxPooling (None, 112, 112, 32)      0          _________________________________________________________________ conv2d_28 (Conv2D)           (None, 112, 112, 64)      18496      _________________________________________________________________ max_pooling2d_25 (MaxPooling (None, 56, 56, 64)        0          _________________________________________________________________ conv2d_29 (Conv2D)           (None, 56, 56, 128)       73856      _________________________________________________________________ max_pooling2d_26 (MaxPooling (None, 28, 28, 128)       0          ================================================================= Total params: 93,248 Trainable params: 93,248 Non-trainable params: 0</pre><p><strong><em>The Decoder</em></strong></p><p>The decoder takes the tensor having the sizes of (batch size, 28, 28, 128) from the encoder and passes it through convolutional layers having the feature map size of 128, 16, and 3. The resulting tensor has a channel size of 3 representing the RGB. Simple upsampling is performed as described earlier.</p><pre>decoder = models.Sequential()</pre><pre>decoder.add(layers.Conv2D(128, 3, strides=1, padding=&#39;same&#39;, activation=&#39;relu&#39;, input_shape=encoder.output.shape[1:]))<br>decoder.add(layers.UpSampling2D(2))</pre><pre>decoder.add(layers.Conv2D(16, 3, strides=1, padding=&#39;same&#39;, activation=&#39;relu&#39;))<br>decoder.add(layers.UpSampling2D(2))</pre><pre>decoder.add(layers.Conv2D(3, 3, strides=1, padding=&#39;same&#39;, activation=&#39;relu&#39;))<br>decoder.add(layers.UpSampling2D(2))</pre><pre>decoder.summary()</pre><pre>Model: &quot;sequential_10&quot; _________________________________________________________________ Layer (type)                 Output Shape              Param #    ================================================================= conv2d_30 (Conv2D)           (None, 28, 28, 128)       147584     _________________________________________________________________ up_sampling2d_3 (UpSampling2 (None, 56, 56, 128)       0          _________________________________________________________________ conv2d_31 (Conv2D)           (None, 56, 56, 16)        18448      _________________________________________________________________ up_sampling2d_4 (UpSampling2 (None, 112, 112, 16)      0          _________________________________________________________________ conv2d_32 (Conv2D)           (None, 112, 112, 3)       435        _________________________________________________________________ up_sampling2d_5 (UpSampling2 (None, 224, 224, 3)       0          ================================================================= Total params: 166,467 Trainable params: 166,467 Non-trainable params: 0</pre><p><strong><em>The Autoencoder</em></strong></p><p>CAE is constructed by setting the computation graph. Model is compiled and trained with <em>Adam </em>optimizer, <em>mean squared error</em> loss, and a batch size of 64 for 40 epochs.</p><pre>conv_autoencoder = Model(inputs=encoder.input, outputs=decoder(encoder.outputs))</pre><pre>conv_autoencoder.compile(optimizer=&#39;adam&#39;, loss=losses.mean_squared_error)</pre><pre>history = conv_autoencoder.fit(x_train, x_train, batch_size=64, epochs=40, validation_data=(x_test, x_test))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/891/1*E3yD1rP-KO0lXfnIdPdokw.png" /></figure><p>A similar approach to what has been done in the previous post for qualitative analysis is followed here as well. Taking two random images and encoding them into latent space, then interpolating 16 more latent vectors between the original vectors, and finally decoding them yielded the results below. We can observe the gradual shift from one dog breed to another one.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0MXAddBqpgQQ-FYA5omijg.png" /></figure><blockquote>Final Remark: Please do not forget to leave some food and water outside your apartment for the street animals.</blockquote><h4>convolutional_autoencoder_tensorflow.ipynb</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/55500cd15635d5f0b98a976651509576/href">https://medium.com/media/55500cd15635d5f0b98a976651509576/href</a></iframe><h4>Conclusion</h4><p>As compared to the Autoencoders with fully connected layers, Convolutional Autoencoders does a better job to encapsulate the underlying patterns in the pixel data. In this article, a more challenging dataset is used with larger image sizes and RGB channels. Still, CAE performs at least as well as a standard Autoencoder through the qualitative analysis.</p><p>Hope you enjoyed it. See you in the following Autoencoder applications.</p><p>Best wishes…</p><p>mrgrhn</p><h4>References</h4><ol><li>Gallinari, P., LeCun, Y., Thiria, S., &amp; Fogelman-Soulie, F. (1987). “Memoires associatives distribuees”. Proceedings of COGNITIVA 87. Paris, La Villette.</li><li>Masci, Jonathan &amp; Meier, Ueli &amp; Ciresan, Dan &amp; Schmidhuber, Jürgen. (2011). “Stacked Convolutional Auto-Encoders for Hierarchical Feature Extraction”. 52–59. 10.1007/978–3–642–21735–7_7.</li><li>Aljaafari, Nura. (2018). “Ichthyoplankton Classification Tool using Generative Adversarial Networks and Transfer Learning”.</li><li><a href="https://zhuanlan.zhihu.com/p/67510665">https://zhuanlan.zhihu.com/p/67510665</a></li><li><a href="https://www.programmersought.com/article/2908798542/">https://www.programmersought.com/article/2908798542/</a></li><li>Khosla, A., Jayadevaprakash, N., Yao, B., &amp; Fei-Fei, L. (2012). “Novel Dataset for Fine-Grained Image Categorization: Stanford Dogs”.</li><li><a href="http://vision.stanford.edu/aditya86/StanfordDogs/">http://vision.stanford.edu/aditya86/StanfordDogs/</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=97e8d8859cbe" width="1" height="1" alt=""><hr><p><a href="https://ai.plainenglish.io/convolutional-autoencoders-cae-with-tensorflow-97e8d8859cbe">Convolutional AutoEncoders (CAE) with Tensorflow</a> was originally published in <a href="https://ai.plainenglish.io/">Artificial Intelligence in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AutoEncoders with TensorFlow]]></title>
            <link>https://medium.com/analytics-vidhya/autoencoders-with-tensorflow-2f0a7315d161?source=rss-87eead11556c------2</link>
            <guid isPermaLink="false">https://medium.com/p/2f0a7315d161</guid>
            <category><![CDATA[compression]]></category>
            <category><![CDATA[reconstruction]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[autoencoder]]></category>
            <dc:creator><![CDATA[mrgrhn]]></dc:creator>
            <pubDate>Sun, 24 Jan 2021 20:01:21 GMT</pubDate>
            <atom:updated>2021-01-26T17:21:59.998Z</atom:updated>
            <content:encoded><![CDATA[<p>Autoencoders are unsupervised neural network models that are designed to learn to represent multi-dimensional data with fewer parameters. Data compression algorithms have been known for a long time however, learning the nonlinear operations for mapping the data into lower dimensions has been the contribution of autoencoders into the literature.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/829/1*ViBG49eTCKqqO2UVRL9mEw.png" /><figcaption>A general scheme of autoencoders (Figure is taken from[1])</figcaption></figure><h4>Introduction</h4><p>Autoencoders provided a very basic approach to extract the most important features of data by removing the redundancy. Redundancy occurs when multiple pieces (a column in a .csv file or a pixel location in an image dataset) of a dataset show a high correlation among themselves. In such cases, just holding one of the columns or disregarding the correlated pixels except one would allow us to store the same data with acceptable information loss.</p><p>In the scope of image compression, one of the most popular techniques was the JPEG algorithm which employs <em>discrete cosine transformation</em>[2] and a linear transformation that yields an image matrix that is mostly occupied by zeros after a simple integer rounding. Basically, holding only the non-zero elements and ignoring the rest would create a representation of the data with fewer parameters. Applying the inverse of the transformations would reconstruct the same image with little losses.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/850/1*32BOM5UIO28mpvwoHqdqAg.png" /><figcaption>JPEG pipeline (Figure is taken from[3])</figcaption></figure><p>A mathematical intuition lies underneath the idea of utilizing discrete cosine transformation and applying a certain linear transformation, however we cannot make sure that this is the best mapping there is. JPEG is a good algorithm that is still commonly used today but what if we come up with a model that learns a better mapping specific to the relevant data. Autoencoders exactly does it by compressing and reconstructing the data by learned parameters. The loss is defined as <em>reconstruction loss</em> in terms of the input data and reconstructed data which is usually <a href="https://afteracademy.com/blog/what-are-l1-and-l2-loss-functions">L1 or L2</a> losses.</p><p>The idea of denoising the data with autoencoders has been proposed by Gallinari &amp; LeCun et al.[4] in 1987 which has been an alternative to the Hoplied network which utilizes <a href="https://igi-web.tugraz.at/lehre/NNB/SS10/Lecture_Hopfield_nets.pdf">associative memory</a> for the task[5]. Today, autoencoders are widely used not only for denoising and compression but also for data generation.</p><h4>A Simple AutoEncoder with Tensorflow</h4><p>Actually, autoencoders are not novel neural networks, meaning that they do not have an architecture with unique properties for themselves. Instead, an autoencoder structure is a pipeline that uses other types of modules (fully connected layers, convolutional layers, copying, cropping, etc.) following each other. The complexity of the models is flexible according to the nature of the data and the task. In this article, a straightforward autoencoder with fully connected layers will be built and tested on the <a href="http://yann.lecun.com/exdb/mnist/">MNIST dataset</a>.</p><p>For a simple implementation, <a href="https://www.tensorflow.org/api_docs/python/tf/keras">Keras API on TensorFlow</a> backend is preferred with <a href="https://colab.research.google.com/">Google Colab</a> GPU services.</p><p>First, needed libraries are imported.</p><pre>import tensorflow as tf<br>import numpy as np<br>import matplotlib.pyplot as plt<br>from tensorflow.keras import datasets, layers, models, losses, Modelfrom random import randint</pre><p><strong><em>The Data</em></strong></p><p>MNIST data is available on Keras and it is imported with one line of code within seconds. Since the purpose of the model will be learning how to reconstruct the data, it is an unsupervised task or with a better term I enjoy, it is self-supervised. Thus, labels are not necessary and not stored while loading the data.</p><p>The train set has 60000 and the test set has 10000 mono-channel images with 28x28 sizes. The test set will be used for validation during training.</p><pre>(x_train, _), (x_test, _)=tf.keras.datasets.mnist.load_data()</pre><p><strong><em>The Encoder</em></strong></p><p>First, the images will be flattened into a vector having 784 (28 times 28) elements. And then, after a hidden layer with 100 neurons, the output of the encoder will have 20 parameters. This is named the latent representation of the data. If the model gets successfully trained, it will be able to represent the MNIST images with only 20 numbers. This is the stage where we compressed the data which is named the <em>bottleneck layer</em>.</p><pre>hidden_size = 100<br>latent_size = 20</pre><pre>input_layer = layers.Input(shape = x_train.shape[1:])</pre><pre>flattened = layers.Flatten()(input_layer)<br>hidden = layers.Dense(hidden_size, activation = &#39;relu&#39;)(flattened)<br>latent = layers.Dense(latent_size, activation = &#39;relu&#39;)(hidden)<br>encoder = Model(inputs = input_layer, outputs = latent, name = &#39;encoder&#39;)</pre><pre>encoder.summary()</pre><p>The hidden layer does not have to be included, however heuristically adding a few layers increase the representation power of the encoder. ReLU activation is chosen for the fully connected layers.</p><pre>Model: &quot;encoder&quot; _________________________________________________________________ Layer (type)                 Output Shape              Param #    ================================================================= input_1 (InputLayer)         [(None, 28, 28)]          0          _________________________________________________________________ flatten (Flatten)            (None, 784)               0          _________________________________________________________________ dense (Dense)                (None, 100)               78500      _________________________________________________________________ dense_1 (Dense)              (None, 20)                2020       ================================================================= Total params: 80,520 Trainable params: 80,520 Non-trainable params: 0</pre><p><strong><em>The Decoder</em></strong></p><p>The decoder aims to undo what the encoder did by reverse operations. Meaning, latent variables will be upsampled to 100 and 784 respectively. Finally, the vector will be reshaped into an image matrix.</p><pre>input_layer_decoder = layers.Input(shape = encoder.output.shape)<br>upsampled = layers.Dense(hidden_size, activation = &#39;relu&#39;)(input_layer_decoder)<br>upsampled = layers.Dense(encoder.layers[1].output_shape[-1], activation = &#39;relu&#39;)(upsampled)<br>constructed = layers.Reshape(x_train.shape[1:])(upsampled)</pre><pre>decoder = Model(inputs = input_layer_decoder, outputs = constructed, name= &#39;decoder&#39;)</pre><pre>decoder.summary()</pre><p>Notice that the input size of the decoder is equal to the output size of the encoder. Similarly, the output size of the final layer is equal to the size of the output of the flattening layer. Above you can see that it is (<em>None, 784)</em>.</p><pre>Model: &quot;decoder&quot; _________________________________________________________________ Layer (type)                 Output Shape              Param #    ================================================================= input_2 (InputLayer)         [(None, None, 20)]        0          _________________________________________________________________ dense_2 (Dense)              (None, None, 100)         2100       _________________________________________________________________ dense_3 (Dense)              (None, None, 784)         79184      _________________________________________________________________ reshape (Reshape)            (None, 28, 28)            0          ================================================================= Total params: 81,284 Trainable params: 81,284 Non-trainable params: 0</pre><p><strong><em>The Autoencoder</em></strong></p><p>The computational graph of the model is structured such that the overall input is the input of the encoder, and the overall output is the output of the decoder as it processes the output of the encoder.</p><pre>autoencoder = Model(inputs = encoder.input, outputs = decoder(encoder.output))<br>autoencoder.summary()</pre><pre>Layer (type)                 Output Shape              Param #    ================================================================= input_1 (InputLayer)         [(None, 28, 28)]          0          _________________________________________________________________ flatten (Flatten)            (None, 784)               0          _________________________________________________________________ dense (Dense)                (None, 100)               78500      _________________________________________________________________ dense_1 (Dense)              (None, 20)                2020       _________________________________________________________________ decoder (Functional)         (None, 28, 28)            81284      ================================================================= Total params: 161,804 Trainable params: 161,804 Non-trainable params: 0</pre><p>The loss is the mean squared error between the input image and the reconstructed image, namely L2 loss. The model is trained for 50 epochs with batches of 64 samples.</p><pre>autoencoder.compile(optimizer=&#39;adam&#39;, loss=losses.MeanSquaredError())</pre><pre>history = autoencoder.fit(x_train, x_train, epochs=50, batch_size=64, validation_data = (x_test, x_test))</pre><pre>Epoch 1/50 938/938 [==============================] - 3s 2ms/step - loss: 3085.7667 - val_loss: 1981.6154 <br>Epoch 2/50 938/938 [==============================] - 2s 2ms/step - loss: 1917.1781 - val_loss: 1790.9906<br>...<br>Epoch 50/50 938/938 [==============================] - 2s 2ms/step - loss: 1456.5116 - val_loss: 1454.8712</pre><p>Since this is not a classification example there is not metric as accuracy and the important metrics to be tracked are the losses. Even though 1456 might seem a big number, the drop in the error compared to the initial epochs implies a learning phase. Moreover, the loss is not an absolute metric like the accuracy of the <a href="https://scikit-learn.org/stable/modules/generated/sklearn.metrics.f1_score.html">F1-score</a>, it should be commented on according to the context.</p><pre>fig, axs = plt.subplots(figsize=(15,15))<br>axs.plot(history.history[&#39;loss&#39;])<br>axs.plot(history.history[&#39;val_loss&#39;])<br>axs.title.set_text(&#39;Training Loss vs Validation Loss&#39;)<br>axs.set_xlabel(&#39;Epochs&#39;)<br>axs.set_ylabel(&#39;Loss&#39;)<br>axs.legend([&#39;Train&#39;,&#39;Val&#39;])</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/897/1*eOCf6JUxl_dVqgMhjBysig.png" /></figure><p>In this article qualitative analysis (eye-test) would be more informative about the success of the model rather than quantitative analysis. First, some random reconstructions are visualized:</p><pre>fig, axs = plt.subplots(3,2,figsize=(10,15))<br>for i in range(3):<br>  sample1 = x_train[randint(0,x_train.shape[0])]<br>  axs[i][0].imshow(sample1, cmap = &#39;gray&#39;)<br>  axs[i][1].imshow(autoencoder(np.expand_dims(sample1,0))[0], cmap = &#39;gray&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/578/1*9Ru0IchcaVIFYsGLPfyCvw.png" /><figcaption>Results for reconstruction (Original on left, Reconstructed on right)</figcaption></figure><p>As you can see, some information loss occurred since it is a very basic autoencoder, however it is still possible to understand which digit is subjected to the model in most cases.</p><p>A more interesting visualization idea is playing on the latent space and observing the results. Each image is first encoded into vectors with a size of 20. What happens if we take the average of two latent vectors and pass it to the decoder? Would the reconstructed image resemble both of the original digits or would something completely meaningless image appear? Further, we can take the weighted averages of the latent variables and visualize the effects of gradual change in the latent vector:</p><pre>sample1_idx = randint(0,x_train.shape[0])<br>sample1 = x_train[sample1_idx]</pre><pre>sample2_idx = randint(0,x_train.shape[0])<br>sample2 = x_train[sample2_idx]</pre><pre>latent1 = encoder(np.expand_dims(sample1,0))<br>latent2 = encoder(np.expand_dims(sample2,0))</pre><pre>weights = np.arange(0.05,0.95,0.15)</pre><pre>fig, axs = plt.subplots(2,4,figsize=(20,10))<br>axs = axs.ravel()</pre><pre>axs[0].imshow(sample2, cmap = &#39;gray&#39;)<br>axs[-1].imshow(sample1, cmap = &#39;gray&#39;)</pre><pre>for i in range(6):<br>  latent = latent1*weights[i] + latent2*(1-weights[i])<br>  constructed = decoder(latent)<br>  axs[i+1].imshow(constructed[0], cmap=&#39;gray&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CI3Kc9FkVufz0ABER7PDJw.png" /><figcaption>The gradual change from one digit to another</figcaption></figure><p>In the image above we changed to weighted importance of two latent vectors from 95%-5% to 5%-95%. Accordingly, the reconstructed image first resembled<em> “0”</em>. When it comes to 50%-50% it becomes completely unrecognizable, it is neither <em>“0”</em> nor <em>“1”</em>. Finally, as the importance of the second latent vector becomes dominant, the decoder produces images that look like <em>“1”</em>.</p><h4>autoencoder_tensorflow.ipynb</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/231c97fac246e5180e00860d46942e19/href">https://medium.com/media/231c97fac246e5180e00860d46942e19/href</a></iframe><h4>Conclusion</h4><p>Autoencoders create an alternative way to compress the data by learning efficient data-specific mappings and reducing the dimensionality. In this article, MNIST, images consisting of 784 pixels have been represented by a vector having a size of 20 and reconstructed back. Even though the results were not perfect it was still possible to see that the latent vector holds a good majority of the information while losing the details. The embedded information in the latent variable decides the success of the reconstruction. By manipulating the latent vector, it is possible to create intermediate results.</p><p>Hope you enjoyed it. See you in the following AutoEncoder applications.</p><p>Best wishes…</p><p>mrgrhn</p><p>For the follow-up post, please visit:</p><p><a href="https://mrgrhn.medium.com/convolutional-autoencoders-cae-with-tensorflow-97e8d8859cbe">Convolutional AutoEncoders (CAE) with Tensorflow</a></p><h4>References</h4><ol><li>Sublime, Jeremie &amp; Kalinicheva, Ekaterina. (2019). “Automatic Post-Disaster Damage Mapping Using Deep-Learning Techniques for Change Detection: Case Study of the Tohoku Tsunami”. Remote Sensing. 11. 1123. 10.3390/rs11091123.</li><li>NN, Ahmed &amp; Natarajan, T. &amp; Rao, Kamisetty. (1974). “Discrete Cosine Transform”. Computers, IEEE Transactions on. C-23. 90–93. 10.1109/T-C.1974.223784.</li><li>Ri, S. &amp; Tsuda, H. &amp; Chang, K. &amp; Hsu, S. &amp; Lo, F. &amp; Lee, T.. (2020). “Dynamic Deformation Measurement by the Sampling Moiré Method from Video Recording and its Application to Bridge Engineering”. Experimental Techniques. 44. 10.1007/s40799–019–00358–4.</li><li>Gallinari, P., LeCun, Y., Thiria, S., &amp; Fogelman-Soulie, F. (1987). “Memoires associatives distribuees”. Proceedings of COGNITIVA 87. Paris, La Villette.</li><li><a href="https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.536.3644&amp;rep=rep1&amp;type=pdf">https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.536.3644&amp;rep=rep1&amp;type=pdf</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2f0a7315d161" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/autoencoders-with-tensorflow-2f0a7315d161">AutoEncoders with TensorFlow</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Essentials of Convolutional Neural Networks with LeNet, AlexNet, VGG, GoogleNet, and ResNet]]></title>
            <link>https://medium.com/swlh/essentials-of-convolutional-neural-networks-with-lenet-alexnet-vgg-googlenet-and-resnet-3f9dd477f666?source=rss-87eead11556c------2</link>
            <guid isPermaLink="false">https://medium.com/p/3f9dd477f666</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[convolutional-network]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[neural-networks]]></category>
            <dc:creator><![CDATA[mrgrhn]]></dc:creator>
            <pubDate>Fri, 22 Jan 2021 15:54:29 GMT</pubDate>
            <atom:updated>2021-01-22T22:27:55.858Z</atom:updated>
            <content:encoded><![CDATA[<p>Convolutional Neural Networks or ConvNets or even in shorter CNNs are a family of neural networks that are commonly implemented in computer vision tasks, however the use cases are not limited to that. Today, CNNs are employed in deep learning research and practical artificial intelligence applications with interests ranging from autonomous driving to medical imaging.</p><p>Before diving into the fundamental glossary and the chronological evolution of CNNs, I will rewind the tape back to the mid-1900s and briefly mention the story of human-like learning attempts in computer science.</p><h4>A Very Brief History of Neural Networks</h4><p>The first model to mimic the biological nervous system is developed by Frank Rosenblatt back in 1958. Consisting of a single neuron named <em>perceptron</em>[1] it was only capable of identifying <a href="https://stackoverflow.com/questions/13976565/neural-networks-what-does-linearly-separable-mean">linearly separable</a> samples in a binary classification problem. The underlying principle was the Hebbian Learning[2] which can be summarized as<em> “Cells that fire together, wire together”</em>. This motto means that when two cells are having high activations simultaneously, the connection between them (the weight links them in artificial networks) should be strengthened. However, due to its low capacity of learning, the findings on perceptron did not draw much attention. In 1982, Hopfield published his study[3] on a type of network that serves as a content-addressable memory. Being the first popular recurrent neural network, it is used for optimization problems. Even though back-propagation[4] was first formulated in 1963, it is reformulated for supporting Multi-Layer Perceptrons (MLPs) by Rumelhart et al. in 1986[5]. The two decades in between did not really witness any major improvements in the context of deep learning research. While there was a bigger excitement in the community thanks to MLPS in the ’80s, there were still many open problems to be solved such as overfitting, being stuck in the local minima, lack of capable hardware for bigger computations, and many more. To top it all, the huge success of Support Vector Machines[6] in the 90s undermined the promising ideas of neural networks.</p><h4>Notable CNN Models</h4><p>The long-waited breakthrough came in 1998 by Yann LeCun as he introduced convolutional neural networks to the literature by releasing LeNet[7]. Convolution, weight sharing, and subsampling were all new ideas back in time and this ignited lots of research in computer vision.</p><p><a href="https://medium.com/analytics-vidhya/lenet-with-tensorflow-a35da0d503df">LeNet with TensorFlow</a></p><p>However, CNNs required an abundance of data to be trained successfully and this need is met by the ImageNet[8] dataset containing more than 3.2 million annotated images in 2009. Yet, more data required more computational power and in 2012, AlexNet[9] became the first CNN to be trained using NVIDIA GeForce 256 GPU. Other key properties of AlexNet was the <a href="https://machinelearningmastery.com/dropout-for-regularizing-deep-neural-networks/">dropout</a> layer that performs a means of regularization during training and ReLU nonlinear activation.</p><p><a href="https://medium.com/swlh/alexnet-with-tensorflow-46f366559ce8">AlexNet with TensorFlow</a></p><p>In 2014 <a href="http://image-net.org/index">ILSVRC</a> of ImageNet hosted two very competitive models and the runner-up was VGGNet[10] with its deeper architecture and 1x1 convolutions for the purpose of reducing the dimensionality of the feature maps.</p><p><a href="https://medium.com/ai-in-plain-english/vggnet-with-tensorflow-transfer-learning-with-vgg16-included-7e5f6fa9479a">VGGNet with TensorFlow (Transfer Learning with VGG16 Included)</a></p><p>The winner of ILSVRC-2014 was GoogLeNet[11] that was developed by a group of computer scientists from Google. Thanks to its branched structure and the auxiliary losses during the training, they managed to design and train a deeper model and obtained a greater learning capacity.</p><p><a href="https://mrgrhn.medium.com/googlenet-inceptionv1-with-tensorflow-9e7f3a161e87">GoogLeNet (InceptionV1) with TensorFlow</a></p><p>A more recent winner of the challenge was ResNet in 2016 and it is absolutely worth mentioning for its residual learning mechanism with skip connections. They developed a very smart solution to tackle the vanishing gradient problem by bypassing certain layers with identity connections in order to reduce the number of operations the gradient is subjected to.</p><p><a href="https://medium.com/swlh/resnet-with-tensorflow-transfer-learning-13ff0773cf0c">ResNet with TensorFlow (Transfer Learning)</a></p><p>These five CNNs are the most fundamental models that changed the paradigm in computer vision research. Today CNNs are utilized in many domains, even on non-pixel data. UNet[14], YOLO[15], SSD[16], and R-CNN[17] are also among significant architectures and pipelines employing convolutional neural networks.</p><h4>A Glossary of Essential CNN Concepts</h4><p><strong><em>Convolution:</em></strong></p><p>Convolution is a very common mathematical operation in signal processing regarding linear time-invariant systems, yet its wide use cases have become popular with the advance of CNNs. In 2-D convolutions, a small mask (usually 3x3, not bigger than 11x11) hovers over the input image and makes an element-wise multiplication between its elements (weights) and covered pixels of the image. The sum of the multiplications is transferred to the next layer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*pwAsbrd-HGr-BzOS.gif" /><figcaption>A visualization of the convolution operation (Animation is taken from[18])</figcaption></figure><p><strong><em>Weight sharing:</em></strong></p><p>In fully connected networks, all the neurons in the consecutive layers are connected to each other and the corresponding weights are updated independently. For instance, between two layers having 100 and 64 neurons, there are 6400 weights and the model is ought to train all of them. However, in CNNs, masks having certain sizes go through the images in two (or more if it is convolutions with more dimensions) directions and the weights are embedded in those masks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/612/0*EG4qgkKFA1dVUyjR.png" /><figcaption>A visualization of the weight sharing (Chart is taken from[19])</figcaption></figure><p>Weight sharing leverages the fact that these masks are looking for common patterns in the image such as edges and corners for low-level motifs or eyes and noses in facial recognition for high-level motifs. Since the same parameters are used for distant locations of the image, the number of trainable parameters is significantly lower compared to fully connected networks.</p><p><strong><em>Stride:</em></strong></p><p>Stride is the step size of the masks (or kernels, filters) as they go through the input image by performing convolution. A smaller stride means more overlap of the receptive fields for the next layers, thus more textual information among the neighboring pixels. A greater stride is used for reducing the image size, which happens at the later layers of a CNN.</p><p><strong><em>Padding:</em></strong></p><p>Padding is the procedure of adjusting the image sizes such that the sizes would be preserved or the sizes become integer numbers before the next layer. Padding is usually done with zeros and in a symmetric way both in height and width. It also lets the masks fit into the images at the edges.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgfycat.com%2Fifr%2FPlasticMenacingDegu&amp;display_name=Gfycat&amp;url=https%3A%2F%2Fthumbs.gfycat.com%2FPlasticMenacingDegu-size_restricted.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=gfycat" width="754" height="678" frameborder="0" scrolling="no"><a href="https://medium.com/media/b60ad0f4cda4d829c399a5279d0f500d/href">https://medium.com/media/b60ad0f4cda4d829c399a5279d0f500d/href</a></iframe><p>Above you can see a very explanatory animation visualizing the concepts above.</p><p><strong><em>Subsampling:</em></strong></p><p>CNN models learn patterns in visual data in a hierarchical manner. In the earlier layers, the masks detect small details and low-level features. As the propagation continues, maks be able to detect high-level features, however to do so they need to cover a larger area. Instead of creating larger masks, the images get shrink by subsampling. Average pooling takes the average of a designated area and conveys it to the next layer whereas max-pooling takes the maximum activation for the sake of stressing the biggest activation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/602/0*oWfsXgIwJeIVWgYQ.png" /><figcaption>A visualization of the max-pooling operation</figcaption></figure><p><strong><em>Receptive fields:</em></strong></p><p>The receptive field means the number of neurons, a neuron in the following layers receives information from. In 3x3 kernels, each neuron receives information from 9 neurons belonging to the previous edge. If we assume stride = 1, the same neuron collects the information of 25 neurons in the layer two steps previous. You can see below why it is true:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/520/0*h8mhtc8JjXklRd9k.png" /><figcaption>A visualization of the max-pooling operation</figcaption></figure><p><strong><em>Dilated convolutions:</em></strong></p><p>While deciding on the size of the filters in CNN layers a trade-off emerges between the ability to obtain details with small filters and the ability to capture the texture with larger filters. Dilated convolutions offer a method to support the exponential expansion of receptive fields through the layers without losing resolution or coverage[21].</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/395/0*OLMLyeIQ1F6m1HWL.gif" /><figcaption>2D convolution using a 3x3 kernel with a dilation rate of 2 and no padding[22]</figcaption></figure><p><strong><em>Vanishing gradients:</em></strong></p><p>During the backpropagation in CNNs and other mainstream neural networks, as the gradient is being flowed backward it is multiplied by the layer weights until it reaches the layer that we want to update. During its journey from the output layer, it gets multiplied by several weights most of them being very small numbers, in deep networks. As the multiplications grow cumulatively, the gradient gets smaller and eventually vanishes with underflow. This is a common problem of deep networks and encountered especially in recurrent neural networks. As a result of the vanishing gradients, the learning becomes very slow in the initial layers. ResNet eliminated this problem by directing the gradient through skip connections.</p><p><strong><em>The MNIST Dataset:</em></strong></p><p><a href="http://yann.lecun.com/exdb/mnist/">The MNIST Dataset</a> consists of 60.000 training and 10.000 test images each having a handwritten digit, size-normalized and centered in a fixed-size image. All the images have a label from 0 to 9, indicating the handwritten digit of the specific image. Images have 28 pixels of width and 28 pixels of height. It is a very practical dataset with its small size in the memory and easy preprocessing. Many models are benchmarked on MNIST as the first step of development.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/319/0*NNmaVHmSdkyMjWND.png" /><figcaption>MNIST digit samples</figcaption></figure><p><strong><em>ImageNet:</em></strong></p><p>ImageNet is one of the earliest examples of large-scale image datasets for the use of computer vision tasks such as object recognition and image classification. Initially having around 3.2 million images, ImageNet now has more than 14 million images with 20 thousand distinct labels. ImageNet has been a milestone in computer vision research by providing the desired data to train deep models. Many research groups still benchmark their networks on ImageNet. Moreover, <em>ImageNet Large Scale Visual Recognition Challenge</em> or <em>ILSVRC</em> had been organized between 2010 and 2017 in different categories. The models that are explained in the other posts of this blog and mentioned above were significant models of this competition.</p><h4>References:</h4><ol><li>Rosenblatt, F. “The perceptron: a probabilistic model for information storage and organization in the brain”. (1958). <em>Psychological Review</em> 65 6: 386–408.</li><li>Hebb, Donald O. (1949). The Organization of Behavior. New York: Wiley, pg. 62.</li><li>Hopfield, J. J. (1982). “Neural networks and physical systems with emergent collective computational abilities”. Proceedings of the National Academy of Sciences. 79 (8): 2554–2558.</li><li>Bryson, Arthur E. (1962). “A gradient method for optimizing multi-stage allocation processes”. Proceedings of the Harvard Univ. Symposium on digital computers and their applications, 3–6 April 1961. Cambridge: Harvard University Press. OCLC 498866871.</li><li>Rumelhart, D., Hinton, G. &amp; Williams, R. (1986). “Learning representations by back-propagating errors”. <em>Nature</em> 323<strong>, </strong>533–536.</li><li>Cortes, Corinna &amp; Vapnik, Vladimir N. (1995). “Support-vector networks”. Machine Learning. 20 (3): 273–297.</li><li>Lecun, Yann (June 1989). “Generalization and network design strategies”. Technical Report CRG-TR-89–4. Department of Computer Science, University of Toronto.</li><li>Deng, Jia &amp; Dong, Wei &amp; Socher, Richard &amp; Li, Li-Jia &amp; Li, Kai &amp; Li, Fei Fei. (2009). “ImageNet: a Large-Scale Hierarchical Image Database”. IEEE Conference on Computer Vision and Pattern Recognition. 248–255. 10.1109/CVPR.2009.5206848.</li><li>Krizhevsky, Alex &amp; Sutskever, Ilya &amp; Hinton, Geoffrey. (2012). “ImageNet Classification with Deep Convolutional Neural Networks”. Neural Information Processing Systems. 25. 10.1145/3065386.</li><li>Simonyan, Karen &amp; Zisserman, Andrew. (2014). “Very Deep Convolutional Networks for Large-Scale Image Recognition”. arXiv 1409.1556.</li><li>Szegedy, Christian &amp; Liu, Wei &amp; Jia, Yangqing &amp; Sermanet, Pierre &amp; Reed, Scott &amp; Anguelov, Dragomir &amp; Erhan, Dumitru &amp; Vanhoucke, Vincent &amp; Rabinovich, Andrew. (2014). “Going Deeper with Convolutions”.</li><li>He, Kaiming &amp; Zhang, Xiangyu &amp; Ren, Shaoqing &amp; Sun, Jian. (2016). “Deep Residual Learning for Image Recognition”. 770–778. 10.1109/CVPR.2016.90.</li><li><a href="http://www.cs.cmu.edu/~10701/slides/Perceptron_Reading_Material.pdf">http://www.cs.cmu.edu/~10701/slides/Perceptron_Reading_Material.pdf</a></li><li>Ronneberger, Olaf &amp; Fischer, Philipp &amp; Brox, Thomas. (2015). “U-Net: Convolutional Networks for Biomedical Image Segmentation”. LNCS. 9351. 234–241. 10.1007/978–3–319–24574–4_28.</li><li>Redmon, Joseph &amp; Divvala, Santosh &amp; Girshick, Ross &amp; Farhadi, Ali. (2016). “You Only Look Once: Unified, Real-Time Object Detection”. 779–788. 10.1109/CVPR.2016.91.</li><li>Liu, Wei &amp; Anguelov, Dragomir &amp; Erhan, Dumitru &amp; Szegedy, Christian &amp; Reed, Scott &amp; Fu, Cheng-Yang &amp; Berg, Alexander. (2016). “SSD: Single Shot MultiBox Detector”. 9905. 21–37. 10.1007/978–3–319–46448–0_2.</li><li>Girshick, Ross &amp; Donahue, Jeff &amp; Darrell, Trevor &amp; Malik, Jitendra. (2013). Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. 10.1109/CVPR.2014.81.</li><li><a href="https://towardsdatascience.com/intuitively-understanding-convolutions-for-deep-learning-1f6f42faee1">https://towardsdatascience.com/intuitively-understanding-convolutions-for-deep-learning-1f6f42faee1</a></li><li>Abdel-Hamid, Ossama &amp; Deng, li &amp; Yu, Dong. (2013). “Exploring Convolutional Neural Network Structures and Optimization Techniques for Speech Recognition”.</li><li><a href="https://cs231n.github.io/assets/conv-demo/index.html">https://cs231n.github.io/assets/conv-demo/index.html</a></li><li>Yu, Fisher &amp; Koltun, Vladlen. (2015). “Multi-Scale Context Aggregation by Dilated Convolutions”.</li><li><a href="https://towardsdatascience.com/types-of-convolutions-in-deep-learning-717013397f4d">https://towardsdatascience.com/types-of-convolutions-in-deep-learning-717013397f4d</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3f9dd477f666" width="1" height="1" alt=""><hr><p><a href="https://medium.com/swlh/essentials-of-convolutional-neural-networks-with-lenet-alexnet-vgg-googlenet-and-resnet-3f9dd477f666">Essentials of Convolutional Neural Networks with LeNet, AlexNet, VGG, GoogleNet, and ResNet</a> was originally published in <a href="https://medium.com/swlh">The Startup</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ResNet with TensorFlow (Transfer Learning)]]></title>
            <link>https://medium.com/swlh/resnet-with-tensorflow-transfer-learning-13ff0773cf0c?source=rss-87eead11556c------2</link>
            <guid isPermaLink="false">https://medium.com/p/13ff0773cf0c</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[tensorflow]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[transfer-learning]]></category>
            <category><![CDATA[resnet]]></category>
            <dc:creator><![CDATA[mrgrhn]]></dc:creator>
            <pubDate>Thu, 21 Jan 2021 20:17:00 GMT</pubDate>
            <atom:updated>2021-02-03T22:29:46.612Z</atom:updated>
            <content:encoded><![CDATA[<p>ResNet owes its name to its <em>residual blocks</em> with <em>skip connections</em> that enable the model to be extremely deep. Even though including skip connections is a common idea in the community now, it was a revolutionary architectural choice and allowed ResNet to reach up to 152 layers with no vanishing or exploding gradient problems during training.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/840/1*4stFJqnyvZxzef3Zmb-uuw.png" /><figcaption>ResNet 34 Architecture (Illustration is taken from the original paper [1])</figcaption></figure><p>For the previous posts, please visit:</p><ul><li><a href="https://medium.com/ai-in-plain-english/vggnet-with-tensorflow-transfer-learning-with-vgg16-included-7e5f6fa9479a">VGGNet with TensorFlow (Transfer Learning with VGG16 Included)</a></li><li><a href="https://mrgrhn.medium.com/googlenet-inceptionv1-with-tensorflow-9e7f3a161e87">GoogLeNet (InceptionV1) with TensorFlow</a></li></ul><h4>Introduction</h4><p>With the developments in hardware technology and the variety of design techniques in deep learning deeper and deeper, models became popular in ImageNet competition. Unlike <a href="https://mrgrhn.medium.com/lenet-with-tensorflow-a35da0d503df">LeNet </a>and <a href="https://mrgrhn.medium.com/alexnet-with-tensorflow-46f366559ce8">AlexNet</a>, <a href="https://medium.com/ai-in-plain-english/vggnet-with-tensorflow-transfer-learning-with-vgg16-included-7e5f6fa9479a">VGG </a>and <a href="https://mrgrhn.medium.com/googlenet-inceptionv1-with-tensorflow-9e7f3a161e87">GoogLeNet </a>managed to deal with larger structures. However, training deeper networks requires some kind of intuition of how gradients flow in the models and a heuristic about how to train models. As models get deeper, research groups needed to push their imaginations and come up with more creative designs. After the <em>Inception </em>module[2] in GoogLeNet, another interesting breakthrough came with the <em>residual learning</em> mechanism of ResNet. Other notable improvements were 1x1 convolutions, dropout layers, and ReLU[3], yet none of them were as daring as these two.</p><p>While it is mentioned in detail in the <a href="https://mrgrhn.medium.com/googlenet-inceptionv1-with-tensorflow-9e7f3a161e87">earlier posts</a>, it is still needed to go through the problem of <em>vanishing or exploding gradients</em>. In gradient-based learning procedures, gradients are calculated in terms of the final loss and the weight space. “In machine learning, an artificial neural network is a model that consists of a directed graph, with weights (real numbers) on the edges of the graph. The parameter space is known as a weight space, and learning consists of updating the parameters, most often by gradient descent or some variant.”[4] In the final layer, the loss is partially differentiated with respect to each of the weights, and the weights are updated in the reverse direction of the differentiation in order to decrease the loss. The step size of this update can be constant or adaptive and it is controlled by the learning rate of the training procedure. For the prior layers, the gradient flowing backward from the following layer is multiplied by the input of the current layer. As the gradient flows backward it is multiplied with weight matrices over and over again, which may result in vanishing or exploding gradients. The risk grows as the number of layers increases since the gradient traverses a longer path. Although exploding gradients can be controlled by <em>batch normalization</em>[6] and <em>gradient clipping</em>, vanishing gradients is much bigger of a problem.</p><p>ResNet introduces bypass connections in the network and allows the gradient to flow without getting multiplied with weight matrices several times.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/406/1*1P0-EfOWD1pujuttVcSIAQ.png" /><figcaption>Skip Connections (Illustration is taken from the original paper [1])</figcaption></figure><p>In a branched network structure, if a layer leads to multiple modules, the gradients coming from all the modules are summed up and backpropagation continues with the <em>chain rule</em>. In the figure above, if the weight layers tend to have very small numbers (in the order of 10^-5 or smaller) at least some of the gradients would be decreased to a millionth of the gradients in the upper layer. But the design of ResNet is providing identity connection by skipping some of the layers, thus making the gradients flow without being subject to any multiplications. That method is a very common architectural choice in recent networks having hundreds of layers.</p><h3>ResNet with Tensorflow</h3><p>Even though skip connections make it possible to train extremely deep networks, it is still a tedious process to train these networks and it requires a huge amount of data. It is also covered in the VGGNet <a href="https://medium.com/ai-in-plain-english/vggnet-with-tensorflow-transfer-learning-with-vgg16-included-7e5f6fa9479a">post</a> that, trying to train these kinds of networks with MNIST data may not lead to convergence and acceptable accuracies. ResNet is originally trained on the ImageNet dataset and using transfer learning[7], it is possible to load pretrained convolutional weights and train a classifier on top of it.</p><p>First, needed libraries are imported.</p><pre>import tensorflow as tf<br>import matplotlib.pyplot as plt<br>from tensorflow.keras import datasets, layers, models, losses, Model</pre><p><strong><em>The Data</em></strong></p><p>Then, the data is loaded as in the LeNet implementation. One important notice is that the original ResNet model receives images with the size 224 x 224 x 3 however, MNIST images are 28 x 28. The images are padded with zeros and the third axis is expanded and repeated 3 times to make image sizes 32 x 32 x 3. When loading the model from Keras, it is possible to indicate the input shape, which will be 32 x 32 x 3 in our case instead of 224 x 224 x 3.</p><pre>(x_train,y_train),(x_test,y_test) = datasets.mnist.load_data()<br>x_train = tf.pad(x_train, [[0, 0], [2,2], [2,2]])/255<br>x_test = tf.pad(x_test, [[0, 0], [2,2], [2,2]])/255<br>x_train = tf.expand_dims(x_train, axis=3, name=None)<br>x_test = tf.expand_dims(x_test, axis=3, name=None)<br>x_train = tf.repeat(x_train, 3, axis=3)<br>x_test = tf.repeat(x_test, 3, axis=3)<br>x_val = x_train[-2000:,:,:,:]<br>y_val = y_train[-2000:]<br>x_train = x_train[:-2000,:,:,:]<br>y_train = y_train[:-2000]</pre><p><strong><em>The Model</em></strong></p><p>The ResNet model consists of lots and lots of convolutional layers each having 3x3 masks (except the first layer with has 7x7 masks). There are a few variations of the model but ResNet-152 was the model that won ILSVRC in 2015 and it will be implemented in this post. Yes, you guessed it right, 152 is the number of layers the model has. The network is over eight times larger than VGGNet while still having lower complexity.[5] The reason for that is using many small kernels in the layers instead of fewer large kernels and having one fully connected layer with 1000 neurons instead of two fully connected layers with 4096 neurons. In many CNNs, most of the parameters are coming from fully connected layers, since convolutional layers have weight sharing property.</p><p>The design is pretty monotonous with only convolutional layers and skip connections other than max-pooling at the beginning and global average pooling at the end.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/579/1*6XuGRu5T3EJpfg-c7ogf6g.png" /><figcaption>A comparison of layer depths</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/527/1*Y_r9qVwfUkzzIDNQ9_Pf-Q.png" /><figcaption>A comparison of layer depths</figcaption></figure><p>As you can see in the visuals above, ResNet-152 is absurdly deep and it is usually a good idea to load the model using <em>Keras </em>or any other deep learning library. We do not include the top, because that is what we want to train ourselves. We only load convolutional weights that are trained on ImageNet data. After loading the model the layers are set “not trainable”, thus frozen.</p><pre>base_model = tf.keras.applications.ResNet152(weights = &#39;imagenet&#39;, include_top = False, input_shape = (32,32,3))</pre><pre>for layer in base_model.layers:<br>  layer.trainable = False</pre><p>The top is added as follows:</p><pre>x = layers.Flatten()(base_model.output)<br>x = layers.Dense(1000, activation=&#39;relu&#39;)(x)<br>predictions = layers.Dense(10, activation = &#39;softmax&#39;)(x)</pre><p>The computational graph is constructed as it begins with the inputs of the base model and ends with a vector having the size of 10, indicating the probabilities of each category of MNIST. The model is compiled such that it will be trained using<em> Adam optimizer </em>for adaptive learning rate and <em>sparse categorical crossentropy </em>loss<em>.</em></p><pre>head_model = Model(inputs = base_model.input, outputs = predictions)</pre><pre>head_model.compile(optimizer=&#39;adam&#39;, loss=losses.sparse_categorical_crossentropy, metrics=[&#39;accuracy&#39;])</pre><p>The model is trained as the backbone layers are frozen. Thus, only fully connected layers that were added afterward will be trained.</p><pre>history = head_model.fit(x_train, y_train, batch_size=64, epochs=40, validation_data=(x_val, y_val))</pre><pre>Epoch 1/40 907/907 [==============================] - 44s 33ms/step - loss: 1.2104 - accuracy: 0.5902 - val_loss: 0.4982 - val_accuracy: 0.8450 <br>Epoch 2/40 907/907 [==============================] - 28s 31ms/step - loss: 0.6148 - accuracy: 0.8014 - val_loss: 0.3869 - val_accuracy: 0.8955<br>...<br>Epoch 40/40<br>907/907 [==============================] - 28s 31ms/step - loss: 0.2082 - accuracy: 0.9326 - val_loss: 0.1713 - val_accuracy: 0.9495</pre><p>Even after two epochs, validation accuracy arrives near 90%. After 40 epochs the model comfortably converges. It is possible to reach up to higher accuracies by adding a couple of more fully connected layers. The successful results with only one hidden fully connected layer mean that ResNet-152 does a pretty good job while extracting features for the classifier even though ImageNet and MNIST contain fairly distant image samples.</p><pre>fig, axs = plt.subplots(2, 1, figsize=(15,15))</pre><pre>axs[0].plot(history.history[&#39;loss&#39;])<br>axs[0].plot(history.history[&#39;val_loss&#39;])<br>axs[0].title.set_text(&#39;Training Loss vs Validation Loss&#39;)<br>axs[0].set_xlabel(&#39;Epochs&#39;)<br>axs[0].set_ylabel(&#39;Loss&#39;)<br>axs[0].legend([&#39;Train&#39;,&#39;Val&#39;])</pre><pre>axs[1].plot(history.history[&#39;accuracy&#39;])<br>axs[1].plot(history.history[&#39;val_accuracy&#39;])<br>axs[1].title.set_text(&#39;Training Accuracy vs Validation Accuracy&#39;)<br>axs[1].set_xlabel(&#39;Epochs&#39;)<br>axs[1].set_ylabel(&#39;Accuracy&#39;)<br>axs[1].legend([&#39;Train&#39;, &#39;Val&#39;])</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/894/1*YagMRqAQbYp80CWnA3VB0w.png" /></figure><p>Test accuracy came out at 92.75%.</p><pre>head_model.evaluate(x_test, y_test)</pre><pre>313/313 [==============================] - 7s 20ms/step - loss: 0.2287 - accuracy: 0.9275<br>[0.22868101298809052, 0.9275000095367432]</pre><h4>resnet_transfer_learning_tensorflow.ipynb</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/84fb78c037d71c8ca67656f544c022dc/href">https://medium.com/media/84fb78c037d71c8ca67656f544c022dc/href</a></iframe><h4>Conclusions</h4><p>ResNet-152 had a valuable contribution to the literature by being the first model to employ residual learning principles. While many other research groups were looking for ways to train deeper models, ResNet managed to do it by adding skip connections to the architecture and swept competitions in 2015 including ILSVRC. Nowadays, skip connections are being used in not only CNNs but in many other types of networks. Moreover thanks to its impressive learning capacity, it becomes a preferable feature extraction module in many tasks ranging from object recognition to autoencoders. In the ImageNet competition, with a 3.6% error rate ResNet even surpassed human performance.</p><p>Hope you enjoyed it. See you in the following Deep Learning articles.</p><p>Best wishes…</p><p>mrgrhn</p><h4>References</h4><ol><li>He, Kaiming &amp; Zhang, Xiangyu &amp; Ren, Shaoqing &amp; Sun, Jian. (2016). “Deep Residual Learning for Image Recognition”. 770–778. 10.1109/CVPR.2016.90.</li><li>Szegedy, Christian &amp; Liu, Wei &amp; Jia, Yangqing &amp; Sermanet, Pierre &amp; Reed, Scott &amp; Anguelov, Dragomir &amp; Erhan, Dumitru &amp; Vanhoucke, Vincent &amp; Rabinovich, Andrew. (2014). “Going Deeper with Convolutions”.</li><li>Krizhevsky, Alex &amp; Sutskever, Ilya &amp; Hinton, Geoffrey. (2012). “ImageNet Classification with Deep Convolutional Neural Networks”. Neural Information Processing Systems. 25. 10.1145/3065386.</li><li><a href="https://www.wikiwand.com/en/Parameter_space">https://www.wikiwand.com/en/Parameter_space</a></li><li><a href="https://web.cs.hacettepe.edu.tr/~aykut/classes/spring2016/bil722/slides/w04-ResNet.pdf">https://web.cs.hacettepe.edu.tr/~aykut/classes/spring2016/bil722/slides/w04-ResNet.pdf</a></li><li>Ioffe, Sergey &amp; Szegedy, Christian. (2015). “Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift”.</li><li>Bozinovski, Stevo &amp; Fulgosi, Ante (1976). “The Influence of Pattern Similarity and Transfer Learning upon Training of a Base Perceptron B2.” (original in Croatian) Proceedings of Symposium Informatica 3–121–5, Bled.</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=13ff0773cf0c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/swlh/resnet-with-tensorflow-transfer-learning-13ff0773cf0c">ResNet with TensorFlow (Transfer Learning)</a> was originally published in <a href="https://medium.com/swlh">The Startup</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[GoogLeNet (InceptionV1) with TensorFlow]]></title>
            <link>https://ai.plainenglish.io/googlenet-inceptionv1-with-tensorflow-9e7f3a161e87?source=rss-87eead11556c------2</link>
            <guid isPermaLink="false">https://medium.com/p/9e7f3a161e87</guid>
            <category><![CDATA[convolutional-neural-net]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[tensorflow]]></category>
            <category><![CDATA[google]]></category>
            <dc:creator><![CDATA[mrgrhn]]></dc:creator>
            <pubDate>Thu, 21 Jan 2021 00:14:55 GMT</pubDate>
            <atom:updated>2021-01-22T18:00:26.052Z</atom:updated>
            <content:encoded><![CDATA[<p>InceptionV1 or with a more remarkable name GoogLeNet is one of the most successful models of the earlier years of convolutional neural networks. Szegedy et al. from Google Inc. published the model in their paper named <em>Going Deeper with Convolutions</em>[1] and won <a href="http://image-net.org/challenges/LSVRC/2014/">ILSVRC-2014</a> with a large margin. The name both signifies the affiliation of most of the contributed scholars with Google and the reference to the LeNet[2] model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*q5eMDjUHKqEyo7qY.png" /><figcaption>GoogLeNet architecture, forward propagation from right to left (Illustration is taken from [3].)</figcaption></figure><p>For the previous posts, please visit:</p><ul><li><a href="https://mrgrhn.medium.com/alexnet-with-tensorflow-46f366559ce8">AlexNet with TensorFlow</a></li><li><a href="https://mrgrhn.medium.com/vggnet-with-tensorflow-transfer-learning-with-vgg16-included-7e5f6fa9479a">VGGNet with TensorFlow (Transfer Learning with VGG16 Included)</a></li></ul><h4>Introduction</h4><p>After analyzing and implementing VGG16[7] (runner-up of ILSVRC-2014), now it is time for the winner of the competition, GoogLeNet. As the name of the paper[1] implies the main intuition of GoogLeNet is obtaining a more capable model by increasing the dept. However, as covered in the previous posts, it is a risky architectural choice since deeper and larger models are notoriously harder to train. The success of GoogLeNet stems from the smart tricks that make the model lighter and easier to train. The original GoogLeNet is also named InceptionV1 and it is a 22-layers-deep network. Other versions of the model have also been developed by some of the writers of the first paper in 2016[3].</p><p>A few new ideas made GoogLeNet superior to the counterpart model VGGNet:</p><ol><li>1x1 convolutions were used for reducing the dimensionality of the layers while making the model deeper.</li><li>Average global pooling[6] is performed before the fully connected layers in order to reduce the number of feature maps.</li><li>Two auxiliary losses were introduced at the earlier layers of the model for the sake of propagating good gradients to the initial layers. It is a very brilliant idea for tackling the <a href="https://deepai.org/machine-learning-glossary-and-terms/vanishing-gradient-problem">vanishing gradient problem</a>.</li><li>Prior models such as LeNet, AlexNet, and VGGNet follow a sequential model architecture. However, GoogLeNet has braches that lead to auxiliary losses.</li><li>Another branchy entity in the model is the Inception module that combines the outputs of differently sized filters. The parallel structure of multiple scales enables the module to capture both smaller and larger motifs in the pixel-data.</li></ol><p>All these ideas will be discussed further throughout the next sections as we build the model using Keras.</p><h3>GoogLeNet with Tensorflow</h3><p>This tutorial is intended for beginners to demonstrate a basic TensorFlow implementation of GoogLeNet on the MNIST dataset. For more information on CNNs and TensorFlow, you can visit the <a href="https://mrgrhn.medium.com/lenet-with-tensorflow-a35da0d503df">previous posts</a> linked at the beginning of this article. The reason for the usage of MNIST instead of ImageNet is simplicity, but the model can be used for any dataset with very few variations in the code.</p><p>First, the libraries that will be needed are imported:</p><pre>import tensorflow as tf<br>import matplotlib.pyplot as plt<br>from tensorflow.keras import datasets, layers, models, losses, Model</pre><p><strong><em>The Data</em></strong></p><p>Then, the data is loaded as in the LeNet implementation. One important notice is that the original GoogLeNet model receives images with the size 224 x 224 x 3 however, MNIST images are 28 x 28. The third axis is expanded and repeated 3 times to make image sizes 28 x 28 x 3. It will be then resized at the first layer of the model to 224 x 224 x 3.</p><pre>(x_train,y_train),(x_test,y_test) = datasets.mnist.load_data()<br>x_train = tf.pad(x_train, [[0, 0], [2,2], [2,2]])/255<br>x_test = tf.pad(x_test, [[0, 0], [2,2], [2,2]])/255<br>x_train = tf.expand_dims(x_train, axis=3, name=None)<br>x_test = tf.expand_dims(x_test, axis=3, name=None)<br>x_train = tf.repeat(x_train, 3, axis=3)<br>x_test = tf.repeat(x_test, 3, axis=3)<br>x_val = x_train[-2000:,:,:,:]<br>y_val = y_train[-2000:]<br>x_train = x_train[:-2000,:,:,:]<br>y_train = y_train[:-2000]</pre><p><strong><em>The Model</em></strong></p><p>GoogLeNet is a deep network and used this advantage to come on top of the ImageNet classification challenge in 2014. But why other researchers did not design their models even deeper than GoogLeNet and achieve higher accuracies? The answer is that there is no direct correlation between the dept of the model and the accuracy it yields in certain tasks. Mostly, there is a sweet spot for the size of the networks, and if it is exceeded it becomes really cumbersome, sometimes infeasible, to train the models. There are two aspects of this problem:</p><ol><li>Deeper networks mean more parameters to be trained, which requires an abundance of data, clever ways of training procedures, and most importantly strong GPUs. The available data and hardware enabled larger and larger networks nowadays, yet that was not the case back in time. GoogLeNet uses <em>1x1 convolutions</em> to reduce the number of feature maps and deepen the network while preserving the image sizes and not introducing many parameters.</li><li>During the propagation, <em>gradients </em>are calculated in terms of the losses and they flow back from the prediction layer to the initial layer. In deeper networks, gradients have to traverse a long way and go through several operations. A common problem is that these gradients get smaller and smaller as they get multiplied with activations between 0 and 1. Also, unbounded activations may blow up the gradients, which poses a similar problem. GoogLeNet introduces two auxiliary losses before the actual loss and makes more sensible gradients flow backward. So, these gradients travel a shorter path and help initial layers converge faster.</li></ol><p>These two approaches enabled GoogLeNet to be deeper and as it gets deeper its learning capacity also increases which leads to higher accuracies.</p><p>Another trick is the use of <em>global average pooling</em>. Convolutional networks are superior to multilayer perceptrons in terms of efficiency thanks to the weight sharing[6] mechanism. However, for classification purposes, fully connected layers are added at the top of the models and they are usually responsible for most of the parameters in the network. GoogLeNet reduces the number of neurons by taking an average among the channels right before the <em>dense </em>layer.</p><p>The parallel flow of the information is another key factor in the success of GoogLeNet. Patterns with different scales get captured and stacked on top of each other. A parallel backpropagation is also efficient compared to a sequential order since the gradients do not suffer vanishing/exploding. 1x1 convolutions control the feature map sizes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/850/0*cNu65t6O5thbHBYe.png" /><figcaption>The illustration is from the original paper[1]</figcaption></figure><p>The inception model is coded as follows:</p><pre>def inception(x,<br>              filters_1x1,<br>              filters_3x3_reduce,<br>              filters_3x3,<br>              filters_5x5_reduce,<br>              filters_5x5,<br>              filters_pool):</pre><pre>  path1 = layers.Conv2D(filters_1x1, (1, 1), padding=&#39;same&#39;,    activation=&#39;relu&#39;)(x)</pre><pre>  path2 = layers.Conv2D(filters_3x3_reduce, (1, 1), padding=&#39;same&#39;, activation=&#39;relu&#39;)(x)<br>  path2 = layers.Conv2D(filters_3x3, (1, 1), padding=&#39;same&#39;, activation=&#39;relu&#39;)(path2)</pre><pre>  path3 = layers.Conv2D(filters_5x5_reduce, (1, 1), padding=&#39;same&#39;, activation=&#39;relu&#39;)(x)<br>  path3 = layers.Conv2D(filters_5x5, (1, 1), padding=&#39;same&#39;, activation=&#39;relu&#39;)(path3)</pre><pre>  path4 = layers.MaxPool2D((3, 3), strides=(1, 1), padding=&#39;same&#39;)(x)<br>  path4 = layers.Conv2D(filters_pool, (1, 1), padding=&#39;same&#39;, activation=&#39;relu&#39;)(path4)</pre><pre>  return tf.concat([path1, path2, path3, path4], axis=3)</pre><p>A big thanks to the <a href="https://www.analyticsvidhya.com/blog/2018/10/understanding-inception-network-from-scratch/">blog post</a>[5] by Faizan Shaik who share some code snippets that were very useful when I was coding this model.</p><p>The rest of the model is carefully built by following the figure at the top of this article. Inception modules, auxiliary outputs, and subsampling layers were all placed in the correct order.</p><pre>inp = layers.Input(shape=(32, 32, 3))</pre><pre>input_tensor = layers.experimental.preprocessing.Resizing(224, 224, interpolation=&quot;bilinear&quot;, input_shape=x_train.shape[1:])(inp)</pre><pre>x = layers.Conv2D(64, 7, strides=2, padding=&#39;same&#39;, activation=&#39;relu&#39;)(input_tensor)<br>x = layers.MaxPooling2D(3, strides=2)(x)</pre><pre>x = layers.Conv2D(64, 1, strides=1, padding=&#39;same&#39;, activation=&#39;relu&#39;)(x)<br>x = layers.Conv2D(192, 3, strides=1, padding=&#39;same&#39;, activation=&#39;relu&#39;)(x)<br>x = layers.MaxPooling2D(3, strides=2)(x)</pre><pre>x = inception(x, filters_1x1=64, filters_3x3_reduce=96, filters_3x3=128, filters_5x5_reduce=16, filters_5x5=32, filters_pool=32)</pre><pre>x = inception(x, filters_1x1=128, filters_3x3_reduce=128, filters_3x3=192, filters_5x5_reduce=32, filters_5x5=96, filters_pool=64)</pre><pre>x = layers.MaxPooling2D(3, strides=2)(x)</pre><pre>x = inception(x, filters_1x1=192, filters_3x3_reduce=96, filters_3x3=208, filters_5x5_reduce=16, filters_5x5=48, filters_pool=64)</pre><pre>aux1 = layers.AveragePooling2D((5, 5), strides=3)(x)<br>aux1 =layers.Conv2D(128, 1, padding=&#39;same&#39;, activation=&#39;relu&#39;)(aux1)<br>aux1 = layers.Flatten()(aux1)<br>aux1 = layers.Dense(1024, activation=&#39;relu&#39;)(aux1)<br>aux1 = layers.Dropout(0.7)(aux1)<br>aux1 = layers.Dense(10, activation=&#39;softmax&#39;)(aux1)</pre><pre>x = inception(x, filters_1x1=160, filters_3x3_reduce=112, filters_3x3=224, filters_5x5_reduce=24, filters_5x5=64, filters_pool=64)</pre><pre>x = inception(x, filters_1x1=128, filters_3x3_reduce=128, filters_3x3=256, filters_5x5_reduce=24, filters_5x5=64, filters_pool=64)</pre><pre>x = inception(x, filters_1x1=112, filters_3x3_reduce=144, filters_3x3=288, filters_5x5_reduce=32, filters_5x5=64, filters_pool=64)</pre><pre>aux2 = layers.AveragePooling2D((5, 5), strides=3)(x)<br>aux2 =layers.Conv2D(128, 1, padding=&#39;same&#39;, activation=&#39;relu&#39;)(aux2)<br>aux2 = layers.Flatten()(aux2)<br>aux2 = layers.Dense(1024, activation=&#39;relu&#39;)(aux2)<br>aux2 = layers.Dropout(0.7)(aux2) <br>aux2 = layers.Dense(10, activation=&#39;softmax&#39;)(aux2)</pre><pre>x = inception(x, filters_1x1=256, filters_3x3_reduce=160, filters_3x3=320, filters_5x5_reduce=32, filters_5x5=128, filters_pool=128)</pre><pre>x = layers.MaxPooling2D(3, strides=2)(x)</pre><pre>x = inception(x, filters_1x1=256, filters_3x3_reduce=160, filters_3x3=320, filters_5x5_reduce=32, filters_5x5=128, filters_pool=128)</pre><pre>x = inception(x, filters_1x1=384, filters_3x3_reduce=192, filters_3x3=384, filters_5x5_reduce=48, filters_5x5=128, filters_pool=128)</pre><pre>x = layers.GlobalAveragePooling2D()(x)<br>x = layers.Dropout(0.4)(x)</pre><pre>out = layers.Dense(10, activation=&#39;softmax&#39;)(x)</pre><p>The computational graph is structured by indicating the input and outputs (including auxiliary outputs):</p><pre>model = Model(inputs = inp, outputs = [out, aux1, aux2])</pre><p>Model is compiled and trained with Adam optimizer and categorical cross-entropy loss for all three outputs. The weighted sum of the losses is adjusted such that auxiliary losses are as important as 0.3 times the actual loss:</p><pre>model.compile(optimizer=&#39;adam&#39;, <br>              loss=[losses.sparse_categorical_crossentropy,<br>                    losses.sparse_categorical_crossentropy,<br>                    losses.sparse_categorical_crossentropy]<br>              loss_weights=[1, 0.3, 0.3],<br>              metrics=[&#39;accuracy&#39;])</pre><pre>history = model.fit(x_train, [y_train, y_train, y_train], validation_data=(x_val, [y_val, y_val, y_val]), batch_size=64, epochs=40)</pre><pre>...<br>...<br>Epoch 40/40 907/907 [==============================] - 148s 163ms/step - loss: 0.0184 - dense_4_loss: 0.0113 - dense_1_loss: 0.0118 - dense_3_loss: 0.0119 - dense_4_accuracy: 0.9961 - dense_1_accuracy: 0.9961 - dense_3_accuracy: 0.9966 - val_loss: 0.0525 - val_dense_4_loss: 0.0321 - val_dense_1_loss: 0.0280 - val_dense_3_loss: 0.0402 - val_dense_4_accuracy: 0.9955 - val_dense_1_accuracy: 0.9970 - val_dense_3_accuracy: 0.9955</pre><p>The result of the dense_4_loss and dense_4_accuracy is what is important for the evaluation since other auxiliary outputs were just useful during training time and meaningless at forward-passing. After 40 epochs validation accuracy reaches up to 99.55%, which is quite impressive.</p><pre>fig, axs = plt.subplots(2, 1, figsize=(15,15))</pre><pre>axs[0].plot(history.history[&#39;loss&#39;])<br>axs[0].plot(history.history[&#39;val_loss&#39;])<br>axs[0].title.set_text(&#39;Training Loss vs Validation Loss&#39;)<br>axs[0].set_xlabel(&#39;Epochs&#39;)<br>axs[0].set_ylabel(&#39;Loss&#39;)<br>axs[0].legend([&#39;Train&#39;,&#39;Val&#39;])</pre><pre>axs[1].plot(history.history[&#39;dense_4_accuracy&#39;])<br>axs[1].plot(history.history[&#39;val_dense_4_accuracy&#39;])<br>axs[1].title.set_text(&#39;Training Accuracy vs Validation Accuracy&#39;)<br>axs[1].set_xlabel(&#39;Epochs&#39;)<br>axs[1].set_ylabel(&#39;Accuracy&#39;)<br>axs[1].legend([&#39;Train&#39;, &#39;Val&#39;])</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/894/1*XE62mON4z06PWJp30yhEJA.png" /></figure><p>Test accuracy came out at 99.29%:</p><pre>model.evaluate(x_test, y_test)</pre><pre>313/313 [==============================] - 10s 27ms/step - loss: 0.0516 - dense_4_loss: 0.0299 - dense_1_loss: 0.0335 - dense_3_loss: 0.0388 - dense_4_accuracy: 0.9929 - dense_1_accuracy: 0.9923 - dense_3_accuracy: 0.9918</pre><h4>googlenet_tensorflow.ipynb</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3b1e69b935da508f44a96fd33fb3a251/href">https://medium.com/media/3b1e69b935da508f44a96fd33fb3a251/href</a></iframe><h4>Conclusion</h4><p>As the winner of the ILSVRC-2014, GoogLeNet offered very avant-garde ideas to the deep learning community. The research group from Google combined bold architectural choices with strong intuitions and managed to design and train a very deep network back in the time. GoogLeNet proves that a linear structure is not a necessity for the network, on the contrary carefully drawn branches help out the training for a great deal. 1x1 convolutions, auxiliary losses, average pooling layers, and an original computational graph makes GoogLeNet totally worth studying and understanding. Unlike VGGNet, GoogLeNet could be trained on MNIST from scratch and yielded almost perfect results.</p><p>Hope you enjoyed it. See you in the following CNN models.</p><p>Best wishes…</p><p>mrgrhn</p><p>For the follow-up post, please visit:</p><p><a href="https://mrgrhn.medium.com/resnet-with-tensorflow-transfer-learning-13ff0773cf0c">ResNet with TensorFlow (Transfer Learning)</a></p><h4>References</h4><ol><li>Szegedy, Christian &amp; Liu, Wei &amp; Jia, Yangqing &amp; Sermanet, Pierre &amp; Reed, Scott &amp; Anguelov, Dragomir &amp; Erhan, Dumitru &amp; Vanhoucke, Vincent &amp; Rabinovich, Andrew. (2014). “Going Deeper with Convolutions”.</li><li>LeCun, Y.; Boser, B.; Denker, J. S.; Henderson, D.; Howard, R. E.; Hubbard, W.; Jackel, L. D. (December 1989). “Backpropagation Applied to Handwritten Zip Code Recognition”. <em>Neural Computation</em>. 1(4): 541–551.</li><li>Szegedy, Christian &amp; Vanhoucke, Vincent &amp; Ioffe, Sergey &amp; Shlens, Jon &amp; Wojna, ZB. (2016). “Rethinking the Inception Architecture for Computer Vision”. 10.1109/CVPR.2016.308.</li><li><a href="https://www.geeksforgeeks.org/understanding-googlenet-model-cnn-architecture/">https://www.geeksforgeeks.org/understanding-googlenet-model-cnn-architecture/</a></li><li><a href="https://www.analyticsvidhya.com/blog/2018/10/understanding-inception-network-from-scratch/">https://www.analyticsvidhya.com/blog/2018/10/understanding-inception-network-from-scratch/</a></li><li>Ott, Jordan &amp; Linstead, Erik &amp; LaHaye, Nick &amp; Baldi, Pierre. (2020). Learning in the machine: “To share or not to share?”. Neural Networks. 126. 10.1016/j.neunet.2020.03.016.</li><li>Simonyan, Karen &amp; Zisserman, Andrew. (2014). “Very Deep Convolutional Networks for Large-Scale Image Recognition”. arXiv 1409.1556.</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9e7f3a161e87" width="1" height="1" alt=""><hr><p><a href="https://ai.plainenglish.io/googlenet-inceptionv1-with-tensorflow-9e7f3a161e87">GoogLeNet (InceptionV1) with TensorFlow</a> was originally published in <a href="https://ai.plainenglish.io/">Artificial Intelligence in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[VGGNet with TensorFlow (Transfer Learning with VGG16 Included)]]></title>
            <link>https://ai.plainenglish.io/vggnet-with-tensorflow-transfer-learning-with-vgg16-included-7e5f6fa9479a?source=rss-87eead11556c------2</link>
            <guid isPermaLink="false">https://medium.com/p/7e5f6fa9479a</guid>
            <category><![CDATA[cnn]]></category>
            <category><![CDATA[receptive-field]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[mrgrhn]]></dc:creator>
            <pubDate>Wed, 20 Jan 2021 10:43:01 GMT</pubDate>
            <atom:updated>2021-01-23T16:07:19.668Z</atom:updated>
            <content:encoded><![CDATA[<p>VGG owes its name to the Visual Geometry Group of Oxford University. After being submitted to ILSVRC in 2014[1], the model VGGNet became as popular as the group itself. It is mostly considered as one step further from AlexNet due to deeper architecture and smaller kernel sizes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/850/0*Tmmor_6qcy3TlQxx.jpg" /><figcaption>VGGNet Architecture (Illustration is taken from [2].)</figcaption></figure><p>For the previous posts, please visit:</p><ul><li><a href="https://mrgrhn.medium.com/lenet-with-tensorflow-a35da0d503df">LeNet with TensorFlow</a></li><li><a href="https://mrgrhn.medium.com/alexnet-with-tensorflow-46f366559ce8">AlexNet with TensorFlow</a></li></ul><h4>Introduction</h4><p>Years after the release of LeNet[3], in 2012 AlexNet paper[4] was published and drew so much attention back to the CNNs. The following models focused on previous ones’ deficiencies and aimed for higher accuracies in more efficient ways. VGGNet borrows a lot from AlexNet yet it is a deeper model in terms of the number of layers. In the original paper, 6 variants of the same architecture are implemented, having the number of layers ranging from 11 to 19. In this post, the one with 16 layers (13 convolutional layers, 3 fully connected layers) will be implemented. It is commonly referred to as VGG16.</p><p>The main contribution of VGG16 compared to AlexNet is using smaller filters on top of each other instead of one single large filter. The intuition behind this is very much related to receptive fields of convolutional filters. In a 7 x 7 kernel, a pixel receives information from 49 pixels in the previous layer. This is the receptive field of this pixel. Higher receptive fields can capture more motifs in a large area while missing the details. Smaller receptive fields, on the other hand, fails to capture general patterns.</p><p>Let us assume that we have a convolutional layer with 5 x 5 kernels and 64 feature maps. The number of parameters to be trained is 5 x 5 x 64 = 1600. If we prefer two layers with 3 x 3 kernels and again 64 feature maps, the number of parameters to be trained would be 2 x 3 x 3 x 64 = 1152. This is 28% fewer parameters than the first case, hence the receptive field remains the same:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*WgYIIFrQGqDg1plt.png" /><figcaption>The receptive field of a 3x3 kernel in 2 layers</figcaption></figure><p>This is why VGGNet could get deeper while keeping the number of parameters in reasonable amounts. It can be considered the first truly <em>deep </em>convolutional neural network, however it is just a relative comparison with the earlier models.</p><h4>VGG16 with TensorFlow</h4><p>This tutorial is intended for beginners to demonstrate a basic TensorFlow implementation of AlexNet on the MNIST dataset. For more information on CNNs and TensorFlow, you can visit the previous post linked at the beginning of this article. The reason for the usage of MNIST instead of ImageNet is simplicity, but the model can be used for any dataset with very few variations in the code.</p><p>First, needed libraries are imported.</p><pre>import tensorflow as tf<br>import matplotlib.pyplot as pltfrom tensorflow.keras import datasets, layers, models, losses</pre><p><strong><em>The Data</em></strong></p><p>Then, the data is loaded as in the LeNet implementation. One important notice is that the original AlexNet model receives images with the size 224 x 224 x 3, however MNIST images are 28 x 28. The third axis is expanded and repeated 3 times to make image sizes 28 x 28 x 3. It will be then resized at the first layer of the model to 224 x 224 x 3.</p><pre>x_train,y_train),(x_test,y_test) = datasets.mnist.load_data()x_train = tf.pad(x_train, [[0, 0], [2,2], [2,2]])/255<br>x_test = tf.pad(x_test, [[0, 0], [2,2], [2,2]])/255x_train = tf.expand_dims(x_train, axis=3, name=None)<br>x_test = tf.expand_dims(x_test, axis=3, name=None)x_train = tf.repeat(x_train, 3, axis=3)<br>x_test = tf.repeat(x_test, 3, axis=3)x_val = x_train[-2000:,:,:,:]<br>y_val = y_train[-2000:]<br>x_train = x_train[:-2000,:,:,:]<br>y_train = y_train[:-2000]</pre><p><strong><em>The Model</em></strong></p><p>VGG16 consists of many convolutional, subsampling, and fully connected layers. Although there are other variations, the model that is implemented in this post employs 3 x 3 and 1 x 1 convolutional filters. 1 x 1 convolutions only correspond to the linear combination of a pixel location among the layers. Of course, nonlinear activation is used and ReLU is preferred as in the AlexNet.</p><p>In the first layer, 32 x 32 MNIST images (after the padding) were resized to 224 x 224 by <em>bilinear interpolation</em>[5]. The architecture continues with convolutional layers having the following specifications:</p><ol><li>Two layers with feature map size = 64, kernel size = 3</li><li>Two layers with feature map size = 128, kernel size = 3</li><li>Two layers with feature map size = 256, kernel size = 3</li><li>One layer with feature map size = 256, kernel size = 1</li><li>Two layers with feature map size = 512, kernel size = 3</li><li>One layers with feature map size = 512, kernel size = 1</li><li>Two layers with feature map size = 512, kernel size = 3</li><li>One layers with feature map size = 512, kernel size = 1</li></ol><p>Each convolutional layer has stride = 1 and pooling option = “same”. Setting the pooling argument <em>“same”</em> means that the input tensor is padded such that after the convolution the resulting tensor shape equals to input tensor shape for stride = 1. For example _,64,64,_ tensor would come out as _,64,64,_ from the layer if stride = 1 and pooling = “same”. The same tensor would come out as _,32,32,_ for stride = 2 and pooling = “same”. After blocks 1, 2, 4, 6, and 8 max-pooling layers are placed stride = 2 and kernel size = 2.</p><p>Convolutional layers are followed by three fully connected layers having the sizes of 4096, 4096, and 10, with a 0.5 dropout rate. ReLU activation is employed throughout the model. Activation can be inserted as a keyword argument in Conv2D function, but I prefer displaying it as an individual layer, which does not alter the model.</p><pre>model = models.Sequential()</pre><pre>model.add(layers.experimental.preprocessing.Resizing(224, 224, interpolation=&quot;bilinear&quot;, input_shape=x_train.shape[1:]))</pre><pre>model.add(layers.Conv2D(64, 3, strides=1, padding=&#39;same&#39;))<br>model.add(layers.Activation(&#39;relu&#39;))</pre><pre>model.add(layers.Conv2D(64, 3, strides=1, padding=&#39;same&#39;))<br>model.add(layers.Activation(&#39;relu&#39;))<br>model.add(layers.MaxPooling2D(2, strides=2))</pre><pre>model.add(layers.Conv2D(128, 3, strides=1, padding=&#39;same&#39;))<br>model.add(layers.Activation(&#39;relu&#39;))</pre><pre>model.add(layers.Conv2D(128, 3, strides=1, padding=&#39;same&#39;))<br>model.add(layers.Activation(&#39;relu&#39;))<br>model.add(layers.MaxPooling2D(2, strides=2))</pre><pre>model.add(layers.Conv2D(256, 3, strides=1, padding=&#39;same&#39;))<br>model.add(layers.Activation(&#39;relu&#39;))</pre><pre>model.add(layers.Conv2D(256, 3, strides=1, padding=&#39;same&#39;))<br>model.add(layers.Activation(&#39;relu&#39;))</pre><pre>model.add(layers.Conv2D(256, 1, strides=1, padding=&#39;same&#39;))<br>model.add(layers.Activation(&#39;relu&#39;))<br>model.add(layers.MaxPooling2D(2, strides=2))</pre><pre>model.add(layers.Conv2D(512, 3, strides=1, padding=&#39;same&#39;))<br>model.add(layers.Activation(&#39;relu&#39;))</pre><pre>model.add(layers.Conv2D(512, 3, strides=1, padding=&#39;same&#39;))<br>model.add(layers.Activation(&#39;relu&#39;))</pre><pre>model.add(layers.Conv2D(512, 1, strides=1, padding=&#39;same&#39;))<br>model.add(layers.Activation(&#39;relu&#39;))<br>model.add(layers.MaxPooling2D(2, strides=2))</pre><pre>model.add(layers.Conv2D(512, 3, strides=1, padding=&#39;same&#39;))<br>model.add(layers.Activation(&#39;relu&#39;))</pre><pre>model.add(layers.Conv2D(512, 3, strides=1, padding=&#39;same&#39;))<br>model.add(layers.Activation(&#39;relu&#39;))</pre><pre>model.add(layers.Conv2D(512, 1, strides=1, padding=&#39;same&#39;))<br>model.add(layers.Activation(&#39;relu&#39;))<br>model.add(layers.MaxPooling2D(2, strides=2))</pre><pre>model.add(layers.Flatten())</pre><pre>model.add(layers.Dense(4096, activation=&#39;relu&#39;))<br>model.add(layers.Dropout(0.5))</pre><pre>model.add(layers.Dense(4096, activation=&#39;relu&#39;))<br>model.add(layers.Dropout(0.5))</pre><pre>model.add(layers.Dense(10, activation=&#39;softmax&#39;))</pre><pre>model.summary()</pre><p>The model summary is as follows:</p><pre>Model: &quot;sequential&quot; _________________________________________________________________ Layer (type)                 Output Shape              Param #    ================================================================= resizing (Resizing)          (None, 224, 224, 3)       0          _________________________________________________________________ conv2d (Conv2D)              (None, 224, 224, 64)      1792       _________________________________________________________________ activation (Activation)      (None, 224, 224, 64)      0          _________________________________________________________________ conv2d_1 (Conv2D)            (None, 224, 224, 64)      36928      _________________________________________________________________ activation_1 (Activation)    (None, 224, 224, 64)      0          _________________________________________________________________ max_pooling2d (MaxPooling2D) (None, 112, 112, 64)      0          _________________________________________________________________ conv2d_2 (Conv2D)            (None, 112, 112, 128)     73856      _________________________________________________________________ activation_2 (Activation)    (None, 112, 112, 128)     0          _________________________________________________________________ conv2d_3 (Conv2D)            (None, 112, 112, 128)     147584     _________________________________________________________________ activation_3 (Activation)    (None, 112, 112, 128)     0          _________________________________________________________________ max_pooling2d_1 (MaxPooling2 (None, 56, 56, 128)       0          _________________________________________________________________ conv2d_4 (Conv2D)            (None, 56, 56, 256)       295168     _________________________________________________________________ activation_4 (Activation)    (None, 56, 56, 256)       0          _________________________________________________________________ conv2d_5 (Conv2D)            (None, 56, 56, 256)       590080     _________________________________________________________________ activation_5 (Activation)    (None, 56, 56, 256)       0          _________________________________________________________________ conv2d_6 (Conv2D)            (None, 56, 56, 256)       65792      _________________________________________________________________ activation_6 (Activation)    (None, 56, 56, 256)       0          _________________________________________________________________ max_pooling2d_2 (MaxPooling2 (None, 28, 28, 256)       0          _________________________________________________________________ conv2d_7 (Conv2D)            (None, 28, 28, 512)       1180160    _________________________________________________________________ activation_7 (Activation)    (None, 28, 28, 512)       0          _________________________________________________________________ conv2d_8 (Conv2D)            (None, 28, 28, 512)       2359808    _________________________________________________________________ activation_8 (Activation)    (None, 28, 28, 512)       0          _________________________________________________________________ conv2d_9 (Conv2D)            (None, 28, 28, 512)       262656     _________________________________________________________________ activation_9 (Activation)    (None, 28, 28, 512)       0          _________________________________________________________________ max_pooling2d_3 (MaxPooling2 (None, 14, 14, 512)       0          _________________________________________________________________ conv2d_10 (Conv2D)           (None, 14, 14, 512)       2359808    _________________________________________________________________ activation_10 (Activation)   (None, 14, 14, 512)       0          _________________________________________________________________ conv2d_11 (Conv2D)           (None, 14, 14, 512)       2359808    _________________________________________________________________ activation_11 (Activation)   (None, 14, 14, 512)       0          _________________________________________________________________ conv2d_12 (Conv2D)           (None, 14, 14, 512)       262656     _________________________________________________________________ activation_12 (Activation)   (None, 14, 14, 512)       0          _________________________________________________________________ max_pooling2d_4 (MaxPooling2 (None, 7, 7, 512)         0          _________________________________________________________________ flatten (Flatten)            (None, 25088)             0          _________________________________________________________________ dense (Dense)                (None, 4096)              102764544  _________________________________________________________________ dropout (Dropout)            (None, 4096)              0          _________________________________________________________________ dense_1 (Dense)              (None, 4096)              16781312   _________________________________________________________________ dropout_1 (Dropout)          (None, 4096)              0          _________________________________________________________________ dense_2 (Dense)              (None, 10)                40970      ================================================================= Total params: 129,582,922 Trainable params: 129,582,922 Non-trainable params: 0</pre><p>Notice that VGG16 has more than 129 million trainable parameters, which means that is more than 6 times the size of AlexNet.</p><pre>model.compile(optimizer=&#39;adam&#39;, loss=losses.sparse_categorical_crossentropy, metrics=[&#39;accuracy&#39;])<br>model.fit(x_train, y_train, batch_size=64, epochs=40, validation_data=(x_val, y_val))</pre><pre>Epoch 1/40 907/907 [==============================] - 628s 682ms/step - loss: 2.3029 - accuracy: 0.1094 - val_loss: 2.3019 - val_accuracy: 0.1070 <br>Epoch 2/40 907/907 [==============================] - 613s 676ms/step - loss: 2.3012 - accuracy: 0.1135 - val_loss: 2.3016 - val_accuracy: 0.1070<br>...<br>Epoch 40/40 907/907 [==============================] - 613s 676ms/step - loss: 2.3011 - accuracy: 0.1147 - val_loss: 2.3017 - val_accuracy: 0.1070</pre><p>After 40 epochs, the model did not even fit the training data. With 11.47% accuracy, it merely surpasses the random accuracy which is 10% in a ten-category classification problem. It also takes a great amount of time to train the model even with the GPU provided by Google Colab. So, there are two questions to be answered:</p><ol><li>Why could not the model learn?</li><li>What should be done to find a way around this problem?</li></ol><p>The original VGGNet models have been trained on the large scale ImageNet dataset before ILSVRC submission in 2014. Training AlexNet on MNIST was not a problem in the previous post, however VGG16 is a much larger model and requires much more data to be well trained. Simply, the MNIST dataset cannot provide the desired statistical variety for the model. The good news is that the pre-trained models are available in Keras and in many other deep learning frameworks and libraries. Trained and adjusted weights are available and it is a common practice to borrow some or all of the layers from these networks and use them as a <em>backbone feature extractor</em> module. This method is called <em>transfer learning</em> and it really speeds up deep learning projects by eliminating the need of training huge models from scratch.</p><p><strong><em>vggnet_tensorflow.ipynb</em></strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/de7a9ec259946e59c378f862c32fc1f9/href">https://medium.com/media/de7a9ec259946e59c378f862c32fc1f9/href</a></iframe><p><strong><em>Transfer Learning with VGG16</em></strong></p><p>As always, needed libraries are imported and data is loaded. Since the pre-trained VGG16 model only accepts images with 3 channels, data dimensions are adjusted accordingly.</p><pre>import tensorflow as tf<br>import matplotlib.pyplot as plt<br>from tensorflow.keras import datasets, layers, models, losses, Model</pre><pre>(x_train, y_train), (x_test, y_test)=tf.keras.datasets.mnist.load_data()<br>x_train = tf.pad(x_train, [[0, 0], [2,2], [2,2]])/255<br>x_test = tf.pad(x_test, [[0, 0], [2,2], [2,2]])/255<br>x_train = tf.expand_dims(x_train, axis=3, name=None)<br>x_test = tf.expand_dims(x_test, axis=3, name=None)<br>x_train = tf.repeat(x_train, 3, axis=3)<br>x_test = tf.repeat(x_test, 3, axis=3)<br>x_val = x_train[-2000:,:,:]<br>y_val = y_train[-2000:]<br>x_train = x_train[:-2000,:,:]<br>y_train = y_train[:-2000]</pre><p>The backbone (or base or feature extractor, however you want to name it) model is loaded as follows. Weights are updated for several epochs using the ImageNet dataset, and we only take the convolutional part of the model by setting <em>“include_top = False”</em>. Input shape is customized and we are choosing 32 x 32 x 3, instead of dealing with interpolation.</p><p>Also as the purpose of transfer learning, convolutional weights are frozen, meaning they are fixed and will not be updated during training. Thus, all the parameters are non-trainable.</p><pre>base_model = tf.keras.applications.VGG16(weights = &#39;imagenet&#39;, include_top = False, input_shape = (32,32,3))</pre><pre>for layer in base_model.layers:<br>  layer.trainable = False</pre><pre>base_model.summary()</pre><pre>Model: &quot;vgg16&quot; _________________________________________________________________ Layer (type)                 Output Shape              Param #    ================================================================= input_1 (InputLayer)         [(None, 32, 32, 3)]       0          _________________________________________________________________ block1_conv1 (Conv2D)        (None, 32, 32, 64)        1792       _________________________________________________________________ block1_conv2 (Conv2D)        (None, 32, 32, 64)        36928      _________________________________________________________________ block1_pool (MaxPooling2D)   (None, 16, 16, 64)        0          _________________________________________________________________ block2_conv1 (Conv2D)        (None, 16, 16, 128)       73856      _________________________________________________________________ block2_conv2 (Conv2D)        (None, 16, 16, 128)       147584     _________________________________________________________________ block2_pool (MaxPooling2D)   (None, 8, 8, 128)         0          _________________________________________________________________ block3_conv1 (Conv2D)        (None, 8, 8, 256)         295168     _________________________________________________________________ block3_conv2 (Conv2D)        (None, 8, 8, 256)         590080     _________________________________________________________________ block3_conv3 (Conv2D)        (None, 8, 8, 256)         590080     _________________________________________________________________ block3_pool (MaxPooling2D)   (None, 4, 4, 256)         0          _________________________________________________________________ block4_conv1 (Conv2D)        (None, 4, 4, 512)         1180160    _________________________________________________________________ block4_conv2 (Conv2D)        (None, 4, 4, 512)         2359808    _________________________________________________________________ block4_conv3 (Conv2D)        (None, 4, 4, 512)         2359808    _________________________________________________________________ block4_pool (MaxPooling2D)   (None, 2, 2, 512)         0          _________________________________________________________________ block5_conv1 (Conv2D)        (None, 2, 2, 512)         2359808    _________________________________________________________________ block5_conv2 (Conv2D)        (None, 2, 2, 512)         2359808    _________________________________________________________________ block5_conv3 (Conv2D)        (None, 2, 2, 512)         2359808    _________________________________________________________________ block5_pool (MaxPooling2D)   (None, 1, 1, 512)         0          ================================================================= Total params: 14,714,688 Trainable params: 0 Non-trainable params: 14,714,688</pre><p>Following that, fully connected layers (top layers) are added to the base model. The combination of the base model and added layers is name <em>head_model</em> and a computational graph is constructed and compiled.</p><pre>x = layers.Flatten()(base_model.output)</pre><pre>x = layers.Dense(4096, activation=&#39;relu&#39;)(x)<br>x = layers.Dropout(0.5)(x)</pre><pre>x = layers.Dense(4096, activation=&#39;relu&#39;)(x)<br>x = layers.Dropout(0.5)(x)</pre><pre>predictions = layers.Dense(10, activation = &#39;softmax&#39;)(x)</pre><pre>head_model = Model(inputs = base_model.input, outputs = predictions)</pre><pre>head_model.compile(optimizer=&#39;adam&#39;, loss=losses.sparse_categorical_crossentropy, metrics=[&#39;accuracy&#39;])</pre><p>The model summary is as follows:</p><pre>head_model.summary()</pre><pre>Model: &quot;model_1&quot; _________________________________________________________________ Layer (type)                 Output Shape              Param #    ================================================================= input_1 (InputLayer)         [(None, 32, 32, 3)]       0          _________________________________________________________________ block1_conv1 (Conv2D)        (None, 32, 32, 64)        1792       _________________________________________________________________ block1_conv2 (Conv2D)        (None, 32, 32, 64)        36928      _________________________________________________________________ block1_pool (MaxPooling2D)   (None, 16, 16, 64)        0          _________________________________________________________________ block2_conv1 (Conv2D)        (None, 16, 16, 128)       73856      _________________________________________________________________ block2_conv2 (Conv2D)        (None, 16, 16, 128)       147584     _________________________________________________________________ block2_pool (MaxPooling2D)   (None, 8, 8, 128)         0          _________________________________________________________________ block3_conv1 (Conv2D)        (None, 8, 8, 256)         295168     _________________________________________________________________ block3_conv2 (Conv2D)        (None, 8, 8, 256)         590080     _________________________________________________________________ block3_conv3 (Conv2D)        (None, 8, 8, 256)         590080     _________________________________________________________________ block3_pool (MaxPooling2D)   (None, 4, 4, 256)         0          _________________________________________________________________ block4_conv1 (Conv2D)        (None, 4, 4, 512)         1180160    _________________________________________________________________ block4_conv2 (Conv2D)        (None, 4, 4, 512)         2359808    _________________________________________________________________ block4_conv3 (Conv2D)        (None, 4, 4, 512)         2359808    _________________________________________________________________ block4_pool (MaxPooling2D)   (None, 2, 2, 512)         0          _________________________________________________________________ block5_conv1 (Conv2D)        (None, 2, 2, 512)         2359808    _________________________________________________________________ block5_conv2 (Conv2D)        (None, 2, 2, 512)         2359808    _________________________________________________________________ block5_conv3 (Conv2D)        (None, 2, 2, 512)         2359808    _________________________________________________________________ block5_pool (MaxPooling2D)   (None, 1, 1, 512)         0          _________________________________________________________________ flatten_1 (Flatten)          (None, 512)               0          _________________________________________________________________ dense_3 (Dense)              (None, 4096)              2101248    _________________________________________________________________ dropout_2 (Dropout)          (None, 4096)              0          _________________________________________________________________ dense_4 (Dense)              (None, 4096)              16781312   _________________________________________________________________ dropout_3 (Dropout)          (None, 4096)              0          _________________________________________________________________ dense_5 (Dense)              (None, 10)                40970      ================================================================= Total params: 33,638,218 Trainable params: 18,923,530 Non-trainable params: 14,714,688<br></pre><p>Now, the model is much easier to train. Compared to 129 million parameters the initial model has, this model has 18 million trainable parameters. Also initial model followed the original VGG16 architecture, whereas smaller images are used in this implementation (32 x 32).</p><pre>history = head_model.fit(x_train, y_train, batch_size=64, epochs=40, validation_data=(x_val, y_val))</pre><pre>Epoch 1/40 907/907 [==============================] - 628s 682ms/step - loss: 2.3029 - accuracy: 0.1094 - val_loss: 2.3019 - val_accuracy: 0.1070 <br>Epoch 2/40 907/907 [==============================] - 613s 676ms/step - loss: 2.3012 - accuracy: 0.1135 - val_loss: 2.3016 - val_accuracy: 0.1070<br>...<br>Epoch 40/40 907/907 [==============================] - 15s 16ms/step - loss: 0.2771 - accuracy: 0.9127 - val_loss: 0.1397 - val_accuracy: 0.9650</pre><p>The convolutional part of this model was already pre-trained to capture the patterns in ImageNet samples, however low-level features are common among many types of visual data such as points, edges, and simple curves. The base model extracts the ImageNet features and on top of it, <em>dense </em>layers perform the classification with those features.</p><pre>fig, axs = plt.subplots(2, 1, figsize=(15,15))</pre><pre>axs[0].plot(history.history[&#39;loss&#39;])<br>axs[0].plot(history.history[&#39;val_loss&#39;])<br>axs[0].title.set_text(&#39;Training Loss vs Validation Loss&#39;)<br>axs[0].set_xlabel(&#39;Epochs&#39;)<br>axs[0].set_ylabel(&#39;Loss&#39;)<br>axs[0].legend([&#39;Train&#39;,&#39;Val&#39;])</pre><pre>axs[1].plot(history.history[&#39;accuracy&#39;])<br>axs[1].plot(history.history[&#39;val_accuracy&#39;])<br>axs[1].title.set_text(&#39;Training Accuracy vs Validation Accuracy&#39;)<br>axs[1].set_xlabel(&#39;Epochs&#39;)<br>axs[1].set_ylabel(&#39;Accuracy&#39;)<br>axs[1].legend([&#39;Train&#39;, &#39;Val&#39;])</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/894/1*ArQM2M3ALsHRBlswH2dpIQ.png" /></figure><p>Since the validation data is relatively small, the results suffer from variance. Still, both curves exhibit satisfactory results.</p><p>Test accuracy came out at 95.05%.</p><pre>head_model.evaluate(x_test, y_test)</pre><pre>313/313 [==============================] - 3s 9ms/step - loss: 0.1821 - accuracy: 0.9505<br>[0.18214020133018494, 0.9505000114440918]</pre><p>Yet, it is kind of bothering that the training accuracy drops behind the validation accuracy by a good margin. It implies that regularization is overdone. The adjusted results after reducing the dropout rate from 0.5 to 0.2 are as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/894/1*uj3-Z5dV65DR6e2ePYSrSg.png" /></figure><p>In addition to the improvement in the training accuracy, test accuracy has also risen by 2.7%. These adjusted results indicate that if excessively used, dropout can limit the learning capacity of models.</p><pre>head_model.evaluate(x_test, y_test)</pre><pre>313/313 [==============================] - 6s 18ms/step - loss: 0.0885 - accuracy: 0.9772<br>[0.08849174529314041, 0.9771999716758728]</pre><p><strong><em>vggnet_transfer_learning_tensorflow.ipynb</em></strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a85daae0c9e4a8f6b6039cd6dc532964/href">https://medium.com/media/a85daae0c9e4a8f6b6039cd6dc532964/href</a></iframe><h4>Conclusion</h4><p>VGGNet leverages the fact that using smaller kernels in a stack is more efficient than using fewer and larger kernels. As an improvement to AlexNet, a deeper model is designed and the model outperformed all contemporary and previous models in ILSVRC-2014. Similar ideas like ReLU activation and dropout were also employed. Unlike AlexNet, <em>local response regularisation</em> is not used since it does not contribute to the final accuracy empirically. In this post, it is demonstrated that it is not a good idea to train this model with the very limited MNIST dataset. Instead, a pre-trained version on ImageNet is loaded and only top layers are trained. This approach resulted in over 95% testing accuracy. LeNet is still a better option for MNIST classification since it is much smaller but for the sake of a fair comparison, all the models are trained, validated, and tested on the same data. For more challenging datasets VGG16 would be a stronger choice.</p><p>Hope you enjoyed it. See you in the following CNN models.</p><p>Best wishes…</p><p>mrgrhn</p><p>For the follow-up post, please visit:</p><p><a href="https://mrgrhn.medium.com/googlenet-inceptionv1-with-tensorflow-9e7f3a161e87">GoogLeNet (InceptionV1) with TensorFlow</a></p><h4>References</h4><ol><li>Simonyan, Karen &amp; Zisserman, Andrew. (2014). “Very Deep Convolutional Networks for Large-Scale Image Recognition”. arXiv 1409.1556.</li><li>Pandiyan, Vigneashwara &amp; Tjahjowidodo, Tegoeh &amp; Caesarendra, Wahyu &amp; Murugan, Pushparaja. (2019). “In-process virtual verification of weld seam removal in robotic abrasive belt grinding process using deep learning”. Robotics and Computer-Integrated Manufacturing. 57. 477–487. 10.1016/j.rcim.2019.01.006.</li><li>LeCun, Y.; Boser, B.; Denker, J. S.; Henderson, D.; Howard, R. E.; Hubbard, W.; Jackel, L. D. (December 1989). “Backpropagation Applied to Handwritten Zip Code Recognition”. <em>Neural Computation</em>. 1(4): 541–551.</li><li>Krizhevsky, Alex &amp; Sutskever, Ilya &amp; Hinton, Geoffrey. (2012). “ImageNet Classification with Deep Convolutional Neural Networks”. Neural Information Processing Systems. 25. 10.1145/3065386.</li><li><a href="https://www.cambridgeincolour.com/tutorials/image-interpolation.htm">https://www.cambridgeincolour.com/tutorials/image-interpolation.htm</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7e5f6fa9479a" width="1" height="1" alt=""><hr><p><a href="https://ai.plainenglish.io/vggnet-with-tensorflow-transfer-learning-with-vgg16-included-7e5f6fa9479a">VGGNet with TensorFlow (Transfer Learning with VGG16 Included)</a> was originally published in <a href="https://ai.plainenglish.io/">Artificial Intelligence in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AlexNet with TensorFlow]]></title>
            <link>https://medium.com/swlh/alexnet-with-tensorflow-46f366559ce8?source=rss-87eead11556c------2</link>
            <guid isPermaLink="false">https://medium.com/p/46f366559ce8</guid>
            <category><![CDATA[cnn]]></category>
            <category><![CDATA[tensorflow]]></category>
            <category><![CDATA[alexnet]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[mrgrhn]]></dc:creator>
            <pubDate>Tue, 19 Jan 2021 17:16:25 GMT</pubDate>
            <atom:updated>2021-01-21T00:20:13.798Z</atom:updated>
            <content:encoded><![CDATA[<p>AlexNet is an important milestone in the visual recognition tasks in terms of available hardware utilization and several architectural choices. After its publication in 2012 by Alex Krizhevsky et al.[1] the popularity of deep learning and, in specific, the popularity of CNNs grew drastically. Below you can see the architecture of AlexNet:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*hhd-belzmau_TMcF.png" /><figcaption>AlexNet Architecture (It is also truncated from the top in the original paper.)</figcaption></figure><p>For the previous post, please visit:</p><p><a href="https://mrgrhn.medium.com/lenet-with-tensorflow-a35da0d503df">LeNet with TensorFlow</a></p><h4>Introduction</h4><p>After the big success of LeNet[5] in handwritten digit recognition, computer vision applications using deep learning came to a halt. Instead of feature extraction methods by filter convolutions, researchers preferred more handcrafted image processing tasks such as wavelets, Gabor filters, and many more. There were several reasons why the community did not push CNNs for many years to undertake more complex tasks. First of all, LeNet was good for MNIST but very few people believed that it is capable of dealing with more challenging data. After a while, it was not very interesting to work with MNIST over and over again. This links us to the second reason which was the lack of annotated data. Training neural networks requires an abundance of data and the need for more data increases as the model complexity (model size) increases. Since it is expensive to annotate data and CNNs were not really popular, large-scale datasets were not available at the time. Last but not least, training large models (assuming enough training data is provided) is a computationally expensive task. Each added layer of a deep learning model increases the number of parameters to be trained which significantly slows down the training. The hardware used in the ’90s took huge amounts of time to train even basic models and it was not feasible to design larger models.</p><p>All those reasons prevented new breakthroughs in the deep learning community for a long time. However, the last decade witnessed many exciting new projects and it feels like we are just getting started. In 2009, the ImageNet dataset is released with the title of <em>ImageNet: A large-scale hierarchical image database</em>[2]. Initially having 3.2 million annotated images, ImageNet now has more than 14 million images that are labeled in more than 20 thousand categories. That much of data encouraged many groups worldwide and excited them for <em>the ImageNet Large Scale Visual Recognition Challenge (ILSRVC) </em>which took place between 2010 and 2017. On the other hand, improvements were accelerated in hardware at the dawn of the new millennium. NVIDIA released the first commercial GPU GeForce 256 in 1999 and GPU technology was first used for deep learning in 2006 by Kumar Chellapilla et al[3]. Later on in 2009, Raina et al.[4] took one step further for the popularity of GPUs on deep learning.</p><p>However, most of the credit goes to AlexNet in terms of the prevalence of GPU supported computation in deep learning literature. Winning the ILSRVC in 2012 by a quite large margin compared to the previous year’s winner, AlexNet proved that from now on, it is a necessity to employ state-of-the-art hardware for training the state-of-the-art models for the most complex task. AlexNet had more layers than LeNet has which brings a greater learning capacity. It is originally trained on the ImageNet dataset.</p><h4>AlexNet with Tensorflow</h4><p>This tutorial is intended for beginners to demonstrate a basic TensorFlow implementation of AlexNet on the MNIST dataset. For more information on CNNs and TensorFlow, you can visit the previous post linked at the beginning of this article. The reason for the usage of MNIST instead of ImageNet is simplicity, but the model can be used for any dataset with very few variations in the code.</p><p>First, needed libraries are imported.</p><pre>import tensorflow as tf<br>import matplotlib.pyplot as plt</pre><pre>from tensorflow.keras import datasets, layers, models, losses</pre><p><strong><em>The Data</em></strong></p><p>Then, the data is loaded as in the LeNet implementation. One important notice is that the original AlexNet model receives images with the size 224 x 224 x 3 however, MNIST images are 28 x 28. The third axis is expanded and repeated 3 times to make image sizes 28 x 28 x 3. It will be then resized at the first layer of the model to 224 x 224 x 3.</p><pre>x_train,y_train),(x_test,y_test) = datasets.mnist.load_data()</pre><pre>x_train = tf.pad(x_train, [[0, 0], [2,2], [2,2]])/255<br>x_test = tf.pad(x_test, [[0, 0], [2,2], [2,2]])/255</pre><pre>x_train = tf.expand_dims(x_train, axis=3, name=None)<br>x_test = tf.expand_dims(x_test, axis=3, name=None)</pre><pre>x_train = tf.repeat(x_train, 3, axis=3)<br>x_test = tf.repeat(x_test, 3, axis=3)</pre><pre>x_val = x_train[-2000:,:,:,:]<br>y_val = y_train[-2000:]<br>x_train = x_train[:-2000,:,:,:]<br>y_train = y_train[:-2000]</pre><p><strong><em>The Model</em></strong></p><p>AlexNet is one of the first examples of <em>deep </em>convolutional neural networks and it swept the competitions thanks to its high complexity (at the time) and the training procedure that is performed on GPUs. However, it was still not easy to train large networks due to vanishing or exploding gradient problems and long training times. In general, gradient-based deep neural networks are inclined to face unstable gradient updates. AlexNet is large enough to learn the patterns in the ImageNet dataset but very shallow compared to today’s models. Why AlexNet could not be designed deeper becomes clearer after understanding the residual connections in ResNet.</p><p>The first layer (resizing layer) of the model does not exist in the original implementation since the data were already compatible with the model in terms of image size. For a compatible representation, MNIST images were interpolated bilinearly such that the new sizes are 224 x 224 (also one more dimension for 3 channels corresponding to RGB channels in ImageNet).</p><p>The first five layers are similar to LeNet layers in a sense. The layers consist of 96 filters with the sizes of 11 x 11, 256 layers with the sizes of 5 x 5, 384 layers with the sizes of 3 x 3, 384 layers with the sizes of 3 x 3, and finally 256 filters with the sizes of 3 x 3, respectively. Unlike LeNet, AlexNet uses ReLU[5] activation function. ReLU does not involve exponentiation operation, which is computationally expensive, thus ReLU is very cheap to calculate. It also reduces the effects of vanishing gradient and resembles the mechanism of biological cells. Yet, ReLU sometimes leads to dying neurons and it is advised to keep track of the activations during the first few epochs of the training.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MZDbeA215VZld2AFCUeJvg.png" /><figcaption>Response graphs of some nonlinear activation functions</figcaption></figure><p>Another new concept is <em>local response normalization</em>, which is a novel trick to make training go smooth and stable. Unlike <em>sigmoid </em>and <em>tanh </em>activations, ReLU has an unbounded response in the positive edge. Thus, activations can blow up and hurt the training procedure as the training continues. This is why some kind of normalization is required. Local response normalization attempts to balance the activation values in a pixel among neighboring channels. Each activation is divided by the sum of the squares of neighboring channels’ activations in the same pixel location. This idea is borrowed from the neurobiological term <em>lateral inhibition</em> and first described in the original AlexNet paper[1] in detail. Nowadays batch normalization is much more pervasive in the literature and it is faster than LRN but it was not available back in 2012.</p><p>After the convolutional layers, the resulting tensor (1 x 1 x 256) is flattened and fed into fully connected layers. After passing two hidden layers having the sizes of 4096, final class scores (for 10 classes) are obtained by softmax operation. Here, <em>dropout </em>is a measure to prevent overfitting. In the model below, half of the neurons in hidden fully connected layers are shut down randomly. During each forward passing of the training samples, a specified fraction of the activations in a layer is set to 0, and in order to keep the mean activation level of the layer the same, the remaining activations are divided by dropout probability (in this case divided by 0.5 or simply multiplied by 2). During test time, dropout is not performed.</p><pre>model = models.Sequential()</pre><pre>model.add(layers.experimental.preprocessing.Resizing(224, 224, interpolation=&quot;bilinear&quot;, input_shape=x_train.shape[1:]))</pre><pre>model.add(layers.Conv2D(96, 11, strides=4, padding=&#39;same&#39;))<br>model.add(layers.Lambda(tf.nn.local_response_normalization))<br>model.add(layers.Activation(&#39;relu&#39;))<br>model.add(layers.MaxPooling2D(3, strides=2))</pre><pre>model.add(layers.Conv2D(256, 5, strides=4, padding=&#39;same&#39;))<br>model.add(layers.Lambda(tf.nn.local_response_normalization))<br>model.add(layers.Activation(&#39;relu&#39;))<br>model.add(layers.MaxPooling2D(3, strides=2))</pre><pre>model.add(layers.Conv2D(384, 3, strides=4, padding=&#39;same&#39;))<br>model.add(layers.Activation(&#39;relu&#39;))</pre><pre>model.add(layers.Conv2D(384, 3, strides=4, padding=&#39;same&#39;))<br>model.add(layers.Activation(&#39;relu&#39;))</pre><pre>model.add(layers.Conv2D(256, 3, strides=4, padding=&#39;same&#39;))<br>model.add(layers.Activation(&#39;relu&#39;))</pre><pre>model.add(layers.Flatten())</pre><pre>model.add(layers.Dense(4096, activation=&#39;relu&#39;))<br>model.add(layers.Dropout(0.5))</pre><pre>model.add(layers.Dense(4096, activation=&#39;relu&#39;))<br>model.add(layers.Dropout(0.5))</pre><pre>model.add(layers.Dense(10, activation=&#39;softmax&#39;))</pre><pre>model.summary()</pre><p>The model summary is as follows:</p><pre>Model: &quot;sequential&quot; _________________________________________________________________ Layer (type)                 Output Shape              Param #    ================================================================= resizing (Resizing)          (None, 224, 224, 3)       0          _________________________________________________________________ conv2d (Conv2D)              (None, 56, 56, 96)        34944      _________________________________________________________________ lambda (Lambda)              (None, 56, 56, 96)        0          _________________________________________________________________ activation (Activation)      (None, 56, 56, 96)        0          _________________________________________________________________ max_pooling2d (MaxPooling2D) (None, 27, 27, 96)        0          _________________________________________________________________ conv2d_1 (Conv2D)            (None, 7, 7, 256)         614656     _________________________________________________________________ lambda_1 (Lambda)            (None, 7, 7, 256)         0          _________________________________________________________________ activation_1 (Activation)    (None, 7, 7, 256)         0          _________________________________________________________________ max_pooling2d_1 (MaxPooling2 (None, 3, 3, 256)         0          _________________________________________________________________ conv2d_2 (Conv2D)            (None, 1, 1, 384)         885120     _________________________________________________________________ activation_2 (Activation)    (None, 1, 1, 384)         0          _________________________________________________________________ conv2d_3 (Conv2D)            (None, 1, 1, 384)         1327488    _________________________________________________________________ activation_3 (Activation)    (None, 1, 1, 384)         0          _________________________________________________________________ conv2d_4 (Conv2D)            (None, 1, 1, 256)         884992     _________________________________________________________________ activation_4 (Activation)    (None, 1, 1, 256)         0          _________________________________________________________________ flatten (Flatten)            (None, 256)               0          _________________________________________________________________ dense (Dense)                (None, 4096)              1052672    _________________________________________________________________ dropout (Dropout)            (None, 4096)              0          _________________________________________________________________ dense_1 (Dense)              (None, 4096)              16781312   _________________________________________________________________ dropout_1 (Dropout)          (None, 4096)              0          _________________________________________________________________ dense_2 (Dense)              (None, 10)                40970      ================================================================= Total params: 21,622,154 Trainable params: 21,622,154 Non-trainable params: 0</pre><p>The model is compiled and trained as in the previous post where LeNet is implemented. The choice of the <em>Adam </em>optimizer and the mathematical background of the optimizers are the topics of another post. However, it should be mentioned that Adam is the most common choice among the deep learning community for adaptively updating the learning rate.</p><pre>model.compile(optimizer=&#39;adam&#39;, loss=losses.sparse_categorical_crossentropy, metrics=[&#39;accuracy&#39;])history = model.fit(x_train, y_train, batch_size=64, epochs=40, validation_data=(x_val, y_val))</pre><pre>Epoch 1/40 907/907 [==============================] - 111s 114ms/step - loss: 1.0786 - accuracy: 0.5796 - val_loss: 0.1184 - val_accuracy: 0.9665 <br>Epoch 2/40 907/907 [==============================] - 103s 114ms/step - loss: 0.1036 - accuracy: 0.9729 - val_loss: 0.0595 - val_accuracy: 0.9845<br>...<br>Epoch 40/40 907/907 [==============================] - 103s 114ms/step - loss: 0.0051 - accuracy: 0.9990 - val_loss: 0.1677 - val_accuracy: 0.9930</pre><p>The accuracy rises up very quickly in the validation set and fluctuates above 98% throughout the training, which means that only one epoch was already enough for the relatively small validation test. Training loss decreases consistently as expected. Using LRN instead of batch normalization slows down the training while not really contributing to the final accuracy.</p><pre>fig, axs = plt.subplots(2, 1, figsize=(15,15))</pre><pre>axs[0].plot(history.history[&#39;loss&#39;])<br>axs[0].plot(history.history[&#39;val_loss&#39;])<br>axs[0].title.set_text(&#39;Training Loss vs Validation Loss&#39;)<br>axs[0].set_xlabel(&#39;Epochs&#39;)<br>axs[0].set_ylabel(&#39;Loss&#39;)<br>axs[0].legend([&#39;Train&#39;, &#39;Val&#39;])</pre><pre>axs[1].plot(history.history[&#39;accuracy&#39;])<br>axs[1].plot(history.history[&#39;val_accuracy&#39;])<br>axs[1].title.set_text(&#39;Training Accuracy vs Validation Accuracy&#39;)<br>axs[1].set_xlabel(&#39;Epochs&#39;)<br>axs[1].set_ylabel(&#39;Accuracy&#39;)<br>axs[1].legend([&#39;Train&#39;, &#39;Val&#39;])</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/894/1*CgoPjnpIA322tahKX110eQ.png" /></figure><p>Test accuracy came out at 98.79%.</p><pre>model.evaluate(x_test, y_test)</pre><pre>313/313 [==============================] - 2s 7ms/step - loss: 0.0809 - accuracy: 0.9879<br>[0.08089840412139893, 0.9879000186920166]</pre><h4>alexnet_tensorflow.ipynb</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0d05b480c14d00354bf809bb664457f6/href">https://medium.com/media/0d05b480c14d00354bf809bb664457f6/href</a></iframe><h4>Conclusion</h4><p>AlexNet started a revolution in the computer vision community and paved the road for many other deep learning models. In the following years, ILSRVC became much more competitive and winner models reached higher and higher accuracies. The use of GPUs, ReLU activations, local response normalization layers, and dropout mechanism were all new compared to the predecessor model, LeNet. In this implementation, a relatively light MNIST dataset is used for fast training and simplicity, for which AlexNet is an overkill. The original model is trained and evaluated on ImageNet in 2012.</p><p>Hope you enjoyed it. See you in the following CNN models.</p><p>Best wishes…</p><p>mrgrhn</p><p>For the follow-up post, please visit:</p><p><a href="https://mrgrhn.medium.com/vggnet-with-tensorflow-transfer-learning-with-vgg16-included-7e5f6fa9479a">VGGNet with TensorFlow (Transfer Learning with VGG16 Included)</a></p><h4>References</h4><ol><li>Krizhevsky, Alex &amp; Sutskever, Ilya &amp; Hinton, Geoffrey. (2012). “ImageNet Classification with Deep Convolutional Neural Networks”. Neural Information Processing Systems. 25. 10.1145/3065386.</li><li>Deng, Jia &amp; Dong, Wei &amp; Socher, Richard &amp; Li, Li-Jia &amp; Li, Kai &amp; Li, Fei Fei. (2009). “ImageNet: a Large-Scale Hierarchical Image Database”. IEEE Conference on Computer Vision and Pattern Recognition. 248–255. 10.1109/CVPR.2009.5206848.</li><li>Chellapilla, Kumar &amp; Puri, Sidd &amp; Simard, Patrice. (2006). “High Performance Convolutional Neural Networks for Document Processing”.</li><li>Raina, Rajat &amp; Madhavan, Anand &amp; Ng, Andrew. (2009). “Large-scale deep unsupervised learning using graphics processors”. Proceedings of the 26th International Conference On Machine Learning, ICML 2009. 382. 110. 10.1145/1553374.1553486.</li><li>Nair, Vinod &amp; Hinton, Geoffrey. (2010). “Rectified Linear Units Improve Restricted Boltzmann Machines”. Proceedings of ICML. 27. 807–814.</li><li>LeCun, Y.; Boser, B.; Denker, J. S.; Henderson, D.; Howard, R. E.; Hubbard, W.; Jackel, L. D. (December 1989). “Backpropagation Applied to Handwritten Zip Code Recognition”. <em>Neural Computation</em>. 1(4): 541–551.</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=46f366559ce8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/swlh/alexnet-with-tensorflow-46f366559ce8">AlexNet with TensorFlow</a> was originally published in <a href="https://medium.com/swlh">The Startup</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[LeNet with TensorFlow]]></title>
            <link>https://medium.com/analytics-vidhya/lenet-with-tensorflow-a35da0d503df?source=rss-87eead11556c------2</link>
            <guid isPermaLink="false">https://medium.com/p/a35da0d503df</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[yann-lecun]]></category>
            <category><![CDATA[tensorflow]]></category>
            <category><![CDATA[cnn]]></category>
            <dc:creator><![CDATA[mrgrhn]]></dc:creator>
            <pubDate>Mon, 18 Jan 2021 16:56:48 GMT</pubDate>
            <atom:updated>2021-01-21T15:53:23.353Z</atom:updated>
            <content:encoded><![CDATA[<p>LeNet is considered to be the ancestor of convolutional neural networks and is a well-known model among the computer vision community.</p><figure><img alt="LeNet for Digit Recognition" src="https://cdn-images-1.medium.com/max/532/0*u-yUNdr5JhIDgmYs" /><figcaption>LeNet for Digit Recognition</figcaption></figure><h4>Introduction</h4><p>LeNet is one of the most fundamental deep learning models that is primarily used to classify handwritten digits. Proposed by Yann LeCun[1] in 1989, LeNet is one of the earliest neural networks that employ the convolution operation. Combining newly developed back-propagation algorithms with convolutional neural networks, LeCun et al. became pioneers of image classification using deep learning. The name LeNet is mostly used interchangeably with LeNet-5 which indicates the kernel size of the convolution masks.</p><p>This tutorial is intended for beginners to demonstrate a basic TensorFlow implementation of LeNet on the MNIST dataset. References to the related papers are shared at the end of the blog post. For a better understanding of the underlying concepts you can go through these videos:</p><p>Backpropagation:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FIlg3gGewQ5U%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DIlg3gGewQ5U&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FIlg3gGewQ5U%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/5dbc3754f254c17bab73ee29247c0cc0/href">https://medium.com/media/5dbc3754f254c17bab73ee29247c0cc0/href</a></iframe><p>Convolutional Neural Networks:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FumGJ30-15_A%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DumGJ30-15_A&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FumGJ30-15_A%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/6560ef8a79d6fe5a25732456d04e2858/href">https://medium.com/media/6560ef8a79d6fe5a25732456d04e2858/href</a></iframe><h4>LeNet with TensorFlow</h4><p>Tensorflow is one of, if not the, most popular deep learning frameworks that enables machine learning enthusiasts to work with general prototype models. Although it is not a good habit to build models by the default settings of TensorFlow without inspecting the consequences of certain design choices, Keras (a deep learning library that performs on TensorFlow backend) makes it fairly easy to test flexible models when utilized properly.</p><p>Without further introduction, we can jump into LeNet implementation with TensorFlow. The code is written as a Jupyter Notebook[5] hosted by Google Colab[6]. The link for the whole notebook is shared just before the References section. In order to utilize free GPUs of Google for your computations, please follow: <em>Runtime &gt; Change runtime time &gt; Hardware accelerator: GPU </em>in your Colab notebook.</p><p>First, needed libraries are imported. TensorFlow and Matplotlib are imported to design the network and visualize the results, respectively. Moreover, Keras offers certain architecture and training templates as well as common datasets, which are handy to use.</p><pre><strong>import</strong> <strong>tensorflow</strong> <strong>as</strong> <strong>tf</strong><br><strong>import</strong> <strong>matplotlib.pyplot</strong> <strong>as</strong> <strong>plt</strong><br><strong>from</strong> <strong>tensorflow.keras</strong> <strong>import</strong> datasets, layers, models, losses</pre><p><strong><em>The Data</em></strong></p><p>After importing the libraries, the dataset is downloaded by a single line of code. Be careful to take the outputs in the correct format as below:</p><pre>x_train,y_train),(x_test,y_test) = datasets.mnist.load_data()<br>x_train.shape</pre><pre>(60000, 28, 28)</pre><p>Original LeNet model receives 32 by 32 images, thus 28 by 28 MNIST images are padded with zeros and 8-bit (0–255 range) pixel values are scaled between 0 and 1:</p><pre>x_train = tf.pad(x_train, [[0, 0], [2,2], [2,2]])/255<br>x_test = tf.pad(x_test, [[0, 0], [2,2], [2,2]])/255<br>x_train.shape</pre><pre>TensorShape([60000, 32, 32])</pre><p>However, most CNN’s accept 4-dimensional tensors as inputs having the dimensions of batch size, height, width, and channel. Since MNIST images are grayscale, the last dimension does not necessarily exist. We need to expand the tensor and create a dummy dimension at axis number 3. (Recall that the tensor initially had axis 0, 1, and 2.)</p><pre>x_train = tf.expand_dims(x_train, axis=3, name=<strong>None</strong>)<br>x_test = tf.expand_dims(x_test, axis=3, name=<strong>None</strong>)<br>x_train.shape</pre><pre>TensorShape([60000, 32, 32, 1])</pre><p>The last 2000 samples of the training set are reserved for the validation set. The validation set is mainly used for tuning of the hyperparameters of the model. The test set is never used before the final evaluation.</p><pre>x_val = x_train[-2000:,:,:,:] <br>y_val = y_train[-2000:] <br>x_train = x_train[:-2000,:,:,:] <br>y_train = y_train[:-2000]</pre><p><strong><em>The Model</em></strong></p><p>LeNet has a pretty simple architecture. There are 3<em> convolutional layers</em> each having 5 by 5 kernels with 6, 16, and 120 feature maps, respectively. In between, there are 2 <em>subsampling layers</em> as <em>average pooling</em>. All these 5 layers use <em>stride </em>1 and average pooling layers use 2 by 2 <em>kernels </em>as default settings. After the convolutional layers, <em>hyperbolic tangent nonlinear activation </em>is used, whereas subsampling layers are followed by <em>sigmoid nonlinearity</em>. After the last convolutional layer, the activations are <em>flattened </em>and fed into<em> fully connected layers</em> having 84 and 10 neurons. The output of the last layer (after <em>softmax </em>operation) represents the probabilities of the classes (from 0 to 9) for the input image.</p><p>Nowadays, <em>tanh </em>and <em>sigmoid </em>activations are rarely used due to saturation problems. Instead, <em>ReLU </em>and<em> Leaky ReLU</em> are much more popular. Also, using two consecutive 3 by 3 layers instead of a single 5 by 5 layer is more preferable, since the same receptive field size is obtained by a fairly lower number of trainable parameters.</p><pre>model = models.Sequential()<br>model.add(layers.Conv2D(6, 5, activation=&#39;tanh&#39;, input_shape=x_train.shape[1:]))<br>model.add(layers.AveragePooling2D(2))<br>model.add(layers.Activation(&#39;sigmoid&#39;))<br>model.add(layers.Conv2D(16, 5, activation=&#39;tanh&#39;))<br>model.add(layers.AveragePooling2D(2))<br>model.add(layers.Activation(&#39;sigmoid&#39;))<br>model.add(layers.Conv2D(120, 5, activation=&#39;tanh&#39;))<br>model.add(layers.Flatten())<br>model.add(layers.Dense(84, activation=&#39;tanh&#39;))<br>model.add(layers.Dense(10, activation=&#39;softmax&#39;))<br>model.summary()</pre><pre>Model: &quot;sequential&quot;<br>_________________________________________________________________<br>Layer (type)                 Output Shape              Param #   <br>=================================================================<br>conv2d (Conv2D)              (None, 28, 28, 6)         156       <br>_________________________________________________________________<br>average_pooling2d (AveragePo (None, 14, 14, 6)         0         <br>_________________________________________________________________<br>activation (Activation)      (None, 14, 14, 6)         0         <br>_________________________________________________________________<br>conv2d_1 (Conv2D)            (None, 10, 10, 16)        2416      <br>_________________________________________________________________<br>average_pooling2d_1 (Average (None, 5, 5, 16)          0         <br>_________________________________________________________________<br>activation_1 (Activation)    (None, 5, 5, 16)          0         <br>_________________________________________________________________<br>conv2d_2 (Conv2D)            (None, 1, 1, 120)         48120     <br>_________________________________________________________________<br>flatten (Flatten)            (None, 120)               0         <br>_________________________________________________________________<br>dense (Dense)                (None, 84)                10164     <br>_________________________________________________________________<br>dense_1 (Dense)              (None, 10)                850       <br>=================================================================<br>Total params: 61,706<br>Trainable params: 61,706<br>Non-trainable params: 0</pre><p>The model is set to be optimized by <em>Adam </em>optimizer[7]. <em>Sparse categorical cross-entropy</em> measures the negative of the natural logarithm of the true class probability. For instance, if the final output of the model is a vector such as [0.03, 0.78, …, 0.05] and the true class of the input image is 1; the loss for this instance will be -ln(0.78) = 0.248. <em>Accuracy </em>metric is reported for each epoch. Model is trained for 40 <em>epochs </em>with a <em>batch size </em>of 64<em>. history </em>object has a property named<em> history </em>which is useful for keeping track of the training phase.</p><pre>model.compile(optimizer=&#39;adam&#39;, loss=losses.sparse_categorical_crossentropy, metrics=[&#39;accuracy&#39;])<br>history = model.fit(x_train, y_train, batch_size=64, epochs=40, validation_data=(x_val, y_val))<br></pre><pre>Epoch 1/40 907/907 [==============================] - 11s 6ms/step - loss: 1.8860 - accuracy: 0.2950 - val_loss: 0.2266 - val_accuracy: 0.9450 <br>Epoch 2/40 907/907 [==============================] - 4s 5ms/step - loss: 0.3354 - accuracy: 0.8943 - val_loss: 0.1769 - val_accuracy: 0.9490<br>...<br>Epoch 40/40 907/907 [==============================] - 5s 5ms/step - loss: 0.0375 - accuracy: 0.9875 - val_loss: 0.0428 - val_accuracy: 0.9915</pre><h4>Results</h4><p>Losses and accuracies for training and validation sets are stored in <em>history </em>object and plotted using Matplotlib library.</p><pre>fig, axs = plt.subplots(2, 1, figsize=(15,15))  axs[0].plot(history.history[&#39;loss&#39;]) axs[0].plot(history.history[&#39;val_loss&#39;]) axs[0].title.set_text(&#39;Training Loss vs Validation Loss&#39;) axs[0].legend([&#39;Train&#39;, &#39;Val&#39;])  axs[1].plot(history.history[&#39;accuracy&#39;]) axs[1].plot(history.history[&#39;val_accuracy&#39;]) axs[1].title.set_text(&#39;Training Accuracy vs Validation Accuracy&#39;) axs[1].legend([&#39;Train&#39;, &#39;Val&#39;])</pre><figure><img alt="Losses and Accuracies for Training and Validation Sets" src="https://cdn-images-1.medium.com/max/880/1*wMh37U3zr6nABv3Td01g6A.png" /><figcaption>Losses and Accuracies for Training and Validation Sets</figcaption></figure><p>Testing accuracy of the model came out at 98.51%, which is pretty satisfactory for this simple task and reasonably slightly lower than training accuracy. The graph above and the testing accuracy indicates that the model is capable of learning the patterns of digit drawings and generalizable enough not to overfit.</p><pre>model.evaluate(x_test, y_test)</pre><pre>313/313 [==============================] - 1s 2ms/step - loss: 0.0468 - accuracy: 0.9851</pre><h4>lenet_tensorflow.ipynb</h4><p>You can check out the whole notebook uninterrupted on Github and Google Colab.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ab9dbb27b78ef153f1003a2ab3438177/href">https://medium.com/media/ab9dbb27b78ef153f1003a2ab3438177/href</a></iframe><h4>Conclusion</h4><p>The aim of contemporary CNNs well exceeded the simple tasks like MNIST classification however, it is indispensable to analyze fundamental models like LeNet on common datasets like MNIST to get the underlying ideas of more complex models. Trying to replicate the simple models and obtain similar results is a good first step for diving deep into new tasks. Additionally, examining the shortcomings of these models equally important and useful. In this post, LeNet architecture is explained and implemented with TensorFlow.</p><p>Hope you enjoyed it. See you in the following CNN models.</p><p>Best wishes…</p><p>mrgrhn</p><p>For the follow-up post, please visit:</p><p><a href="https://mrgrhn.medium.com/alexnet-with-tensorflow-46f366559ce8">AlexNet with TensorFlow</a></p><h4>References</h4><ol><li>LeCun, Y.; Boser, B.; Denker, J. S.; Henderson, D.; Howard, R. E.; Hubbard, W.; Jackel, L. D. (December 1989). “Backpropagation Applied to Handwritten Zip Code Recognition”. <em>Neural Computation</em>. 1(4): 541–551.</li><li>Lecun, Yann (June 1989). “Generalization and network design strategies”. Technical Report CRG-TR-89–4. Department of Computer Science, University of Toronto.</li><li>LeCun, Y.; Boser, B.; Denker, J. S.; Henderson, D.; Howard, R. E.; Hubbard, W.; Jacker, L. D. (June 1990). “Handwritten digit recognition with a back-propagation network”. Advances in Neural Information Processing Systems 2: 396–404.</li><li>Lecun, Y.; Bottou, L.; Bengio, Y.; Haffner, P. (1998). “Gradient-based learning applied to document recognition”. Proceedings of the IEEE. 86 (11): 2278–2324.</li><li><a href="https://jupyter-notebook.readthedocs.io/en/stable/notebook.html">https://jupyter-notebook.readthedocs.io/en/stable/notebook.html</a></li><li><a href="https://colab.research.google.com/notebooks/intro.ipynb">https://colab.research.google.com/notebooks/intro.ipynb</a></li><li>Kingma, Diederik &amp; Ba, Jimmy. (2014). “Adam: A Method for Stochastic Optimization”. International Conference on Learning Representations.</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a35da0d503df" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/lenet-with-tensorflow-a35da0d503df">LeNet with TensorFlow</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>