<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Chinesh Doshi on Medium]]></title>
        <description><![CDATA[Stories by Chinesh Doshi on Medium]]></description>
        <link>https://medium.com/@chinesh4?source=rss-c4f2657b8262------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*yMK2kJPKWGZmJJhGdGaUBQ.jpeg</url>
            <title>Stories by Chinesh Doshi on Medium</title>
            <link>https://medium.com/@chinesh4?source=rss-c4f2657b8262------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 04:40:03 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@chinesh4/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Generalized way of Interpreting CNNs!!]]></title>
            <link>https://medium.com/@chinesh4/generalized-way-of-interpreting-cnns-a7d1b0178709?source=rss-c4f2657b8262------2</link>
            <guid isPermaLink="false">https://medium.com/p/a7d1b0178709</guid>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[interpretability]]></category>
            <dc:creator><![CDATA[Chinesh Doshi]]></dc:creator>
            <pubDate>Sat, 22 Aug 2020 10:16:47 GMT</pubDate>
            <atom:updated>2020-08-22T11:15:45.196Z</atom:updated>
            <content:encoded><![CDATA[<h3>Generalized way of Interpreting CNNs using Guided Gradient Class Activation Maps!!</h3><p>Convolutional Neural Networks are very good at extracting features from the images and performing tasks like Image classification, Image Segmentation, Object Detection, Image captioning and visual question answering etc. While these models enable superior performance, their lack of decomposability into individually intuitive components makes them hard to interpret. You might have come across a situation where you are unable to find why the state of the art model is not giving satisfactory results for your use case which leads you in various speculations about the code, adversarial attack, model architecture, hyper-parameters, and Dataset, etc. Interpretability matters a lot to build trust in intelligent systems and move towards their meaningful integration into our everyday lives.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*82j9C_0XJ9oJ4Y0V.png" /><figcaption>CNNs are widely used in Self Driving Cars</figcaption></figure><p>There typically exists a trade-off between accuracy and interpretability. Classical rule-based models such as Random Forest , Decisoin Trees, Linear REgression etc. are highly interpretable but not very accurate. By using deep models, we sacrifice interpretable modules for uninterpretable ones that achieve greater performance through greater abstraction (more layers). For example, deep residual networks (ResNets) have over 200-layers deep and have shown state-of-the-art performance in several challenging tasks. Such complexity makes these models hard to interpret. In this article, we will understand various works that have been done to interpret the Computer Vision Algorithms using Guided Backpropagation, Class Activation Maps, Grad CAM and Guided Grad-CAM.</p><p>Code: <a href="https://github.com/chinesh/Guided_Grad_CAM">https://github.com/chinesh/Guided_Grad_CAM</a></p><p>Mentioning some of the terms which we will be using to explain the algorithm later in the blog:</p><h4>Global Average Pooling (GAP)</h4><p>Similar to max pooling layers, GAP layers are used to reduce the spatial dimensions of a three-dimensional tensor. However, GAP layers perform a more extreme type of dimensionality reduction, where a tensor with dimensions <em>h</em>×<em>w</em>×<em>d </em>is reduced in size to have dimensions 1×1×<em>d</em>. GAP layers reduce each <em>h</em>×<em>w</em> feature map to a single number by simply taking the average of all <em>hw</em> values. Experts have used global average pooling (GAP) layers to minimize overfitting by reducing the total number of parameters in the model by removing the Fully Connected Layer which is considered to have the highest number of parameters in a network.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*H12J-kmHF4bid56N" /><figcaption>Global Average Pooling</figcaption></figure><p>Why we are using GAP, though Global Max Pooling (GMP) is also there:</p><p>GAP loss encourages the network to identify the extent of the object as compared to GMP which encourages it to identify just one discriminative part. This is because, when doing the average of a map, the value can be maximized by finding all discriminative parts of an object as all low activations reduce the output of the particular map. On the other hand, for GMP, low scores for all image regions except the most discriminative one do not impact the score as you just perform a max. This is experimentally verified on ILSVRC dataset that GMP achieves similar classification performance as GAP, but GAP outperforms GMP for localization.</p><p><strong>Guided Backpropagation:</strong></p><p>Guided Backpropagation combines vanilla backpropagation at ReLUs (leveraging which elements are positive in the preceding feature map) with DeconvNets (keeping only positive error signals). We are only interested in what image features the neuron detects. So when propagating the gradient, we set all the negative gradients to 0. We don’t care if a pixel “suppresses’’ (negative value) a neuron somewhere along the part to our neuron. Value in the filter map greater than zero signifies the pixel importance which is overlapped with the input image to show which pixel from the input image contributed the most.</p><p><strong>Given below is the example of how guided backpropagation works:</strong></p><p>Relu Forward Pass ( Flow the value which are greater than 0):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/0*yBqno-BGNARrgQ8-" /></figure><p>Relu Backward Pass ( flow the value as it is where value is greater than zero in the filter (h_l) during forward propagation.)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/737/0*UHMN2lGscccqr6tL" /></figure><p>Deconvolution for Relu: Flow the values backward as it is where value in the filter is greater than 0.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/753/0*STbOsROvbJGCHy2_" /></figure><p>Guided Backpropagation: Taking the intersection of the concept of Backward pass and the deconvolution.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/702/0*K8kRB_m2meWQV4C1" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/358/0*P7Ky-cE6uMv_yuGW" /></figure><h3><strong>Class Activation Mapping (CAM )</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/808/0*_99JUuwM4tZYnJG8" /><figcaption>Class Activation Mapping</figcaption></figure><p><em>Class Activation Mapping: the predicted class score is mapped back to the previous convolutional layer to generate the class activation maps (CAMs). The CAM highlights the class-specific discriminative regions.</em></p><p>Modifying the base network to remove all fully-connected layers at the end, and including a tensor product (followed by softmax), which takes as input the Global-Average-Pooled convolutional feature maps, and outputs the probability for each class. We can identify the importance of the image regions by projecting back the weights of the output layer on the convolutional feature maps obtained from the last Convolution Layer. This technique is known as Class Activation Mapping. A class activation map for a particular category indicates the discriminative image regions used by CNN to identify that category<strong><em>. The dot product of the extracted weights from the final layer and the feature map is calculated to produce the class activation map. The class activation map is upsampled by using Bi-Linear Interpolation and superimposed on the input image to show the regions which the CNN model is looking at.</em></strong></p><p>This approach of modifying image classification CNN architectures replacing fully-connected layers with convolutional layers and global</p><p>average pooling, thus achieving class-specific feature maps. A drawback of CAM is that it requires feature maps to directly precede softmax layers, so it is only applicable to a particular kind of CNN architectures performing global average pooling over convolutional maps immediately prior to prediction (i.e. conv feature maps → global average pooling → softmax layer).</p><p>Here comes Grad CAM to solve the problem where we combine the feature maps using the gradient signal that does not require any modification in the network architecture. This approach can be applied to off-the-shelf CNN-based architectures, including those for image captioning and visual question answering. For a fully-convolutional architecture, CAM is a special case of Grad-CAM</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Y6zisgTBMPX3v3MI" /></figure><h3><strong>Grad CAM — Gradient-weighted Class Activation Mapping</strong></h3><p>Grad-CAM makes CNN-based models more transparent by visualizing input regions with high-resolution details that are important for predictions. Grad-Cam, unlike CAM, uses the gradient information flowing into the last convolutional layer of the CNN to understand each neuron for a decision of interest. Grad-CAM works by finding the final convolutional layer in the network and then examining the gradient information flowing into that layer. In order to obtain Grad-CAM of width u and height v for any class c, we first compute the gradient of the score for class c, yc (before the softmax), with respect to feature map activations Ak of a convolutional layer, i.e. ∂yc / ∂Ak . These gradients flowing back are <em>global-average-pooled</em> over the width and height dimensions (indexed by i and j respectively) to obtain the neuron importance weights <em>α c k</em> :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/383/0*AqNq7wfWJNp28Sys" /></figure><p>Where yc:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/288/0*bgynKndBcJu7Gypz" /></figure><p>weight <em>α c k </em>represents a partial linearization of the deep network downstream from A, and captures the ‘importance’ of feature map k for a target class c. We perform a weighted combination of forward activation maps, and follow it by a ReLU to obtain,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/341/0*QpyA-Zp1JyK_oejR" /></figure><p>This results in a coarse heatmap of the same size as the convolutional feature maps (14 × 14 in the case of last convolutional layers of VGG. We apply a ReLU to the linear combination of maps because we are only interested in the features that have a positive influence on the class of interest, i.e. pixels whose intensity should be increased in order to increase yc. Negative pixels are likely to belong to other categories in the image. But you might see from the below output that Grad-CAM does not tell you precise or fine-grained information of the pixel activation. Hence, guided Grade — CAM was introduced.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/688/0*KdScbxoVI0aers8w" /></figure><h3><strong>Guided Grad-CAM</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Ymoztp-krf0wWPE1" /></figure><p>While Grad-CAM is class-discriminative and localizes relevant image regions, it lacks the ability to highlight fine-grained details like pixel-space gradient visualization methods. Guided Backpropagation visualizes gradients with respect to the image where negative gradients are suppressed when backpropagating through ReLU layers. Intuitively, this aims to capture pixels detected by neurons, not the ones that suppress neurons. Grad-CAM can easily localize the cat; however, it is unclear from the coarse heatmap why the network predicts this particular instance as ‘tiger cat’. In order to combine the best aspects of both, we fuse Guided Backpropagation and Grad-CAM visualizations via element-wise multiplication (L c Grad-CAM is first upsampled to the input image resolution using bilinear interpolation). This visualization is both high-resolution (when the class of interest is ‘tiger cat’, it identifies important ‘tiger cat’ features like stripes, pointy ears and eyes) and class-discriminative (it highlights the ‘tiger cat’ but not the ‘boxer (dog)’)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/688/0*gxahI9qf0o64Nf3W" /></figure><p>If you wish to try it by yourself, please find the code attached on the following link: <a href="https://github.com/chinesh/Guided_Grad_CAM">https://github.com/chinesh/Guided_Grad_CAM</a></p><p>References:</p><p><a href="https://arxiv.org/pdf/1610.02391.pdf">https://arxiv.org/pdf/1610.02391.pdf</a></p><p><a href="http://cnnlocalization.csail.mit.edu/Zhou_Learning_Deep_Features_CVPR_2016_paper.pdf">http://cnnlocalization.csail.mit.edu/Zhou_Learning_Deep_Features_CVPR_2016_paper.pdf</a></p><p><a href="https://arxiv.org/pdf/1802.10171.pdf">https://arxiv.org/pdf/1802.10171.pdf</a></p><p><a href="https://glassboxmedicine.com/2019/10/06/cnn-heat-maps-gradients-vs-deconvnets-vs-guided-backpropagation/#:~:text=are%20deconvolutional%20layers%3F-,Guided%20Backpropagation,%2Fpytorch%2Dcnn%2Dvisualizations)%3A">https://glassboxmedicine.com/2019/10/06/cnn-heat-maps-gradients-vs-deconvnets-vs-guided-backpropagation</a>/</p><p><a href="https://alexisbcook.github.io/2017/global-average-pooling-layers-for-object-localization/">https://alexisbcook.github.io/2017/global-average-pooling-layers-for-object-localization/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a7d1b0178709" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Custom Activation Function in Tensorflow for Deep Neural Networks from scratch, tutorial.]]></title>
            <link>https://medium.com/@chinesh4/custom-activation-function-in-tensorflow-for-deep-neural-networks-from-scratch-tutorial-b12e00652e24?source=rss-c4f2657b8262------2</link>
            <guid isPermaLink="false">https://medium.com/p/b12e00652e24</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[tensorflow]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Chinesh Doshi]]></dc:creator>
            <pubDate>Mon, 15 Jul 2019 18:18:53 GMT</pubDate>
            <atom:updated>2019-07-15T18:18:53.618Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/570/0*r1iUc2W15txWzqTP" /><figcaption>Neural Network</figcaption></figure><p><strong><em>Activation</em></strong><em> functions</em> are really important for an Artificial Neural Network to learn and make sense of something really complicated and Non-linear complex functional mappings between the inputs and response variable. <em>It</em> <em>introduces non-linear properties to our Network</em>. <strong><em>Their main purpose is to convert an input signal of a node in an A-NN to an output signal.</em></strong> That output signal now is used as an input in the next layer in the stack.</p><p>We can choose from a huge list of popular activation functions from popular Deep Learning frameworks, like <a href="https://en.wikipedia.org/wiki/Rectifier_(neural_networks)">ReLU</a>, <a href="https://en.wikipedia.org/wiki/Sigmoid_function">Sigmoid</a>, <a href="https://en.wikipedia.org/wiki/Hyperbolic_function">Tanh</a>, and many others. However, to create a state of the art model, customized particularly for your task, you may need to use a custom activation function, which is absent in Deep Learning framework you are using.</p><p>Tensorflow is a great tool that works with deep learning by making the implementation abstract and there are a lot of operations that you can easily implement to work for too complex problems such as in Computer Vision, Natural Language Processing, speech recognition, machine translation, etc. In this post, I am introducing a combination of Relu 6 and Leaky Relu activation function, which is not available as a pre-implemented function in TensorFlow Library. The activation function is defined as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/597/1*0l55p-qvr1wUmO2Rwvb-_w.jpeg" /><figcaption>Custom Activation Function</figcaption></figure><p>The power of TensorFlow and Keras is that, though it has a tendency to calculate the differentiation of the function, but what if you have an activation function which changes over the range of input. That is where this implementation comes into the picture. Let us begin implementing it in our code.</p><p>Import the required Libraries.</p><pre>from __future__ import absolute_import, division, print_function<br>import sys<br>import tensorflow as tf<br>import numpy as np<br>from tensorflow.python.framework import ops</pre><pre>tf.logging.set_verbosity(tf.logging.INFO)</pre><p>Define the activation function as a normal python function.</p><pre>def leaky_relu_6(x):</pre><pre>if x &gt;=0.0 and x &lt;= 6.0:<br>        return x*1.0<br>    elif x &gt; 6.0:<br>        return 6.0<br>    else:<br>        return 0.2 * x</pre><p>Vectorize the python function.</p><pre>np_leaky_relu_6= np.vectorize(leaky_relu_6)</pre><p>Why Vectorizing the function?</p><p>A vectorized function takes a nested sequence of objects or numpy arrays as inputs and returns a single or tuple of numpy array as output. Vectorizing a function helps us to deal with any input shape and size for generalizing our implementation. The data type of the output of <em>vectorized</em> is determined by calling the function with the first element of the input.</p><p>Now let us define the derivative of our activation function so that it can be used during backpropagation.</p><pre>def d_leaky_relu_6(x):<br>    if x &gt;=0.0 and x &lt; 6.0:<br>        return 1.0<br>    elif x &gt; 6.0:<br>        return 0.0<br>    else:<br>        return 0.2<br>np_d_leaky_relu_6 = np.vectorize(d_leaky_relu_6)</pre><p>Gradient Function: A gradient is a <strong>vector</strong>. Its components consist of the partial derivatives of a function and it points in the direction of the greatest rate of increase of the function. So for example, if you have the function f(x1,…xn), its gradient would consist of n partial derivatives and would represent the vector field.</p><pre>def relu_grad(op, grad):<br>    x = op.inputs[0]</pre><pre>n_gr = tf_d_leaky_relu_6(x)    #defining the gradient.<br>    return grad * n_gr</pre><p><strong>op</strong> contains all passed variables; grad — the flown gradient from the backpropagation.</p><pre>def py_func(func, inp, Tout, stateful=True, name=None, grad=None):</pre><pre># Need to generate a unique name to avoid duplicates:<br>    rnd_name = &#39;PyFuncGrad&#39; + str(np.random.randint(0, 1E+2))</pre><pre>tf.RegisterGradient(rnd_name)(grad)<br>    g = tf.get_default_graph()<br>    with g.gradient_override_map({&quot;PyFunc&quot;: rnd_name}):<br>        return tf.py_func(func, inp, Tout, stateful=stateful, name=name)</pre><p><a href="https://www.tensorflow.org/api_docs/python/tf/RegisterGradient"><strong>tf.RegisterGradient</strong></a> allows you to override the gradient.</p><p>gradient_override_map(<strong>op_type_map</strong>) where <strong>op_type_map is </strong>a dictionary ({“PyFunc”: rnd_name}) mapping op type strings to alternative op type strings.</p><p>Gradient_override_map, context manager is used to mapping the “rnd_name” function (PyFuncGrad in our case) to the gradient of the “PyFunc” function within the scope of the context.</p><p>The tf.py_func() operator enables you to run arbitrary Python code in the middle of a TensorFlow graph. It is particularly convenient for wrapping custom NumPy operators for which no equivalent TensorFlow operator (yet) exists. Adding tf.py_func() is an alternative to using sess.run() calls inside the graph. Find more about tf.py_func <a href="https://www.tensorflow.org/api_docs/python/tf/py_func">here</a>.</p><p><strong>Combining it all together:</strong> Now that we have all the pieces, we can combine them all together:</p><pre>np_leaky_relu_6_32 = lambda x: np_leaky_relu_6(x).astype(np.float32)</pre><pre>def tf_leaky_relu_6(x,name=None):<br>    with tf.name_scope(name, &quot;leaky_relu_6&quot;, [x]) as name:</pre><pre>y = py_func(np_leaky_relu_6_32,   #forward pass function<br>                        [x],<br>                        [tf.float32],<br>                        name=name,<br>                         grad= relu_grad) #the function that overrides gradient<br>        y[0].set_shape(x.get_shape())     #when using with the code, it is used to specify the rank of the input.<br>        return y[0]</pre><pre>np_d_leaky_relu_6_32 = lambda x: np_d_leaky_relu_6(x).astype(np.float32)</pre><pre>def tf_d_leaky_relu_6(x,name=None):<br>    with tf.name_scope(name, &quot;d_leaky_relu_6&quot;, [x]) as name:<br>        y = tf.py_func(np_d_leaky_relu_6_32,<br>                        [x],<br>                        [tf.float32],<br>                        name=name,<br>                        stateful=False)<br>        return y[0]</pre><p>CNN Model:</p><pre>def cnn_model_fn(features, labels, mode):<br>  &quot;&quot;&quot;Model function for CNN.&quot;&quot;&quot;<br>  # Input Layer<br>  input_layer = tf.reshape(features[&quot;x&quot;], [-1, 28, 28, 1])<br>  #conv1_act = tf.placeholder(tf.float32, [4,])</pre><pre># Convolutional Layer #1<br>  conv1 = tf.layers.conv2d(<br>      inputs=input_layer,<br>      filters=32,<br>      kernel_size=[5, 5],<br>      padding=&quot;same&quot;,<br>      activation=tf.nn.relu)</pre><pre>#conv1_act = tf_leaky_relu_6(conv1)</pre><pre>pool1 = tf.layers.max_pooling2d(inputs=conv1, pool_size=[2, 2], strides=2)</pre><pre># Convolutional Layer #2 and Pooling Layer #2<br>  conv2 = tf.layers.conv2d(<br>      inputs=pool1,<br>      filters=64,<br>      kernel_size=[5, 5],<br>      padding=&quot;same&quot;) #to try other activation function replcae relu6 by leaky_relu or relu<br>    <br>  conv2_act = tf_leaky_relu_6(conv2)<br>  <br>  pool2 = tf.layers.max_pooling2d(inputs=conv2_act, pool_size=[2, 2], strides=2)</pre><pre># Dense Layer<br>  pool2_flat = tf.reshape(pool2, [-1, 7 * 7 * 64])<br>  dense = tf.layers.dense(inputs=pool2_flat, units=1024, activation=tf.nn.relu6) #to try other activation function replcae relu6 by leaky_relu or relu<br>  dropout = tf.layers.dropout(<br>      inputs=dense, rate=0.4, training=mode == tf.estimator.ModeKeys.TRAIN)</pre><pre># Logits Layer<br>  logits = tf.layers.dense(inputs=dropout, units=10)</pre><pre>predictions = {<br>      # Generate predictions (for PREDICT and EVAL mode)<br>      &quot;classes&quot;: tf.argmax(input=logits, axis=1),<br>      # Add `softmax_tensor` to the graph. It is used for PREDICT and by the<br>      # `logging_hook`.<br>      &quot;probabilities&quot;: tf.nn.softmax(logits, name=&quot;softmax_tensor&quot;)<br>  }</pre><pre>if mode == tf.estimator.ModeKeys.PREDICT:<br>    return tf.estimator.EstimatorSpec(mode=mode, predictions=predictions)</pre><pre># Calculate Loss (for both TRAIN and EVAL modes)<br>  loss = tf.losses.sparse_softmax_cross_entropy(labels=labels, logits=logits)</pre><pre># Configure the Training Op (for TRAIN mode)<br>  if mode == tf.estimator.ModeKeys.TRAIN:<br>    optimizer = tf.train.GradientDescentOptimizer(learning_rate=0.001)<br>    train_op = optimizer.minimize(<br>        loss=loss,<br>        global_step=tf.train.get_global_step())<br>    return tf.estimator.EstimatorSpec(mode=mode, loss=loss, train_op=train_op)</pre><pre># Add evaluation metrics (for EVAL mode)<br>  eval_metric_ops = {<br>      &quot;accuracy&quot;: tf.metrics.accuracy(<br>          labels=labels, predictions=predictions[&quot;classes&quot;])<br>  }<br>  return tf.estimator.EstimatorSpec(<br>      mode=mode, loss=loss, eval_metric_ops=eval_metric_ops)</pre><p>Load training and eval data</p><pre>((train_data, train_labels),<br> (eval_data, eval_labels)) = tf.keras.datasets.mnist.load_data()</pre><pre>train_data = train_data/np.float32(255)<br>train_labels = train_labels.astype(np.int32)  # not required</pre><pre>eval_data = eval_data/np.float32(255)<br>eval_labels = eval_labels.astype(np.int32)  # not required</pre><p># Create the Estimator</p><pre>mnist_classifier = tf.estimator.Estimator(<br>    model_fn=cnn_model_fn)</pre><pre># Set up logging for predictions<br>tensors_to_log = {&quot;probabilities&quot;: &quot;softmax_tensor&quot;}</pre><pre>logging_hook = tf.train.LoggingTensorHook(<br>    tensors=tensors_to_log, every_n_iter=50)</pre><pre># Train the model<br>train_input_fn = tf.estimator.inputs.numpy_input_fn(<br>    x={&quot;x&quot;: train_data},<br>    y=train_labels,<br>    batch_size=100,<br>    num_epochs=None,<br>    shuffle=True)<br># train one step and display the probabilties<br>#mnist_classifier.train(input_fn=train_input_fn, steps=1,hooks=[logging_hook])</pre><pre>mnist_classifier.train(input_fn=train_input_fn, steps=1000)</pre><pre>#Evaluate Input</pre><pre>eval_input_fn = tf.estimator.inputs.numpy_input_fn(<br>    x={&quot;x&quot;: eval_data},<br>    y=eval_labels,<br>    num_epochs=1,<br>    shuffle=False)</pre><pre>eval_results = mnist_classifier.evaluate(input_fn=eval_input_fn)<br>print(eval_results)</pre><p>The proposed activation function may not be better than existed versions of Relu. Try running the code on your system. Do post your questions and suggestions in the comments. I would be happy to answer your questions. All code from this tutorial is available on <a href="https://github.com/chinesh/custom_activation_function">GitHub</a>. Other examples of implemented custom activation functions for <a href="https://pytorch.org/">PyTorch</a> and <a href="https://keras.io/">Keras</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b12e00652e24" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why Relu? Tips for using Relu. Comparison between Relu, Leaky Relu, and Relu-6.]]></title>
            <link>https://medium.com/@chinesh4/why-relu-tips-for-using-relu-comparison-between-relu-leaky-relu-and-relu-6-969359e48310?source=rss-c4f2657b8262------2</link>
            <guid isPermaLink="false">https://medium.com/p/969359e48310</guid>
            <category><![CDATA[relu]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[activation-functions]]></category>
            <dc:creator><![CDATA[Chinesh Doshi]]></dc:creator>
            <pubDate>Sat, 29 Jun 2019 07:42:12 GMT</pubDate>
            <atom:updated>2019-06-29T07:42:12.579Z</atom:updated>
            <content:encoded><![CDATA[<p>A neural network without an activation function is essentially just a linear regression model. The activation function does the non-linear transformation to the input making it capable to learn and perform more complex tasks. Today we will be discussing the most commonly used activation function in the neural networks that is Relu. Relu stands for Rectified Linear Unit.</p><p>A(x) = max(0,x) , where x is the output of hidden layer.</p><p>The ReLu function is as shown above. It gives an output x if x is positive and 0 otherwise</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/311/1*Xul22nQfzRNFFeICETrZxA.jpeg" /><figcaption>Relu</figcaption></figure><p>At first look, this would look like having the same problems of the linear function, as it is linear in the positive axis. First of all, ReLu is nonlinear in nature. And combinations of ReLu are also non linear! ( in fact it is a good approximator. Any function can be approximated with combinations of ReLu). Great, so this means we can stack layers. It is not bound though. The range of ReLu is [0, inf). This means it can blow up the activation.</p><p>Another point that I would like to discuss here is the sparsity of the activation. Imagine a big neural network with a lot of neurons. Using a sigmoid or tanh will cause almost all neurons to fire in an analog way. That means almost all activations will be processed to describe the output of a network. In other words the activation is dense. This is costly. We would ideally want a few neurons in the network to not activate and thereby making the activations sparse and efficient.</p><p>A general problem with both the sigmoid and tanh functions is that they saturate. This means that large values snap to 1.0 and small values snap to -1 or 0 for tanh and sigmoid respectively. Further, the functions are only really sensitive to changes around their mid-point of their input, such as 0.5 for sigmoid and 0.0 for tanh.</p><p>The limited sensitivity and saturation of the function happen regardless of whether the summed activation from the node provided as input contains useful information or not. Once saturated, it becomes challenging for the learning algorithm to continue to adapt the weights to improve the performance of the model.</p><ul><li>(+) It was found to greatly accelerate (e.g. a factor of 6 in <a href="http://www.cs.toronto.edu/~fritz/absps/imagenet.pdf">Krizhevsky et al.</a>) the convergence of stochastic gradient descent compared to the sigmoid/tanh functions. It is argued that this is due to its linear, non-saturating form.</li><li>(+) Compared to tanh/sigmoid neurons that involve expensive operations (exponentials, etc.), the ReLU can be implemented by simply thresholding a matrix of activations at zero.</li></ul><p>Because of the horizontal line in ReLu( for negative X ), the gradient can go towards 0. For activations in that region of ReLu, the gradient will be 0 because of which the weights will not get adjusted during descent. That means those neurons which go into that state will stop responding to variations in error/ input ( simply because gradient is 0, nothing changes ). This is called dying ReLu problem. This problem can cause several neurons to just die and not respond making a substantial part of the network passive.</p><p>For example, you may find that as much as 40% of your network can be “dead” (i.e. neurons that never activate across the entire training dataset) if the learning rate is set too high. With a proper setting of the learning rate this is less frequently an issue.</p><p>There are variations in ReLu to mitigate this issue by simply making the horizontal line into the non-horizontal component . for example y = 0.01x for x&lt;0 will make it a slightly inclined line rather than horizontal line. This is leaky ReLu. The main idea is to let the gradient be non zero and recover during training eventually. <strong>Leaky ReLUs</strong> are one attempt to fix the “dying ReLU” problem. Instead of the function being zero when x &lt; 0, a leaky ReLU will instead have a small negative slope (of 0.01, or so). That is, the function computes <em>f</em>(<em>x</em>)=𝟙(<em>x</em>&lt;0)(<em>αx</em>)+𝟙(<em>x</em>&gt;=0)(<em>x</em>) where α is a small constant. Some people report success with this form of activation function, but the results are not always consistent. The slope in the negative region can also be made into a parameter of each neuron, as seen in PReLU neurons, introduced in <a href="http://arxiv.org/abs/1502.01852">Delving Deep into Rectifiers</a>, by Kaiming He et al., 2015. However, the consistency of the benefit across tasks is presently unclear.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*wi7cGWx0TWIoUsmCXzBlxw.png" /><figcaption>Leaky — Relu</figcaption></figure><p><strong>Relu6</strong></p><p>From the paper: <a href="http://www.cs.utoronto.ca/~kriz/conv-cifar10-aug2010.pdf">cs.utoronto.ca/~kriz/conv-cifar10-aug2010.pdf</a></p><p>The range of ReLu is [0, inf). This means it can blow up the activation, which also is not a favorable condition for a network. If you multiply a bunch of terms which are greater than 1, they will tend towards infinity, hence exploding gradient as you get further from the output layer if you have activation functions which have a slope &gt; 1.</p><p>One solution to this problem is by clipping the gradient at some value n.</p><p>From the paper: <a href="http://www.cs.utoronto.ca/~kriz/conv-cifar10-aug2010.pdf">cs.utoronto.ca/~kriz/conv-cifar10-aug2010.pdf</a>, First, we cap the units at 6, so our ReLU activation function is y = min(max(x, 0), 6). In our tests, this encourages the model to learn sparse features earlier. In the formulation, this is equivalent to imagining that each ReLU unit consists of only 6 replicated bias-shifted Bernoulli units, rather than an infinite amount. We will refer to ReLU units capped at n as ReLU-n units. Also Relu-6 does not have any activation value for x &lt; 0. I am still wondering what if we combine Leaky-Relu and Relu-6 to overcome all the disadvantages noted till date.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*C7KdIsgmKq2EHFBHG1irtw.png" /><figcaption>RELU — 6</figcaption></figure><p>I have implemented the image classification for MNIST dataset using the different type of Relu activation function. What I found is that the best accuracy is given by the Relu6 activation function .</p><h3>Tips for Using the Rectified Linear Activation</h3><p>In this section, we’ll take a look at some tips when using the rectified linear activation function in your own deep learning neural networks.</p><h3>Use ReLU as the Default Activation Function</h3><p>For a long time, the default activation to use was the sigmoid activation function. Later, it was the tanh activation function.</p><p>For modern deep learning neural networks, the default activation function is the rectified linear activation function. Most papers that achieve state-of-the-art results will describe a network using ReLU.<em> </em>If in doubt, start with ReLU in your neural network, then perhaps try other piecewise linear.</p><p><strong>Use ReLU with MLPs, CNNs, but Probably Not RNNs</strong></p><p>The ReLU can be used with most types of neural networks. It is recommended as the default for both Multilayer Perceptron (MLP) and Convolutional Neural Networks (CNNs).Traditionally, LSTMs use the tanh activation function for the activation of the cell state and the sigmoid activation function for the node output. Given their careful design, ReLU were thought to not be appropriate for Recurrent Neural Networks (RNNs) such as the Long Short-Term Memory Network (LSTM) by default.</p><p><em>At first sight, ReLUs seem inappropriate for RNNs because they can have very large outputs so they might be expected to be far more likely to explode than units that have bounded values.</em> Nevertheless, there has been some work on investigating the use of ReLU as the output activation in LSTMs, the result of which is a careful initialization of network weights to ensure that the network is stable prior to training.</p><p>Code: <a href="https://github.com/chinesh/mnist-relu-classificatin">https://github.com/chinesh/mnist-relu-classificatin</a></p><p>Training accuracy on MNIST dataset (Please find the network in the GitHub Link):</p><p>Relu : 67.59%</p><p>Leaky Relu: 72.10%</p><p>Relu6: 71.55%</p><p>Though after trying Relu, Leaky Relu and Relu6 as the activation function, Leaky Relu gave the best accuracy, I am still skeptical why the standard/ benchmark networks such as DGN (Deep Graph Network), RetinaNet, VGG-16, Resnet use Relu in place of Leaky-Relu. Try running the code on your system. Do post your questions and suggestions in the comments. I would be happy to answer your questions.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=969359e48310" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>