<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Praditya Raudi on Medium]]></title>
        <description><![CDATA[Stories by Praditya Raudi on Medium]]></description>
        <link>https://medium.com/@raudipra?source=rss-4e64b52d5bb6------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*-675t8loaHY5ataEusvZrA.jpeg</url>
            <title>Stories by Praditya Raudi on Medium</title>
            <link>https://medium.com/@raudipra?source=rss-4e64b52d5bb6------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 04:40:54 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@raudipra/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Vulnerability of Deep Learning on Covariate Shift of Data by Praditya Raudi and Rifqi Luthfan]]></title>
            <link>https://medium.com/@raudipra/vulnerability-of-deep-learning-on-covariate-shift-of-data-by-praditya-raudi-and-rifqi-luthfan-bb1726578adf?source=rss-4e64b52d5bb6------2</link>
            <guid isPermaLink="false">https://medium.com/p/bb1726578adf</guid>
            <category><![CDATA[generative-model]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[adversarial-attack]]></category>
            <dc:creator><![CDATA[Praditya Raudi]]></dc:creator>
            <pubDate>Fri, 12 Nov 2021 06:06:33 GMT</pubDate>
            <atom:updated>2021-11-12T23:43:43.353Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>Current state-of-the-art neural networks are capable of achieving high performance in accuracy, making them the first choice for complicated applications. Ongoing research and development are continuously improving performance for different domains and this field is now more popular than ever. One of the drawbacks that it has, however, is robustness issues, which arise due to the missing guarantee that such performance would generalize well on permutations and unseen data. Robustness is as crucially important as performance, especially in product development, since neural networks are applied to more and more safety-critical systems (such as manufacturing automation to autonomous driving), and this property must, therefore, be discussed.</p><h3>Intriguing Properties of Neural Networks</h3><p>This paper was submitted in 2013 and revised in 2014, and until the time of making this post, it has been cited 8,000+ times. This is a very interesting paper, aiming at providing evidence that neural networks acting in ways that contradict notions of generalization, challenging the prior assumptions that people had about the <em>uninterpretable</em> neural networks. In summary, adversarial attacks do exist in neural networks, challenging the assumption that classifiers are robust, and semantic information is contained in the space of activation functions, not in the space of individual units as previously assumed.</p><h4>Existence of Adversarial Examples</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/685/0*5AOxjRNQ1wqB9Ahh" /><figcaption>Image from C. Szegedy et al., Intriguing properties of neural networks, 2014.</figcaption></figure><p>Let’s use the image above as an example. The left image is correctly classified as a truck by the neural network (AlexNet). Adding specific changes to the image (using L-BFGS formula which will be discussed later in the post) represented by the middle image, the left image becomes the right image, which for our eyes looks the same but labelled by the classifier as “ostrich”. Imagine if we have a self-driving car that can be fooled, then misclassifying a stop sign as a speed limit sign, it becomes a serious matter and as long as this is the case, we don’t want them out on our streets.</p><h4>L-BFGS</h4><p>This kind of image is called an adversarial example. For the case of image above, it is generated using Limited-memory Broyden–Fletcher–Goldfarb–Shanno or L-BFGS algorithm, which approximated box-constrained optimization problem below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*y9WNJMWss7kuKpKe" /></figure><p>This algorithm is white-box, and still slow, but it is enough to show how neural networks can be fooled. Even the author can manage to find an average minimum distortion <em>r</em> value that can have a fooling rate of 100% on MNIST for several network architectures.</p><h4>Analysis of Adversarial Examples</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_ZHVynzNsUJO3b1x.png" /><figcaption>Table 1 from C. Szegedy et al., Intriguing properties of neural networks, 2014.</figcaption></figure><p>Then the authors proceed to change the regularization lambda parameter. We can see from the results that the larger lambda the larger perturbation needed (av. min. distortion), making attack more difficult. However, it impacts the performance, as the accuracy is lower (test set error increases) and the trade-off is still not good enough, as the distortion is still essentially imperceptible for our eyes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*tqLm5Rg_wc2F3TDw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*diG8cvYbT2Fb_g6O.png" /><figcaption>Table 2–4 from C. Szegedy et al., Intriguing properties of neural networks, 2014.</figcaption></figure><p>What is more worrying is that adversarial examples might transfer to other models, even without training overlap. From table 2 in the paper (which can be considered as a confusion matrix), we can see that adversarial examples might transfer to other models (diagonal represents the same model, hence 100%). Moving from left to the right of the table the models are becoming more complex, same thing for up to bottom row. Here we can also see that model complexity affects transfer rate of adversarial examples, where less complex models have lower transfer rate to more complex models and vice versa. In table 2, the experiment is done using the same subset of MNIST. The authors then proceed to do experiments on different subsets of MNIST, where the results are captured in table 3–4 in the paper. The results on table 3 shows that adversarial examples also might transfer to other models with no training dataset overlap, but with lower transfer rate. When adversarial samples ratio is amplified, the fooling rate increases as shown in table 4.</p><h4>Unit Level Inspection</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*f9bSBxo2o0ARTOTc.png" /><figcaption>Image from C. Szegedy et al., Intriguing properties of neural networks, 2014.</figcaption></figure><p>These attacks are possible due to semantic information that is used for prediction is spread among units, as opposed to individual units (previous assumption). The authors prove this by collecting the image that each unit likes, using the natural basis and random basis (combination of units). We can see that both left image (individual units) and right image (random direction, combination of units) gave similarly interpretable semantic properties. Semantic information is likely spread among units, as individual units are not complete in data representation. Since the neural network is not locally stable (the author proved this using Lipschitz constant), the network can then misclassify when given small perturbations.</p><h3>Explaining and Harnessing Adversarial Examples</h3><p>The expectation of how deep neural network could generalize and being robust over unseen dataset was refuted by adversarial example case above. Previous paper suggested it is due to extreme nonlinearity of deep neural networks, combined with insufficient regularization of the supervised learning problem. One year later, the exact same authors suggested a new explanation which contradicted the existed one.</p><h4>Perturbation-based Adversarial Examples on Linear Model</h4><p>How does small perturbation affect linear model? Consider a linear model y = wx, let η be the perturbation, m be the average magnitude of w, ε be a random very small value and n be the size of x dimension.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1012/1*bcxtZu8N9wgrHNfXaDtpQg.png" /></figure><p>From the equation above we can see that even though we limit the magnitude of perturbation to be under ε, as long as we have sufficient dimension m, the impact would be sufficient to affect our model. Therefore, perturbation based adversarial example works well on image dataset, given the big size of input dimension compared to any structured dataset.</p><h4>Adversarial Examples on Deep Neural Network Model</h4><p>We know that this perturbation can affect linear model, but why it can also fool the super non linear deep neural network model? Do you really think deep neural network is non linear?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*56AlnrvkCvKxXQIqfhUuYQ.png" /><figcaption>Source: <a href="https://towardsdatascience.com/activation-functions-neural-networks-1cbd9f8d91d6">https://towardsdatascience.com/activation-functions-neural-networks-1cbd9f8d91d6</a></figcaption></figure><p>If you look at the figure above, ReLu is basically an “if else” linear function, meanwhile Sigmoid, although it is clearly non linear, however, in practice you want your model to learn the linear part of it to avoid vanishing and exploding gradient problem. From this cue we can say that turns out the so called non linear deep neural network is somewhat linear. Finally, it explains why perturbation attack works on deep neural network.</p><p>Intriguing transferability properties of adversarial examples on deep neural network:</p><ul><li>Adversarial examples generated from one model are often misclassified by other models trained on even different architectures and training set.</li><li>Misclassification by these models often agree with each other on a certain class.</li></ul><p>But how come? A hypothesis is conveyed by Szegedy, Goodfellow, et al [2]. Due to the linearity and generalization property of deep neural network, the model tends to learn the same “weights” when trained on even different training set. This consistency encourages the stability of adversarial examples.</p><h4>Fast Gradient Sign Method</h4><p>By utilizing the knowledge above the authors introduced a new method to generate adversarial examples, it is called fast gradient sign method or FGSM. The idea is simple, we calculate the gradient of loss function over x, meaning we are looking for the direction where loss function is sensitive to the changes in the pixel input. Then we multiply it by ε.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/588/1*IhM0MuAQxJ1wcXiYLloV_g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YQBOONc_n7-LpIFTz3S32A.png" /></figure><p>In the previous paper, the authors mentioned that conducting adversarial training by generating adversarial examples per training iteration is computationally impractical. However using FGSM above, we could easily incorporate the adversarial examples by modifying the loss function.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GPmm7jLQDwXFwQffo3ZpeA.png" /></figure><p>As you can see above, the loss function for adversarial training comprises of two parts, the loss function for normal images, and perturbed images. This adversarial training was able to reduce the error rate of generated adversarial examples on MNIST dataset from 89.4% to 17.9%.</p><h3>DeepSafe: A Data-driven Approach for Checking Adversarial Robustness in Neural Networks</h3><p>So after knowing that it’s possible to trick neural networks, how should we defend against this? Preventing these kinds of attacks is still an on-going area of research, and no one is entirely sure yet. It is worth thinking about scenarios where neural networks applications are used so that we can at least lessen the risk that this kind of attack would cause damage to our business. Because we still cannot guarantee that a network is robust against all kinds of attacks, we can try to give guarantees that it is safe against specific targeted attacks and the approach in this next paper is trying to do just that.</p><p>In summary, DeepSafe approach tries to automatically identify targeted safe regions of the input space, within which the network is robust against adversarial perturbations. This approach is data-guided, relying solely on a clustering algorithm to identify well-defined geometric regions as candidates of safe regions. The clustering approach is black-box in the sense that it relies solely on the data distribution to determine the safe regions (and not on the network parameters), making it applicable to a wide range of networks with various topologies and architectures. Then, the resulting clusters are analyzed and verified in order to group which regions are completely safe regions, targeted safe regions, or robustness level unknown.</p><h4>Data-guided Clustering</h4><p>An extension of the k-means algorithm used to cluster training inputs. We use both the similarity of the data-points (similarity measure is domain specific, can be L1 or L2 norm depending on dataset) <em>and their labels</em> for doing the clustering. This algorithm then outputs a set of dense clusters within the input space that are close to each other and are known to share the same label. <em>Such clusters are assumed to be safe regions</em>, in which all inputs should be labeled consistently. The main hypothesis is that for a given cluster <em>C</em>, with centroid <em>cen</em> and radius <em>r</em>, any input <em>x</em> within distance <em>r</em>from <em>cen</em> should have the same true label <em>l</em> as that of the cluster.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*YwEi2A69mXAtdW3W" /></figure><h4>Cluster Analysis</h4><p>The resulting clusters are then analyzed and verified. The analyzed properties are density and centroid behavior of the cluster. For density, we should disregard clusters with low density in order to increase the chances of examining only valid safe regions, and hence of detecting only valid adversarial perturbations. The main hypothesis is assumed to hold in clusters with high density, since it contains a large number of instances at close proximity to each other, but not in clusters with low density, where fewer inputs belong to the same label and are spread out. We then check the centroid behavior, since the centroid of a cluster is considered as a representative for the behavior of the network over that cluster. We will use the neural network’s assigned score for each possible label in each cluster centroid to look for <em>targeted safe regions</em> (to determine targeted robustness).</p><h4>Cluster Verification</h4><p>The filtered clusters are assumed to be <em>completely safe</em> at first. Then we need to verify if these clusters/regions are actually completely safe or only safe against specific types of attack. For this verification purposes, <em>Reluplex tool</em>is used on a per label basis to verify a formula representing the negation of the main hypothesis.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*6zpHPQQdjpFjFdNX" /></figure><p>The labelling should still be consistent for at least one type of attack. We will only consider the safe regions that pass this verification process. If the region/cluster is safe against the attacks generated by the Reluplex tool, it will be considered as <em>completely safe</em>. On the other hand, if it is at least safe against one type of attack generated, the cluster/region will be considered as <em>targeted safe</em>. Other clusters will then be deemed as not robust.</p><h3>Experiment</h3><p>Preliminary experiments using ACAS Xu and MNIST dataset highlight the potential of the DeepSafe in providing formal guarantees about the targeted robustness of neural networks. DeepSafe successfully identified 80 clusters for MNIST, with 7 regions completely safe and 63 regions targeted safe and 210 clusters for ACAS Xu, with 125 regions completely safe and 52 regions targeted safe. Using DeepSafe is scalable, since when restricted to a small sub-domain of the input space, these functions may present purely linear behavior. Additionally, it is applicable to a wide range of networks with various topologies and architectures since it only relies on data distribution.</p><h3>Constructing Unrestricted Adversarial Examples with Generative Models</h3><p>For the context, perturbation-based adversarial example is also called restricted adversarial example due to the restricted perturbation value of the method. This paper introduced a new unrestricted approach to generate adversarial examples that is not only visually more convincing or similar to the original image, but also able to fool the most robust defense mechanism to date.</p><h4>Unrestricted Adversarial Examples</h4><p>The main idea is to use AC-GAN, as the generative model, to generate convincing images that capable to fool the image classifier.</p><figure><img alt="Source: https://www.geeksforgeeks.org/understanding-auxiliary-classifier-gan/" src="https://cdn-images-1.medium.com/max/646/1*y9ZMlHrezkQDG6O2hkzF0w.png" /><figcaption>Source: <a href="https://www.geeksforgeeks.org/understanding-auxiliary-classifier-gan/">https://www.geeksforgeeks.org/understanding-auxiliary-classifier-gan/</a></figcaption></figure><p>Given generator g, discriminator c, and target model f. Find latent vector z that minimize L.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/748/1*xlfuQRvwm2Ci16ZM44cirw.png" /></figure><p>L0 was devised to encourage the model to generate image belongs to y source class that is misclassified by f as y target. L1 meant for ensuring the diversity of images generated from any random z. Lastly, L2 encouraged the discriminator to not be fooled by generator, hence any image generated from g would be convincing. For multi class classifier we could generate targeted or untargeted adversarial examples, the only difference would be on L0 i.e. y target can be anything class besides y source. Also it is optional to add small noise or perturbation after generating the images, yet it doesn’t significantly change the effectiveness.</p><p>What does it look like compared to perturbation-based adversarial example?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*metc3ZrT3qaR8wA7VIIVnQ.png" /></figure><p>Figure above shows the visual comparison of restricted adversarial examples (top) and unrestricted adversarial examples (bottom). Both samples have exactly same success attack rate of 86% over MNIST dataset. Statistically speaking, annotators of the images could correctly tell the restricted adversarial examples were indeed an artificially generated images by 92.9%, meanwhile for the unrestricted only 76.8%. Meaning the unrestricted adversarial examples is visually more convincing and similar to the actual dataset distribution.</p><h4>Unrestricted Adversarial Examples vs Model Robustness Defense Mechanism</h4><p>This new approach was tested against two best defense mechanism for adversarial examples, among others, certified defense and PGD adversarial training.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*o6EyTD1HKJJ_QhPQLoU0tw.png" /></figure><p>Table above shows the success rate of unrestricted adversarial examples attack on two certified defense based classifiers using 10 different set of datasets. The first and second classifiers are supposedly able to ensure the success rate of any restricted attack with ε = 0.1 to be always under 35% and 5.8%. However the unrestricted attack were able to achieve overall 86.6% and 88.8% respectively.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tE0UbBU6asKr0f4IFSVwzA.png" /></figure><p>Table above shows the success rate of unrestricted adversarial examples attack on three models, each adversarially trained on different dataset. These three adversarially trained models were able to reduce the success rate of PGD restricted attack into 10.4%, 59.9%, and 20.5%. Yet once again the unrestricted attack could achieve success rate higher than 84% for all those datasets.</p><h3>Key Takeaways</h3><ul><li>Analyzing adversarial methods on deep neural network not only exposes security vulnerability of the technique, but also open up the opportunity to deepen our understanding towards this blackbox predictor.</li><li>Both traditional perturbation-based attacks and the new unrestricted method exploit current classifiers’ vulnerability to covariate shift.</li><li>Various approaches have been studied for determining adversarial robustness, now in a more scalable way in operating not only on individual input points.</li><li>Other non linear transformation using conditional generative model over the input had successfully fooled the currently most robust deep neural network.</li></ul><h3>References:</h3><p><a href="https://arxiv.org/pdf/1312.6199.pdf">Intriguing Properties of Neural Networks</a></p><p><a href="https://arxiv.org/pdf/1412.6572.pdf">Explaining and Harnessing Adversarial Examples</a></p><p><a href="https://arxiv.org/pdf/1710.00486.pdf">DeepSafe: A Data-driven Approach for Checking Adversarial Robustness in Neural Networks</a></p><p><a href="https://arxiv.org/pdf/1805.07894.pdf">Constructing Unrestricted Adversarial Examples with Generative Models</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bb1726578adf" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>