<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[ML Summaries - Medium]]></title>
        <description><![CDATA[Summarize everything related to Machine Learning, mainly Research papers, Blogs and Videos. - Medium]]></description>
        <link>https://medium.com/ml-summaries?source=rss----fcfb4aadd7f---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>ML Summaries - Medium</title>
            <link>https://medium.com/ml-summaries?source=rss----fcfb4aadd7f---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 07:39:54 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/ml-summaries" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Diffusion model mini-summaries #1 — DreamBooth: Fine Tuning Text-to-Image Diffusion Models for…]]></title>
            <link>https://medium.com/ml-summaries/diffusion-model-mini-summaries-1-dreambooth-fine-tuning-text-to-image-diffusion-models-for-a5ed97aeeb30?source=rss----fcfb4aadd7f---4</link>
            <guid isPermaLink="false">https://medium.com/p/a5ed97aeeb30</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[diffusion-models]]></category>
            <category><![CDATA[diffusion]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[generative-art]]></category>
            <dc:creator><![CDATA[Gowthami Somepalli]]></dc:creator>
            <pubDate>Thu, 12 Jan 2023 16:04:43 GMT</pubDate>
            <atom:updated>2023-01-12T16:05:00.676Z</atom:updated>
            <content:encoded><![CDATA[<h3>Diffusion mini-summaries #1 — DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*RjQ3jO0A8Mq8JQNO" /></figure><p>DreamBooth: Assign a rare sequence of tokens as the subject’s identifier and fine-tune the diffusion model on the small set of images with the “subject”.</p><blockquote>Paper: <a href="https://arxiv.org/abs/2208.12242">https://arxiv.org/abs/2208.12242</a></blockquote><p>The authors use the Imagen model in this paper which uses T5-XXL language model to encode the text guidance to generate a small 64x64 image first and then use a super-resolution model to blow it up to 1024x1024.</p><p>The authors observed that fine-tuning all the modules (including the Super-resolution module) results in the best performance.</p><p>To prevent overfitting of the model to the small input dataset, the authors use prior-preservation loss, which is like “distillation from the original pretrained model”. In the following loss term, “c” is prompt with the subject’s identifier and c_pr is without it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*6qZHdx8Qwb78JR3C" /></figure><p>So the overall fine-tuning pipeline is as follows. The yellow module refers to the first loss term and the second yellow refers to the second loss term from the above tweet. Once the text2image part is finetuned on the input set of images, the authors then fine-tune the SR module.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*UzSj8AjZnQc8a5aZ" /></figure><p>The model is quite versatile and can do lots of things, like content change wrt the background or the object itself, style change, and so on.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*l6F3rMl6LO65Dmae" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*q7E7gdybMwAx9n85" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*U64F36avhTUIFE-z" /></figure><p>Prior preservation loss is important, otherwise, the model will output only the “subject” for a given noun.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*5SgrTUv5-TxQvPSq" /></figure><p><strong>Failure cases: </strong>Might not work on rare categories, might overfit and output the training data for some instances, and appearance might change based on certain text prompts.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_wDqxmqxF6FIOYFy" /></figure><p><strong>Some useful resources</strong>: Finetuning your own dreambooth model using huggingface — <a href="https://huggingface.co/docs/diffusers/training/dreambooth">https://huggingface.co/docs/diffusers/training/dreambooth</a> Unofficial implementation for SD — <a href="https://github.com/XavierXiao/Dreambooth-Stable-Diffusion">https://github.com/XavierXiao/Dreambooth-Stable-Diffusion</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a5ed97aeeb30" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ml-summaries/diffusion-model-mini-summaries-1-dreambooth-fine-tuning-text-to-image-diffusion-models-for-a5ed97aeeb30">Diffusion model mini-summaries #1 — DreamBooth: Fine Tuning Text-to-Image Diffusion Models for…</a> was originally published in <a href="https://medium.com/ml-summaries">ML Summaries</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Diffusion mini-summaries #2: An Image is Worth One Word: Personalizing Text-to-Image Generation…]]></title>
            <link>https://medium.com/ml-summaries/diffusion-mini-summaries-2-textual-inversion-29110d18664a?source=rss----fcfb4aadd7f---4</link>
            <guid isPermaLink="false">https://medium.com/p/29110d18664a</guid>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[diffusion]]></category>
            <category><![CDATA[diffusion-models]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[generative-art]]></category>
            <dc:creator><![CDATA[Gowthami Somepalli]]></dc:creator>
            <pubDate>Thu, 12 Jan 2023 16:04:08 GMT</pubDate>
            <atom:updated>2023-01-12T16:04:10.342Z</atom:updated>
            <content:encoded><![CDATA[<h3>Diffusion mini-summaries #2: An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*aPxYudze3BDJV20Z" /></figure><p>Textual Inversion: Learn the word for a given subject by optimizing the language encoder which enables us to generate new images of the given subject in new settings.</p><blockquote>Paper: <a href="https://arxiv.org/abs/2208.01618">https://arxiv.org/abs/2208.01618</a></blockquote><p>If someone asks you what the following image is, you say “teapot”. If I ask you for more details, maybe you say, “colorful teapot”. But I will be imagining a completely different colorful teapot in my head. Text-to-image models also face this same problem.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*vhoF3IarY8s355zB" /></figure><p>This paper tries to learn the “word” which encompasses a bit more information on the subject, such that we can generate more images of the given subject in many new scenarios.</p><p>In typical T2I models, there is a text encoder and there is an image generation module. In this paper, the authors freeze the image generation part and *fine-tune only the text module* using the “subject” images.</p><p>For a given “subject” image, we create a template caption like, “A photo of S*”. And we want to learn the embeddings of S* by optimization such that when we use “A photo of S*” text for image generation, we should get back the training image.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*rmb2ztjVLgvSEPhS" /></figure><p>The optimization objective is very simple. Train the model on regular diffusion loss on the small subset of “subject” images. And v* is the embedding of S*. Find the best embedding such that the loss is minimized on the training images.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*QC6eFDWyx6mGQL78" /></figure><p>The authors conducted experiments on stable diffusion and they fine-tune for 5000 steps with this new optimization objective. The authors also initialize the v* with a coarse description of the object. Like “teapot” for this example we have seen before.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*QP0Fg126mlfnmeA0" /></figure><p>We can now use the S* as vocabulary to describe the subject and can generate in different settings, like the following example.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Z7_D7h2BXnvZR19B" /></figure><p>We can also use this technique to learn a word to represent a style, and can generate more images in that style!!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*-nFNovO0KAR2yfsg" /></figure><p>Some useful links: <br>Official implementation — <a href="https://github.com/rinongal/textual_inversion">https://github.com/rinongal/textual_inversion</a><br>Huggingface colab — <a href="https://colab.research.google.com/github/huggingface/notebooks/blob/main/diffusers/sd_textual_inversion_training.ipynb">https://colab.research.google.com/github/huggingface/notebooks/blob/main/diffusers/sd_textual_inversion_training.ipynb</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=29110d18664a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ml-summaries/diffusion-mini-summaries-2-textual-inversion-29110d18664a">Diffusion mini-summaries #2: An Image is Worth One Word: Personalizing Text-to-Image Generation…</a> was originally published in <a href="https://medium.com/ml-summaries">ML Summaries</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[My first in-person conference experience! CVPR 2022]]></title>
            <link>https://medium.com/ml-summaries/my-first-in-person-conference-experience-cvpr-2022-1883344cf886?source=rss----fcfb4aadd7f---4</link>
            <guid isPermaLink="false">https://medium.com/p/1883344cf886</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[conference]]></category>
            <category><![CDATA[cvpr]]></category>
            <dc:creator><![CDATA[Gowthami Somepalli]]></dc:creator>
            <pubDate>Fri, 01 Jul 2022 23:36:44 GMT</pubDate>
            <atom:updated>2022-07-01T23:36:44.876Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lSW5-HC4jyvbHNCNnH_bvg.png" /></figure><p>Hi there,</p><p>Sorry, this post is a bit of an anomaly compared to my other recent blogs. I am going to share my first in-person conference experience. I attended CVPR in New Orleans last week, and it was fun!</p><p>Instead of writing long-ass paragraphs about my experience, I will just split my experience into 4 segments with bullet points.</p><h4>What I liked:</h4><ol><li>Man, there was a lot of energy and excitement all around. Seeing other people excited makes me more excited.</li><li>There is a little bit of something for everyone. If you work in a niche field, bam, there’s a workshop, if you are interested in learning some new techniques, bam! there’s a tutorial. You will not get bored, and you can spend 8–5 at the convention center.</li><li>People are nice in general and they are open to talking. Unless you want to meet some super busy hot-shot prof, you can reach out to a researcher and they are generally down to meet you.</li><li>You will get a quick overview of what’s happening in the field. I absorbed more information in this conference than I ever did in virtual conferences in the last 2 years.</li><li>You can make friends in other universities. I was contacted on Twitter (Thanks Vikram if you are reading this) to have lunch together, and I ended up meeting the whole gang and I met folks from CMU, MILA, MIT, and so on. All of them work in diverse fields and it was a lot of fun talking to them.</li></ol><h4>What I did not like so much:</h4><ol><li>Food. :| I wish the conference food was a bit better. As a vegetarian who does not eat mushrooms, it was a bit hard for me.</li><li>Coffee. There are only fixed times where we can get coffee, and it sucked. I wish conference organizers made sure there is coffee all the time.</li><li>A lot of interesting workshops/oral sessions were happening at the same time. So it was a bit hard to decide on which one to go to.</li><li>Meeting people and also attending talks is a bit exhausting. I tried to do both this time, but next time I’ll plan better.</li><li>Special 2022 entry — Covid &amp; Masks</li></ol><h4>Things I wish there were more of:</h4><ol><li>There was a student mentorship session that happened and it had very few spots, I wish there are more interactive sessions like that.</li><li>At the end of Day 2, after NeRF workshop, a bunch of people spontaneously walked to the Marriot bar next door and did an informal gathering of sorts. I met a lot of interesting people there. I just happened to be at the right place. I wish conferences organize such after-hour informal meetings where we can just chat and meet people.</li><li>Have all workshops, and tutorials recorded and share them later.</li></ol><h4>Takeaways for the next conference:</h4><ol><li>Do not carry your laptop/other electronics if you are not going to use them. I carried them every day thinking I will need them, but I did not, and it unnecessarily exhausted me.</li><li>Make a list of sessions you want to attend before the conference or at least make a rough plan.</li><li>Do not be too aggressive with the plan, don’t fill up all 8–5 slots with something. Talking to people/Learning new things is exhausting.</li><li>Make a list of people you want to meet and message them before the conference and schedule the meetings.</li><li>The strategy of twos: This is a good one. I realized I am best at socializing if I have a friend around. So if you are on a mission to socialize, going with one friend might work out better than going alone.</li><li>Don’t have lunch with friends. Instead, sit at a random table. I met some awesome folks that way.</li><li>Have some fruit/energy bar on you at all times.</li></ol><p>Overall, I had a lot of fun. And I am looking forward to going to more in-person conferences in the future. :)</p><p>I hope you found this blog post helpful. If you have any tricks to optimize attending a conference, please drop them in the comments. I will add them to the post and attribute them to you. :)</p><p><em>Thank you </em><a href="https://twitter.com/kamalgupta09"><em>Kamal Gupta</em></a><em> for contributing some of the points, and proofreading the post.</em></p><p>Cheers, G 💯.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1883344cf886" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ml-summaries/my-first-in-person-conference-experience-cvpr-2022-1883344cf886">My first in-person conference experience! CVPR 2022</a> was originally published in <a href="https://medium.com/ml-summaries">ML Summaries</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Universal Adversarial Training — Paper Summary]]></title>
            <link>https://medium.com/ml-summaries/universal-adversarial-training-paper-summary-e9150fd7aa1c?source=rss----fcfb4aadd7f---4</link>
            <guid isPermaLink="false">https://medium.com/p/e9150fd7aa1c</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[adversarial-attack]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[robustness]]></category>
            <dc:creator><![CDATA[Gowthami Somepalli]]></dc:creator>
            <pubDate>Mon, 07 Mar 2022 11:45:18 GMT</pubDate>
            <atom:updated>2022-03-08T00:32:03.092Z</atom:updated>
            <content:encoded><![CDATA[<h3>Universal Adversarial Training — Paper Summary</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bkY3uCV6h4p5IKT3gVyMjg.png" /></figure><p><strong>Paper</strong>: Universal Adversarial Training<br><strong>Link</strong>: <a href="https://ojs.aaai.org/index.php/AAAI/article/view/6017/5873">https://ojs.aaai.org/index.php/AAAI/article/view/6017/5873</a><br><strong>Authors</strong>: Ali Shafahi, Mahyar Najibi, Zheng Xu, John Dickerson, Larry S. Davis, Tom Goldstein<br><strong>Tags</strong>: Adversarial attack, Universal attack, White-box attack<br><strong>Code</strong>: -<br><strong>Misc. info</strong>: Accepted to AAAI’20</p><h4>What?</h4><p>In this paper, the authors propose an optimized way to find the universal adversarial examples (first introduced in Moosavi-Dezfooli et al.[1]) for a given model. The authors also propose a low-cost algorithm to robustify the model against such perturbations.</p><h4>Why?</h4><p>Universal Adversarial Perturbations (UAP) are cheap — a single noise can be used to cause the model to mislabel lots of images. (Unlike the usual attacks where perturbations are generated per-image basis. These are more effective though). UAPs are also found to transfer across different models, hence they can be used in black-box attack setting too. Hence it is important to study them.</p><h4>Pre-requisites:</h4><p><strong>UAP vs Adversarial Perturbation: </strong>To attack the given model, in a common adversarial attack case, we find a unique delta for each image so that the model misclassifies it. In the UAP case, we find a delta and use it for all the images.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/293/1*t6eSq2_mWPVMnDzddkfWjg.png" /></figure><p><strong>UAP computation in [1]:</strong> UAP is first introduced in [1]. It is a simple technique, however, there are no convergence guarantees. The authors iterate across images and keep updating delta till ξ percent of images are misclassified. And the update in each iteration is computed with DeepFool [2]. The attack formulation and algorithm are shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/373/1*A6hn2TEw0TnhBn7Ke3HemQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/634/1*D9LZFYxeUzYWzGA5J5bH6Q.png" /></figure><p><strong>Adversarial training:</strong> To make the models robust against adversarial attacks, Madry et al proposed adversarial training, wherein every iteration, we generate adversarial examples and then calculate the loss on them and then update the weights on that loss. The formulation is as follows. (Z is the perturbed image)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/507/1*EmLX2hcRJCfLexEyw5Qx8Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ctdtHiiSb2uzs71BA4M3Bg.png" /><figcaption>Some examples of images with UAP added in them and the predicted class. Refer to Figure 3 in [1].</figcaption></figure><p>We are all done with the pre-requisites. Beyond this, I will discuss the paper’s contributions.</p><h4>How?</h4><p><strong>Improved UAP computation:</strong> In this paper, the authors simplified the formulation to find the delta that maximizes the loss. This way we can update the δ with an optimizer. The loss formulation is as follows. Since the loss is unbounded above, the authors propose a clipped version of this loss. This formulation searches for a universal perturbation that maximizes the training loss and thus forcing images into the wrong class.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/530/1*sGs_Gmce4r-K3c39LD4ScQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/408/1*bGCIuj6dkjjFmiFKS91bng.png" /><figcaption>(left) The formulation proposed in this paper to find UAPs (right) Clipping the loss. Beta is hyperparameter.</figcaption></figure><p>The above optimization problem can be solved by a stochastic gradient method, get loss gradient wrt δ, say g, update the δ to δ +l.r.*g, then project back δ to ε l_p ball.</p><p><strong>Improved UAP adversarial training:</strong> Similarly the authors propose to find a UAP for a given batch and train the model with perturbed inputs (x_i + δ)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/474/1*u0rcGWoFv6gsNru92aLv7A.png" /></figure><p>The authors also explore the fast adversarial training case where the weights and δ are backproped simultaneously. It gives decent enough performance. We can see the algorithms for both styles of UAP adversarial training in the below figures.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/908/1*i95kTFr73MH_OZ7GtPTgwA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/916/1*-wyOlhghHQe7yN6g57l28A.png" /><figcaption>(left) Algorithm for above adversarial training formulation, (right) Fast adversarial training formulation</figcaption></figure><h4>Results:</h4><ul><li>On CIFAR-10 dataset, with WideResnet-32 architecture, for ε = 8, 42.56% for the SGD perturbation, 13.30% for ADAM, and 13.79% for PGD. The clean test accuracy of the WRN is 95.2%.</li><li>In the below set of figures, we can see what the UAP looks like for CIFAR-10, from normally trained and robustly trained models. The robust models seem to have low-frequency UAPs than clean models. Also, the UAP can vary a lot with optimizer.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/539/1*0iw4gxQQmvGcZ6Vyh0xQXQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/756/1*6n6Y01LYSfeRWGosdUkQeA.png" /><figcaption>(Left) WRN 32–10, UAP after 160 iterations across different optimizers. (Right) with 400 iterations) UAPs on CIFAR-10 robust models: adversarially trained with FGSM or PGD, and universally adversarially trained with FGSM (uFGSM) or SGD (uSGD).</figcaption></figure><ul><li>The attack formulation is more successful than previous methods, as we can see from below left table.</li><li>The right table below shows that the adversarial training proposed in the paper is more robust to UAP perturbations and also has better generalization behavior. Interesting to note that the most ubiquitous adversarial training method (denoted by PGD) is not quite robust to UAPs.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/754/1*7MGN9DvzM282xLjYrFEJqw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/788/1*LXvZMlI2_waGNbZY_7WOYA.png" /><figcaption>(Left) Test accuracy of clean trained models on UAP attacked test images of ImageNet (Right) The test accuracy of adversarially trained models on CIFAR-10 UAP attacked test images.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DCVHJxbwlyqxRSsPCkmLSw.png" /><figcaption>3.9%, 42%, 28.3% are the accuracies on UAP attacked test images of ImageNet.</figcaption></figure><ul><li>The UAPs generated from the UAP-trained model vary significantly compared to the low-cost UAP-trained model. It somehow feels like the natural model’s (not adversarially trained) UAP looks high-frequency while the UAP-trained model has the lowest frequency, with the low-cost model’s UAP looking somewhere in between.</li></ul><p><strong>Comments:</strong></p><p>Overall it’s an interesting paper. The formulation for UAP computation introduced by this paper is computationally efficient and the perturbations are more effective. I wonder how the norm of the UAPs generated through this method compare to the UAPs computed using [1].</p><p>This training method seems to have better clean test performance than the usual adversarial training, one thing I wish they addressed is, how effective is this UAP adversarial training against the usual evasion attacks.</p><p>Bibliography:<br>[1] — Moosavi-Dezfooli, Seyed-Mohsen, et al. “Universal adversarial perturbations.” <em>Proceedings of the IEEE conference on computer vision and pattern recognition</em>. 2017.<br>[2] — Moosavi-Dezfooli, Seyed-Mohsen, Alhussein Fawzi, and Pascal Frossard. “Deepfool: a simple and accurate method to fool deep neural networks.” <em>Proceedings of the IEEE conference on computer vision and pattern recognition</em>. 2016.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e9150fd7aa1c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ml-summaries/universal-adversarial-training-paper-summary-e9150fd7aa1c">Universal Adversarial Training — Paper Summary</a> was originally published in <a href="https://medium.com/ml-summaries">ML Summaries</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentations —…]]></title>
            <link>https://medium.com/ml-summaries/when-vision-transformers-outperform-resnets-without-pre-training-or-strong-data-augmentations-eed97f99988?source=rss----fcfb4aadd7f---4</link>
            <guid isPermaLink="false">https://medium.com/p/eed97f99988</guid>
            <category><![CDATA[mlp-mixer]]></category>
            <category><![CDATA[vision-transformer]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[transformers]]></category>
            <category><![CDATA[optimization]]></category>
            <dc:creator><![CDATA[Gowthami Somepalli]]></dc:creator>
            <pubDate>Mon, 21 Feb 2022 19:58:43 GMT</pubDate>
            <atom:updated>2022-02-21T20:05:42.360Z</atom:updated>
            <content:encoded><![CDATA[<h3>When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentations — Paper Summary</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vgGQqzQT3YOoOq3gth__fw.png" /></figure><p><strong>Paper</strong>: When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentations<br><strong>Link</strong>: <a href="https://openreview.net/forum?id=LtKcMgGOeLt">https://openreview.net/forum?id=LtKcMgGOeLt</a><br><strong>Authors</strong>: Xiangning Chen, Cho-Jui Hsieh, Boqing Gong<br><strong>Tags</strong>: Vision Transformer, Optimization, SAM, Analysis<br><strong>Code</strong>: <a href="https://github.com/google-research/vision_transformer">https://github.com/google-research/vision_transformer</a> <br><strong>Misc. info</strong>: Accepted to ICLR’22</p><h4>What?</h4><p>Training a ViT is hard, computationally expensive, and they tend to underperform ResNets of comparable size without lots of pretraining/ strong data augmentations. This paper proposes that using a principled optimizer like SAM [1] can make ViTs on par with Conv-models without the need for the aforementioned tricks.</p><h4>Why?</h4><p>Vision Transformers are ubiquitous these days. However, training them requires a large amount of computing since they are highly sensitive to hyperparameters and the amount of training data. This paper tries to understand why ViTs behave this way from the loss-landscape perspective.</p><h4>How?</h4><p><strong>Prior results: </strong>I will add some results/ conclusions from the prior work on which the authors built up the results in this paper.</p><ol><li>[2] showed that the trainability of NN can be characterized by the associated NTK condition number. Smaller the number, easier it is to train the model</li><li>SGD and Adam are first-order optimizers and they ignore the curvature information of the loss surface. They only try to find the minima, they do not care whether the minima is flat/ sharp. It has been shown in prior works that flatter minima leads to better generalization (check paper page 4 for the list of references for this claim)</li><li>SAM works by optimizing the following objective. This essentially means we want to find “w” which minimizes the loss over the neighborhood. Please check out the paper for how the min-max is solved.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/642/1*uzkxCAFysh12WZOH670TjA.png" /><figcaption>Optimization objective of SAM. \rho is the size of the neighborhood ball.</figcaption></figure><p><strong>Experimental Setup: </strong>The authors evaluated ResNet-152, Vit, Mixer (B,S versions). All the models are trained from scratch on Imagenet for 300 epochs with the basic Inception-style preprocessing. Please check the paper for complete training details.</p><p>Hessian max eigenvalue, λ_max, computation is done using 10% of the ImageNet training images via <a href="https://en.wikipedia.org/wiki/Power_iteration">power iteration method</a> [used 100 iters to ensure convergence]</p><p><strong>Main Results:</strong></p><p>1. The ViT and Mixer models end up in sharper minima when trained with first-order optimizers compared to ResNet, however when we used SAM for optimization, we ended up in wider minima. [Figure 1]</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HqiHj4RaL9V60pYREy1Fqg.png" /><figcaption>Fig 1.</figcaption></figure><p>2 (a). The computed NTK condition number and the maximum eigenvalue, λ_max, of hessian, the values are in the order ResNet &lt; ViT &lt; Mixer, implying the ResNets are easiest to train as well as their minima are wider. While Mixers are at the worst end of the spectrum. [Top row of Table 1]</p><p>2 (b). In the bottom section of Table1, we can see that the ViTs with SAM outperform the ResNets [with or w/o SAM]. Mixers benefit the most with SAM while ResNets benefit the least. Authors conjecture that the inductive bias inherent to Conv-nets like translation invariance and locality might be responsible for wider minimas of ResNets.</p><p>2 (c). We also see that ViTs trained with SAM perform more robustly when evaluated on ImageNet-C [3] dataset which corrupts images with noise, blur etc.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VGeLRL297gcBW8Cy00maDw.png" /><figcaption>Table 1</figcaption></figure><p>3 (a). The authors tried to understand which part of the architecture is responsible for such high λ_max (~ max curvature) values. Hence they broke down the hessian into small diagonal Hessians and calculated the λ_max for each of the layers. Results for a few blocks are shown in the following table (Tab 2).</p><p>3 (b). In table 2, we can see, the embedding layer is the one incurring the sharpest geometry. As we get deeper, move from Block 1 to Block 12, the λ_max reduces. SAM reduced the dominant eigenvalues in all the layers (drastically for early layers in fact!)</p><p>3 (c). We can also see the norm of the weight vector goes up in the presence of SAM implying, weight decay (which we typically use in first-order optimizers) might be quite helpful with regularization in ViTs or Mixers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hzMJjucWrfe8sp8aAeaLdA.png" /><figcaption>Table 2</figcaption></figure><p>4 (a). In most cases, SAM improves performance much more than training with Augmentations even in low data regimes for ViTs and Mixers. However, Augmentations work better for ResNets in the low-data case (as seen in i1k 1/10 case). See Table 3 below for the complete results.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BRJYvcaSkgYkrjY03HKMBQ.png" /><figcaption>Table 3</figcaption></figure><p>4 (b). In the above table, we see the results of ViT+AUG are really close to that of ViT+SAM in most cases. The authors ask, are the Augmentations flatten the loss surface too? The analysis shows, they do but in different ways. When we look at λ_max for ViT+AUG (Fig2, right), is even worse than ViT without augmentation. However, when we look at loss surfaces (Fig 2, left), the surface seems flat near the convergence point for ViT-AUG, but when we venture far, it’s not.</p><blockquote>The authors say, “The difference is that SAM enforces the smoothness by reducing the largest curvature via a minimax formulation to optimize the worst-case scenario, while augmentations ignore the worse-case curvature and instead smooth the landscape over the directions concerning the inductive biases induced by the augmentations.”</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hXQJMAU1lZjqGXIqsVxmMA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/506/1*gMgsy7vGmi7ZVPhGb1F8GQ.png" /><figcaption>Fig 2. (Left) Cross entropy loss surfaces of ViT in different settings. (Right) Max eigenvalue of Hessian for each of the models.</figcaption></figure><p><strong>Mis. results:</strong> There are a couple of observations the authors made, a bit unrelated to the main topic of the paper. They are as follows —</p><ol><li>ViT activations are sparse, hence there is huge potential for pruning. Less than 10% of neurons have values greater than zero for most layers. In ResNet the number is &gt; 50%.</li><li>Also, the ViT+SAM qualitatively gave better segmentation maps.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Bqb3vcjr0BOlCwRHfcw9mA.png" /></figure><p><strong>Comments:</strong></p><p>Overall good paper. Well written and not too dense, and the authors conveyed their message forward clearly with interesting analysis. A few questions/ issues I had —</p><ol><li>I wish the authors showed how computationally expensive using SAM is, compared to augmentations.</li><li>Will this claim, “SAM helps with training &amp; generalization of ViTs” hold for smaller datasets say on CIFAR-10? [they showed how the transferability is, but not the training-from-scratch case]</li></ol><p>Bibliography:<br>[1] — Foret, Pierre, et al. “Sharpness-aware minimization for efficiently improving generalization.” <em>arXiv preprint arXiv:2010.01412</em> (2020).<br>[2] — Xiao, Lechao, Jeffrey Pennington, and Samuel Schoenholz. “Disentangling trainability and generalization in deep neural networks.” <em>International Conference on Machine Learning</em>. PMLR, 2020.<br>[3] — Hendrycks, Dan, and Thomas Dietterich. “Benchmarking neural network robustness to common corruptions and perturbations.” <em>arXiv preprint arXiv:1903.12261</em> (2019).</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=eed97f99988" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ml-summaries/when-vision-transformers-outperform-resnets-without-pre-training-or-strong-data-augmentations-eed97f99988">When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentations —…</a> was originally published in <a href="https://medium.com/ml-summaries">ML Summaries</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Deep Learning Through the Lens of Example Difficulty — Paper Summary]]></title>
            <link>https://medium.com/ml-summaries/deep-learning-through-the-lens-of-example-difficulty-paper-summary-3d048e9e6dd0?source=rss----fcfb4aadd7f---4</link>
            <guid isPermaLink="false">https://medium.com/p/3d048e9e6dd0</guid>
            <category><![CDATA[curriculum-learning]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[neurips]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[google-brain]]></category>
            <dc:creator><![CDATA[Gowthami Somepalli]]></dc:creator>
            <pubDate>Sun, 13 Feb 2022 22:03:41 GMT</pubDate>
            <atom:updated>2022-02-13T22:03:41.607Z</atom:updated>
            <content:encoded><![CDATA[<h3>Deep Learning Through the Lens of Example Difficulty — Paper Summary</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OokQGeUQL_g8MQ8CMdpGqg.png" /></figure><p><strong>Link</strong>: <a href="https://openreview.net/forum?id=fmgYOUahK9">https://openreview.net/forum?id=fmgYOUahK9</a><br><strong>Authors</strong>: Robert John Nicholas Baldock, Hartmut Maennel, Behnam Neyshabur<br><strong>Tags</strong>: Deep Learning, Example Difficulty, Dataset Difficulty, Curriculum learning<br><strong>Code</strong>: — <br><strong>Video: </strong><a href="https://papertalk.org/papertalks/37152">https://papertalk.org/papertalks/37152</a><strong> </strong><br><strong>Misc. info</strong>: Accepted to NeurIPS’21</p><h4>What?</h4><p>In this paper, the authors try to understand the “difficult” examples in the training. Difficult here means, given an example, how much model capacity is needed to fit it to a given label. The authors also try to understand what makes a given example difficult? Is the example ambiguous? Does it look like a different class? and so on..</p><h4>Why?</h4><p>Most of the deep learning research is focused on trying to understand the inductive biases of deep learning architectures or to propose new networks to improve the performance on benchmark tasks. We take the datasets for granted. This paper takes the dataset perspective and try to make some conclusions on different types of images present in the data.How?</p><h4>How?</h4><p><strong>Experimental setup:</strong> For a given dataset (say CIFAR-10), the authors keep the network constant (say ResNet-18). The authors experimented on multiple datasets like CIFAR-10, CIFAR-100, FMNIST, SVHN, and with different architectures like ResNet-18, VGG16, and MLP. MLP consists of 7 hidden layers of width 2048 with ReLU activations.</p><p>Let’s define a few terms before we move on to the main results of the paper.</p><p><strong>Example difficulty: </strong>There are many ways to define the difficulty of the example. In this paper, the authors define difficulty as the smallest depth beyond which the representation of a given example starts being classified in the correct class. The authors use a k-NN probe to do the classification at each layer.</p><p><strong>Consistency score: </strong>For a given example, let’s say we train ‘n’ models with different initialization seeds and on train data excluding the example, consistency score is defined as % of models where the prediction is same as the assigned label.</p><p><strong>Consensus class:</strong> For a given example, for ‘n’ trained models, the consensus class is given by the majority of the classifiers.</p><p><strong>Consensus-Consistency score: </strong>For a given example, this is the % of models where the prediction is the same as the consensus label.</p><p><strong>Prediction Entropy:</strong> For a given example, when we predict the class using ‘n’ models, we calculate the entropy on each of the class probabilities.</p><p><strong>Example: </strong>To understand the above 3 concepts, let’s take a simple example. Let’s take a cat image, and we trained 1classifier with the given cat image in training and we do k-NN at different prediction depths, and we notice at depth 6 and beyond, the image representation is always classified as ‘cat’, so the example difficulty is 6 here.</p><p>Now let’s train 100 more classifiers with different initialization seeds, with this cat image omitted from the training data. Now, 70 classifiers predicted this image as ‘dog’, 30 classifiers predicted as ‘cat’. So consistency score is 0.3, and the consensus class is ‘dog’. And the consensus-consistency score would be 0.7.</p><p>The prediction entropy is -(0.7log_2 0.7 + 0.3log2 0.3) [since we got 0 predictions for rest of the classes.]</p><h4>Main analyses and takeaways:</h4><ol><li>Some datasets are harder than other datasets, as in most examples fall in the higher depth ranges. CIFAR-100 is much harder than Fashion MNIST. (See Fig 1, left)</li><li>The correlation between prediction depths of a given dataset between 2 models if the architectures have similar inductive bias, i.e. ResNet and VGG are more correlated than VGG and MLP since the former pair are convolution architectures. (Fig 1, right)</li><li>For easy examples, the consistency can be either high or low, but the consensus-consistency is high! It implies, the model can fit these examples very easily, however, the prediction label might not match the ground-truth label in absence of a label. (Fig.2)</li><li>In hard examples (ones with higher prediction depth), some examples can be fit confidently to the correct label, however, some are hard to fit in absence of a label. (Fig.2)</li><li>The prediction depth can be a lower bound for consensus-consistency of all the examples. (i.e. prediction depth can give a rough estimate on the uncertainty of the example, but it is uncertainty around consensus class, not the true class) (Fig.2)</li><li>The easy examples are learned earlier in the process too (Fig 3; left)</li><li>Data with smaller prediction depths have both larger input and output margins on average and that variances of the input and output margins decrease as the prediction depth increases. (Fig 3; center, right) [ output margin is the difference between the largest and second-largest logits and the adversarial input margin is the smallest norm required for an adversarial perturbation in the input to change the model’s class prediction]</li><li>The prediction entropy is lowest for examples with lower prediction depth. (Fig 4)</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/337/1*BVMjabcbJpzjdDH4Ua5sOw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/422/1*6NMdSW93QcR8c-agItxT3g.png" /><figcaption>Fig 1. (Left) The fraction of points at different prediction depths. Depending on the dataset the distribution of points change. (Right) How does the prediction depth of all the points correlate across different models when they are in train or validation splits. When the same model is compared with itself, they are initialized with different seeds.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/858/1*yVkQinT3yHVuEG8_5r7lHA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UI9j7eedBdS217qRuPauoA.png" /><figcaption>Fig. 2. (Left) 250 ResNet-18 models trained with different train, validation splits. Each point in the plot is an example evaluated on all the models with the point in the validation split. (Center, Right) Condensed plots of points for different datasets. The size of the bubble indicates the number of points. Easy examples are more consistent and are more likely to be accurate.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/590/1*WO5c2-j7ALqpEyqk-KaX0Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kMIeANQFb3FzkyayBxxKsg.png" /><figcaption>Fig. 3 — (Left) Are the easy examples learned fast too? It seems to be true. The figure shows results on VGG (Center, Right) If we plot the margins of examples compared to prediction depths, they are correlated too. Easy examples seem to have higher margins. On VGG, CIFAR-10.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KTnMtRnx1MXwUtSVThQBGA.png" /><figcaption>Fig.4 — When the uncertainty of predictions on an example is high, then the entropy is higher. Intuitively, most of the points are of lower entropy in FMNIST and are of higher entropy for CIFAR-100.</figcaption></figure><h4>Why are some examples difficult?</h4><p>The authors also try to understand the difficulty of examples by asking the following 3 questions.</p><blockquote>“Does this example look mislabeled?”; “Is classifying this example only easy if the label is given?”; “Is this example ambiguous both with and without its label?”.</blockquote><p>To understand this question, we need to look at example difficulty when the example is part of training and when it is not part of training. The following plots show, the average prediction depth on 250 ResNet-18 models (90:10 train, val split — 25 models where an example is missing, and 225 models where an example is present)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*w75NmD-qSS_AqJDs04v_Xg.png" /></figure><p>We can decompose the left plot into 2 plots where Consensus class =GT and when it’s not. Let’s examine the right plot. The points with high depth at both train and val splits are the most ambiguous ones, with or without labels they are hard to recognize. We see some examples from bird class.</p><p>When we look at examples where consensus class != GT, there are 2 types, (1) points easily fit in val but not in train — these are points which look like different class/ perhaps mislabeled [points closest to decision boundary perhaps], like in the above figure, the birds look very much like a plane. (2) points easily fit in train but not in val — these are ambiguous points without label. They are easy to fit with labels, however, they are hard to classify when similar points don&#39;t exist in training data.</p><p>Hence we can divide all points into ~4 categories [PD -&gt;prediction depth]</p><ol><li>Easy examples: (Low PDVal., Low PDTrain)</li><li>Looks like a different class: (Low PDVal., High PDTrain)</li><li>Ambiguous unless the label is given: (High PDVal., Low PDTrain)</li><li>Ambiguous: (High PDVal., High PDTrain)</li></ol><h4>Comments:</h4><p>It is an interesting analysis. The paper essentially introduces a new difficulty metric for examples and the authors build a story around it and try to intuitively explain what the edge case points are.</p><p>The paper is a bit too dense though. There are a looooot of experiments and a lot of plots, but I wish authors cut down on analysis and spent more time on building intuition on why we are observing such trends.</p><p>Another concern I had is, we are calculating the consensus class from a set of models and are using the same models to calculate consensus-consistency. it somehow feels like cheating. But again, we are not working with binary class but with multi-class problems, so it might be ok...</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3d048e9e6dd0" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ml-summaries/deep-learning-through-the-lens-of-example-difficulty-paper-summary-3d048e9e6dd0">Deep Learning Through the Lens of Example Difficulty — Paper Summary</a> was originally published in <a href="https://medium.com/ml-summaries">ML Summaries</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why Do Better Loss Functions Lead to Less Transferable Features? — Paper Summary]]></title>
            <link>https://medium.com/ml-summaries/why-do-better-loss-functions-lead-to-less-transferable-features-paper-summary-8577a59920e?source=rss----fcfb4aadd7f---4</link>
            <guid isPermaLink="false">https://medium.com/p/8577a59920e</guid>
            <category><![CDATA[transfer-learning]]></category>
            <category><![CDATA[resnet50]]></category>
            <category><![CDATA[machine-learning-tools]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Gowthami Somepalli]]></dc:creator>
            <pubDate>Mon, 07 Feb 2022 03:01:29 GMT</pubDate>
            <atom:updated>2022-02-07T03:01:29.149Z</atom:updated>
            <content:encoded><![CDATA[<h3>Why Do Better Loss Functions Lead to Less Transferable Features? — Paper Summary</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LXdJkrTlsMqIACAPNQiHNA.png" /></figure><p><strong>Paper</strong>: Why Do Better Loss Functions Lead to Less Transferable Features?<br><strong>Link</strong>: <a href="https://openreview.net/forum?id=8twKpG5s8Qh">https://openreview.net/forum?id=8twKpG5s8Qh</a><br><strong>Authors</strong>: Simon Kornblith, Ting Chen, Honglak Lee, Mohammad Norouzi<br><strong>Tags</strong>: Computer Vision, Transfer Learning, Centered Kernel Alignment, Analysis<br><strong>Code</strong>: — <br><strong>Misc. info</strong>: Accepted to NeurIPS’21</p><h4>What?</h4><p>The choice of loss function for a given model can impact the classification performance on Imagenet, however also impacts the transfer learning capabilities. The authors systematically study multiple losses and try to understand the reason behind this observation</p><h4>Why?</h4><p>CrossEntropy is the typical loss used in classification models, however, some techniques like Label smoothing are proposed to improve the performance of the models. It is good to understand if the improvement in accuracy is worth the trade-off, especially if we want to use the model for other downstream tasks.</p><h4>How?</h4><p><strong>Losses studied: </strong>Softmax cross-entropy, Label smoothing, Dropout on the penultimate layer, Final layer regularization (L2 on weights of the final layer), Logit penalty (L2 on logits themselves), Logit normalization (Applying softmax on normalized logits with some temperature scaling), Cosine softmax (Softmax cross- entropy loss applied on the similarity between weight vector and the penultimate layer plus some bias), Sigmoid cross-entropy (binary CE loss on each of the logits and sum them up across all the classes), Squared error (like the name says, MSE + some tricks). Check out pages 2–3 of the paper for more detailed mathematical notations and explanations along with references to the papers who used/ proposed them.</p><p><strong>Experimental setup: </strong>ResNet-50 trained on Imagenet, all the hyperparameters for all the losses are decided based on some validation dataset. Two transfer tasks are considered — Linear transfer (freeze the network, add a final layer and train only that), and k-Nearest Neighbors (Find the features of all training data and test data and classify using k-NN). 8 datasets are used in transfer context — CIFAR-100, CIFAR-10, Food and others.</p><p>A couple of concepts we need to know before we look at the main results</p><p><strong>Centered kernel alignment (CKA): </strong>This metric measures how similar weights of 2 layers are, and is invariant to rotation and isotropic scaling of the weights. Let X and Y are weight matrices from different layers/ same layer from different runs. CKA can be defined as follows —</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1w8MXkontghaexzCJXh4hw.png" /></figure><p><strong>Class separation (R²):</strong> Measure how within class variance and overall dataset variance change wrt to each other, and calculated using following formula. Essentially measures how seperated the classes are.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*imE6ZWrcMCICnWy4u5G_EA.png" /></figure><p>where sim(.,.) is scaled dot product.</p><h4>Main results</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ww7N5NyKimwfII5hwmQATQ.png" /><figcaption>Table 1</figcaption></figure><ol><li>Losses which lead to higher ImageNet accuracy lead to less transferable features. Softmax loss leads to best transferable features. (See above — Table 1)</li><li>When fine-tuned on other datasets, loss doesn&#39;t matter, they almost perform the same</li><li>The choice of loss affects the deeper features as observed by CKA scores (Fig 1) and the sparsity of ReLU activations across the layers (Fig 2)</li><li>The models with highest R² have the least transferable features (Figures 4 and 5)</li><li>The models whose representations have greater class separation are “overfit,” not to the pretraining datapoints, but to the pretraining classes — they perform better for classifying these classes, but worse when the downstream task requires classifying different classes. (This gives me an idea, check out my comments at the end)</li><li>Augmentation can improve accuracy without changing the class separation much.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uDP5mnuvExhnPyDfxivYNA.png" /><figcaption>Fig 1— CKA between different models trained on different loss functions. Lower layers similarity is much higher, while later layers are very different.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CosAANMUSIajLYVLyGzzDA.png" /><figcaption>Fig 2— Amount of Relu (non)-sparsity. Values don&#39;t change much in the early layers, but they diverge a lot. We see the highest sparsity in Squared-error while lowest in Softmax and others.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SNxCabxhDIgOstff9DBHEA.png" /><figcaption>Fig 3 — (Image from Appendix) — Similarity computed on agreement on mislabeled points. The right plot gives Dendrogram based on a similarity matrix.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Qlfs5-7ppeSRv0ZTujZsgA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oUknfSar2okSj4al1So81A.png" /></figure><p><strong>Comments:</strong></p><ul><li>For the overfitting point, I wonder if we train a 10 way classifier, and transfer to another 10 way classification dataset, will these results hold? Is it the # of classes that matter? or do we need the datasets (pertaining and transfer) to have the same/similar-ish distribution?</li><li>Overall, very interesting paper. I love how the paper reads really well. All the experiments are well thought and paper puts forward a clear message.</li><li>I guess takeaway is, CE is still the best loss, but I wonder if there are any applications where squared error loss can be useful..</li><li>Check out <a href="https://arxiv.org/abs/1805.08974">this prior work</a> from the same authors.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8577a59920e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ml-summaries/why-do-better-loss-functions-lead-to-less-transferable-features-paper-summary-8577a59920e">Why Do Better Loss Functions Lead to Less Transferable Features? — Paper Summary</a> was originally published in <a href="https://medium.com/ml-summaries">ML Summaries</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[SLIP: Self-supervision meets Language-Image Pre-training — Paper Summary]]></title>
            <link>https://medium.com/ml-summaries/slip-self-supervision-meets-language-image-pre-training-paper-summary-650c69e46058?source=rss----fcfb4aadd7f---4</link>
            <guid isPermaLink="false">https://medium.com/p/650c69e46058</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[pytorch]]></category>
            <category><![CDATA[self-supervised-learning]]></category>
            <category><![CDATA[clip]]></category>
            <category><![CDATA[vision-transformer]]></category>
            <dc:creator><![CDATA[Gowthami Somepalli]]></dc:creator>
            <pubDate>Mon, 03 Jan 2022 02:26:12 GMT</pubDate>
            <atom:updated>2022-01-03T02:26:12.532Z</atom:updated>
            <content:encoded><![CDATA[<h3>SLIP: Self-supervision meets Language-Image Pre-training — Paper Summary</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2DPTDlBZyagACMN_LE9UOw.png" /></figure><p><strong>Paper</strong>: SLIP: Self-supervision meets Language-Image Pre-training<br><strong>Link</strong>: <a href="https://arxiv.org/abs/2112.12750">https://arxiv.org/abs/2112.12750</a><br><strong>Authors</strong>: Norman Mu, Alexander Kirillov, David Wagner, Saining Xie<br><strong>Tags</strong>: Vision Transformers, CLIP, Self-supervision, Language pre-training<br><strong>Code</strong>: <a href="https://github.com/facebookresearch/SLIP">https://github.com/facebookresearch/SLIP</a><br><strong>Misc. info</strong>: From the same folks as Moco V3</p><h4>What?</h4><p>CLIP[1] loss + SimCLR[2] loss for pre-training a Vision Transformer.</p><h4>Why?</h4><p>While CLIP is essentially pre-training with a contrastive loss between language and image embeddings of the same concept, SLIP explores if adding additional loss — contrastive pre-training on images will add any value to the pipeline! (Spoiler: It does!)</p><h4>How?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/914/1*i1kX3lY75NdvWgXwGJfIxg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/718/1*jglYFG2rHVq_WftIg4tO8w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/724/1*Qa0rFZXnFxgIQ6hssNkgqw.png" /></figure><p><strong>So how does it work?</strong> The left code snippet makes it really easy to understand the idea behind the paper. For a given image, text pair. Essentially we generate 3 views for the image. 2 views are used in SimCLR loss and the third view and the text are used in CLIP loss. The final loss term is the sum of SimCLR and CLIP losses with some weighting. In this algorithm, <em>fi</em> and <em>ft</em> are the encoders, what are <em>hi</em>, <em>ht</em>, and <em>hs</em>? They are projectors from high to low dimensional spaces. <em>hi</em>, and <em>ht</em> are learned linear projectors while <em>hs</em> is a 3-layer MLP head (with ReLU and <a href="https://pytorch.org/docs/stable/generated/torch.nn.SyncBatchNorm.html">SyncedBatchNorm</a>). The middle code snippet shows how the clip loss works and the right code snippet is the implementation of SimCLR loss.</p><p>Implementation details are as follows.</p><p><strong>Architecture</strong>: It is similar to CLIP with one image encoder and one language encoder. The authors explored ViT-B/16, ViT-S/16, ViT-L/16 as image encoders. For the text encoder, they used the smallest model from CLIP paper.</p><p><strong>Augmentations</strong>: CLIP loss image augmentation is only cropping (between 50 and 100%). For SimCLR loss branch, however, the authors use Augmentations from MoCo v3 [3] such as random resized cropping, horizontal flipping, color jittering, grayscale conversion, blurring, and<br>solarization.</p><h4>Results</h4><p>The model is evaluated on 3 different tasks. 1) Zero-shot transfer (i.e. no Finetuning at all) 2) Linear Classification (Adding a classification layer and tuning it with new dataset) 3) End-to-end Finetuning (The whole model is tuned with new dataset). All the models are pre-trained on the YFCC15M dataset (15M subset of YFCC100M consisting of English-only titles and descriptions [1])and the models are evaluated on the ImageNet dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ak1ZwEhdlApV5_BHdVeO5Q.png" /></figure><p>The above figure shows SLIP is doing better than the CLIP model or a ViT pre-trained using just MoCo V3 or SimCLR in all 3 tasks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/912/1*LxLd8uaGEt5xe-RWfhQBYw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/946/1*6deEMI5wkW_ISD0jl6gSWg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/926/1*TzHk9Hik_1VrwkRxfWyb7g.png" /></figure><p>So, the authors check different ablations to see the effectiveness of the additional Self-Supervision term. In Table 6, the authors try different SSL methods for image pre-training, we see that SimCLR seems to be performing the best.</p><p>In Table 8, the authors check whether adding additional augmentations to the CLIP model can achieve the same level of performance as SLIP, the results show that they do not! The additional pre-training is clearly improving the encodings learned by the image encoder!</p><p>In Table 7, the authors check if we take a pre-trained image encoder from SimCLR and then train a CLIP model vs SLIP model performance. SLIP did well in this case too.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eTzVK4FxAAyvTts70USHfQ.png" /></figure><p>In Table 4, the authors show the Zero-shot performance of CLIP and SLIP models on various image datasets. SLIP seems to be outperforming CLIP on an average.</p><p><strong>Comments:</strong></p><ul><li>Overall, it is a great engineering paper. The idea is simple, but the evaluation is quite thorough. I wish authors had a better illustration of how it works, the one they have is not quite clear. The algorithm snippet is quite helpful though. (I love these PyTorch pseudo-code snippets in papers! 😁)</li><li>Also, I wish the authors gave a better intuition on why the image representations of SLIP are better than CLIP by visualizing them. Table 4 is quite surprising for me actually! CLIP being trained on so much more data, I expected it to generalize better!</li></ul><p>Bibliography:<br>[1] — Radford, Alec, et al. “Learning transferable visual models from natural language supervision.” <em>arXiv preprint arXiv:2103.00020</em> (2021).<br>[2] — Chen, Ting, et al. “A simple framework for contrastive learning of visual representations.” <em>International conference on machine learning</em>. PMLR, 2020.<br>[3] — Chen, Xinlei, Saining Xie, and Kaiming He. “An empirical study of training self-supervised vision transformers.” <em>arXiv preprint arXiv:2104.02057</em> (2021).</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=650c69e46058" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ml-summaries/slip-self-supervision-meets-language-image-pre-training-paper-summary-650c69e46058">SLIP: Self-supervision meets Language-Image Pre-training — Paper Summary</a> was originally published in <a href="https://medium.com/ml-summaries">ML Summaries</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Suggest papers or topics to cover!]]></title>
            <link>https://medium.com/ml-summaries/suggest-papers-or-topics-to-cover-7a8432ac22ae?source=rss----fcfb4aadd7f---4</link>
            <guid isPermaLink="false">https://medium.com/p/7a8432ac22ae</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[suggestion-box]]></category>
            <dc:creator><![CDATA[Gowthami Somepalli]]></dc:creator>
            <pubDate>Sun, 19 Dec 2021 22:02:40 GMT</pubDate>
            <atom:updated>2022-01-03T02:43:58.666Z</atom:updated>
            <content:encoded><![CDATA[<p>Hi there,<br>If you have a specific topic or paper in mind, please suggest in the following Typeform.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Ftbvk8v1dlk8.typeform.com%2Fto%2FZY2tF615%3Ftypeform-embed%3Doembed%26typeform-medium%3Dembed-oembed%26format%3Djson&amp;display_name=Typeform&amp;url=https%3A%2F%2Ftbvk8v1dlk8.typeform.com%2Fto%2FZY2tF615&amp;image=https%3A%2F%2Fimages.typeform.com%2Fimages%2FFYUps4mFKPYK%2Fimage%2Fdefault&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=typeform" width="900" height="600" frameborder="0" scrolling="no"><a href="https://medium.com/media/81d8060fbee50ae51242408f9193f7d4/href">https://medium.com/media/81d8060fbee50ae51242408f9193f7d4/href</a></iframe><p>Cheers,<br>ML Summaries folks!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7a8432ac22ae" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ml-summaries/suggest-papers-or-topics-to-cover-7a8432ac22ae">Suggest papers or topics to cover!</a> was originally published in <a href="https://medium.com/ml-summaries">ML Summaries</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[MLP-Mixer: An all-MLP Architecture for Vision — Paper Summary]]></title>
            <link>https://medium.com/ml-summaries/mlp-mixer-an-all-mlp-architecture-for-vision-paper-summary-e50fa915e04d?source=rss----fcfb4aadd7f---4</link>
            <guid isPermaLink="false">https://medium.com/p/e50fa915e04d</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[mlp-mixer]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[vision-transformer]]></category>
            <dc:creator><![CDATA[Gowthami Somepalli]]></dc:creator>
            <pubDate>Tue, 31 Aug 2021 01:18:15 GMT</pubDate>
            <atom:updated>2021-08-31T01:35:26.726Z</atom:updated>
            <content:encoded><![CDATA[<h3>MLP-Mixer: An all-MLP Architecture for Vision — Paper Summary</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OSFjv1O4k-ACArsY-vIq-w.png" /></figure><p><strong>Paper</strong>: MLP-Mixer: An all-MLP Architecture for Vision<br><strong>Link</strong>: <a href="https://arxiv.org/abs/2105.01601">https://arxiv.org/abs/2105.01601</a><br><strong>Authors</strong>: Ilya Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, Mario Lucic, Alexey Dosovitskiy<br><strong>Tags</strong>: Machine Learning, Deep Learning, MLP architectures, Deep Learning architectures<br><strong>Code</strong>: <a href="https://github.com/google-research/vision_transformer">https://github.com/google-research/vision_transformer</a><br><strong>Misc. info</strong>:</p><h3>What?</h3><p>A competitive all MLP based architecture (without convolutions or attention), which performs well on Image-net classification benchmarks.</p><h3>Why?</h3><p>Simpler architecture with even fewer biases than a Vision Transformer[1]. Also unique deep learning architectures are all the rage right now!</p><h3>How?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wyeNqapnVsTJwHSrHB8jWg.png" /><figcaption>Figure 1: MLP-Mixer architecture illustration.</figcaption></figure><h4><strong>Architecture</strong></h4><p>The input to the model are ‘S’ number of square image patches, which is similar to Vision transformer. Unlike ViT[1] where convolutions might be used to generate the patch or “token” representations, in MLP-Mixer, the patches are linearly projected into C dimensions using the same projection matrix. So the embedding dimension would be SxC.</p><p>The main contribution of the paper is the Mixer layer which has 2 main steps. 1) MLPs applied for a given patch across channels, known as channel-mixing MLPs (MLP1 in the above figure) 2) MLPs applied at a given channel across tokens, which are known as token-mixing MLPs (MLP2 in the above figure). Note that for a given Mixer layer, MLP1 and MLP2 are initialized only once, and are shared across different channels/ tokens. Also layer norm is applied on each channel before both the MLP operations. Like Self-Attention layers in transformer, the dimension of the output is same as that of input in Mixer layers too.</p><p>Each MLP is 2 fully connected layers with one non-linearity, GeLU[2]. (A bit of trivia, one of the authors claimed the usage of GeLU has no particular significance, and they did not try with other non-linearities —<a href="https://twitter.com/neilhoulsby/status/1390253098288001027"> link</a>)</p><p>D_s and D_c are hidden layer widths of token-mixing and channel-mixing MLPs, respectively. Since both of these are hyper parameters, the complexity of the model is linear in terms on input size. (Unlike attention-based models which scale quadratically.)</p><blockquote>In a way, channel mixing part of the Mixer-MLP can be seen as a type of Conv-net with 1x1 filter. The token-mixing part of the Mixer actually increases the model’s receptive field to the complete image, as compared to Conv-nets who have smaller receptive fields in the beginning.</blockquote><p>Post the Mixer layers, we just take the average pooling across the channels which leads to S-dimensional embedding which is further passed on to a Fully connected network to give the final class prediction!</p><h4>Main results</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fOfzeo1Cm_cQ2GYBnWEXlQ.png" /><figcaption>Table1: Fine tuning results.</figcaption></figure><p>In Table 1, the authors present fine-tuned results on Imagenet, and average of 5 datasets (ImageNet, CIFAR-10, CIFAR-100, Pets, Flowers) after pre-training on ImageNet-21k and the Google’s proprietary datasets. The authors also present the speed of inference of each of the models, as well as the duration of pre-training in scale of 1000 TPUv3 days! 😱.</p><p>The blue dots represent models with attention, yellow dots are models with convolutions and purple are mixer models. It seems like Mixer-MLP results are very close to that of other standard models if not better, but they are faster than ViTs in terms of inference, but they need longer pre-training to reach similar level of performance!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KxH9_atmMehRx5zZSIAxWA.png" /><figcaption>Table 2: Various Mixer-MLP configurations</figcaption></figure><p>Similar to ViTs, there are multiple Mixer models with increasing complexity with higher MLP widths and increasing number of mixer layers as described in Table2.</p><p>In addition, when we pre-train with increasing amount of data, after certain point, the performance of Mixer-MLPs exceed Vision transformers as seen in Figure 2</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/808/1*ODrsMn2u3FE9so_U_YvVBw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EVXsgxQLKQszwXyTS0DWKA.png" /><figcaption><strong>Left</strong>: Figure 2 — Validation accuracy with increasing pretraining data size on Mixer models, ViTs and Convolutional models. <strong>Right</strong>:Figure 3 — Even if we shuffle the patches of data, Mixer model performs the same, while Resnet accuracy drops drastically.</figcaption></figure><p>The authors also performed an additional experiment by modifying the training data as follows (1) shuffle patches and shuffle pixels within in a fixed pattern (2) Global shuffling of all the pixels (An example can be seen in above Figure 3). Mixer-MLPs, performance is exactly the same in the first case, while the performance drops a bit in second experiment albeit less than ResNet’s[3] case! The author’s explain it as due to the Mixer’s invariance to the order of the patches and the pixels within them.</p><blockquote><strong>In the patch shuffling case, since the permutation applied is same for all images, for an MLP mixer, it is same as learning an additional permutation matrices with in the patch and across the patches. But in case of CNNs, who have the inbuilt prior of nearby pixels being related to each other, the shuffling of pixels disrupts the proper usage of the prior.</strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9V-bxvcRu8bNPWT7RDI1Vg.png" /><figcaption>Figure 4: Visualizations of the hidden units of first, second and third token mixing MLPs of Mixer-B/16 models trained on JFT dataset.Each pixel in each of the blocks represent the weight for a patch in the original image.</figcaption></figure><p>In Figure 4, the authors visualize the pre-trained hidden units of token-mixing MLPs . For better visualization, the authors sorted all hidden units according to a heuristic that tries to show low frequency filters first. The authors also curated the visualization in such a way that each unit followed by its closest inverse.</p><p>An important point to note is, these visualizations are heavily dependent on the dataset (and augmentations) the model is pre-trained on. Check out the appendix of the paper to see the visualizations compared against different pre-training scenarios.</p><h4>Comments:</h4><p>Overall it is an interesting paper, with a new and simple architecture with surprisingly good results. Following are couple of questions I had while reading, I don’t have answers to them right now, feel free to comment if you have some thoughts around them.</p><ol><li>In the permutation experiment, I wonder if ViT’s also behave same as Mixers, since the pattern is constant through the dataset.</li><li>I wonder how the performance would be without any pre-training.</li></ol><p>Do let me know what you think in the comments!</p><p>Bibliography:<br>[1] — Dosovitskiy, Alexey, et al. “An image is worth 16x16 words: Transformers for image recognition at scale.” <em>arXiv preprint arXiv:2010.11929</em> (2020).<br>[2] — Hendrycks, Dan, and Kevin Gimpel. “Gaussian error linear units (gelus).” <em>arXiv preprint arXiv:1606.08415</em> (2016).<br>[3] — He, Kaiming, et al. “Deep residual learning for image recognition.” <em>Proceedings of the IEEE conference on computer vision and pattern recognition</em>. 2016.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e50fa915e04d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ml-summaries/mlp-mixer-an-all-mlp-architecture-for-vision-paper-summary-e50fa915e04d">MLP-Mixer: An all-MLP Architecture for Vision — Paper Summary</a> was originally published in <a href="https://medium.com/ml-summaries">ML Summaries</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>