<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[One Minute Machine Learning - Medium]]></title>
        <description><![CDATA[Short summaries (1–2 minutes reading time) to help you (and me) understand and remember important papers/concepts about machine learning and related topics. &quot;If you can’t explain it simply, you don’t understand it well enough” — Einstein, maybe. - Medium]]></description>
        <link>https://medium.com/one-minute-machine-learning?source=rss----cce48f05fc2f---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>One Minute Machine Learning - Medium</title>
            <link>https://medium.com/one-minute-machine-learning?source=rss----cce48f05fc2f---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 18:34:37 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/one-minute-machine-learning" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[CLIP Paper Explained Easily in 3 Levels of Detail]]></title>
            <link>https://medium.com/one-minute-machine-learning/clip-paper-explained-easily-in-3-levels-of-detail-61959814ad13?source=rss----cce48f05fc2f---4</link>
            <guid isPermaLink="false">https://medium.com/p/61959814ad13</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[computer-vision]]></category>
            <dc:creator><![CDATA[Jeffrey Boschman]]></dc:creator>
            <pubDate>Mon, 17 Jul 2023 13:11:30 GMT</pubDate>
            <atom:updated>2023-07-17T13:11:30.688Z</atom:updated>
            <content:encoded><![CDATA[<h4>🤔 And the key points to remember</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YLazFpgkr5dGNFAXV2xFNw.png" /></figure><h4>Can you answer these questions easily?</h4><ol><li>CLIP is a pre-trained model for telling you how well a given <em>___ </em>and a given <em>___</em> fit together.</li><li>In training CLIP, the similarity scores of the correct image-text pairs are found on the <em>___ </em>of the similarity score matrix for the current batch.</li></ol><p>If not, keep reading! (see answers at the bottom of this post)</p><h3>One Sentence Summary</h3><p>CLIP is a <strong><em>pretrained</em></strong> model for telling you how well a given<strong> <em>image</em></strong> and a given<strong> <em>text</em> </strong>fit together.</p><h3>One Minute Summary</h3><p>CLIP, which stands for Contrastive Language-Image Pre-training, is a model for telling you how well a given image and a given text caption fit together.</p><p>In <strong><em>training</em></strong>, it tries to <strong><em>maximize</em> </strong>the “cosine similarity” between <strong><em>correct</em></strong> image-caption vector pairs, and <strong><em>minimize</em></strong> the similarity scores between all <strong><em>incorrect</em></strong> pairs.</p><p>In <strong><em>inference</em></strong>, it calculates the similarity scores between the vector of a <strong><em>single image</em></strong> with <strong>a <em>bunch of possible caption </em></strong>vectors, and picks the caption with the highest similarity.</p><p>Note that CLIP is <strong><em>not</em></strong> a <strong><em>caption generation</em></strong> model, it can only tell you if some existing text caption fits well with an existing image or not.</p><h3>Five Minute Summary</h3><p>CLIP is a pre-trained model for telling you how well a given image and a given text caption fit together, introduced by the paper “Learning Transferrable Visual Models from Natural Language Supervision” (2021) from OpenAI. It was <strong><em>trained</em></strong> <strong><em>contrastively</em></strong> on a huge amount (400 million) of web scraped data of image-caption pairs (one of the first models to do this). It is useful because this pre-trained model can be used for a lot of downstream tasks; instead of just associating an image with a class label out of a set of class labels, it can <strong><em>associate an image with</em></strong><em> </em>a text caption containing <strong><em>any words from the English language</em></strong>.</p><h4>🏋🏻 <strong>Training</strong></h4><p><strong>Input:</strong> A bunch of image-caption(text) pairs (all encoded to be vectors).</p><p><strong>Output:</strong> The “cosine similarity” scores between <strong><em>all</em></strong> the image vector and caption vector combinations.</p><p><strong>Objective function:</strong> A contrastive function that will modify the weights of the model such that correct image-caption pairs get a high similarity score, and incorrect pairs get low similarity scores.</p><p>Note — during training, the model requires that a huge batch of image-text pairs is fed at once (e.g., 20,000 pairs). That way, each batch contains 20,000*20,000 = 400,000,000 possible pairs, with only 20,000 being correct pairs. For efficient processing, the similarity scores of all possible pairs are computed at once to yield a 20,000 by 20,000 matrix, with the values in the <strong><em>diagonal</em></strong> being the similarity scores for the <strong><em>correct image-text pairs</em></strong>. That way, the objective function can have the goal to maximize the scores in the diagonal and minimize all the scores not in the diagonal. See the figure below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BkWsQpgE2byO0xgoUzgNKA.png" /><figcaption>Overview of how CLIP works during training.</figcaption></figure><h4><strong>🤔 Inference</strong></h4><p><strong>Inputs:</strong> the vector for a single image, and the vectors for a bunch of different possible text captions.</p><p><strong>Output:</strong> the similarity scores of the single image to all the different text captions.</p><p><strong>Goal:</strong> select the text which has the highest similarity with the image.</p><p>Note — the authors used this strategy to use CLIP for <strong><em>classification</em></strong> inference on the ImageNet dataset. They turned the label of each ImageNet class into a sentence, and used these sentences as the possible captions for a given image. For example, instead of using the ImageNet label “cat”, they created a sentence like “A photo of a cat” because this is the type of text that CLIP is used to. Then they compared a given ImageNet image with the set of sentences that correspond to the different classes, and picked the sentence that fit the image the best.</p><p>The paper says they have <strong><em>high zero-shot performance</em></strong> — this is because even though the model might not have been trained on any examples of the classes in the ImageNet dataset, it still performs well because it could kind of figure out what the words of the classes mean and associate that with the images.</p><h4><strong>Other Notes</strong></h4><p>Although CLIP itself is not a caption generator model, the pre-trained CLIP model can be used to calculate similarity scores between images and captions, which could therefore be useful <strong><em>as part of</em></strong> caption generator models.</p><h3><strong>Answers to questions at the beginning</strong></h3><ol><li><em>Image, text (or caption)</em></li><li><em>Diagonal</em></li></ol><h3>Links</h3><ul><li><a href="https://www.youtube.com/watch?v=dh8Rxhf7cLU">https://www.youtube.com/watch?v=dh8Rxhf7cLU</a> — this YouTube video does a great job explaining the main points of CLIP visually.</li><li><a href="https://openai.com/research/clip">https://openai.com/research/clip</a> — This is the official post about CLIP from OpenAI, which contains links to the paper and code.</li></ul><h3>Hey, have you become a Medium member yet?</h3><p>If not, please consider <a href="https://jeffreyboschman.medium.com/subscribe">signing up</a> with my referral link to get access to ALL my articles, plus the articles from other smart and interesting people on the platform.</p><h4>👉 <a href="https://jeffreyboschman.medium.com/subscribe">SIGN UP HERE</a> 👈</h4><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=61959814ad13" width="1" height="1" alt=""><hr><p><a href="https://medium.com/one-minute-machine-learning/clip-paper-explained-easily-in-3-levels-of-detail-61959814ad13">CLIP Paper Explained Easily in 3 Levels of Detail</a> was originally published in <a href="https://medium.com/one-minute-machine-learning">One Minute Machine Learning</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[SimCLR Explained in Simple Terms]]></title>
            <link>https://medium.com/one-minute-machine-learning/simclr-explained-in-simple-terms-3fa69af45ff9?source=rss----cce48f05fc2f---4</link>
            <guid isPermaLink="false">https://medium.com/p/3fa69af45ff9</guid>
            <category><![CDATA[self-supervised-learning]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Jeffrey Boschman]]></dc:creator>
            <pubDate>Wed, 21 Dec 2022 22:09:29 GMT</pubDate>
            <atom:updated>2022-12-21T22:09:29.651Z</atom:updated>
            <content:encoded><![CDATA[<h4>SimCLR explained simply in 1 sentence, 1 minute, and 5 minutes</h4><p><strong><em>SimCLR</em></strong><em> stands for a </em><strong><em>Sim</em></strong><em>ple Framework for </em><strong><em>C</em></strong><em>ontrastive </em><strong><em>L</em></strong><em>earning of </em><strong><em>V</em></strong><em>isual </em><strong><em>R</em></strong><em>epresentations. There are a few steps involved and the </em><a href="https://arxiv.org/abs/2002.05709"><em>original paper</em></a><em> (by </em><strong><em>Ting Chen</em></strong><em> and </em><strong><em>Geoffrey Hinton</em></strong><em>, among others) uses some complex words so it might seem daunting, but once you understand it the concept is quite simple and intuitive. So, what is SimCLR?</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/576/1*-313X8y2e-Q2bCakWjV77g.gif" /><figcaption>Illustration of SimCLR from <a href="https://ai.googleblog.com/2020/04/advancing-self-supervised-and-semi.html">the author’s blog post</a>.</figcaption></figure><h3>One Sentence Summary</h3><p><em>If I had to describe SimCLR in one sentence, I would say something like:</em></p><blockquote><em>SimCLR is a strategy that trains model weights (to eventually be used for a separate downstream computer vision task like classification or detection) by training on the task of </em><strong><em>identifying the two images that are augmented views of each other </em></strong><em>out of a batch of other random images.</em></blockquote><p><em>However, there is still a lot to unpack before this makes complete sense, so allow me to go into a bit more detail.</em></p><h3>One Minute Summary</h3><p><em>Now I will try to explain SimCLR in the way I would for example to a peer, maybe in about one minute.</em></p><p>SimCLR is a <strong>self-supervised learning</strong> strategy. You would use SimCLR when you want to train the weights of a model to be really good before you eventually train it further for a separate computer vision task, like classification of images.</p><p>The SimCLR framework has four major components:</p><ol><li>A <strong>Data Augmentation Module,</strong> which transforms a given image randomly in two ways, yielding two correlated views of the same example.</li><li>A <strong>Base Encoder</strong>, such as the ResNet-50 architecture (although you can use almost any encoder) to extract representation vectors from augmented data examples. This is the model whose weights you are training for the eventual downstream task.</li><li>A <strong>Projection Head</strong>, which is an MLP (with one hidden layer) that just makes the representation vectors smaller before the loss function.</li><li>A <strong>Contrastive Loss Function</strong>, which is low if the two correlated vectors (from the same original image) are similar to each other AND dissimilar to the the vectors from a set of “negative” examples (all the other examples in the mini-batch; not from the same original example), and high otherwise.</li></ol><p>During training, each image in a mini-batch goes through the first 3 steps (the Data Augmentation Module, the Base Encoder, and the Projection Head), which yields <strong>two representation vectors per original image</strong>.</p><p>Then, the Contrastive Loss is calculated for each “positive” pair of vectors (from the same input image) by comparing them to one “negative” representation vector from all the other images in the mini-batch. The total loss is the average loss across all positive pairs in the mini-batch.</p><blockquote><em>As the loss gets propagated backwards through the network and weights are updated, the Base Encoder (and Projection Head, but we don’t care about this) should </em><strong><em>learn to make the representation vectors from the same input image more similar to each other, and more different from the representation vectors of the other images</em></strong><em>.</em></blockquote><h3>Five Minute Summary</h3><p><em>So now, I will go into a bit more detail. Imagine if I was talking to someone and they had more questions about what I was explaining.</em></p><h4>What is self-supervised learning?</h4><p><em>Let me run through a scenario to try to explain this.</em></p><p>Let’s say you want to train a model to do classification of images — maybe to classify medical images as containing cancer or not. If you can only train using <strong>example images that have already been annotated or labelled</strong> by a human doctor (as cancerous or not-cancerous), then you are doing <strong>supervised learning</strong>, right? However, you are limited by the number of example images that have been labelled. If you train an initial model using only supervised learning and realize it does not perform well, you might conclude you need to get more labeled images for training. Unfortunately, getting an expert doctor to annotate enough images to make your model perform better is very time consuming and expensive.</p><p><em>So what can you do? Can you do something besides supervised learning?</em></p><p>Yes, you can! This is where <strong>self-supervised learning</strong> comes in.</p><blockquote><em>Instead of supervised training using labelled data for a task like classification, </em><strong><em>self-supervised learning</em></strong><em> is a way of training on a </em><strong>different task</strong><em> </em><strong><em>beforehand</em></strong><em> using </em><strong><em>“labels” related to the metadata</em></strong><em>.</em></blockquote><p>Generally, computer vision strategies that use self-supervised learning involve performing two tasks, a pretext task and a downstream task.</p><p>The <strong>downstream task</strong> is the real task you care about, like classification or detection.</p><blockquote><em>The </em><strong><em>pretext task</em></strong><em> is the self-supervised learning task solved to train model weights that produce good representation vectors for input images, with the aim of using these model weights for the downstream task.</em></blockquote><p>In self-supervised scenarios, “labels” are not the same kind of labels as in supervised learning. Labels in supervised learning come from a human annotator, like a doctor who goes through a bunch of medical images and labels each as containing cancer or not.</p><blockquote><strong><em>In self-supervised learning, “labels” are inherent to the data without any human intervention</em></strong><em>. For example, in SimCLR, you have images that are augmented views of each other from the same source image, as well as augmented images that come from other source images. Comparing two augmented images, the “labels” could therefore be “from the same source image” or “</em><strong><em>not</em></strong><em> from the same source image”.</em></blockquote><p>Training with self-supervised learning does not <strong>explicitly</strong> help with the downstream task (e.g., classification), BUT it does help train the model weights to be better in general (i.e., it would make the representation vectors more <em>representative</em> of the images they are from), which should thereby help with the downstream task anyways.</p><p>And the best thing is that you don’t need data annotated by a human labeller for self-supervised tasks like SimCLR — you can use completely unlabelled datasets which expands the amount of data that you can use to train your model!</p><p><em>So, what is self-supervision in regards to SimCLR? In conclusion:</em></p><blockquote>SimCLR is a self-supervised strategy for using unlabelled data to train your model to produce good visual representations before a real, downstream task.</blockquote><h4><strong>Data Augmentation Module Details</strong></h4><p>As mentioned, the Data Augmentation Module transforms a given image randomly in two ways. There are three transforms that are done:</p><ul><li>random cropping (and resizing back to the original size)</li><li>random colour distortion</li><li>random Gaussian blur</li></ul><p>The authors chose these three after trying a bunch of different augmentations and found this combination to be the best.</p><h4>Base Encoder Details</h4><p>The Base Encoder extracts representation vectors (also sometimes called feature vectors) from the augmented images. If you’ve ever read/heard someone talk about “transforming an image into a lower-dimension space”, that’s what this Base Encoder does.</p><p>In simpler terms:</p><ul><li>Input: One of the augmented images from the previous step (e.g., the input image might have a size of 256*256 pixels and have 3 colour channels, for a total of 256*256*3=196,608 numbers per image)</li><li>Output: A vector that somehow represents the input image, but using less numbers (e.g., the ResNet-50 output feature vector is made up of 2,048 numbers)</li></ul><blockquote><em>Ideally, with SimCLR, the output vectors for </em><strong><em>similar</em></strong><em> images should be </em><strong><em>similar</em></strong><em>, while the output vectors for </em><strong><em>dissimilar</em></strong><em> images should be </em><strong><em>dissimilar</em></strong><em>.</em></blockquote><p>How well the Base Encoder accomplishes this depends on how well the weights are trained — essentially that is the whole point of SimCLR (and training deep learning models in general).</p><p>The Base Encoder is the model that we will use later (after training with SimCLR) to do the actual task we want to do, like classification or regression.</p><p>The original paper uses the commonly used ResNet-50 architecture (without the last layer) for the Base Encoder, but notes that pretty much any model that takes an image and yields an output vector could be used.</p><h4>Projection Head Details</h4><p>The Projection Head might sound fancy, but it is actually just a multilayer perceptron (MLP) with one hidden layer (i.e., two fully connected layers) that transforms the representation vector from the previous step to a different, smaller representation vector.</p><ul><li>Input: A representation vector that is the output of the base encoder (length 2,048)</li><li>Output: A different representation vector (length 128)</li></ul><p>After you have trained with SimCLR, you remove the Projection Head and only use the Base Encoder for the downstream task.</p><p>The reason you need the Projection Head is because it puts the representation vector into the format/size that you will apply the Contrastive Loss Function.</p><p>It seems like the authors only included the Projection Head because they tried both having it and not having it, and found that SimCLR worked better when it was included. Perhaps the weights that are learned in these two Projection Head layers negatively impact the downstream task.</p><h4>Contrastive Loss Function Details</h4><p>The formula(s) for the Constrastive Loss Function looks like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/894/1*qbzwUr34JwBkz9suHx4IOQ.png" /><figcaption>The steps to calculate the total loss L in SimCLR (from <a href="https://arxiv.org/abs/2002.05709">the original paper</a>). z is a representation vector after the Projection Head. N is the number of original images in a mini-batch.</figcaption></figure><p><em>I know this looks really complicated, but really the main thing to understand is that:</em></p><blockquote><em>The loss is </em><strong><em>low</em></strong><em> if the two correlated vectors (from the same example) are similar to each other </em><strong><em>AND</em></strong><em> dissimilar to the the vectors from a set of “negative” examples (all the other examples in the mini-batch; not from the same original example).</em></blockquote><blockquote><em>The loss is </em><strong><em>high</em></strong><em> otherwise.</em></blockquote><p><em>Anyways, I will still try to explain in more detail.</em></p><p><strong>Step 1:</strong></p><p>The first step in calculating the total loss is to calculate the pairwise similarity for all combinations of representation vectors. If there are <em>N</em> original images in the mini-batch, then there are <em>2N</em> representation vectors (because of the Data Augmentation Module taking each image and randomly transforming it in two ways). That is what the following formula says:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/882/1*41uh1a5N2ADK2tw2m3C2rw.png" /><figcaption>Calculating the pairwise similarity for all combinations of representation vectors. From <a href="https://arxiv.org/abs/2002.05709">the original paper</a>.</figcaption></figure><p><strong>Step 2:</strong></p><p>The next step is to define the actual Contrastive Loss Function. To be specific, the Contrastive Loss Function in SimCLR is called NT-Xent (normalized temperature-scaled cross entropy loss). The formula is the following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/592/1*qTbvDt-4a8AG-DNLL3_U7A.png" /><figcaption>The NT-Xent loss function. From <a href="https://arxiv.org/abs/2002.05709">the original paper</a>.</figcaption></figure><p>The numerator (the top of the fraction) is a measure of how similar the current vectors being compared are. If they are from the same original image, we want the similarity to be high (the highest similarity value is 1.0).</p><p>The denominator (the bottom of the fraction) is the sum of how similar other vectors are. Usually this is higher than the numerator, and thus the fraction is less than 1.0. So to make the fraction higher (closer to 1.0), you want the denominator value to be low. But it is a sum of a bunch of values, so how can it be low? You want all those values to be low.</p><p>The best case is that when <em>i</em> and <em>j </em>are from the same original image (which is always should be because of the third equation), you have a similarity of 1.0, and <em>i</em> compared to any other vector has a similarity of 0.0. That way, the total fraction will be 1.0 — which is the largest number it can be, and since we are taking the negative log, this is the smallest loss you can possibly get (0.0).</p><p><strong>Step 3:</strong></p><p>The third step is to actually calculate and sum all the losses.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/742/1*Y-0d43HeSu_cmAxDi_oOxA.png" /><figcaption>The total loss of a forward pass. From <a href="https://arxiv.org/abs/2002.05709">the original paper</a>.</figcaption></figure><p>As I mentioned a bit earlier, we are only calculating the losses for when <em>i</em> and <em>j</em> (in the second equation) are from the same image (because the (<em>2k)</em>th and (<em>2k-1)</em>th are the two corresponding representation vectors from the same original image). And we do this for all original images, and sum up all the losses to get the total loss.</p><h4>That’s it! I hope all of this helps someone out there!</h4><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3fa69af45ff9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/one-minute-machine-learning/simclr-explained-in-simple-terms-3fa69af45ff9">SimCLR Explained in Simple Terms</a> was originally published in <a href="https://medium.com/one-minute-machine-learning">One Minute Machine Learning</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What do doctors think about the ethics of machine learning in healthcare?]]></title>
            <link>https://medium.com/one-minute-machine-learning/what-do-doctors-think-about-the-ethics-of-machine-learning-in-healthcare-cf5ab7cbb8ab?source=rss----cce48f05fc2f---4</link>
            <guid isPermaLink="false">https://medium.com/p/cf5ab7cbb8ab</guid>
            <category><![CDATA[ethics]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[healthcare]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[2021]]></category>
            <dc:creator><![CDATA[Jeffrey Boschman]]></dc:creator>
            <pubDate>Thu, 28 Oct 2021 14:00:19 GMT</pubDate>
            <atom:updated>2021-10-28T14:00:19.108Z</atom:updated>
            <content:encoded><![CDATA[<h4>“AI will not replace doctors, [but] doctors who use AI will replace doctors who do not”</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*QYrlmOGgquXsq9IC" /><figcaption>Photo by <a href="https://unsplash.com/@nci?utm_source=medium&amp;utm_medium=referral">National Cancer Institute</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>If you were to ask a doctor what they thought of the ethics of AI in healthcare, what is the first thing that would come to their mind?</p><p>A recent 2021 study (“<a href="https://www.sciencedirect.com/science/article/pii/S0933365721001834">A healthy debate: Exploring the views of medical doctors on the ethics of artificial intelligence</a>”) from researchers at TU Delft surveyed medical doctors about their opinions on AI in healthcare, especially the ethics of it. Most previous studies about AI in healthcare focussed on image-based diagnostics, but not enough is known about the views of <em>ethics</em>. From 77 doctors (from 13 different specialities) from The Netherlands, Portugal, and the U.S., the authors identified four specific opinions. Although the opinions have overlap, there are four distinct ideas that the groups <strong><em>focus</em></strong> on:</p><h4><em>1. AI is a helpful tool: Let physicians do what they were trained for</em></h4><p>The main focus of doctors in this group is that AI will be helpful for repetitive, easy tasks to free up time for physicians to work on more complex situations. However, they think the doctor should definitely have the final decision; AI is simply a tool. A good quote from the paper is: “[these doctors think that] AI will not replace doctors, [but] doctors who use AI will replace doctors who do not”</p><p>They do not have strong opinions on private corporations, other than the thought that medical doctors should be involved in the technology development process.</p><h4><em>2. Rules &amp; Regulations are crucial: Private companies only think about money</em></h4><p>When asked about the ethics of AI in healthcare, the doctors in this group focus on their distrust of private corporations that develop the AI technology, thinking that they need to be very regulated. These doctors are very worried about patient privacy, companies becoming monopolistic, and generally think that the values of healthcare do not align with the tech industry (i.e. money-focussed, “fail fast and fix later”). They are of the opinion that technology companies should definitely hold the liability.</p><h4><em>3. Ethics is enough: Private companies can be trusted</em></h4><p>This group focussed on the fact that current health systems already rely heavily on technology and automation, and therefore we already have a system in place for dealing with the ethics of AI in healthcare. In the current system, private companies that develop technology for healthcare are generally trusted and do not need extensive rules and regulations, and thus the same should apply to companies who produce AI technologies for healthcare.</p><h4><em>4. Explainable AI tools: Learning is necessary and inevitable</em></h4><p>The doctors in this group focussed heavily on the necessity for clinicians to understand how the AI decisions are made (i.e. AI cannot be a “black box”). They also believe doctors should be involved in the AI design process.</p><p>They do not have strong opinions on private corporations, other than the thought that medical doctors should be involved in the technology development process.</p><p><em>You may also like:</em></p><ul><li><a href="https://medium.com/one-minute-machine-learning/integration-of-machine-learning-and-genome-scale-metabolic-modeling-identifies-multi-omics-aae6087b13a7">Integration of machine learning and genome-scale metabolic modeling identifies multi-omics…</a></li><li><a href="https://medium.com/one-minute-machine-learning/bert-bidirectional-encoder-representations-from-transformers-one-minute-summary-c5e667e8630e">BERT (Bidirectional Encoder Representations from Transformers) | one minute summary</a></li></ul><p><em>If you would like to get unlimited access to my posts, please consider joining Medium with my referral link (disclaimer: part of the monthly fee will directly support me).</em></p><p><a href="https://jeffreyboschman.medium.com/membership">Join Medium with my referral link - Jeffrey Boschman</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cf5ab7cbb8ab" width="1" height="1" alt=""><hr><p><a href="https://medium.com/one-minute-machine-learning/what-do-doctors-think-about-the-ethics-of-machine-learning-in-healthcare-cf5ab7cbb8ab">What do doctors think about the ethics of machine learning in healthcare?</a> was originally published in <a href="https://medium.com/one-minute-machine-learning">One Minute Machine Learning</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Integration of machine learning and genome-scale metabolic modeling identifies multi-omics…]]></title>
            <link>https://medium.com/one-minute-machine-learning/integration-of-machine-learning-and-genome-scale-metabolic-modeling-identifies-multi-omics-aae6087b13a7?source=rss----cce48f05fc2f---4</link>
            <guid isPermaLink="false">https://medium.com/p/aae6087b13a7</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[2021]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[medicine]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Jeffrey Boschman]]></dc:creator>
            <pubDate>Fri, 15 Oct 2021 05:00:34 GMT</pubDate>
            <atom:updated>2021-10-15T05:01:06.071Z</atom:updated>
            <content:encoded><![CDATA[<h3>Integration of machine learning and genome-scale metabolic modeling identifies multi-omics biomarkers for radiation resistance (2021) | Abstract summary</h3><h4>Explaining an application of machine learning in medicine, line-by-line</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*I49UUzdiUzCMXKnj9MHN1A.png" /><figcaption>Lewis, J.E., Kemp, M.L. Integration of machine learning and genome-scale metabolic modeling identifies multi-omics biomarkers for radiation resistance. <em>Nat Commun</em> <strong>12, </strong>2700 (2021). <a href="https://www.nature.com/articles/s41467-021-22989-1">https://doi.org/10.1038/s41467-021-22989-1</a></figcaption></figure><p><a href="https://www.nature.com/articles/s41467-021-22989-1/figures/6">This article</a>, published in Nature Communications in May 2021, has a lot of biological jargon that makes it hard to read for the average machine learning practitioner. Here is a breakdown of all the sentences of the abstract, hopefully providing enough background information and context to understand the general idea completely.</p><h3>Abstract line-by-line</h3><p><strong><em>Resistance to ionizing radiation, a first-line therapy for many cancers, is a major clinical challenge.</em></strong></p><ul><li>For many people diagnosed with cancer, one of the first suggested treatments is “<a href="https://www.cancer.org/content/dam/CRC/PDF/Public/6151.00.pdf">ionizing radiation</a>”, also often called “radiotherapy”. This treatment works by sending waves of high energy particles (imagine like from an X-ray machine) towards cancerous areas in order to damage the genes of cancer cells and prevent them from growing and dividing anymore.</li><li>However, <em>this treatment doesn’t work for all cancer patients (about one-fifth)</em>, and recommending this treatment to people for whom it doesn’t work is bad for their health and also wastes the precious time of the patient and resources of the hospitals.</li></ul><p><strong><em>Personalized prediction of tumor radiosensitivity is not currently implemented clinically due to insufficient accuracy of existing machine learning classifiers.</em></strong></p><ul><li>Although scientists are working on developing machine learning models that could predict whether or not radiotherapy is worthwhile for any given patient/tumor, the <em>current models are not good enough to be used in actual hospital settings</em>. Current machine learning models use things like tumor histology (tissue sample images), tumor grade (how seriously the cancer has spread), and genomic information (the specific patient’s DNA), but none combine them with metabolomic data (the concentrations of small molecules in the area).</li></ul><p><strong><em>Despite the acknowledged role of tumor metabolism in radiation response, metabolomics data is rarely collected in large multi-omics initiatives such as The Cancer Genome Atlas (TCGA) and consequently omitted from algorithm development.</em></strong></p><ul><li>Genomics is the study of DNA and genetic information within a cell, transcriptomics is the study of RNA and differences in mRNA expression, proteomics is the study of protein activity, and <em>metabolomics is the study of small molecules (like the substrates or products of metabolism)</em>. The types and concentrations of small molecules are influenced both by genetics and the cell environment (like a tumor environment).</li><li>The Cancer Genome Atlas (TCGA) is huge database for researchers to use, comprised of the genomic, epigenomic, transcriptomic, and proteomic data of a bunch of different cancer samples (and their comparative normal tissue samples). It is extremely useful for cancer research. Unfortunately, it does not have a lot of metabolomic data, especially for <em>individual samples</em>.</li></ul><p><strong><em>In this study, we circumvent the paucity of personalized metabolomics information by characterizing 915 TCGA patient tumors with genome-scale metabolic Flux Balance Analysis models generated from transcriptomic and genomic datasets.</em></strong></p><ul><li>In this study, the authors were able to calculate the concentrations of small molecules (metabolites) for <em>individual patients/samples</em> by combining the <em>transcriptomic and genomic data of the individual samples (from the TCGA)</em>with <em>metabolic information about human’s in general</em>. They did this with a mathematical approach called <a href="https://www.nature.com/articles/nbt.1614">Flux Balance Analysis</a>.</li></ul><p><strong><em>Metabolic biomarkers differentiating radiation-sensitive and -resistant tumors are predicted and experimentally validated, enabling integration of metabolic features with other multi-omics datasets into ensemble-based machine learning classifiers for radiation response.</em></strong></p><ul><li>After the authors calculated the concentrations of various small molecules (metabolomic biomarkers) for all of their 915 samples, they <em>identified which of them were highly present or absent</em> in the tumor samples that responded well to radiotherapy (radiation-sensitive) vs. the tumor samples that were resistant to radiotherapy (radiation-resistant).</li><li>They also validated that their Flux Balance Analysis results were reasonable by they analyzing four samples and measuring the actual small molecule concentrations to compare with their calculated concentrations.</li><li>Then, they trained a couple of <em>different</em> machine learning classifier models <em>separately</em>. In one, the inputs were the calculated small molecule concentrations (metabolomic information) of all the samples to a machine learning classifier. In others, the inputs were either clinical, genomic, or transcriptomic information. For all of them, the outputs were the probability of classifying a sample as radiotherapy-sensitive or radiotherapy-resistant based on the inputs. Finally, they <em>combined the outputs for the different individual classifiers to get the overall prediction of radiation response (this is ensemble learning)</em>.</li></ul><p><strong><em>These multi-omics classifiers show improved classification accuracy, identify clinical patient subgroups, and demonstrate the utility of personalized blood-based metabolic biomarkers for radiation sensitivity.</em></strong></p><ul><li>The classifier strategy described above (combining the outputs from the clinical, metabolomic, genomic, and transcriptomic classifiers) provided the <em>best results so far (AUROC = 0.904) for predicting whether individual patients would respond well to radiotherapy or not</em>. AUROC refers to the Area Under Receiver Operating Characteristic curve, which is a measure for classification models where 1 means a perfect score.</li><li>The authors also took it a step further and were able to identify where the results came from for each patient. For example, for some patients the clinical information contributed the most to the final radiotherapy-response-prediction, whereas for other patients the metabolomic information was important. By identifying these patient subgroups, the authors showed that radiation-response can be predicted for some patients easily if they have certain clinical features (e.g. certain cancer types, cancer grade, etc), but others might need more tests.</li></ul><p><strong><em>The integration of machine learning with genome-scale metabolic modeling represents a significant methodological advancement for identifying prognostic metabolite biomarkers and predicting radiosensitivity for individual patients.</em></strong></p><ul><li>The new technique of calculating metabolomic data (the small molecule concentrations) with Flux Balance Analysis was shown to be pretty good for figuring out which small molecules are associated with whether a patient would respond well to radiotherapy or not, and helped make machine models trying to solve this problem more accurate in general.</li></ul><p>—</p><p>You may also like:</p><ul><li><a href="https://medium.com/one-minute-machine-learning/recurrent-neural-networks-one-minute-summary-36832a7e3bd4">Recurrent Neural Networks | one minute summary</a></li><li><a href="https://medium.com/one-minute-machine-learning/bert-bidirectional-encoder-representations-from-transformers-one-minute-summary-c5e667e8630e">BERT (Bidirectional Encoder Representations from Transformers) | one minute summary</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=aae6087b13a7" width="1" height="1" alt=""><hr><p><a href="https://medium.com/one-minute-machine-learning/integration-of-machine-learning-and-genome-scale-metabolic-modeling-identifies-multi-omics-aae6087b13a7">Integration of machine learning and genome-scale metabolic modeling identifies multi-omics…</a> was originally published in <a href="https://medium.com/one-minute-machine-learning">One Minute Machine Learning</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Properly Interpret Error Bars]]></title>
            <link>https://medium.com/one-minute-machine-learning/how-to-properly-interpret-error-bars-4a6410625656?source=rss----cce48f05fc2f---4</link>
            <guid isPermaLink="false">https://medium.com/p/4a6410625656</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[significance]]></category>
            <dc:creator><![CDATA[Jeffrey Boschman]]></dc:creator>
            <pubDate>Wed, 08 Sep 2021 03:32:41 GMT</pubDate>
            <atom:updated>2021-09-08T03:32:41.085Z</atom:updated>
            <content:encoded><![CDATA[<h4>“The meaning of error bars is often misinterpreted, as is the statistical significance of their overlap.”</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dN8WBC83Oldwd41d5Dt-3w.png" /><figcaption><a href="https://www.nature.com/articles/nmeth.2659">Krzywinski, M., Altman, N. Error bars. <em>Nat Methods</em> <strong>10, </strong>921–922 (2013). https://doi.org/10.1038/nmeth.2659</a></figcaption></figure><p>Given a figure comparing some mean values with <strong>error bars</strong>, the author is likely trying to do one of two things: 1. show the<em> variability of observations in that sample</em> (<strong>descriptive statistics</strong>), or 2. show <em>how well the sample mean represents the population mean</em> (<strong>inferential statistics</strong>; where the term “statistical significance” comes into play).</p><p>Unfortunately, people often forget this and misinterpret error bars — for example, people often assume that a gap between bars of two sample means implies a statistically significance difference (i.e. <strong>statistical significance</strong>, given a standard P value of 0.05, means that it is less than 5% likely that the difference between the points is only due to random chance), but this isn’t always true because<em> it depends on what type of error bar is being used</em>.</p><p>The second article from Nature Method’s <a href="https://www.nature.com/collections/qghhqm">Points of Significance</a>, “<a href="https://www.nature.com/articles/nmeth.2659">Error Bars</a>”, talks about the differences between three common types of error bars— <strong>standard deviation, standard error of the mean, </strong>and<strong> confidence interval</strong>.</p><p><em>Prerequisite info: </em><a href="https://medium.com/one-minute-machine-learning/importance-of-being-uncertain-2013-one-minute-summary-69cf5cf4fcde">Sampling and estimation</a></p><h3>Standard Deviation</h3><p>Error bars based on <strong>standard deviation (s.d.)</strong> only tell us about the <em>spread of the population </em>(i.e. what values new samples taken from the population might have; part of descriptive statistics). They <em>do not</em> technically indicate anything about the uncertainty of a mean or statistical significance. You can think of a sample s.d. as an estimate of the variability of observations.</p><h3>Standard Error of the Mean</h3><p>Error bars based on <strong>standard error of the mean (s.e.m.)</strong> <em>do </em>reflect the uncertainty of the mean (i.e. indicate the <em>reliability of a measurement</em>; part of inferential statistics). You can think of s.e.m. as an estimate of the variability of possible values of means of samples, <em>as long as the sample size is large enough</em> (beware of small sample sizes). With non-overlapping s.e.m. error bars, you <em>cannot</em> assume that the <em>P</em> value is less than 0.05 (i.e. there is a significant difference between the means); as you can see in Figure 3 (below) from the article, s.e.m. bars are not overlapping even at <em>P</em> = 0.1.</p><h3>Confidence Intervals</h3><p>Error bars based on <strong>confidence intervals (C.I.)</strong> also indicate the <em>reliability of a measurement</em> (i.e. part of inferential statistics) — they indicates a range that would capture the population mean CI% of the time (e.g. if you have a 95% C.I., then if you collect 20 samples and calculate their means and C.I. ranges, then 19 of the ranges would contain the true population mean). You can relate C.I. to s.e.m. using the <em>t</em>-statistic, and for large sample size the s.e.m. is approximately a 67% C.I. With non-overlapping error bars, you <em>can</em> probably assume that the <em>P</em> value is less than 0.05, but this is a bit overkill as 95% C.I. bars only start to overlap at around 0.005 (Figure 3 from the article).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*A_ZXhRq4eE3URl652u-gDA.png" /><figcaption>Again, from <a href="https://www.nature.com/articles/nmeth.2659">Krzywinski, M., Altman, N. Error bars. <em>Nat Methods</em> <strong>10, </strong>921–922 (2013). https://doi.org/10.1038/nmeth.2659</a></figcaption></figure><h3>Summary</h3><p>You <em>cannot</em> get any information about statistical significance from s.d. error bars. You <em>can</em> draw conclusions about statistical significance from s.e.m. or 95% C.I. error bars, but whether or not they less than 0.05 p-value is <em>not intuitive</em>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4a6410625656" width="1" height="1" alt=""><hr><p><a href="https://medium.com/one-minute-machine-learning/how-to-properly-interpret-error-bars-4a6410625656">How to Properly Interpret Error Bars</a> was originally published in <a href="https://medium.com/one-minute-machine-learning">One Minute Machine Learning</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Recurrent Neural Networks | one minute summary]]></title>
            <link>https://medium.com/one-minute-machine-learning/recurrent-neural-networks-one-minute-summary-36832a7e3bd4?source=rss----cce48f05fc2f---4</link>
            <guid isPermaLink="false">https://medium.com/p/36832a7e3bd4</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[basics]]></category>
            <dc:creator><![CDATA[Jeffrey Boschman]]></dc:creator>
            <pubDate>Tue, 17 Aug 2021 22:31:05 GMT</pubDate>
            <atom:updated>2021-08-17T22:30:39.108Z</atom:updated>
            <content:encoded><![CDATA[<h4>This is a recurring concept that you should make sure you understand</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zR9YqwT53pM54OPpVdFkng.png" /><figcaption>Image idea from: <a href="https://blog.floydhub.com/a-beginners-guide-on-recurrent-neural-networks-with-pytorch/">https://blog.floydhub.com/a-beginners-guide-on-recurrent-neural-networks-with-pytorch/</a></figcaption></figure><p>Although Recurrent Neural Networks (RNNs) are relatively old (from the 1980&#39;s), they are still the basic model underlying a lot of speech, text, audio, and financial data applications. This article is an introduction to the basic idea of RNNs.</p><ol><li><em>Why?</em> Data like images can often simply be processed one at a time by a feed-forward network. However <em>sometimes data is not independent of other data</em>, and you actually need to: 1. input multiple pieces of data together because they rely on each other (e.g. words in a sentence), or 2. the current data is influenced by past inputs (e.g. financial data over time).</li><li><em>What?</em> A <strong>recurrent neural network (RNN)</strong> is a model that loops the information of multiple inputs back into the network in order to capture <strong>sequential</strong> or <strong>temporal data</strong>.</li><li><em>How?</em> In RNNs, an input is split into multiple elements (a.k.a. <strong>time-steps</strong>) and fed into the model one at a time, with the output of each element being the “<strong>hidden state</strong>” that gets fed back into the model along with the next input element. <em>The actual weights of an RNN network stay the same across all the elements of a training example</em>, until the final output is calculated and that error used to update the weights.</li></ol><p><em>Related articles:</em></p><ul><li><a href="https://medium.com/one-minute-machine-learning/do-you-remember-the-key-points-of-long-short-term-memory-e49770f1272e">Long Short-Term Memory (1997) | one minute summary</a></li><li><a href="https://medium.com/one-minute-machine-learning/sequence-to-sequence-learning-with-neural-networks-2014-one-minute-summary-bce5e24c5e0c">“Sequence to Sequence Learning with Neural Networks” (2014) | one minute summary</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=36832a7e3bd4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/one-minute-machine-learning/recurrent-neural-networks-one-minute-summary-36832a7e3bd4">Recurrent Neural Networks | one minute summary</a> was originally published in <a href="https://medium.com/one-minute-machine-learning">One Minute Machine Learning</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[L1 Regularization | one minute summary]]></title>
            <link>https://medium.com/one-minute-machine-learning/l1-regularization-one-minute-summary-b407940e52f?source=rss----cce48f05fc2f---4</link>
            <guid isPermaLink="false">https://medium.com/p/b407940e52f</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[basics]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Jeffrey Boschman]]></dc:creator>
            <pubDate>Fri, 30 Jul 2021 01:44:16 GMT</pubDate>
            <atom:updated>2021-07-30T01:44:16.328Z</atom:updated>
            <content:encoded><![CDATA[<h4>Have you wrangled with concept of LASSO Regression?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*F5bZcgWIb8412dKusunCiQ.png" /><figcaption>L1 Regularization / LASSO Regression encourages sparsity</figcaption></figure><p><strong>L1 Regularization</strong> (also called <strong>LASSO regression</strong>) is used less often than L2 Regularization, but has some key advantages in some situations; the 80–20 Rule (a.k.a. the Pareto Principle), “80% of the consequences come from 20% of the causes”, comes to mind.</p><p><em>Prerequisite Info:</em> <a href="https://medium.com/one-minute-machine-learning/regularization-one-minute-introduction-1d81f42791fd">Regularization</a>, <a href="https://medium.com/one-minute-machine-learning/l2-regularization-one-minute-summary-c5c60b57e7a6">L2 Regularization</a></p><ol><li><em>Why?</em> For any given model, some weights will be more important than others. However, random noise during training will cause some of the less important weights to have influence. Therefore, one way to prevent a model from overfitting to noise (and <em>also make it easier to see which features are actually important</em>) is to get rid of weights that are less predominant.</li><li><em>What?</em> <strong>L1 Regularization</strong> is a technique to reduce model complexity by <em>zeroing out some less important weights</em> (i.e. it encourages <strong>sparsity</strong>), thereby making a model more visually <strong>interpretable.</strong></li><li><em>How?</em> L1 Regularization adds the <em>absolute value of a weight</em> as the penalty term to the loss function (multiplied by a <strong>lambda</strong> hyperparameter). This means that during gradient descent, all weights are repeatedly penalized, such that <em>only the weights that are important</em> <em>(i.e. are repeatedly increased in the positive direction because training examples use them, not just because of random noise) will survive</em>, and the rest will go to 0.</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b407940e52f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/one-minute-machine-learning/l1-regularization-one-minute-summary-b407940e52f">L1 Regularization | one minute summary</a> was originally published in <a href="https://medium.com/one-minute-machine-learning">One Minute Machine Learning</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[L2 Regularization | one minute summary]]></title>
            <link>https://medium.com/one-minute-machine-learning/l2-regularization-one-minute-summary-c5c60b57e7a6?source=rss----cce48f05fc2f---4</link>
            <guid isPermaLink="false">https://medium.com/p/c5c60b57e7a6</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[basics]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Jeffrey Boschman]]></dc:creator>
            <pubDate>Thu, 29 Jul 2021 04:39:47 GMT</pubDate>
            <atom:updated>2021-07-29T04:39:47.433Z</atom:updated>
            <content:encoded><![CDATA[<h4>You too can understand L2</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YZhjhbz_ttvR2-h_a2V_-g.png" /><figcaption>Math to really understand how L2 Regularization / Weight Decay works</figcaption></figure><p>One of the most common technique to prevent overfitting is called <strong>L2 Regularization</strong> (because it uses the L2 norm). It is also known as <strong>Ridge Regression</strong> (from the original <a href="https://www.math.arizona.edu/~hzhang/math574m/Read/RidgeRegressionBiasedEstimationForNonorthogonalProblems.pdf">1970 paper</a>) or <strong>Weight Decay</strong> (in deep learning frameworks, because that’s essentially what it does).</p><p>Prerequisite Info: <a href="https://medium.com/one-minute-machine-learning/regularization-one-minute-introduction-1d81f42791fd">Regularization</a></p><ol><li><em>Why?</em> <em>Large weights</em> in a neural network are often a sign of a <em>too complex network that has overfit the training data</em>. Therefore, one way to prevent a model from becoming too complex is to stop weights from becoming too large.</li><li><em>What?</em> <strong>L2 Regularization</strong> is a technique to <em>reduce model complexity</em> by lowering the weights of a model proportional to the square of each weight (therefore especially penalizing those that are highest), but <em>does not make them 0</em>.</li><li><em>How?</em> L2 Regularization adds the<em> squared magnitude of a weight</em> as the penalty term to the loss function (multiplied by a lambda hyperparameter). Because of the derivative, this means that during gradient descent <em>the weights that are higher get penalized more</em>, whereas weights that are lower won’t change that much (therefore preventing them from going to 0).</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c5c60b57e7a6" width="1" height="1" alt=""><hr><p><a href="https://medium.com/one-minute-machine-learning/l2-regularization-one-minute-summary-c5c60b57e7a6">L2 Regularization | one minute summary</a> was originally published in <a href="https://medium.com/one-minute-machine-learning">One Minute Machine Learning</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Dropout | one minute summary]]></title>
            <link>https://medium.com/one-minute-machine-learning/dropout-one-minute-summary-2a2b20de02f4?source=rss----cce48f05fc2f---4</link>
            <guid isPermaLink="false">https://medium.com/p/2a2b20de02f4</guid>
            <category><![CDATA[influential]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[basics]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Jeffrey Boschman]]></dc:creator>
            <pubDate>Thu, 15 Jul 2021 01:09:07 GMT</pubDate>
            <atom:updated>2021-07-15T01:38:54.805Z</atom:updated>
            <content:encoded><![CDATA[<h4>A regularization strategy motivated by a theory of the role of sex in evolution</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LslkNciBkAP4pfwF8Qr8rA.png" /><figcaption>Image modified from <a href="https://www.cs.toronto.edu/~hinton/absps/JMLRdropout.pdf">Srivastava, Nitish, et al. “Dropout: a simple way to prevent neural networks from overfitting.” <em>The journal of machine learning research</em> 15.1 (2014): 1929–1958.</a></figcaption></figure><p>The purpose of dropout is alluded to in the title of its 2014 paper by Srivastava et al: “<a href="https://www.cs.toronto.edu/~hinton/absps/JMLRdropout.pdf">Dropout: A Simple Way to Prevent Neural Networks from Overfitting</a>”. It has since become one of the most popular <a href="https://medium.com/one-minute-machine-learning/regularization-one-minute-introduction-1d81f42791fd">regularization</a> techniques.</p><p><em>Prerequisites: </em><a href="https://medium.com/one-minute-machine-learning/regularization-one-minute-introduction-1d81f42791fd"><em>Regularization</em></a></p><ol><li><em>Why?</em> <em>Theoretically</em>, a great way to make predictions with a given model is to train many versions of it separately and then average the outputs. However, this is a <em>really computationally expensive</em> idea.</li><li><em>What?</em> <strong>Dropout</strong> is a <em>regularization technique</em> that prevents overfitting by <em>approximating training multiple different neural network architectures in parallel</em>.</li><li><em>How?</em> During training, some hidden units are randomly ignored (“dropped out”). For each training pass, different hidden units are randomly dropped, <em>yielding a different “thinned” network each time</em>. This makes the network more robust by <em>preventing units from co-adapting</em> (correcting each other’s mistakes), and therefore helps prevent overfitting. There is no dropout during testing, and thus the weights are scaled down to compensate for more nodes being active.</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2a2b20de02f4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/one-minute-machine-learning/dropout-one-minute-summary-2a2b20de02f4">Dropout | one minute summary</a> was originally published in <a href="https://medium.com/one-minute-machine-learning">One Minute Machine Learning</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Vision Transformers | one minute summary]]></title>
            <link>https://medium.com/one-minute-machine-learning/vision-transformers-one-minute-summary-c1af0aa33c27?source=rss----cce48f05fc2f---4</link>
            <guid isPermaLink="false">https://medium.com/p/c1af0aa33c27</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[influential]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[basics]]></category>
            <dc:creator><![CDATA[Jeffrey Boschman]]></dc:creator>
            <pubDate>Fri, 09 Jul 2021 00:43:10 GMT</pubDate>
            <atom:updated>2021-07-09T17:54:03.150Z</atom:updated>
            <content:encoded><![CDATA[<h4>New idiom: “An Image is Worth 16x16 Pixels”</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rGi2u0IUNhQSm6CC8z2mdg.png" /><figcaption>Image modified from Dosovitskiy, Alexey, et al. “An image is worth 16x16 words: Transformers for image recognition at scale.” <em>arXiv preprint arXiv:2010.11929</em> (2020).</figcaption></figure><p>The 2020 paper: “<a href="https://arxiv.org/abs/2010.11929">An Image is Worth 16x16 Pixels: Transformers for Image Recognition at Scale</a>” by Dosovitskiy, A., et al (Google) introduced the Vision Transformer, which at first just seemed like a cool extension of NLP Transformers but which has now proved to be very effective for computer vision tasks.</p><p><em>Prerequisites: </em><a href="https://medium.com/one-minute-machine-learning/attention-is-all-you-need-2017-one-minute-summary-f3c7509f2144"><em>Transformers</em></a></p><ol><li><em>Why?</em> For computer vision tasks, all the best models have typically been ConvNets (e.g. <a href="https://medium.com/one-minute-machine-learning/deep-residual-learning-for-image-recognition-2015-one-minute-summary-aa94949b8fcf">ResNet</a>, <a href="https://medium.com/one-minute-machine-learning/very-deep-convolutional-networks-for-large-scale-image-recognition-2014-one-minute-summary-44a8f04586ab">Vgg</a>, <a href="https://medium.com/one-minute-machine-learning/going-deeper-with-convolutions-2014-one-minute-summary-dd0e11c4152">Inception</a>). But with Transformers being successful for NLP tasks, can they be used for computer vision as well?</li><li><em>What?</em> The Vision Transformer (ViT) is a modified NLP Transformer (encoder only) for image classification <em>without any convolutional layers</em></li><li><em>How?</em> An image is split into patches (16x16x3), which are then <em>flattened</em> and put into a<em> lower-dimensional embedding space</em>. An extra token is added to <em>each vector </em>(i.e. <strong>linearly-embedded</strong> patch) to denote its relative location in the image (i.e. <strong>positional embedding</strong>), and another, learnable token is added to the <em>entire sequence of vectors</em> to denote the class. The sequence of vectors is fed to the standard Transformer encoder, which has been modified with an extra fully-connected layer at the end for doing classification.</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c1af0aa33c27" width="1" height="1" alt=""><hr><p><a href="https://medium.com/one-minute-machine-learning/vision-transformers-one-minute-summary-c1af0aa33c27">Vision Transformers | one minute summary</a> was originally published in <a href="https://medium.com/one-minute-machine-learning">One Minute Machine Learning</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>