<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Sairam Sundaresan on Medium]]></title>
        <description><![CDATA[Stories by Sairam Sundaresan on Medium]]></description>
        <link>https://medium.com/@artofsaience?source=rss-a99fbdf99073------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*w24u5Ug1TxMXMyLMdZC0Bg.jpeg</url>
            <title>Stories by Sairam Sundaresan on Medium</title>
            <link>https://medium.com/@artofsaience?source=rss-a99fbdf99073------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 15:57:28 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@artofsaience/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[ParanorML activity: Avoiding ML casualties]]></title>
            <link>https://medium.com/@artofsaience/paranorml-activity-avoiding-ml-casualties-9251130d283f?source=rss-a99fbdf99073------2</link>
            <guid isPermaLink="false">https://medium.com/p/9251130d283f</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[ml-so-good]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Sairam Sundaresan]]></dc:creator>
            <pubDate>Thu, 16 Feb 2023 17:44:28 GMT</pubDate>
            <atom:updated>2023-02-16T17:44:28.609Z</atom:updated>
            <content:encoded><![CDATA[<h4>How to effectively safeguard your ML experiments</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*nHD_abG2S3lGTzlL" /><figcaption>Photo by <a href="https://unsplash.com/@clemhlrdt?utm_source=medium&amp;utm_medium=referral">Clément Hélardot</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p><em>Many years ago…</em></p><p>I was staring at my screen, scrutinizing every little wiggle on my Tensorboard. Something was off. Problem was, I couldn’t put my finger on it. I had been trying to train a model for a couple of weeks now (Yes, that’s not a typo. Weeks). The validation accuracy would rise consistently for the first 30 epochs of training and then BAM! It would plunge into the depths of <a href="https://en.wikipedia.org/wiki/NaN">NaN-sanity</a>.</p><p>It made no sense. I had checked and re-checked everything too many times. The model architecture, the hyperparameters, the dataset, the training script, and whatever else you can imagine. No matter what I changed, the same behavior persisted. At the 31st epoch, things would go ballistic.</p><p>At this point, half exasperated, I turned to a colleague and asked them to look through it. Fresh eyes spot things that fatigued retinas miss. My colleague didn’t see anything out of the ordinary but asked me a question that changed my approach to training these models forever.</p><blockquote>“What are your tests telling you?”</blockquote><p>Yes, I had logged everything meticulously, visualized all the key plots and metrics, and documented the code to an inch of its life.</p><p>But I hadn’t comprehensively tested it.</p><p>Eventually, I figured out the issue (it involved a stray random tensor that was conditionally invoked at the 30th epoch). But, I lost weeks of time and effort in the process. There began a practice that I continue to this day and would like to share with you so that you can avoid the frustrations and head-banging (not the fun kind) that I went through.</p><p>The art of <strong>ParanorML Practitioning</strong>.</p><p>Before we dive into what it is, let’s understand why it’s needed in the first place.</p><h3>Traditional software vs Machine learning</h3><p>In traditional software engineering, you write a piece of code, and you know what output to expect. If you don’t get said output, there are deterministic ways to figure out why and fix any errors in the process.</p><p>Why? You write the instructions/rules for the computer to follow. If it doesn’t work, your instructions are wrong. The logic to solve the task is the input.</p><p>With machine learning, you write a piece of code to define a model, share with it lots of data repeatedly, and the expected output. If you don’t get said output, there isn’t a deterministic way to figure out why.</p><p>Why? The model learns the instructions/rules to follow by itself from the data you provide. If it doesn’t work, your implementation could be wrong, the data might be wrong, the output might be underspecified, or a ton more reasons. The logic to solve the task isn’t the input anymore.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*gJ35cx-5TFwfN24E" /><figcaption>Photo by <a href="https://unsplash.com/fr/@thisisengineering?utm_source=medium&amp;utm_medium=referral">ThisisEngineering RAEng</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>In the words of Andrej Karpathy, <a href="http://karpathy.github.io/2019/04/25/recipe/">“It fails silently”</a>.</p><p>Silent failures cost you. Not just in terms of the result but the time and cost to run these experiments. Most experiments take a few days to run, if not weeks, and small issues are incredibly hard to uncover until much later. Compute power isn’t cheap either and the less said about deadlines, the better. Planning machine learning experiments within a fixed timeframe is a logistical nightmare.</p><p>Blindly trusting that everything will work is like <a href="https://en.wikipedia.org/wiki/The_Ring_(2002_film)">watching a cursed videotape</a>. Your ML experiment will perish in 7 days.</p><p>The most <em>amazing</em> thing about modern machine learning is that it somehow finds a way to work. Even if there’s a problem.</p><p>The most <em>infuriating</em> thing about modern machine learning is that it somehow finds a way to work. Even if there’s a problem.</p><p>So how do you tackle this devilish beast? Who you <a href="https://www.youtube.com/watch?v=Fe93CLbHjxQ">gonna call</a>?</p><h3>The art of being paranoid</h3><p>The core of the problem here is that failure in ML is silent, non-deterministic, and costs a lot of time. To combat it, we must strive to make even the slightest error loud and as deterministic as possible while minimizing turnaround time.</p><p>The best way I’ve found to do this is by being paranoid when designing experiments.</p><p><strong>I do this through three overarching principles:</strong></p><ol><li>Test-driven development</li><li>Embrace simplicity and delay abstraction until needed</li><li>Prioritize small and quick experiments before scaling up</li></ol><p>Now, these aren’t new and definitely not the best thing since sliced bread, but by adopting these principles (which I call ParanorML Practitioning), I’ve cut down wasted time and effort from days to hours to minutes.</p><h3>A test is the first user of your code</h3><p>The heading above is one of my favorite quotes from the book “The Pragmatic Programmer”. In it, the authors argue that by shining the light of a test on your code, things become clearer. Test-driven development or TDD is a school of programming that focuses on writing tests first, seeing them fail, and then writing code to make the tests pass. The intuition behind this is that if thinking about testing is so beneficial, then why not write them upfront?</p><p>Diving into the nuances of TDD is beyond the scope of this article. Instead, I’ll focus on how I use it for my ML work.</p><p>Let’s say I’m building an image classifier. Years ago, I would have jumped straight into model architecture, training, and scaling up the pipeline to run on many compute devices.</p><p>Today, I know better. I first start with some questions. What will the end use case look like? Who will call the API I write (another team/ a customer / my grandma)? I write a test to verify that call. Obviously, it will fail since there’s no actual model code written. Then, I write code to make this test pass.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*7R35fYyC0oyIIFGu" /><figcaption>Photo by <a href="https://unsplash.com/@towfiqu999999?utm_source=medium&amp;utm_medium=referral">Towfiqu barbhuiya</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>I look at the data next and validate some hypotheses:</p><ul><li>The data is balanced</li><li>It is unbiased (LOL, this is never true, but I do it just for vibes)</li><li>The labels are correct</li><li>The data splits are stratified</li><li>There is no data leakage</li></ul><p>The list goes on…</p><p>By being paranoid about the tiniest things that can go wrong, you can save yourself from a world of pain later. Say your dataset has images of airplanes, ships, and mitochondria (I don’t know, it’s a weird dataset). I can bet you a fortune that the model will confuse images of airplanes as ships. Why? The background for images of ships is usually blue since they float on the ocean. The background for airplanes in flight is usually blue since they are surrounded by the blue sky. Now your model doesn’t know apriori that both sky and water are blue. Thus when it sees the color blue, it biases toward either a plane or a ship depending on which class has more images.</p><p>When the inevitable model failure occurs, you’ll be busy debugging the code when the issue is actually with the data. Be paranoid instead.</p><p>The same holds true when you write code for the model and the overall pipeline.</p><p>When you code up the model architecture, check if the tensor shapes are correct. You can do that by passing a dummy tensor through the model. If you use an activation function before returning the output from the network, make sure you use the correct loss function. You can write simple unit tests for each of these situations and verify that things work as they should.</p><p>Your tests serve as the scaffolding around which you build your architecture (pun intended).</p><p>Given how “templatized” training and validation scripts have become, it can be tempting to copy over the pipeline from another project and just run with it. Regardless of whether you give in to that temptation or not, add a bunch of tests to check that the pipeline is doing exactly what you want. For example, Is the data preprocessed correctly? Why? If you use augmentations, you want to make sure that things look good after the data has been modified. Also, it’s generally good practice to normalize the data. So, naturally, check if it’s been normalized properly. Are the metrics the right ones to use? Are the outputs properly generated? Don’t assume. Write tests first and validate!</p><p>I can keep on listing things like these but you get the drift.</p><p>We live in a world where Co-pilot and ChatGPT have become our rubber duckies. They write code for us as soon as we think of which keys to press on our fancy Keychron Q1 pro mechanical keyboards. Resist the temptation to blindly trust their <em>memorized</em> wisdom. Use them, but with a healthy serving of ParanorML Practitioning.</p><h3>Delayed gratification abstraction</h3><p>The second trap I used to fall into was to “fancify” every piece of code I wrote. Forgive me, but I come from a C++ background, where beautiful object-oriented code is celebrated by the <a href="https://www.goodreads.com/book/show/85009.Design_Patterns">four horsemen</a> who gave us design patterns.</p><p>Python, at least to my knowledge, doesn’t exactly play by those rules. Members of a class can be manipulated from anywhere if we’re not careful. This can lead to unexplained behavior like my stray tensor issue from years ago. Writing code with the sheer aim of making it beautiful and abstract is like garnishing a moldy paella with fresh organic cilantro. From a distance, it looks rustic and artisan. Up close, the code smells.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*McJ2WTmzjDw4R12S" /><figcaption>Photo by <a href="https://unsplash.com/@steven__chan?utm_source=medium&amp;utm_medium=referral">Ting Tian</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>That isn’t rustic paella… it smells</p><p>These days, I focus on writing functions and then spruce them up depending on the use case. Combined with my paranoid test-driven approach, this yields consistently reliable results. When you need to refactor code from a Jupyter notebook to beautiful OOP files, the tests you’ve written will help validate your conversion efforts. Win-win!</p><p>Now, I’m not saying abstracted code is bad by any stretch of the imagination. I’m just saying it’s better to use it when needed. When starting out, always bias toward simplicity and gradually take the ferry towards complexity island.</p><h3>The unreasonable effectiveness of small bets</h3><p>Each day, ML research unveils shiny new models that do incredible things. Do you remember the time when we were gobsmacked by a model generating a high-resolution <a href="https://futurism.com/the-byte/google-ai-burgers-gan">image of a burger</a>? How naïve were we?</p><p>The problem is though, these new models take up tons of compute. Even with the equivalent of half a nation’s servers, these take weeks to train before producing meaningful results. Naturally, replicating these experiments or trying new ones ourselves at this scale is infeasible and often leads to wasted effort, many cups of coffee (not a bad thing), and raccoon eyes.</p><p>The solution is to hedge our bets on smaller-scale experiments — Try a bunch of things and see what works. Then take the most promising ideas and scale them up. I’ve found this saves me so much time and allows me to iterate very quickly given my limited compute resources and time.</p><p>Jeremy Howard, the creator of fast.ai religiously follows this approach. It’s something he taught to students in his recently concluded Stable Diffusion course. Here’s an <a href="https://www.youtube.com/watch?v=HhGOGuJY1Wk">excerpt from a podcast</a> where he talks about this approach.</p><blockquote>I’ve been working on is exactly what you describe. Which is, how to train and play with a state-of-the-art image-generative model in a notebook on a single GPU. I’m doing all my work — just about — on the Fashion-MNIST dataset. Which, rather than being 512x512 pixel images of literally anything in the world, including artworks, in three channels, Fashion-MNIST is 28x28, single-channel images of 1 of 10 types of clothing. I always tell people — whether you’re doing a Kaggle competition, or a project at work, or whatever — the most important two steps are to <strong>“Create a rapid feedback loop where you can iterate and test fast”</strong>, and to <strong>“Have a test which is highly correlated with the final thing you’re going to be doing”</strong>.<strong> </strong>If you have those two things, you can quickly try lots of ideas, and see if they’re probably going to work on the bigger dataset, or the harder problem, or whatever.</blockquote><p>By working on small-scale experiments, you enable a rapid feedback loop. It also allows you to verify that you have tests that correlate with your end goal and make adjustments as needed.</p><p>So there you have it. My secret sauce for fewer machine learning casualties. Test ahead, embrace simplicity and bet heavily on small-scale experiments.</p><p>Most importantly, be paranoid and become a ParanorML Practitioner.</p><h3>🤖💪 Want more ideas to be a productive ML practitioner?</h3><p>Each week, I send out a newsletter with practical tips and resources to level up as a machine learning practitioner. <a href="https://newsletter.artofsaience.com/"><strong>Join here for free →</strong></a></p><h4><a href="https://mlearning.substack.com/about"><strong>BECOME a WRITER at MLearning.ai</strong></a></h4><p><a href="https://medium.com/mlearning-ai/mlearning-ai-submission-suggestions-b51e2b130bfb">Mlearning.ai Submission Suggestions</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9251130d283f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[“Practically” Building an Image Classifier]]></title>
            <link>https://medium.com/data-science/practically-building-an-image-classifier-531443b3b483?source=rss-a99fbdf99073------2</link>
            <guid isPermaLink="false">https://medium.com/p/531443b3b483</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Sairam Sundaresan]]></dc:creator>
            <pubDate>Tue, 17 Jan 2023 19:39:29 GMT</pubDate>
            <atom:updated>2023-01-17T19:39:29.252Z</atom:updated>
            <content:encoded><![CDATA[<h4>Building an image classifier the practical way (based on a true story)</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*F0gdftRm3h1RBAwQ" /><figcaption>Photo by <a href="https://unsplash.com/@miracleday?utm_source=medium&amp;utm_medium=referral">Elena Mozhvilo</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Most practitioners first learn deep learning through the lens of image classification. If you’ve taken a programming course, then consider this to be the equivalent of the “hello world” exercise they give you at the start. While many experts say image classification is solved (and there may be some truth in that), there’s a lot of nuance and craft you can learn from this type of problem. With that being said, over the next couple of editions, I’ll share how I build, evaluate and analyze these models through the form of a story. Initially, I considered consolidating everything into a single article. But to spare you from reading my response to Tolstoy’s “War and peace”, I’ve broken it down into pieces. <strong>Note:</strong> All the code used in this pose can be found <a href="https://github.com/ssundar6087/practically_building_an_image_classifier">here</a>.</p><h3>So you think you can classify?</h3><p>To motivate this problem, let’s consider that you’ve been hired as a Computer Vision (CV) engineer at Petpoo Inc, a pet shampoo company. Now you can ask me why in the world would a pet shampoo company need a CV engineer. Even more importantly, why for all the gold on earth would they call themselves Petpoo? I will answer the first question below, but I honestly don’t know the answer to the second.</p><p>On your first day, your manager meets you and walks you through your project.</p><p>“Hi INSERT_YOUR_NAME_HERE! We’re thrilled to have you here. Petpoo is scaling up and one of the most requested features from our customers is custom shampoo recommendations for their pets. Our customers are social media savvy and want their petfluencers to have the best perm in reels and selfies that they share. They also don’t have time to fill out lengthy questionnaires because they’re busy teaching their pawed companions the perfect <a href="https://en.wikipedia.org/wiki/Duck_face">duck face</a>. So we’re building a solution to recognize pets based on their pictures. Once we know what kind of breed a pet is, we can recommend custom shampoos from our product lines that will make the pet and its owner happy.</p><p>Since a majority of our customers have cats or dogs, we’d like you to build the image classifier to recognize if a pet is a cat or a dog. For starters, let’s keep it at this level. Based on the traction we receive after launch, we can add more features to the solution. Good luck, we’re all counting on you.”</p><p>You ask, “What data do we have for prototyping?”.</p><p>“Good question. You can start with this <a href="https://www.robots.ox.ac.uk/~vgg/data/pets/">dataset</a>”, he replies.</p><p>“How are we going to evaluate performance?”, you question.</p><p>“For the prototype, use your best judgment.”, he responds and marches off.</p><p>Ok, so you need to build an image classifier to distinguish between cats and dogs. You have a dataset to start with and have to figure out how to evaluate the prototype.</p><p>You decide to look at the data first. After all, if the data is bad, no amount of machine learning shampoo can save it.</p><p>At first glance, this looks like properly processed data that you can start playing with right away. There are nearly 7500 images in total, so that should be sufficient for fine-tuning. The dataset has classification labels for specific pet breeds so you’ll need to modify it so that each image is associated with a cat or dog label instead. Ok, a simple function to do that and we’re all set.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1018/0*cBFByd-dWRImQL2T.png" /><figcaption>Sample images and labels: Image created by author</figcaption></figure><p>So a cat image will have the label “True” while a dog image will have the label “False”.</p><p>Before starting your experiments, you split the data into training and validation sets. You’re smart enough to know that simplicity is always better than sophistication. So, as a first step, you take a well known deep learning model, a ResNet-18 as a baseline. You then turn off all the fancy knobs like schedulers, data augmentation, pretrained weights, and regularization, and pass a batch of data through it. When you check the predictions of the model, you see something like this:</p><pre>(TensorBase([0.0056, 0.9944]), TensorBase(1.))</pre><p>Good, the probabilities add up to 1. That means that the basic setup is ok. Feeling confident, you train the model for a few epochs and watch what happens. The loss seems to decrease, but for such a simple problem with such a <em>clean</em> dataset, the accuracy is underwhelming.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1013/0*OTawFL0Q0wK25GFm.png" /><figcaption>Results after 10 epochs (No transfer learning): Image created by author</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/372/0*GB7GAPvRN2htjMLQ.png" /><figcaption>Training curves: Image created by author</figcaption></figure><p>Wanting to make a good first impression, you visualize the examples that were misclassified and also plot the confusion matrix.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/278/0*1s3c0oHKYszJEaaH.png" /><figcaption>Classification results: Image created by author</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/683/0*UEPoY8OREsW8M2_G.png" /><figcaption>Misclassified examples: Image created by author</figcaption></figure><p>Hmmm, it seems like a whole bunch of dogs are being confused as cats by the model. Strange. Very strange.</p><p>You reason that this is because there are insufficient training examples for training from scratch. So you decide to use a technique that has served you well in your career — transfer learning.</p><p>After initializing the Resnet-18 with pretrained weights, you repeat the same experiment, keeping everything else intact. The results from this run look awesome. Almost zero errors and wonderful training curves.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/996/0*VXsf6DvH2V4qZna0.png" /><figcaption>Results after 10 epochs (with transfer learning): Image created by author</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/372/0*ei7N45cuIAmJs-dj.png" /><figcaption>Training curves with transfer learning: Image created by author</figcaption></figure><p>You repeat the same visualization as before and find that only a handful of dogs are being confused with cats.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/278/0*HISq83sFp_6C6hUm.png" /><figcaption>Confusion matrix looks much better! Image created by author</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/683/0*8jpJ88SWlriqPEpA.png" /><figcaption>Misclassified examples: Image created by author</figcaption></figure><p>Happy that your new job is off to a good start, you save all your work and go home in the evening.</p><p>Later, much later. 1:30 am… You wake up startled as beads of sweat trickle down your forehead.</p><p>You’ve missed a very important detail while setting up the experiment. Accuracy might not be a good metric for this problem at all.</p><p>Why?</p><p>On the next day, you rush to work and check if your intuition was right. While you inspected sample images and calculated the total size of the dataset, in your rush to get results quickly, you forgot to check how many cat and dog images there were. The dataset was originally designed to classify different breeds of dogs and cats. It may have been balanced (similar number of examples for each class) for that task.</p><p>You never checked if it had a similar number of images for cats vs dogs. Feverishly typing on your keyboard, the deafening silence that follows the clackety-clack of the keys confirms your suspicions.</p><p><strong>Cats:</strong> 2400 or 32.4% of the total data</p><p><strong>Dogs:</strong> 4990 or 67.6% of the total data</p><p>That explains why the first model you trained sucked. The data imbalance confused the model. There were more than double the number of dogs as there were cats in the dataset. The model couldn’t gather enough information to distinguish cats and dogs because most of the images it saw were dogs. But why did the second model you trained work almost flawlessly? You initialized it with weights that were trained on 1.2+ million images many of which were of dogs and cats.</p><p>Problem solved! No worries right? Not exactly. Transfer learning (what you did for the second model) doesn’t always work. Transfer learning might fail when there’s a domain mismatch between the pretraining dataset (what the model was originally trained on) and the target dataset (what you are training it on). You got lucky because the pretraining dataset had a similar distribution to the pets dataset you were working with (phew!).</p><p>Here’s the next mistake — Defaulting to accuracy as a metric. Look at the confusion matrices side by side. On the left, you have your original experiment, and on the right, you have the transfer learning experiment. For reference, the rows are the ground truth. The first row corresponds to actual images of dogs, and the second row corresponds to actual images of cats. The first column represents what the model *thinks* are dogs and the second column represents what the model considers to be cats.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*913X57Y2Far5FENs.png" /><figcaption>Confusion matrics side by side: Image created by the author</figcaption></figure><p>If you looked at the raw accuracy numbers, you’d have seen a score of 86% accuracy without transfer learning and a score of 99.7% with transfer learning. Not too bad right?</p><p>However, in the confusion matrix on the left, you can see that 44 dogs were classified as cats by the model, and 162 cats were classified as dogs. That’s a huge problem you’d have missed by blindly relying on accuracy alone. Why does this happen?</p><p>Imagine a worst-case scenario where you only had 90 dog images and 10 cat images and no access to pretrained weights. If you guessed dog as the answer for all 100 images, you’d have an accuracy of 90%. You’d also have a completely useless model.</p><p>Thus, on another occasion, using accuracy as a metric would have hidden this fundamental flaw in your model until it was too late. I mean, what if chihuahua totting socialite Haris Pilton had used this, found it useless, and then condemned the product on her social media channels?</p><p>As you thank your lucky stars and consolidate results from transfer learning, your manager comes by. “Hey there! How’s your work coming along?”, he asks. “Good, I think I’ve got something.”, you reply nervously. You show him your work with transfer learning and the 0.003 error rate. He is floored. “How soon can we build a demo for this?”, he asks. “I can start putting one together this week”, you respond.</p><p>“Our marketing team has connections with top influencers like Haris Pilton, and we’d like for them to try this out and give us feedback.”, he grins. “Umm… Ok”, you gulp as you think back to your gaffe. “Alright, let’s aim for the end of next week. Chop Chop!”, he says, patting you on the shoulder, walking away with delight.</p><p>The scribble of pen on paper is punctuated by pregnant pauses. Even though you’re working on designing the app, your thoughts are elsewhere. What should I have used to evaluate the model? How could I have balanced this wonky dataset? Why did I miss these basic steps?</p><p>Over the next few days, you build a simple app interface to test the model and see how it works on images from the real world. After all, what’s the point of training a model if you can’t use it for practical applications?</p><p>Happy with the initial prototype, you divert your attention to the pressing questions in the back of your head. Opening up a reference textbook, you look through F1-score, ROC, <a href="https://towardsdatascience.com/precision-and-recall-88a3776c8007">precision, and recall</a>, and all those ideas come flooding back. You rerun the experiments, but in addition to measuring accuracy, you also measure precision and recall. The results make way more sense now — The precision and recall numbers are low for the model without transfer learning while the accuracy hides these flaws. On the other hand, precision and recall are really high for the transfer learned model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*sIDLIBeuQ7Cm62w-.png" /><figcaption>Without transfer learning — Meh precision and recall: Image created by author</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*M64Cn_yvxXmJ-dCk.png" /><figcaption>With transfer learning — Awesome precision and recall: Image created by author</figcaption></figure><p>You decide to write a checklist of how one should train a model and evaluate it — just in case you or someone else on your team needs a reference in the future.</p><p>But before you can start, your manager comes by. You show him the app-in-progress and he’s delighted. “This is great! Our marketing team spoke to some of the influencers, and they’d like the app to be even more customized — recommendations per breed. I tried talking them out of it, but they’ve committed to having that ready when the influencers visit HQ by the end of next week. Think you have that ready by then? I understand this is going to take some hardcore engineering.”, he says apologetically. “Granular classification? By the end of next week? That’s a tough ask. Can I have another pair of hands to work with me on this?”, you ask surprised by the sudden change in requirements and the receding timeline. “Umm…”, he pauses…</p><p>(The conclusion of this series will drop sometime in 2023 — It’s still in pre-production)</p><p><strong>Here are my questions for you:</strong></p><p>How would you handle this new task? What do you think the best approach is given these constraints? What should you be mindful of?</p><p>There are no right answers, only tradeoffs. Drop me a note with your ideas or leave a comment in this post.</p><h4>Dataset details:</h4><p><a href="https://www.robots.ox.ac.uk/~vgg/data/pets/"><strong>Oxford Pets dataset</strong></a><strong> — </strong>The dataset is available to download for commercial/research purposes under a <a href="https://creativecommons.org/licenses/by-sa/4.0/">Creative Commons Attribution-ShareAlike 4.0 International License.</a> The copyright remains with the original owners of the images.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=531443b3b483" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/practically-building-an-image-classifier-531443b3b483">“Practically” Building an Image Classifier</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A generational year for AI]]></title>
            <link>https://medium.com/@artofsaience/a-generational-year-for-ai-585d5174b4a7?source=rss-a99fbdf99073------2</link>
            <guid isPermaLink="false">https://medium.com/p/585d5174b4a7</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[2022]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[ml-so-good]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Sairam Sundaresan]]></dc:creator>
            <pubDate>Wed, 04 Jan 2023 18:48:01 GMT</pubDate>
            <atom:updated>2023-01-04T18:48:01.479Z</atom:updated>
            <content:encoded><![CDATA[<h4>Recapping the best of AI from a whirlwind 2022</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*FSa0GRvAuMEwv5DB" /><figcaption>Photo by <a href="https://unsplash.com/@benwhitephotography?utm_source=medium&amp;utm_medium=referral">Ben White</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>A buzzing sense of unease and tension lingers in the air. No one dares to move a muscle. Well over two hours have elapsed, yet the outcome is still unclear. This will be settled in the space of 12 yards. Bearing the weight of his nation on his bruised shoulders, Gonzalo Montiel walks up to the penalty spot hoping to write history for Argentina and in the process, rectify his error in the 117th minute of a breathless game that led to this moment. The football gods have given him a second chance to redeem himself. Calmly, he slots the balls to the right of opponent Hugo Lloris who guesses wrong and agonizingly watches the ball nestle in the corner of the net with a satisfying thump. The lighter of the two blues erupts in celebration. In the background, A 24-year-old Kylian Mbappe shakes his head in grief and disbelief. His hat trick wasn’t enough for France on this day. His club teammate and legend, Lionel Messi shakes his head disbelievingly in joy. He has finally completed his medal collection. His legacy is assured.</p><p>The 2022 football world cup was a spectacle that captured the imagination of millions around the world from the casual fan to the devout analyst. A score of 3–3 after extra time and only the third penalty shootout in the history of the world cup finals to decide the winner of an epic battle. What a game.</p><p>No, you’re not reading a sports article. The machine learning community witnessed this kind of excitement and non-stop action over a span of 12 months, not just for 120 minutes. Machine learning research has made strong inroads in protein folding, solved a 53-year-old math question, and generated incredible images, conversations, and music with the coolness of Emi Martinez, Argentina’s heroic goalkeeper in the shootouts. There has been no paucity of action.</p><p>If the 2022 world cup belongs to the generational talent of Messi, the 2022 equivalent in machine learning belongs to generative AI.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*VhwpAyGqBhl9vrcA.png" /><figcaption>Midjourney’s interpretation of AI artist : Created by Author with Midjourney</figcaption></figure><h3>Imagen Dall-E back to life</h3><p>It’s hard to find a good starting point to reflect on this whirlwind of a year, but I’ll venture a guess that most people would start with image-generating models. So let’s start there.</p><p>Until recently, Generative Adversarial Networks (GANs) were the kings of the hill when it came to image generation. It took researchers an idea from a completely unrelated field, <a href="https://arxiv.org/abs/1503.03585">non-equilibrium statistical physics</a>, to knock these networks off the top spot. The result was a new class of models called Diffusion models. Although an exciting breakthrough, they remained in the shadow of GANs for many years.</p><p>But in April 2022, OpenAI introduced DaLL-E 2 leveraging the latent power of diffusion, and Pandora’s box was opened. Millions of users beta-tested the model, and soon, the internet was flooded with incredible imagery that was almost indistinguishable from “real” images. Whether it was an avocado on the moon or recreations of the Mona Lisa, nothing was impossible anymore.</p><p>Not to be left behind, Google released not one but two models, <a href="https://imagen.research.google/">Imagen</a> and <a href="https://parti.research.google/">Parti</a>. Both models produced excellent results but had fundamental differences in their underlying architectures. Imagen used the diffusion approach while Parti was an auto-regressive model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*4OzHZyukb9eKZ2tT.jpeg" /><figcaption>Imagen from Google [1]</figcaption></figure><p>While this research allowed for exciting creative possibilities and the rise of products like LensaAI and AvatarAI, it wasn’t without controversy. Using Midjourney (a solution for users to create images from text), a Colorado man won a local state fair leading to huge outcries from the artistic community. Traditional artists were outraged when they learnt that their work had been used to train these models without their permission. What made things worse was that these generated works of art were flooding websites where these artists sold their art, and they weren’t being compensated for it in any way. Lensa AI’s app produced derogatory images when women tried the app.</p><p>In 2023, researchers will be looking to further the nascent efforts in <a href="https://makeavideo.studio/">text-to-video</a> and <a href="https://dreamfusion3d.github.io/">text-to-3D</a>. The practical implications of these models will be an interesting area to follow, especially when it comes to the metaverse, deep fakes, and copyright issues. Creatives might be scared that their jobs could be stolen. However, I feel that this will unlock the latent creativity and imagination of many who finally have the tools to create from simple expression. In fact, I feel experienced creators have the most to gain from this. Concept art will be infinitely easier to create, and “creator’s block” will no longer be an issue.</p><p>For researchers, there are still many interesting problems to solve — How to make generation faster, how to deploy these models on edge devices (your phones), how to improve the quality of generation and how to make prompts closer to conversational English.</p><p>Finally, NeRFs (Neural Radiance Fields) made some great headway in a number of areas. NeRFs can synthesize new views of a scene from a limited number of example views. Say I show you a few pictures of a scene, then hide them. If I ask you to close your eyes and imagine how that scene would look from a different angle, you’d probably be able to do it right? That’s kind of what NeRFs do. Previously, NeRFs were limited in what they could “imagine”. Google’s Mip-NERF 360 found a way to generate unbounded views of a scene in all directions, something which was unheard of. Just look at the video below to see what I mean:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FzBSH-k9GbV4%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DzBSH-k9GbV4&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FzBSH-k9GbV4%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/8ea4f222a756792ab7a8601e02224f48/href">https://medium.com/media/8ea4f222a756792ab7a8601e02224f48/href</a></iframe><p>Another limitation of NeRFs is the sheer amount of time they take to generate these new views. Plenoxels shortened this time significantly (Traditional NeRFs were slooooooooooowwwwwww) — from days to minutes.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FKCDd7UFO1d0%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DKCDd7UFO1d0&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FKCDd7UFO1d0%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/a495fa058ab79cb06a19409a46a20d16/href">https://medium.com/media/a495fa058ab79cb06a19409a46a20d16/href</a></iframe><p>My amazing collaborators (and yours truly) used NeRFs to generate new views of our Sun from satellite measurements.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FK_U37sRLoqU%3Fstart%3D3034%26feature%3Doembed%26start%3D3034&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DK_U37sRLoqU&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FK_U37sRLoqU%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/2853c8fee82ac9902a858496944dd209/href">https://medium.com/media/2853c8fee82ac9902a858496944dd209/href</a></iframe><p>Later in the year, researchers were able to generate <a href="https://ai.googleblog.com/2022/11/infinite-nature-generating-3d.html">“fly-throughs” using NeRFs</a> that led to some really amazing moving scenery.</p><p>Can you imagine Google maps powered by this technology? How easy it would be to find a mom-and-pop store on a random street corner if you could “fly through” the route on your phone. That possibility might be closer than you think.</p><p>All right, all right, you’re asking why I haven’t mentioned a peep about ChatGPT. Let’s switch gears to language models.</p><h3>Conversational AI — waddle aside rubber ducky</h3><p>A lot of programmers I know use the rubber ducky method. When you run into a bug while coding (and you will because that’s life), you simply speak aloud sharing your thought process with a rubber ducky in front of you. Magically, you find the error and the solution.</p><p>In 2022, the rubber ducky was replaced with two smarter duckies — <a href="https://github.com/features/copilot">Github Copilot</a> and ChatGPT. Copilot is an autocomplete system for code that is trained on tons of open-source code. Github opened access to it this year for all users. ChatGPT upped the bar significantly, learning from feedback and becoming much more conversational than models past (If you’re curious about how it works, I wrote about it just <a href="https://newsletter.artofsaience.com/p/gpt-gets-chatty-the-future-of-artificial">last week here</a>).</p><p>There were other significant notables I’d be remiss if I didn’t mention. <a href="https://www.deepmind.com/blog/competitive-programming-with-alphacode">Alphacode</a> from DeepMind, launched early this year was able to solve challenging coding problems. After all, it was trained on code submitted to competitive programming contests in a dozen programming languages. In fact, DeepMind found that it was within the top 54% of participants in programming competitions!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*2CGuPz22WW70mC5S.png" /><figcaption>Midjourney’s version of conversational AI: Created by the Author with Midjourney</figcaption></figure><p>If you’re like me, you’ll know how challenging Leetcode interview problems are to solve under severe time constraints. Now, imagine solving problems that make these look like a piece of cake and finishing within the top half of coding experts. That’s a pretty nifty achievement if you ask me. While these models won’t replace human programmers anytime soon, they’re wonderful tools to help us write and debug code more easily.</p><p>Meta released <a href="https://arxiv.org/abs/2208.03299?utm_campaign=The%20Batch&amp;utm_source=hs_email&amp;utm_medium=email&amp;_hsenc=p2ANqtz-8k041XCLsyq6zi214DpkngslM2K3AmJL6ObTN1AYHTLwA0klJ5yV3eF33N8WFD5OoL3c9O">Atlas</a> and later, <a href="https://www.technologyreview.com/2022/11/18/1063487/meta-large-language-model-ai-only-survived-three-days-gpt-3-science/">Galactica</a>. Atlas was a question-answering model that primarily retrieved information from a database of documents. Galactica on the other hand only survived <a href="https://www.technologyreview.com/2022/11/18/1063487/meta-large-language-model-ai-only-survived-three-days-gpt-3-science/">3 days online</a> before being taken down. It was a language model trained on scientific and technical subjects. But, it was prone to generating fake information and citing sources that didn’t exist.</p><p>Google search won’t be replaced anytime soon, folks.</p><p>Reducing these models’ propensity to generate misinformation and hallucination will be a key challenge for researchers going into next year.</p><h3>LOTR meets large models</h3><p>In addition to the amazing progress made above, researchers have also tried to scale up these models to perform more than one function — hundreds of tasks in fact.</p><p>Simply put, imagine if you had one model to rule them all. You’d have to train such a model once and then finetune it as needed for various tasks. That would be so beneficial not only from a generalizability perspective but also in reducing carbon emissions. These large models can chalk up electricity bills that dwarf the annual bills of a few countries.</p><p>In this context, two notable models come to mind this year. First, Google released <a href="https://ai.googleblog.com/2022/04/pathways-language-model-palm-scaling-to.html">PaLM</a> which could show state-of-the-art performance on several language understanding and generation tasks. It could outperform humans in some. The other was <a href="https://www.deepmind.com/blog/a-generalist-agent">Gato</a> from DeepMind. This could learn over 600 different tasks. Whether it was playing atari games, generating captions for images, and more.</p><p>This is still nascent work, but there’s a clear indication that these large models have tremendous potential to generalize and be true multipurpose solutions.</p><p>I can’t wait to see where this thread of research goes in 2023. Think of a good Thanos with all the infinity stones. If you had that power, what would you achieve with a snap of your fingers?</p><p>There were a lot of more interesting breakthroughs that came out this year, but these are the ones that caught my eye. Which piece of research made you do a double-take?</p><h3>References</h3><ol><li>Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding: <a href="https://arxiv.org/abs/2205.11487">https://arxiv.org/abs/2205.11487</a></li></ol><p><a href="https://medium.com/mlearning-ai/mlearning-ai-submission-suggestions-b51e2b130bfb">Mlearning.ai Submission Suggestions</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=585d5174b4a7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[You can’t spell Diffusion without U]]></title>
            <link>https://medium.com/data-science/you-cant-spell-diffusion-without-u-60635f569579?source=rss-a99fbdf99073------2</link>
            <guid isPermaLink="false">https://medium.com/p/60635f569579</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[computer-vision]]></category>
            <dc:creator><![CDATA[Sairam Sundaresan]]></dc:creator>
            <pubDate>Fri, 23 Dec 2022 07:09:53 GMT</pubDate>
            <atom:updated>2022-12-23T07:09:53.419Z</atom:updated>
            <content:encoded><![CDATA[<h3>You Can’t Spell Diffusion without U</h3><h4>Understanding one of the core building blocks of Stable Diffusion</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*BQZFmE9s3RNm-2L4" /><figcaption>Photo by <a href="https://unsplash.com/@mikepetrucci?utm_source=medium&amp;utm_medium=referral">Mike Petrucci</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Just recently, <a href="https://stability.ai/blog/stable-diffusion-v2-release">StabilityAI</a>, the company behind Stable Diffusion, released the newest version of their image generator. If generating unbelievably good images from plain text wasn’t already enough, the model can now generate higher resolution images and can do really fancy stuff like use depth maps to produce novel images.</p><p>In fact, since its release, no other piece of software has been adopted faster by developers.</p><p>Look at the stable diffusion <a href="https://github.com/Stability-AI/stablediffusion#news">release notes</a>. I want to draw your attention to one line in particular:</p><blockquote>New stable diffusion model (Stable Diffusion 2.0-v) at 768x768 resolution. Same number of parameters in the U-Net as 1.5, but…</blockquote><p>The same number of parameters in the U-Net. What is this U-Net all about? How is it that all the machinery around the U-Net has changed, but it remains a constant in this model’s engine room? That’s where our story starts today.</p><p>The U-Net is one of the core building blocks of this image generation model. Here’s an oversimplified explanation. The U-Net in stable diffusion takes encoded text (plain text processed into a format it can understand) and a noisy array of numbers as inputs. Over many iterations, it produces an array containing imageable information from this noisy array it received. The output of the U-Net is then used by another network called a decoder to create the pictures we all see as output.</p><p>But this isn’t what the U-Net was designed for at all. In fact, it was designed for medical imaging, all the <a href="https://arxiv.org/pdf/1505.04597.pdf">way back in 2015</a>! So how did it make its way here into a state-of-the-art image generation software? Let’s deconstruct the U-Net and find out!</p><h3>Understanding the U-Net</h3><p>Why is it called U-Net? One look at its architecture will tell you the answer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ec3V_Ox2-CrLfhxKXOlOEQ.png" /><figcaption>From [1]</figcaption></figure><p>The U-Net takes just an image as input and produced another image as output. More specifically, given the image on the left below, the U-Net produces the image on the right.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-Ub9QyJilcUSOc5VYozjrg.png" /><figcaption>From [1]</figcaption></figure><p>This task is called semantic segmentation. Given an image, assign every pixel (smallest atomic unit of an image) to a class that it belongs to.</p><p>So, as you can see, the structure of this neural network is in the shape of a U. There’s a reason for this symmetry. The first half of the “U” shrinks the input image and extracts meaningful information (called features) from it at each stage. It does so through the use of the basic building blocks of convolutional neural networks — <a href="https://paperswithcode.com/method/convolution">convolution layers</a>, <a href="https://paperswithcode.com/method/max-pooling">pooling layers</a>, and, <a href="https://paperswithcode.com/method/relu">activation layers</a>. The second half of the “U” reverses this process, expanding the intermediate results (called feature maps) to larger and larger sizes until the final output is the same size as the input image. To do so, it uses a layer known as a transposed convolution (<a href="https://datascience.stackexchange.com/questions/6107/what-are-deconvolutional-layers">don’t say deconvolution</a>) in addition to the pooling and activation layers.</p><p>Thus, Convolutional network + Mirror Image of Convolutional Network = U-Net. But, it has another special characteristic. Notice the gray arrows connecting parts of the first half of the network to the second? They help in the expansion process.</p><p>Let’s use a childhood game to understand this further.</p><h3>What did you say?</h3><p>As a kid, I’m sure you’d have played the <a href="https://www.wikihow.com/Play-the-Telephone-Game">telephone game</a> (a.k.a grapevine communication game). A group of children stand in a line. The first child whispers something in the second child’s ear. The second child whispers it to the third child and so on. Eventually, the last child has to guess what was originally whispered. More often than not, the secret text changes so much from kid to kid that the final version barely resembles the original.</p><p>Now, let’s take this game and add a twist. Instead of whispering a phrase, the first child has to ask a question. The last child has to answer that question. As the question is passed from child to child, it morphs, and eventually, the last child answers “42” to the question “What is the purpose of life?”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DAikWbsGC46SM7sHHJA4aA.png" /><figcaption>Image created by author</figcaption></figure><p>Now, what if we change the rules to help the kids out? (I mean, they are kids after all) Per the new rules, A kid much earlier in the chain can communicate with a kid later in the chain. But, they can only ask the question as they heard it. Wouldn’t that make it easier for the last child to answer the original question?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*faE6P7Pwc7ppw-o5YKqF5A.png" /><figcaption>Image created by author</figcaption></figure><p>That is the intuition behind the gray connectors in the U-Net architecture.</p><p>They help the latter half of the model guess what the missing pixels can be. Why do I say missing pixels? In the first half of the U-Net, the model has the full picture (pun intended). Each layer shrinks the picture and effectively removes raw information that was available to it before passing it on to the next layer.</p><p>The second half of the network has the difficult task of expanding a small picture (feature map) back to the same size as the original picture that the network received. But how do you expand something without knowing what to fill into the expanded areas? While this is a topic that deserves its own post, it’s safe to say that it’s easier to do this <em>if</em> you have context. That context comes from the gray connectors (Kids earlier in the chain in our game).</p><h3>The Reverse Journey</h3><p>Traditionally, unique neural networks were proposed for image-related problems first in the computer vision community and then made their way to other domains like medical imaging, etc.</p><p>The U-Net is one of the exceptions to this rule. It was intended to solve medical imaging problems but was so versatile and useful that it made its way into a number of computer vision problems — From winning Kaggle competitions to extracting roads or buildings from satellite imagery.</p><p>Today, it sits in the core of stable diffusion where its ability to map a high-dimensional image to another allows us to create incredible imagery by just typing a few words.</p><h3>References</h3><ol><li>U-Net: Convolutional Networks for Biomedical Image Segmentation<a href="https://arxiv.org/abs/1505.04597v1"> https://arxiv.org/abs/1505.04597v1</a></li><li>Stable Diffusion Adoption Rate: <a href="https://a16z.com/2022/11/16/creativity-as-an-app/">https://a16z.com/2022/11/16/creativity-as-an-app/</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=60635f569579" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/you-cant-spell-diffusion-without-u-60635f569579">You can’t spell Diffusion without U</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[An Introduction To Knowledge Distillation — The Obi-Wan Skywalker Algorithm]]></title>
            <link>https://medium.com/@artofsaience/an-introduction-to-knowledge-distillation-the-obi-wan-skywalker-algorithm-bc117d80d69?source=rss-a99fbdf99073------2</link>
            <guid isPermaLink="false">https://medium.com/p/bc117d80d69</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[ml-so-good]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Sairam Sundaresan]]></dc:creator>
            <pubDate>Thu, 15 Dec 2022 22:56:18 GMT</pubDate>
            <atom:updated>2022-12-15T22:56:18.002Z</atom:updated>
            <content:encoded><![CDATA[<h3>An Introduction To Knowledge Distillation — The Obi-Wan Skywalker Algorithm</h3><p>Close your eyes and imagine for a moment that you’re in Tatooine. <a href="https://www.youtube.com/watch?v=sSTr1usny3E">You’re sitting next to</a> a young Luke Skywalker and C3PO, searching for R2D2. Yes, the room inside the vehicle is a bit cramped, but hey, you’re in the Star Wars saga so you’ll take what you get. The desert air blows across your face as arid peaks and valleys whiz by. You squint into the vast expanse as Luke asks you if you see anything. Finally, you spot R2D2. But before you can celebrate, Tusken Raiders attack and your crew are hopelessly overpowered. Remember, this is Luke before he knew what the “Force” was. But wait! Out of nowhere, a hooded figure comes to save the day, driving away the pesky Raiders and rescuing you all.</p><p>I’ll stop with the blow-by-blow account of Star Wars here, but that hooded figure becomes Luke’s mentor, Obi-Wan Kenobi. He then teaches Luke the ways of the “Force” and we all know what happened thereafter.</p><p>Have you ever wondered if Luke would have discovered the ways of the “Force” if not for Obi-Wan? Imagine the plight of the galaxy if that were the case. The Jedi way of teaching from master to apprentice is potent. For one, it helps the <a href="https://starwars.fandom.com/wiki/Padawan">padawan</a> skip over many years of mistakes and false starts and become one with the Force sooner. The student also receives continuous feedback to improve and become a better Jedi.</p><p>Haven’t you seen this in your own life? When you learn a new skill from someone who’s ahead of you, you learn much faster than if you self-study. Wouldn’t it be awesome if we could use this principle in machine learning? Turns out, we can.</p><p>This technique is called <a href="https://arxiv.org/abs/1503.02531">Knowledge distillation</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/740/1*EKAiWk1ZK8H_UnExcHnemA.png" /><figcaption>Distillation of the “force”</figcaption></figure><h3>What is knowledge distillation?</h3><p>Knowledge distillation is a clever way of transferring knowledge from a computationally expensive teacher model to a smaller, lightweight student model. The large teacher model is really good at the task you want to solve. But it’s really good for a reason. Consequently, you can’t deploy it in an app or use it on mobile (edge) devices, or in situations where resources are constrained. Instead of giving up, you can use knowledge distillation to transfer some of this hard-earned knowledge from this large model over to a smaller model that you know <em>can </em>work in all of the scenarios above.</p><p>Do you know what’s crazy though? The smaller model probably wouldn’t have succeeded on its own. But when you perform knowledge distillation, you get the best of both worlds in that you get a very lightweight but smart model.</p><p>Another way to think about knowledge distillation is to consider it as a <a href="https://tivadardanka.com/blog/how-to-compress-a-neural-network">variation of compression</a>, except that we compress by transference as opposed to pruning and removing neurons from an existing model.</p><h3>Components of knowledge distillation</h3><p>There are three components to knowledge distillation — The teacher, which is this large model that you meticulously trained. Alternatively, this could also be a very good pretrained model you found somewhere. The second component is the student. This is the lightweight model that you plan to use eventually but isn’t good enough yet and needs to learn. What does lightweight mean here? Well, it could be a shallower version of the teacher model with fewer layers and neurons per layer, a quantized version of the teacher, a different smaller network, and so on. The third and final component is the method used to transfer the knowledge over to the student model.</p><h3>Understanding how knowledge distillation works</h3><p>Let’s go back to our Star Wars analogy — If Luke had to learn how to use the Force on his own, he probably would have after a lot of trial and error (after all he’s one of the main heroes in the Star Wars saga). But think of how long it would’ve taken him to <em>really</em> understand the Force and get so good to the point where he could save the day multiple times. Obi-Wan, the teacher, on the other hand, is very experienced and knows the Force in and out. Through his mentorship, Luke’s learning was accelerated significantly.</p><p>Luke got two sources of feedback. Firstly, he felt a connection with the Force. In machine learning terms, that would be the ground truth that the model doesn’t see but that we use to measure its performance. The second source of feedback for Luke is from Obi-Wan telling him whether he’s using the Force well or not. This is complimentary feedback to the first that he can use to improve his Jedi skills. In knowledge distillation terms, this is the signal from the teacher model to the student.</p><p>Naturally, the question is how does Obi-Wan give Luke feedback? For that, we need to first understand what kind of knowledge is worth transferring over from mentor to mentee.</p><h3>Types of knowledge</h3><p>There are broadly two kinds of knowledge that can be used as feedback signals. The first is just the outcome itself. In this case, the teacher and student would both be asked to do the same thing, and we’d compare their results. The student would then learn how different their result was from their teachers. Now, a special case of this is when we need to perform complex tasks. Instead of just learning from the final result, intermediate results could also be used as feedback signals. For example, Obi-Wan could either perform a Jedi mind trick (<a href="https://www.youtube.com/watch?v=gALR3SnDEWw">“You don’t need to see our identification”</a>) as Luke watches, or explain each part of the process to him.</p><p>In knowledge distillation terms, these are called <strong>Response-based knowledge</strong> (just the outcome) and <strong>Feature-based knowledge</strong> (the intermediate results too).</p><p>The second broad kind captures the relationship between the intermediate steps — How the steps tie together. This is called <strong>Relation-based knowledge</strong>, and, in knowledge distillation parlance would be the relationships between <a href="https://towardsdatascience.com/convolutional-neural-network-feature-map-and-filter-visualization-f75012a5a49c#:~:text=Feature%20maps%20are%20generated%20by,Convolutional%20layers%20in%20the%20model.">feature maps of a network</a>.</p><h3>Modes of knowledge distillation</h3><p>Now that we know the types of knowledge worth transferring, how can we transfer the knowledge over?</p><p>The first way is <strong>offline distillation</strong>, where you have a teacher model already pre-trained and available for you and only the student learns. The second is called <strong>online distillation</strong>. This is a rarer form in which you don’t have access to a pre-trained model. Therefore, you have to train the teacher yourself. While doing so, you can also set up the student model side by side, and train both. The third is <strong>self-distillation</strong>, where the same model is used as teacher and student. Here, deeper layers of the model can teach earlier layers, or, a model from earlier epochs can be used to train a model deeper in its learning journey. Very meta, I agree.</p><h3>Knowledge distillation in the wild</h3><p>Here are two notable stories where knowledge distillation was applied IRL. The first is <a href="https://arxiv.org/abs/1910.01108v4">DistilBERT</a> where knowledge from a HUGE language model, BERT, was distilled successfully into a smaller model. This resulting model was 40% smaller (66M vs 110M parameters) and faster by 60% than its teacher. What’s even more impressive is that this smaller model was 97% as good as the teacher in specific tasks. The second is actually knowledge distillation used in <a href="https://arxiv.org/abs/1904.01624">Amazon Alexa</a>, where a teacher-student model combination helped label over 1 million hours of unlabeled speech from just 7000 hours of labeled examples.</p><p>So the next time you’re in a pinch with compute resources but really want a good model to use, think of knowledge distillation.</p><h3>🤖💪 Want more ideas to be a productive ML practitioner?</h3><p>Each week, I send out a newsletter with practical tips and resources to level up as a machine learning practitioner. <a href="https://newsletter.artofsaience.com/"><strong>Join here for free →</strong></a></p><p><a href="https://medium.com/mlearning-ai/mlearning-ai-submission-suggestions-b51e2b130bfb">Mlearning.ai Submission Suggestions</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bc117d80d69" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding Large Language Models]]></title>
            <link>https://medium.com/@artofsaience/understanding-large-language-models-6664be71988e?source=rss-a99fbdf99073------2</link>
            <guid isPermaLink="false">https://medium.com/p/6664be71988e</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[ml-so-good]]></category>
            <dc:creator><![CDATA[Sairam Sundaresan]]></dc:creator>
            <pubDate>Thu, 08 Dec 2022 19:45:55 GMT</pubDate>
            <atom:updated>2022-12-08T19:45:55.746Z</atom:updated>
            <content:encoded><![CDATA[<p>Take a look at this Twitter thread by productivity Youtuber Ali Abdaal:</p><h3>Ali Abdaal on Twitter: &quot;In 5 years of YouTube, the most common question I still get from my viewers is &quot;how are you so productive?&quot;So here&#39;s a thread of 15 actionable tips that help me do more of the things that matter to me, without burning out. 🧵 / Twitter&quot;</h3><p>In 5 years of YouTube, the most common question I still get from my viewers is &quot;how are you so productive?&quot;So here&#39;s a thread of 15 actionable tips that help me do more of the things that matter to me, without burning out. 🧵</p><p>At first glance, nothing seems awry and by the looks of it, it seems to be a really popular thread. Here’s the catch — It was almost entirely written by a large language model (LLM) pretending to be him. In fact, it did such a good job that it got over 1 million views and nearly 25k engagements shortly after it was published.</p><p>If your immediate thought is “Does this mean AI is coming for our jobs?”, don’t worry. It’s not quite there. Yet.</p><p>The model that wrote Ali’s thread is called GPT-3 (Generative Pretrained Transformer). With no context of his personality, other than a few words that he entered to get it started, it promptly churned out a <em>pro</em> twitter thread most of us might be proud of.</p><p>Unveiled by OpenAI in July 2020, GPT-3 might be the most well-known LLM given how widespread it has become, but there is an entire family of these models that are just as capable if not more. For example, <a href="https://blog.google/products/search/search-language-understanding-bert/">Google uses one</a> of these in its search engine. That’s right. If you googled something today, you harnessed the power of an LLM called BERT (Bidirectional Encoder Representations from Transformers). I remember reading somewhere that specific search results improved by 7% since Google started using it. Baidu uses another of these species called <a href="https://arxiv.org/abs/1905.07129">ERNIE </a>in their online products and services.</p><p>I can sense a lot of questions in your mind — “Why are these called Large language models?”, “How do they work?”, and most importantly, “Why do they remind me of Sesame Street?”. I’ll try to break these down in the next few sections. Yes, the Sesame street question too.</p><h3>What are LLMs?</h3><p>Large language models are neural networks based on a specific breakthrough from 2017, the Transformer (no pun intended). The reason they are large is because a) They are several gigabytes in size and b) They are trained on enormous (petabyte scale) datasets. These models have billions of parameters. Think of parameters like knobs that a model can tweak to learn things. Therefore, the more knobs it has, the more complex representations it can learn. Over time, these models have become larger and larger, and have also become significantly more powerful.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*DI9VIjmhiwhZjS1P.jpeg" /><figcaption>The rise of transformers</figcaption></figure><p>So we know why they’re called large language models. Are they really such a big deal? Well, kind of. They can do everything from writing essays, creating charts, and writing code (yup, scary), with limited or no supervision. Fundamentally though, all they are trained to do is predict the next word in a sentence — Yes, similar to autocomplete on your phone. So why do you need a large model and datasets for something this trivial? Let’s look at that next.</p><h3>Why are they large?</h3><p>Imagine you have a magic mouse (I did say imagine). By showing it examples of 5 tasks that you want to be done, no matter how difficult or different, you can train it to be a world-class expert at them. You don’t need to show a lot of examples. Just a few will suffice. How awesome would that be?</p><p>Now imagine that you are given a magic cow. This cow can do the same 5 tasks that your magic mouse can, but better. Also, it can do 5 more new tasks that the mouse can’t do. Even better right?</p><p>Humor me once more and imagine that the gods have bestowed upon you a magic dinosaur. This can one-up the cow (duh <a href="https://www.youtube.com/watch?v=MalLny7MT6E">it’s morphin time </a>) and do even better on the 10 tasks that the cow could do and do another 5 new ones. Do you see where I’m going with this?</p><p>LLMs are remarkable because a single model can be used for several tasks. As the size of the model increases (in parameters), it can learn more complex representations, and, therefore can do more tasks better than a smaller model. What’s more, they can learn new tasks from just a handful of examples. These tasks can be any of the things I mentioned before and more! That’s why researchers are trying to scale these models up.</p><p>The natural question that emerges next is how on earth do these models learn.</p><h3>How do they learn?</h3><p>These models learn in two stages — Pretraining and fine-tuning.</p><h3>Pretraining</h3><p>In the pretraining stage, the goal is to teach the model the semantics, structure, and grammar of language by showing it a lot of examples (think billions). As is the case with machine learning, we simply show examples to the model and let it learn the rules by itself.</p><p>Pretraining is the longest and most computationally expensive part of the learning process. A model is shown examples over and over again and eventually learns the rules. So, how does it learn?</p><p>Here are a couple of common pre-training methods used in practice today:</p><ul><li><strong>What comes next in the sentences below?</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*D2OCfp6ZM5J3rgyJ.png" /></figure><p>This is how the GPT family of models is pretrained. Given a partial sentence, they have to predict the next word in the sentence and then the next, and so on. Even though we may have a full sentence that we can show the model, we don’t. This prevents the model from cheating and looking into the next set of words that may occur in the sentence.</p><ul><li><strong>Fill up the blanks in the sentences below</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*fZjvdCwCrYr3WdKN.png" /></figure><p>You might have had luck guessing the first two, but the third might be tricky. This is called masked language modeling. In this approach, we randomly blank out words in a sentence and have the model guess what those words might be. Unlike the previous approach, we present a full (but blanked out) sentence to the model so it can learn from words on either side of the blanks. Models like BERT are pre-trained using this approach.</p><p>By doing this over and over again on a ridiculously large dataset, these models learn rich representations of language. But of what use is that if it can’t be put into a practical application?</p><p>That’s where fine-tuning comes in.</p><h3>Fine-tuning</h3><p>In the fine-tuning phase, we can take a pretrained model and show it examples of the task we want it to solve. This can be question answering (who is Neil Armstrong?), sentiment analysis (was that tweet mean?), conversation, and so on. This stage usually takes much lesser time and data compared to the pretraining phase which is why these pretrained models are invaluable.</p><p>But that raises the question — “Where do you get all that data for pretraining?”</p><h3>What do they learn from?</h3><p>If you take all the books in the world, all of Wikipedia, and all of the text you can get by crawling the web and ingesting that knowledge, how smart would you be? Well, that’s what these models are fed. In the pretraining phase, these models are trained on all of these datasets (amounting to just short of the collective human consciousness). That’s another reason why they are large by design. Smaller models are limited in what they can learn. In the fine-tuning phase, smaller in-house datasets relevant to your application are used.</p><h3>Where do we go from here?</h3><p>LLMs have some limitations though. First, only a few well-funded organizations can train these models since they need so much compute (hundreds of thousands of dollars). Second, they aren’t as eco-friendly since the energy it takes to train these models amounts to several years of your electricity bill. More importantly, these models are trained on information contained on the internet. That means that they can be biased, learn toxic things, and much more. Given how good they are at writing, the spread of disinformation is a big worry. Thankfully, researchers across the globe are working on addressing these issues.</p><p>That’s the ELI5 version of LLMs for you. Finally, I haven’t yet told you about the Sesame Street connection. Well, <a href="https://www.theverge.com/2019/12/11/20993407/ai-language-models-muppets-sesame-street-muppetware-elmo-bert-ernie">here you go</a>.</p><h3>🤖💪 Want more ideas to be a productive ML practitioner?</h3><p>Each week, I send out a newsletter with practical tips and resources to level up as a machine learning practitioner. <a href="https://newsletter.artofsaience.com/"><strong>Join here for free →</strong></a></p><p><a href="https://medium.com/mlearning-ai/mlearning-ai-submission-suggestions-b51e2b130bfb">Mlearning.ai Submission Suggestions</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6664be71988e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Where do you fit in the Machine learning industry?]]></title>
            <link>https://medium.com/@artofsaience/where-do-you-fit-in-the-machine-learning-industry-48af7cad0780?source=rss-a99fbdf99073------2</link>
            <guid isPermaLink="false">https://medium.com/p/48af7cad0780</guid>
            <category><![CDATA[jobs]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[ml-so-good]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[technology]]></category>
            <dc:creator><![CDATA[Sairam Sundaresan]]></dc:creator>
            <pubDate>Thu, 01 Dec 2022 23:09:31 GMT</pubDate>
            <atom:updated>2022-12-01T23:09:31.518Z</atom:updated>
            <content:encoded><![CDATA[<p><strong><em>“When patterns are broken, new worlds emerge.” — Tuli Kupferberg</em></strong></p><p>The machine learning (ML) market was valued at $15.44 billion in 2021. By 2029, it’s estimated to be valued at around <a href="https://www.fortunebusinessinsights.com/machine-learning-market-102226">$209.91 billion</a>. Astronomical growth, won’t you agree?</p><p>As you can see, I’ve done my due diligence by typing some keywords into Google and quoting the results as an authoritative entity. But besides this feigned bravado, what’s the point of this?</p><p>There’s never been a better time to transition careers into the ML space. Seriously. The best time was 10 years ago.</p><p>The second best time is now!</p><p>There are a few common misconceptions I’d like to tackle before getting to the various roles that might interest you as a practitioner.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*9-5ysz7kntrQUzc9" /><figcaption>Photo by <a href="https://unsplash.com/@homajob?utm_source=medium&amp;utm_medium=referral">Scott Graham</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>Misconception #1: You need to have a PhD</h3><p>The short answer is that you don’t. The long answer is you don’t for 99% of the roles. A Ph.D. in machine learning is needed IF and only IF you want to break into the top tech research labs as a research scientist. For just about any other role, you don’t need one (unless you really want one and would like to be addressed as Dr. INSERT_YOUR_NAME_HERE).</p><p>The primary reason a Ph.D. is valuable in my opinion is that going through a rigorous Ph.D. program <em>teaches</em> you<em> </em>how to do systematic research, and how to formulate abstract ideas into concrete problems that can be solved.</p><p>I don’t have a Ph.D., and I’ve transitioned from a systems engineer to a research engineer to a research scientist (humble brag, sorry). The reason I bring it up is that you need to know that this isn’t a barrier to entry. You can and you should apply to roles that interest you, AND for which you have the skills to deliver on the job.</p><p>What you need is a curiosity itch, the ability to code, and the willingness to learn new things. These three skills will help you break into big tech, and grow in the machine learning space.</p><p>So, repeat after me — “I don’t need a Ph.D. unless I absolutely want to go out and get one.”</p><h3>Misconception #2: You need to be really good at math</h3><p>Machine learning is math-heavy. I agree. But, you don’t need to know all the math in the world to get started. In fact for some roles in ML, you’ll be so far removed from any kind of math that you’ll wonder why you didn’t transition sooner. I won’t lie, there’s some high school math that’s really important. You can learn that on the job as and when you need it.</p><p>Don’t let this stop you either. Please.</p><h3>Misconception #3: Your past career experience is useless</h3><p>One of the most beneficial skills to acquire in any kind of machine-learning role is domain expertise. So whether you’ve been an accountant, or have worked in pharmaceuticals or any other field that seems remote to ML, know this — Your domain skills will actually help you, not hinder you.</p><p>Here’s why.</p><p>The hardest skill to acquire is business intuition. Not ML. If you can’t communicate why you need to use machine learning, or how it would solve your business problem, you won’t get buy-in from all the stakeholders.</p><p>Your domain expertise will let you speak to non-ML folks in a language they understand!</p><p>So treat your past experience as a superpower, not as wasted effort.</p><p><em>With these out of the way (read them aloud if you aren’t convinced yet), let’s look at the two broad categories of machine learning roles:</em></p><h3>Machine Learning Research:</h3><p>The goal of research is to push the boundaries of the field further and to solve fundamental problems. Essentially, the team may go, “Alright folks, let’s see if this claim is valid or not and go down this rabbit hole”. Some research may lead to products, BUT, that isn’t why research is pursued.</p><p>Companies also have a flavor of research called applied research. This is usually directed at solving practical problems the company has, and may have shorter time frames.</p><p>Research roles are of two kinds:</p><ul><li><strong>Research Scientist: </strong>This is probably the <em>only role </em>in ML that usually requires a Ph.D. There are circumstances where a high-performing engineer with a good track record of top-tier publications gets into this role, but those are far and few. A Research Scientist comes up with novel ideas, puts them to the test, and publishes them. Additionally, they act as advisors to other engineers on the team.</li><li><strong>Research Engineer: </strong>This role isn’t present in all companies. Sometimes a company might use Research Engineer and Research Scientist interchangeably. The Research Engineer usually works in collaboration with the Scientists and runs experiments that evaluate ideas they come up with. This role doesn’t require a Ph.D. or top-tier publications.</li></ul><h3>Machine Learning in Production:</h3><p>As the name suggests, the goal of production is to come up with a product — be it a good or service. Machine learning in production usually deals with a need that the company’s users have. Naturally, the constraints are different. In research, you might focus on beating the state of the art. In production, however, your goals may be ensuring that the ML system runs fast, reliably, and works well with the other parts of the product.</p><p>Thus there are more hats to don, and more opportunities to enter the field.</p><p>Below are a few different production roles I’ve seen in the industry:</p><ul><li><strong>Machine Learning Engineer (MLE): </strong>The MLE is a software engineer who has ML skills or can learn ML skills needed for the role. Their goal is to design, build and productionize ML models for business needs. In this day and age, this is one of the coolest roles to have. Your work directly impacts the product.</li><li><strong>Data Scientist: </strong>A data scientist needs to form their own questions about the data and use machine learning or predictive modeling to answer them. Data science is a deep and rich field. As a data scientist, you’ll turn data into business insights and help the organization make better decisions. The key phrase is unlocking meaning from data.</li><li><strong>Data Engineer: </strong>They make it easier for others in the organization like Data Scientists, and MLEs to use data for their needs. To do this, a data engineer builds systems to collect, manage, organize and analyze data at scale. Raw data is hard for anyone to use, and in this age of big data, managing and creating an effective structure for the rest of the organization to use is an invaluable skill.</li></ul><p>There are some newer roles emerging like ML ASIC Engineers, Infrastructure engineers and so on which might also interest you depending on your background.</p><p>Overall, there’re a ton of roles that you can choose from depending on your skills and interests. You just need to start.</p><h3>Key Takeaways:</h3><ul><li>You need just three things to get into the ML field:</li><li>A willingness to learn</li><li>The ability to code</li><li>Curiosity</li><li>There are many opportunities for you — Find the right starting point</li><li>Don’t wait to learn everything and then start. Learn what you need. Learn the rest on the job.</li></ul><h3>🤖💪 Want more ideas to be a productive ML practitioner?</h3><p>Each week, I send out a newsletter with practical tips and resources to level up as a machine learning practitioner. <a href="https://www.newsletter.artofsaience.com/"><strong>Join here for free →</strong></a></p><p><a href="https://medium.com/mlearning-ai/mlearning-ai-submission-suggestions-b51e2b130bfb">Mlearning.ai Submission Suggestions</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=48af7cad0780" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Learning Machine Learning Through Stories]]></title>
            <link>https://medium.com/@artofsaience/learning-machine-learning-through-stories-43de0a6524c6?source=rss-a99fbdf99073------2</link>
            <guid isPermaLink="false">https://medium.com/p/43de0a6524c6</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[ml-so-good]]></category>
            <dc:creator><![CDATA[Sairam Sundaresan]]></dc:creator>
            <pubDate>Fri, 25 Nov 2022 08:01:46 GMT</pubDate>
            <atom:updated>2022-11-25T17:02:06.048Z</atom:updated>
            <content:encoded><![CDATA[<p>The Painter and the Curator</p><p>His ragged clothing was covered with splotches of paint — Burnt Sienna, Cerulean Blue, and Ochre to be specific. The Mediterranean sun was beating down on his back, as he wiped off beads of sweat from his forehead. He stared at his work. The canvas was a mess, and there was nothing meaningful on it. Even the most lenient art critics wouldn’t call it modern art. He threw the canvas aside and started afresh.</p><p>In a tiny cottage by the sea, Benjamin had been slogging for hours trying to paint something original. The trouble was — He had no inspiration today. His whole life had been one of trying to find his purpose — He was a vagabond. But each time he picked up a brush to paint, a surge of electricity flowed through him as if this was what he was meant to do.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*h_MuOQpzeI4InFF4" /><figcaption>Courtesy of Midjourney Prompting</figcaption></figure><p>He wasn’t alone though. His friend and roommate, William had gone out to get food. Starved for food, but not companionship. Unlike Ben (as he liked to be called), William wasn’t an artist, but an appreciator of art. He could sense when there was life in a painting and when it was just an imitation.</p><p>William worked as an assistant at the local museum — His dream was to become a curator, but that remained a distant dream.</p><p>The pair had met at a local cafe. Over a shared plate of toasted bread smothered with olive oil and crushed tomatoes, the two of them connected over their shared passion for art. One loved to create, the other loved to critique.</p><p>Since that day, Will had helped Ben become a better painter, and Ben had helped Will sharpen his observational skills. Starting from a blank canvas was difficult, so Will nudged Ben to start with a splatter of colors on the pristine canvas. This would usually spark his imagination.</p><p>Ben’s initial creations were ordinary — They didn’t inspire or resemble the great artists of the time. Will would give Ben feedback on things he got wrong like the arc of the brushstrokes, the harmony of colors, and the lack of perspective, and, each time, Ben would take copious notes.</p><p>He didn’t take the feedback lightly. He devoured it. Over time, he added a variety of techniques to his toolbelt. Will on the other hand used Ben’s work as a way to sharpen his eye on the differences between amateur and master painters. In recent months though, it was getting harder. Ben was getting good, and Will really needed to inspect the painting over and over before finding out the flaws.</p><p>A knock on the door announced William’s return. The smell of the fresh paella was a dead giveaway. He needn’t have knocked.</p><p>“What do you have today?”, asked Will. “The same. No inspiration”, glumly replied Ben. “Let me be the judge of that”, said Will as he walked towards the canvas. His jaw dropped as he saw the painting. It was nothing like he had ever seen before. The colors jumped off the canvas, and the emotions of the brush as it danced on the canvas moved Will. He looked closer. Surely, there was something he could find to critique. Surely? There was nothing. It was flawless.</p><p>“What have you done?”, asked Will. “I know right? I’m losing my grip.”, grimaced Ben. “You’re mad, my friend. This is one of the most original works of art I’ve ever seen. I can’t find a single flaw”. A weak smile creased Ben’s unshaven face. These words meant the world to him. He had finally arrived.</p><p><em>The end.</em></p><p>You might be wondering what this story has to do with learning machine learning. It has everything to do with it. This story is a fictional description of a concept in deep learning called <a href="https://en.wikipedia.org/wiki/Generative_adversarial_network">Generative Adversarial Networks </a>(or GANs for short).</p><p>It’s the technique behind all the fake faces of people that don’t exist you see online. There are two networks involved in this, the Generator and the Discriminator.</p><p>The Generator network creates “art” for the Discriminator network to critique. The Discriminator compares the generator’s work with real “art” and learns to distinguish between the two. The Generator’s goal is to fool the Discriminator into thinking that what it produced is real, while the Discriminator’s goal is to not be fooled and find flaws in the Generator’s work.</p><p>The training of these two networks happens simultaneously and since each network is trying to outsmart the other, it is “adversarial” training. The training stops when the Generator fools the Discriminator.</p><p>In our story, Ben is the generator, Will is the discriminator and eventually, Ben fools Will. The Generator doesn’t start from a blank slate. It starts with noise. In our story, Will asks Ben to start with a splatter of colors instead of a blank canvas.</p><p>The point I’m trying to drive home?</p><p>You can make ANY machine learning concept memorable by crafting a story for it. Make the characters and setting vivid, and their goals and aspirations concrete. Map the concept to the story.</p><p>Even if you forget the technical terms of the concept, you’ll never forget the fundamental principles. The reason this works is that we are visual learners. We remember pictures better than we remember words. To learn more about this, read<a href="https://www.amazon.com/Moonwalking-Einstein-Science-Remembering-Everything/dp/0143120530"> “Moonwalking with Einstein”</a>.</p><p>So, what story do you plan to craft?</p><h3>🤖💪 Want more ideas to be a productive ML practitioner?</h3><p>Each week, I send out a newsletter with practical tips and resources to level up as a machine learning practitioner. <a href="https://www.newsletter.artofsaience.com/"><strong>Join here for free →</strong></a></p><p><a href="https://medium.com/mlearning-ai/mlearning-ai-submission-suggestions-b51e2b130bfb">Mlearning.ai Submission Suggestions</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=43de0a6524c6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Deconstruct Machine Learning Papers to Implement them:]]></title>
            <link>https://medium.com/@artofsaience/how-to-deconstruct-machine-learning-papers-to-implement-them-1d8b9f2e6adf?source=rss-a99fbdf99073------2</link>
            <guid isPermaLink="false">https://medium.com/p/1d8b9f2e6adf</guid>
            <category><![CDATA[research]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[coding]]></category>
            <category><![CDATA[ml-so-good]]></category>
            <dc:creator><![CDATA[Sairam Sundaresan]]></dc:creator>
            <pubDate>Thu, 17 Nov 2022 20:59:43 GMT</pubDate>
            <atom:updated>2022-11-25T17:01:43.780Z</atom:updated>
            <content:encoded><![CDATA[<p>Reading ML papers is really important for your growth as a practitioner. In fact, I’ve <a href="https://youtu.be/8BlcQEJEdq8">made a video </a>explaining how you can read these papers effectively. But, what’s really important is knowing how to map these papers into code when you need to. This may be either because you’re assigned a paper to implement as homework, or maybe you need it for a work project, or maybe you’re really excited about learning to implement it.</p><p>Whatever the reason, here is an effective strategy that works for me time and time again when I need to implement papers.</p><h3>Step 1: Progressively summarize the paper</h3><p>In order to implement a paper and validate its claims, you’ll need to know the TL;DR of the paper. Specifically, read the paper and progressively summarize it in your own words. At the end of this step, you want to have the following information at your fingertips:</p><ul><li>What are the <strong>main claims</strong> — <em>“We achieve X results”, “We improve on Y previous work”, etc.</em></li><li>What are the new <strong>proposals</strong> — <em>“We propose ABC architecture”, “We use C instead of D”, and so on</em></li><li>What are the <strong>hyperparameters</strong><em> — “We use the Adam Optimizer with a learning rate of …”, “We use a batch size..”, etc.</em></li><li>What are the <strong>hardware</strong> requirements<em> — “We use 1000 GPUs and 500 TPUs” </em>😂</li><li>What are the <strong>target tasks</strong> and <strong>datasets</strong> — <em>“We evaluate our method on DEF dataset” and so on</em></li></ul><p>Identifying these first will help you build a mental landscape of what to expect, what’s different from things you’ve previously implemented, and help you decide if you can re-implement the paper as is. Sometimes, the size of the model you need to implement, or the hardware requirements might prohibit you from replicating the work. In those cases, you’d need to scale down the method or abandon the pursuit altogether if you don’t have the compute needed to run the algorithm.</p><p>This step will help you avoid a lot of wasted effort and mentally prepares you for the actual coding part!</p><h3>Step 2: Identify the core algorithm(s) in the paper</h3><p>The next step is to find what you need to implement in the paper. One of the many challenges new practitioners face when reading these papers is that there is a lot of math <strong>not directly related</strong> to what they need in order to implement the paper.</p><p>This can both confuse and discourage practitioners from trying to implement these papers. Most of this math is usually something that serves as a theoretical proof of the method, or some adjacent theory needed to really understand the foundations of the paper.</p><p>Give permission to yourself to skip these equations! Please. It will save you a lot of pain.</p><p>Here’s what you need — The flowchart or the main method which most papers have. Let’s take an example. The image below is from the paper that proposed the <a href="https://arxiv.org/abs/1412.6980v9">Adam Optimizer </a>(You don’t need to implement this algorithm these days).</p><p>This table in the paper summarizes everything you need to implement Adam on your own:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*fG4xvMhqTBQblswX" /><figcaption>The Adam Optimizer in Pseudocode</figcaption></figure><p>There’s a LOT of math in the paper other than this and if you get sucked into reading and understanding that (not a bad thing but unnecessary to implement), you’ll never implement it.</p><p>If the paper you’re reading doesn’t have this, try to identify the core equations and write down a flowchart like this on your own. Usually, you can find the “main” equations in the methods section of the paper. Extracting this from the paper will reduce the back and forth you need to implement it. This is also good practice to build your equation-to-code muscles for the future.</p><h3>Step 3: Audit and Simplify</h3><p>While it might be tempting to jump into coding right away, avoid that temptation, and do this next. Go line by line through the flowchart you’ve created or identified and audit all the things you need to implement. Auditing here means finding out if you need to write something in code yourself, or if can borrow an existing implementation.</p><p>Most of the code you need to write is usually already available as functions from your favorite libraries (Pytorch, Tensorflow, etc.) or boils down to high school math. Seriously.</p><p>Let’s take the example from above. Here’s what the equations look like without the flowchart:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/610/0*9stv3uPQ2hGKdHnU" /></figure><p>This might look scary. So, replace each of these fancy <a href="https://www.artofsaience.com/my-crusade-against-weapons-of-mass-confusion/">squigglies </a>with the English alphabet as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*6s7e2_rcw7oKrUO3" /><figcaption>A simplified view of the Adam Optimizer</figcaption></figure><p>This doesn’t look as menacing. Essentially, you need to do some addition, subtraction, multiplication, and division. The most complicated thing in this is taking a square root! You might ask me, “Well what about the <strong><em>t</em></strong> and <strong><em>t-1 </em></strong>in the subscripts?”. That’s simply to denote the iteration. You just need to do these steps in a loop and overwrite the values in these variables accordingly.</p><p>While there will be papers with more complicated equations, the principles remain the same. Remember, these equations are just supposed to be concise ways to represent ideas. They always boil down to high school math.</p><p>Now, you’re ready to code and test out the paper’s claims.</p><p>I hope you try this method out on the next paper you implement. Let me know how it works for you.</p><h3>🤖💪 Want more ideas to be a productive ML practitioner?</h3><p>Each week, I send out a newsletter with practical tips and resources to level up as a machine learning practitioner. <a href="https://www.newsletter.artofsaience.com/"><strong>Join here for free →</strong></a></p><p><a href="https://medium.com/mlearning-ai/mlearning-ai-submission-suggestions-b51e2b130bfb">Mlearning.ai Submission Suggestions</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1d8b9f2e6adf" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How would I learn machine learning today?]]></title>
            <link>https://medium.com/@artofsaience/how-would-i-learn-machine-learning-today-697eae482045?source=rss-a99fbdf99073------2</link>
            <guid isPermaLink="false">https://medium.com/p/697eae482045</guid>
            <category><![CDATA[learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Sairam Sundaresan]]></dc:creator>
            <pubDate>Sun, 13 Nov 2022 18:40:08 GMT</pubDate>
            <atom:updated>2022-11-25T17:01:23.212Z</atom:updated>
            <content:encoded><![CDATA[<p>One of the most frequent questions I get is <em>“How would you learn machine learning if you were starting today?”. </em>I thought that it might be worth it to actually write it out and share it with all of you. Here’re 5 things I’d do if I were starting over today…</p><h3>Tip #1: Get Going</h3><p>If you’re unsure where to start and feel intimidated by the math, take a <a href="https://course.fast.ai/">code-first course</a>. If you feel excited by theory, take a <a href="https://www.deeplearning.ai/courses/deep-learning-specialization/">theory first course</a>. The most important thing is that <strong>you start</strong>.</p><h3>Tip #2: Build your Fundamentals</h3><p>Don’t worry about which framework to learn. Pick whichever one you like and run with it. What’s really important is that you learn the fundamentals. Pytorch, Tensorflow, and Jax are all frameworks that help you solve problems, but the fundamentals that they implement are the same. Focus on those first. Every time.</p><blockquote><strong>Frameworks come and go, fundamentals come and grow.</strong></blockquote><h3>Tip #3: Read and Implement Papers</h3><p>Forget trying to keep up with all the advances in the field. Machine learning moves way too fast. Instead, understand the implementations of papers you’re reading and reimplement them yourself. That will help you tremendously and compound your learning.</p><blockquote><strong>The best papers of today will become the textbooks of tomorrow.</strong></blockquote><h3>Tip #4: Learn Publicly</h3><p>Write about what you learn and share it publicly. You don’t need to be an expert. There are a million others who have the same questions you do. Make their lives easier. Share your knowledge. This is the step that everyone skips. Learning in public will build connections that you previously may have missed.</p><h3>Tip #5: Be a T-shaped learner</h3><p>Dive Deep into your area of interest like Computer Vision but explore other areas like Natural Language Processing, Recommenders, and Graph learning.</p><blockquote><strong><em>Great research ideas are the byproduct of cross pollination.</em></strong></blockquote><p>If you’re not sure where to start, Begin your <a href="https://www.kaggle.com/">Kaggle journey</a> seriously. Competing with the best in the world will raise your level and will accelerate your learning.</p><h3>🤖💪 Want more ideas to be a productive ML practitioner?</h3><p>Each week, I send out a newsletter with practical tips and resources to level up as a machine learning practitioner. <a href="https://www.newsletter.artofsaience.com/"><strong>Join here for free →</strong></a></p><p><a href="https://medium.com/mlearning-ai/mlearning-ai-submission-suggestions-b51e2b130bfb">Mlearning.ai Submission Suggestions</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=697eae482045" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>