<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Yusaku Sako on Medium]]></title>
        <description><![CDATA[Stories by Yusaku Sako on Medium]]></description>
        <link>https://medium.com/@u39kun?source=rss-321747ff5910------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*tyZ7imrlwFfJJqq2LMqIGw.png</url>
            <title>Stories by Yusaku Sako on Medium</title>
            <link>https://medium.com/@u39kun?source=rss-321747ff5910------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 00:32:21 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@u39kun/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Inevitable downfall of a Kaggle Grandmaster who tried to win at any cost]]></title>
            <link>https://medium.com/@u39kun/how-a-top-kaggle-grandmaster-cheated-and-got-permanently-banned-c86ca431f499?source=rss-321747ff5910------2</link>
            <guid isPermaLink="false">https://medium.com/p/c86ca431f499</guid>
            <dc:creator><![CDATA[Yusaku Sako]]></dc:creator>
            <pubDate>Sun, 12 Jan 2020 23:24:38 GMT</pubDate>
            <atom:updated>2020-01-16T05:14:55.908Z</atom:updated>
            <content:encoded><![CDATA[<p>On Jan 10, 2020, the Kaggle community was shocked by <a href="https://www.kaggle.com/c/petfinder-adoption-prediction/discussion/125436">an announcement</a> that a top Grandmaster (then ranked #3 out of more than 120-thousand competitors worldwide) was found to have cheated on a competition to win 1st place; his team was disqualified retroactively from the competition. It was also announced that this Grandmaster was permanently banned from the platform.</p><p>This was devastating and upsetting news to the Kaggle community. <a href="https://www.kaggle.com/u39kun">Being a Kaggle competitor myself</a>, I sincerely hope that damages to the Kaggle brand will be minimal from this, and Kaggle continues to attract many organizations to host competitions on its the platform AND top-notch, world-class competitors who diligently and passionately work on them. I truly believe in Kaggle’s mission and the Kaggle community has had tremendous impact on myself and countless others. I would hate for this incident to deter Kaggle and the community from advancing AI forward.</p><p><strong>Background<br></strong>Back in December 2018, a Malaysian pet wellness site <a href="https://www.kaggle.com/c/petfinder-adoption-prediction/overview">Petfinder.my hosted a Kaggle competition</a> which attracted more than 2000 teams to compete over a 3-month period to work on the following noble goal of helping homeless pets by leveraging AI:</p><p><em>“In this competition you will be developing algorithms to predict the adoptability of pets — specifically, how quickly is a pet adopted? If successful, they will be adapted into AI tools that will </em><strong><em>guide shelters and rescuers around the world on improving their pet profiles’ appeal, reducing animal suffering and euthanization.</em></strong><em>”</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yw4Zcpr6892nZIhE6QGM1w.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@snowboardinec?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Max Baskakov</a> on <a href="https://unsplash.com/s/photos/grumpy-cat?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p><strong>Discovery of Cheating<br></strong>This competition completed more than 9 months ago in March 2019. Why did it take this long to discover that the 1st place team was cheating?<br>In Kaggle competitions, the Kaggle staff and the host perform verification of the models submitted by the competitors. In this verification phase, cheats get discovered and get disqualified; this unfortunately happens from time to time. But the 1st place team passed this check and was awarded the prize money of $10K USD. The discovery came when <a href="https://www.kaggle.com/bminixhofer">Benjamin Minixhofe</a>, one of top finishers in this competition, was volunteering to help Petfinder.my productionize the top teams’ models. He found that 1st place team’s model does not work well and he started investigating their code. It turns out that this was non-trivial as the cheat was specifically designed to evade discovery through obscurity.</p><p><strong>Summary of Cheating<br></strong><a href="https://www.kaggle.com/c/petfinder-adoption-prediction/discussion/125436">The competition host from Petfinder.my summarized the cheat</a> as follows:</p><p><em>Here is what the Bestpetting team did in the PetFinder.my contest:</em></p><ul><li><em>They fraudulently obtained adoption speed answers for the private test data (possibly by scraping our website)</em></li><li><em>These data and answers were then encoded, obfuscated and hashed into an ID field that was disguised as part of their external “cute-cats-and-dogs-from-pixabaycom” dataset</em></li><li><em>When processing the data, these hashed IDs were decoded and answers were retrieved for the predictions</em></li><li><em>Only some of the encoded answers were used, so as to keep their final score “realistic”</em></li><li><em>These processing codes were meticulously hidden and obfuscated under many nested layers of functions and codes, intentionally designed to be highly unreadable and seemingly mundane</em></li></ul><p><a href="https://www.kaggle.com/bminixhofer/how-bestpetting-cheated">Benjamin did a great job performing the detective work and described the cheat in detail in his Kaggle Notebook</a>.</p><p>The summary is that they collected the answers to the problem by violating an explicitly-stated official competition rule to not use any data from Petfinder.my (<a href="https://www.kaggle.com/c/petfinder-adoption-prediction/rules">see EXTERNAL DATA section</a>), hid these answers, and used these answers only some of the time to boost their score. Benjamin stated in the Notebook that if they didn’t cheat, they would have placed around ~100th in the competition.</p><p><strong>History of Cheating and Testing the Boundary<br><em>Google Landmark Retrieval 2019 Incident</em></strong><br>It was not the first time that this now-banned Grandmaster was disqualified due to cheating. In the <a href="https://www.kaggle.com/c/landmark-retrieval-2019">Google Landmark Retrieval 2019 competition</a> (which I also competed in), his team placed 1st but later found to have cheated and disqualified. <a href="https://www.kaggle.com/c/landmark-retrieval-2019/discussion/95136">There is a thread where he and other competitors debate the practice of finding answers through scraping</a>. In his words:</p><p>“totally agree, it’s not useful and not scientific at all, as well as poorly designed competitions<br>my logic is always the same: if there is a flaw or leak, somebody will definitely try to use it and if rules will allow it afterward then having an honest solution will be a disappointing disadvantage. so there are only two options left: not to participate or use the flaw”</p><p>“nevertheless, I haven’t seen a lot of leaks/flaws posted during the competitions — people are trying to win by any edge they could find, it’s a competition after all”</p><p>“no, what I’m trying to say: there is a flaw in competition design at Kaggle and there is already a solution to this problem by Google. until they fix it, we will always see solutions like mine and will never be sure about the rest of the LB (only winning solutions were checked)”</p><p><strong><em>Quora Insincere Questions Incident<br></em></strong>On the Quora Insincere Questions competition, <a href="https://www.kaggle.com/c/quora-insincere-questions-classification/discussion/80665">he publicly disclosed how his team placed 1st on the public test set by simply scraping for answers from quora.com.</a> This disclosure happened after the competition was already finished.<br>He claims that the reason his team did this was because they didn’t like the restrictions imposed by the competition rules, so they “decided to have fun”. <br>The technique did not work for the private test set which determines the final standing, so they ended up finishing 1023rd in the competition. Looking at the discussion thread, folks who responded to the thread were mostly about being impressed and praising their cleverness, and perhaps such positive reactions were encouraging and enabling. Had their method worked on the private dataset and placed them near the top, had the cheat been discovered? Maybe not, as we can now see from the Petfinder.my competition incident that post competition verification is not foolproof if the cheat is cleverly hidden. Or if the cheat could not be hidden due to the competition format (no external dataset), they might have tried to place well enough but not so well to stay under the radar. If they could have gotten away with it, I doubt he would have posted and disclosed the method but that’s just my guess. AFAIK there were no consequences from this incident as their entry was never removed from the leaderboard.</p><p><strong>Aftermath<br></strong><a href="https://twitter.com/h2oai/status/1215806481339056128">He was swiftly let go from his employer H2O.ai</a> after getting permanently banned from Kaggle. <a href="https://www.h2o.ai/company/team/">H2O.ai has a number of Kaggle Grandmasters on its roster and advertise this heavily to demonstrate their expertise and establish trust</a>. Employing a Grandmaster known to have cheated and banned would be a PR nightmare for the company. It must suck to be permanently banned from something he cared so much about and to lose a job at the same time but actions have consequences…</p><p><a href="https://twitter.com/ppleskov/status/1215983188876709888">He publicly apologized via Twitter</a> saying:</p><p>“1. We, as a team, would like to apologize to <br><a href="http://twitter.com/myPetFinder">@myPetFinder</a> <a href="http://twitter.com/kaggle">@kaggle</a> and the DS community for all the wrongdoings. I would also like to apologize to <a href="http://twitter.com/h2oai">@h2oai</a> and all the Kaggle competitors for putting their reputation at risk.”</p><p><strong>“2. We will return the prize money asap. For me, it was never about the money but rather about the Kaggle points: a constant struggle of becoming #1 in rating had compromised my judgment.”</strong></p><p>“3. Now, after the ban, I feel almost relieved that the race is over, and there are still a lot of thoughts to process and reflect upon. I hope at least some of you forgive me and hope other competitors will learn from my mistakes.”</p><p>We are human and make mistakes, especially when we start chasing numbers (money, rank, likes, views, or whatever) and forget why we are doing the thing we do to begin with. I applaud him for publicly apologizing and I hope he makes sound choices going forward to avoid such mistakes.</p><p><strong>Thoughts<br></strong>There’s no question that this individual is smart, creative, and very determined to accomplish his goals. It is unfortunate that he deliberately chose to use his intelligence and creativity to improve his Kaggle world ranking even if it meant resorting to cheating and at the expense of others. I wish he could have applied his talent toward productive goals of developing models that work well to solve important problems or to focus on pushing the boundaries of the state of the art by developing new techniques, rather than treating Kaggle as a game to be won.</p><p>It is unfortunate that his cheating has had so much negative impact. For one, it prevented appropriate rewards to legitimate and hardworking competitors who were playing by the rules. For example, because of the disqualification, a competitor on the fringe got bumped up a place and earned his Grandmaster title that he should have received to begin with had cheating not taken place. I am sure there are many others who missed their in-the-money or gold medal by one spot. On a bigger scale, what he did was damaging to the reputation of Kaggle as a platform and the reputation/motivation of Kaggle competitors.</p><p>The Kaggle community attracts many smart people from around the world. It is not unusual for competitors to find exploitable aspects of some competitions. The common practice has been to report such exploits publicly and/or to the organizer for fairness to all participants as soon as they are discovered. I hope that this practice is followed by everyone going forward.</p><p>There will always be exploits even if much care and effort is taken to prevent exploits. However, there are ways to improve the system to be less exploitable and deter cheating. For example, we can allow top finishers to check each other’s solution after the competition for up to X days and have an appeal process (competitors would be VERY motivated to do thorough reviews to uncover cheating for those who finished above them). <a href="https://www.kaggle.com/c/generative-dog-images/discussion/98229">Or we can give everyone a chance to review fellow competitors code (at least for some type of competitions), as I have previously suggested</a> on Kaggle.</p><p>I believe in Kaggle’s mission and future. I have benefited so much personally from competing, learning, and meeting new people, as did hundreds of thousands of others. I hope the ban of the high-profile Grandmaster sends the right message to maintain the integrity of the platform and that any negative impact is contained.</p><p>May the Kaggle community keep thriving to push AI forward!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c86ca431f499" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Perfect lung cancer detections in a $1 million ML competition with an ingenious hack — how it was…]]></title>
            <link>https://medium.com/@u39kun/i-find-machine-learning-competitions-exciting-and-addicting-438fe95b33f5?source=rss-321747ff5910------2</link>
            <guid isPermaLink="false">https://medium.com/p/438fe95b33f5</guid>
            <category><![CDATA[kaggle]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Yusaku Sako]]></dc:creator>
            <pubDate>Mon, 06 Aug 2018 06:50:07 GMT</pubDate>
            <atom:updated>2020-04-25T23:50:08.132Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Perfect lung cancer detections in a $1 million ML competition with an ingenious hack — how it was done</strong></p><p>I find Machine learning competitions exciting and addicting. Competitors from around the world fight for prize money and prestige on a grand challenge by applying ingenious and oftentimes state-of-the-art methods against thousands who are trying to do the same. Certain peculiar situations and tactics arise in the world of machine learning competitions that would not happen in the real world due to the way such competitions are structured, such as data leaks and leaderboard probing (one interesting case of which I will describe here.) While such peculiarities can be used to criticize the practical utility of machine learning competitions, I find it interesting how people find and exploit “loopholes” in ingenious ways (as long as they are openly sharing such “vulnerability” in competition as they discover them).</p><p><strong>THE BATTLE GROUND — 2017 DATA SCIENCE BOWL</strong></p><p>I participated in <a href="https://www.kaggle.com/c/data-science-bowl-2017">2017 Data Science Bowl, a Kaggle machine learning competition hosted by Booze Allen for early detection of lung cancer based on CT scans of patients</a>. The objective was to determine whether a patient will be diagnosed with lung cancer within 6 months from the time the CT scan was taken. The total prize purse was whopping $1 million; competitors were motivated to employ all sorts of tactics to gain a slightest edge. A total of 1972 teams ended up competing. Although it has been more than a year since this competition closed, I wanted to share with you the most interesting thing that happened in this competition, at least in my mind.</p><p><strong>COMPETITION SETUP</strong></p><p>This competition consisted of two stages:<br>* In Stage 1, competitors would train their models using the official training set of around 1400 CT scans (and optionally any external public dataset), submit their lung cancer predictions on the public test set, and get their score back from the system. The public test set consisted of CT scans of 198 patients, whose cancer diagnosis results were known to the competition host but unknown to the competitors. The best score from each competitor is posted on the public leaderboard (this is commonly known as the public LB score in Kaggle.) This is a way to determine how well one is doing with respect to other competitors while the competition is ongoing. By the end of Stage 1, competitors were to submit their models and freeze the code to enter Stage 2.<br>* Stage 2 was one week long. The previously undisclosed labels for the public test set in Stage 1 were revealed to all the competitors in Stage 2. Also, the private test set (CT scans without ground truth labels) for Stage 2 were made available to make predictions on. Competitors were allowed a maximum of two different submissions whose scores will not be revealed until the end of Stage 2. Stage 1 scores were simply ignored and Stage 2 scores determined the final standings. Competitors were allowed to retrain their model by treating Stage 1 public test set as extra training data before performing inference on the Stage 2 private test set so long as no changes to their models or hyperparameters (including the learning rate schedule and number of epochs to train, etc.) were made (unless such changes were automated in the frozen code.)</p><p>Some advantage is gained by competitors who were able to mine the ground truth for the Stage 1 test data set while in still in Stage 1. They can use the mined ground truth on the test set and use them as extra training data (~14% more over the official training set) to tune their hyperparameters, train a vast number of models for ensembling, set up more reliable local validation results, figure out optimal learning schedules, etc., on this extra “training” data <em>before</em> they needed to submit and freeze the code at the end of Stage 1. Also there is definitely a “feel good” factor of beating out other competitors during Stage 1 even if that does not directly contribute to the final standing.</p><p><strong>THE IMPOSSIBLE HAPPENS</strong></p><p>Soon after the competition launched, an amazing thing happened.</p><p>While the top competitors at the time were scoring worse than 0.5 (lower scores are better), there was one gentleman named Oleg Trott, who had a *perfect score* of 0.00000 on the public test set after only after 16 submissions over 6 days (a maximum of 3 submissions per day were allowed in this competition.)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/835/1*Stce9BSay88WWthJnBF4MA.png" /></figure><p>This seemed impossible.<br><a href="https://www.kaggle.com/c/data-science-bowl-2017/discussion/31619#175345">A radiologist resident who was also competing in the competition reported that his manual examination yielded around 0.30–0.38 on various subsets</a>.<br>A brute force method of figuring out one target label per submission would have taken at least 66 days (198 labels with 3 submissions a day.)</p><p><strong>THE REVEAL</strong></p><p>Oleg graciously revealed his ingenious technique for getting the 0.00000 perfect score and shared his code with others; he voluntarily made it fair to all the competitors by giving them a chance to mine the labels for the public test set. Since the public test data were to be released to everyone anyway at the beginning of Stage 2, the competition organizer did not ban this technique, though such perfect scores were removed from the public leaderboard for hygiene.</p><p>So how was this done? You can look at <a href="https://www.kaggle.com/olegtrott/the-perfect-score-script">Oleg’s explanation here</a>. His explanations are concise and come complete with the code he used. When I first read it, there was definitely some magic there that I did not understand — that led me to perform a deeper analysis.</p><p><strong>HIGH-LEVEL OVERVIEW OF THE METHOD</strong></p><p>A submission file in Stage 1 consisted of 198 pairs of scan IDs and predictions on the degree of confidence for cancer diagnosis, like the following:</p><p>id,cancer<br>026470d51482c93efc18b9803159c960,0.2037569618639993<br>031b7ec4fe96a3b035a8196264a8c8c3,0.34115835466290567<br>03bd22ed5858039af223c04993e9eb22,0.11568318622008714<br>06a90409e4fcea3e634748b967993531,0.15048923567505673<br>07b1defcfae5873ee1f03c90255eb170,0.30526565977678183<br>…</p><p>Based on how well your predictions are, you get some score back from the system, say 0.55213. If you submit different prediction values, then the score could go up or down.</p><p>The method for extracting the ground truth exploited the fact that the score returned by the system is relatively high precision at 5 decimal places. <em>By carefully engineering prediction values in each submission, it was possible for the system to spit out a unique score for every possible combination of the actual ground truth for a subset of the test set. Then multiple submissions could be made in a rolling manner to reveal the ground truth for all 198 samples.</em></p><p>Here’s what Oleg did.</p><ul><li>Set the prediction values for 15 of the 198 samples to the values below and all 183 others to 0.5:<br>[4.994803e-01, 4.989605e-01, 4.979210e-01, 4.958421e-01, 4.916848e-01,<br> 4.833741e-01, 4.667850e-01, 4.338618e-01, 3.699983e-01, 2.564604e-01,<br> 1.063198e-01, 1.395599e-02, 2.002820e-04, 4.012894e-08, 1.610332e-15]</li><li>Submit this file to retrieve the score back from Kaggle</li><li>Based on this score, the exact ground truth label for the 15 samples can be figured out with some math wizardry. For example, a score of 0.67220 would imply that the 15 labels are [1 1 0 0 0 1 0 0 0 0 0 0 0 0 0], while a score of 0.91589 means that [0 0 1 1 0 0 1 1 0 1 0 1 1 0 1]</li><li>Repeat in a rolling fashion for the rest of the samples (submit the same prediction values as above, but for a different set of 15) until all 198 labels are figured out</li></ul><p>How does this even work? Read on…</p><p><strong>A DEEP-DIVE ON THE METHOD</strong></p><p><strong><em>Caution! The rest of the post is a bit math heavy!</em></strong></p><p>For each prediction made on a sample, the score (or the loss — these two terms may be used interchangeably) is computed by the log loss:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/656/1*lll2Wwu0XZnLKRkNRP9kog.png" /><figcaption>Sample loss</figcaption></figure><p>where <em>y_i</em> is the ground truth label (either 0 or 1), and <em>p_i</em> is the prediction (between 0 and 1), for sample <em>i</em>.</p><p>Written in another way:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/828/1*FPQPWLXlveKeuCy8CUZbvg.png" /></figure><p>If the ground truth is 1 and the prediction is 1 (or if the ground truth is 0 and the prediction is 0), then the sample loss is 0. If the ground truth is 0 and the prediction is 1 (or if the ground truth is 1 and the prediction is 0), then the loss is infinite; being absolutely certain and being wrong punishes infinitely. However, in this competition <a href="https://www.kaggle.com/c/data-science-bowl-2017#evaluation">prediction values were automatically forced into the range [10^-15 , 1–10^–15] for score evaluation</a>, so the maximum loss one can incur on a sample is -ln(10^-15), which is about 34.53878. The minimum loss, on the other hand, is -ln(1–10^-15), which is around 9.99201e-16 (basically 0.)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vlPXWZ5zbuVXo_BdxS_5bQ.png" /></figure><p>To compute the overall score <em>Loss</em>, all individual losses are simply averaged across all N samples.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/314/1*4s-S4eSoYL9kl-mrjheGhg.png" /><figcaption>Overall loss</figcaption></figure><p>This results in the score range of 0.00000 and 34.53877.</p><p>Making a prediction of 0.5 for all samples (basically saying 50% of cancer or “I have no freakin’ idea”), would give a score of 0.69314 (-ln 0.5.)</p><p>Oleg exploited the fact that the scores returned by Kaggle had a relatively high precision with 5 decimal places.</p><p>If we ignore decimal places, this is equivalent to the system giving out a score from 0 to 3,453,877, based on the predictions you submit; there could be 3,453,878 distinct scores that the system could yield based on different predictions submitted.</p><p>In information theoretic terms, this means that the score can potentially “give out” 21.7 bits of information since log2 3,453,878 ~= 21.7.</p><p>So in theory it is possible to extract up to 21 labels per each submission, if one could encode submitted prediction values in such a way that each of different combinations of predictions yield to a distinct score.</p><p>I say “in theory”, because there are practical concerns, like side effects of floating-point math such as under/overflow, rounding, and as well as being able to come up with an ideal encoding.</p><p>We know that submitting a prediction of 0.5 yields a sample-wise score of -ln 0.5, <em>regardless of the underlying ground truth label</em> (which is either 0 or 1.) Since we can only extract up to 21 bits of information, we can “target” at most 21 labels to be extracted by setting all other non-target sample prediction to 0.5 because we already know the exact loss that these samples would contribute. Then we can focus on mining the target labels. From now on I will refer to this number of target labels we are trying to extract per submission as T and the number of total samples as N (e.g., T=21 and N=198.) With T target labels to extract, the overall loss can be rewritten as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/666/1*h-7rUtin36WXg57M-Fe6sQ.png" /></figure><p>We want to come up with a scheme where each of the 2^T different combinations of target labels would give a distinct score, so that we can map the received score back to to the actual labels in an unambiguous manner.</p><p>Note that each prediction p_i can be any real number between 0 to 1. Setting a prediction to 0.5 gives no information since it yields a constant of -ln 0.5 regardless of the ground truth label.</p><p>Let’s take a simple example of extracting one ground truth label from the score.<br>Suppose we set the prediction for exactly one target sample to 0.1 and 0.5 for all others. For that one target sample, its score is:</p><ul><li>-ln 0.1 ~= 2.30259 if the ground truth label is 1</li><li>-ln 0.9 ~= 0.10536 if the ground truth label is 0</li></ul><p>For all others, it is -ln 0.5 ~= 0.69314</p><p>If we submitted this set of predictions, the total score would be:</p><p>(-ln 0.1 + -ln 0.5 * 197)/198 ~= 0.70128 if the ground truth label is 1</p><p>(-ln 0.9 + -ln 0.5 * 197)/198 ~= 0.69018 if the ground truth label is 0</p><p>So the score revealed the ground truth label for that one particular sample. But we can be much more efficient than that.</p><p>To extract multiple target labels per submission, we want to set the predictions for each target sample differently such that we can determine their exact labels from the resulting score unambiguously. We can do that by setting the prediction values for the T target samples such that every possible combination of 2^T scores (e.g., at T=21, this would be 2,097,152 different combinations) is unique.</p><p>Since the system gives out the score up to the 5th decimal place, we want the total score for each binary combination to be separated by 0.00001 to evenly fill the score space, starting from 0.00000 to 20.97151 (at T=21). Coming up with such an efficient, ideal encoding may not be feasible in practice, though we can get close enough. Oleg used T=15. I believe he set T=15 to be large enough to get a low number of submissions but safe enough to leave a big enough margin for floating point / score calculation side effects (otherwise he would have wasted more submissions trying to find the optimal T than using a conservative number.) I have experimented with T values greater than 15 which worked just as well.</p><p>What prediction values p_i’s can we set to make all scores unique? Here’s what Oleg used:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/478/1*DIaJUdpa3-PZ4Jv-D_S6iw.png" /><figcaption>Encoding function</figcaption></figure><p>where i goes from 1 to T corresponding to each of T targets to be extracted, N is the number of total samples in the submission, and <em>epsilon</em> is a scale factor to control the granularity and range of resulting scores.</p><p>Also, <a href="https://www.kaggle.com/olegtrott/the-perfect-score-script#168319">Oleg claimed</a> that:</p><blockquote>With a little bit of math, and ignoring truncation during I/O and rounding errors on the server, you should be able to show that, when you use the sigmoid expression, the loss equalsconst + epsilon * X where X is the integer whose binary representation is the 15 labels.</blockquote><p>The encoding formula and its loss seem a bit magical, so let’s unpack this.</p><p>Sigmoid is commonly used as an activation function in neural networks and is called the squashing function since it takes any real number and outputs a number between (0, 1). Sigmoid is defined as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/354/1*9YgKZH3crk5CfnNJtHF8hw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*c6TwMq-rrtPiPWkLJH2yrw.png" /></figure><p>For now, let’s forget the N and <em>epsilon</em> in the encoding function for p_i for simplicity; we’ll come back to those later. Then we are left with:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/380/1*B6SNXxY60OXu1dEJAITikA.png" /></figure><p>where i ranges from 1 to T. Note that we are using powers of 2’s *-1. This is to force the sample loss for the y_i = 1 case to be greater than when y_i = 0. This is for arithmetic nicety to follow the standard binary encoding convention where 0,0,0,..,0 takes lowest value and 1,1,1,…,1 takes the highest value.</p><p>Looking at the case where i = 1:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/596/1*uMaZYwN2L-KOXQoHlRXbgQ.png" /></figure><p>The loss for target sample 1 is:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hxcBGnvBoWSSETbUH1BkvA.png" /></figure><p>For the y_1=0 case, we used the fact that -ln(1-sigmoid(-1))=-ln(sigmoid(1)) because 1-sigmoid(x) = sigmoid(-x). A proof can be found at the end of this post.</p><p>Similarly, the loss for target sample 2 is:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*29S8KYAuMOZVclMoBxw4PQ.png" /></figure><p>In general, for any target sample <em>i</em>, the loss is:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DoR16UrwqF2lnuCyaK-62g.png" /></figure><p>Let’s simplify the notation and derivation a bit by introducing the <a href="http://papers.nips.cc/paper/1920-incorporating-second-order-functional-knowledge-for-better-option-pricing.pdf">softplus function</a>.</p><p>softplus(x) = -ln(sigmoid(-x)) = ln (1+e^x)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TRhsnzRwUwki7_1w4Sq4NQ.png" /></figure><p>Softplus has this neat property:</p><p>softplus(x) - softplus(-x) = x</p><p>This is <a href="https://www.deeplearningbook.org/contents/prob.html">Equation 3.41 in the book <em>Deep Learning</em> by I. Goodfellow at el</a>. To see why this is so, see the bottom of this post for a proof.</p><p>If we rewrite the sample loss in terms of softplus:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*msZG8IZqDfVG-qO--hmacw.png" /></figure><p>What’s the difference between the sample loss when the ground truth label is 1 vs 0 for any given sample <em>i</em>?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2gNUH8mb_WNu4l_c42rRNQ.png" /></figure><p>So the sample loss difference ranges from 1, 2, 4, 8, …, up to 2^(T-1) based on which sample it is.</p><p>We are focusing on the loss difference from the label being 1 vs 0 for each sample <em>i</em>, because unlike standard binary encoding, the label (or the “bit”) being 0 actually contributes to the total by the amount softplus(-2^i-1). This effectively adds a constant to the loss. <em>The target loss is at its minimum when all the target ground truth labels are 0’s. Let’s call this constant Z (for all Zeros.)</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/438/1*nW6cfK29XYXhfW_O3Liftw.png" /></figure><p>Similarly, the target loss is at its maximum when all the target ground truth labels are 1’s. We can call this O (for all Ones.)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/368/1*xWquET5Xi9kuWQO5J24l5g.png" /></figure><p>For anything “in between” Z and O (i.e., any combination of 1s and 0s for T samples), sample <em>i</em>’s label being 1 positively offsets Z by 2^i-1. This means that all losses will be unique based on every combination of ground truth labels as in standard binary encoding.</p><p>With the above, we can encode predictions and map back to the exact ground truth labels based on the score. But there are a couple of big problems. sigmoid(-2*i) for i &gt; 9 causes underflow and the expression evaluates to zero. In addition, the range of the score is much bigger than the maximum score of -ln 10^-15 ~= 34.53878 (this max is due to truncation of individual sample predictions performed by the scoring system as mentioned earlier.) Furthermore, the granularity of the score based on this is 1 (no two scores are separately by less than 1.) We can make this closer to 1e-5, which is the precision of the score given by the system to scale the range of the score down to fit. All of the above problems are solved by scaling the predictions by <em>epsilon </em>as shown in the original encoding formula. Oleg set <em>epsilon</em> to 1.05e-5. Using 1.05e-5 rather than 1e-5 avoids any potential floating-point math precision/rounding issues. And let’s not forget that the total score is the average of all sample losses. So we need to counter the fact that a factor of 1/N is applied to all sample losses. This is the reason why a factor of N appears in the encoding function.</p><p>Let’s revisit Oleg’s claim that:</p><blockquote>the loss equals<em>const + epsilon * X</em> where <em>X</em> is the integer whose binary representation is the 15 labels</blockquote><p>The <em>const</em> referred is Z as defined above, and the <em>epsilon * X </em>term arises because the score increases monotonically as <em>X</em> increases and the step size in neighboring scores is <em>epsilon</em>.</p><p>I hope that this shed some light on Oleg’s ingenious technique for getting the perfect score in Stage 1 of Data Science Bowl 2017. It seems that more recent competitions hosted on on Kaggle are very much aware of this exploit and have significantly limited the precision of scores made available to competitors, though I’m not sure about other platforms.</p><p>Sorry if some parts felt belabored, but I wanted to err on the side of making it more obvious than not (to me, at least!) I hope you enjoyed this article as much as I had fun writing it!</p><p><strong>APPENDIX</strong></p><p><strong><em>Prove that 1-sigmoid(x) = sigmoid(-x)</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6MhZyLpODDISgebgsNFfAg.png" /></figure><p>You can see why this is the case visually since sigmoid and 1-sigmoid’s symmetry about the y-axis (x=0) and y=0.5:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aJenl_V21Me3llPssdXlrg.png" /></figure><p><strong><em>Prove that softplus(x)- softplus(-x) = x</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HDidaM5HvUDf2YPsq5K1Yw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/532/1*cxpATXhG4FncyBfWoQQV7g.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=438fe95b33f5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[“Is the term “softmax” driving you nuts?”]]></title>
            <link>https://medium.com/@u39kun/is-the-term-softmax-driving-you-nuts-ee232ab4f6bd?source=rss-321747ff5910------2</link>
            <guid isPermaLink="false">https://medium.com/p/ee232ab4f6bd</guid>
            <category><![CDATA[python-programming]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[mathematics]]></category>
            <dc:creator><![CDATA[Yusaku Sako]]></dc:creator>
            <pubDate>Sat, 02 Jun 2018 18:41:10 GMT</pubDate>
            <atom:updated>2018-08-09T07:32:07.438Z</atom:updated>
            <content:encoded><![CDATA[<p>The <em>softmax</em> activation function is a basic building block that we use often in machine learning. Got a multi-class classification problem? Just whip out <em>softmax</em> to cast the vector to a probability distribution.</p><p>But the term <em>softmax</em> never sat well with me and really bugged the heck out of me every time I type it or read it!<br>The “soft” part is clear… It is continuous and differentiable so that you can do gradient descent to optimize the loss function. But “max”? C’mon, really?</p><p><em>Max</em> returns the largest element of the input.<br>max([-2, 1, 5, 0]) =&gt; 5. This operation preserves the magnitude of the largest element.</p><p><em>Softmax</em>, on the other hand, transforms an input vector to a probability distribution:<br>softmax([-2, 1, 5, 0]) =&gt; [0.0009, 0.0179, 0.9747, 0.0066]<br>This is not really telling uswhat the <em>max</em> of the input is. But it seems to tell us which element is the largest. Isn’t that <em>argmax?</em></p><p><em>Argmax </em>returns which element in the input is the largest:<br>argmax([-2, 1, 5, 0]) =&gt; 2 since element at index 2 is the largest (assuming 0-based index). This would be expressed in one-hot encoding as:<br>[0, 0, 1, 0]</p><p>This looks a lot like the <em>softmax</em> output: <em>softmax</em> is a soft version of one-hot encoded <em>argmax, </em>not<em> max.</em></p><p>So what would be a “soft” version of <em>max</em>?<br>softmax(x) * x.T<br>Following the example above:<br>softmax([-2, 1, 5, 0]) * [-2, 1, 5, 0].T =&gt;[0.0009, 0.0179, 0.9747, 0.0066] * [-2, 1, 5, 0].T =&gt; 4.8896<br>This is close to the actual <em>max</em> of 5.</p><p>This misnomer of <em>softmax </em>is also mentioned briefly in the book <em>Deep Learning</em> by Ian Goodfellow, Yoshua Bengio, and Aaron Courville. The authors said:</p><blockquote>It would perhaps be better to call the softmax function “softargmax” but the current name is an entrenched convention.</blockquote><p>I have heard folks refer to the naming of <em>softmax</em> in contrast to <em>hardmax, </em>as if <em>hardmax</em> is the term that came before <em>softmax</em> to represent one-hot-encoded argmax<em>.</em> However, I can’t find any basis for this. The first usage of the word <em>softmax</em> seems to have come from “<a href="https://papers.nips.cc/paper/195-training-stochastic-model-recognition-algorithms-as-networks-can-lead-to-maximum-mutual-information-estimation-of-parameters.pdf">Training Stochastic Model Recognition Algorithms as Networks can Lead to Maximum Mutual Information Estimation of Parameters</a>” by John S. Birdie in 1990. To quote:</p><blockquote>This transformation can be considered a multi-input generalisation of the logistic, operating on the whole output layer. It preserves the rank order of its input values, and is a differentiable generalisation of the ‘winner-take-all’ operation of picking the maximum value. <strong>For this reason we like to refer to it as soft max.</strong></blockquote><p>Note that there was no mention of <em>softmax</em> being the soft version of <em>hardmax.</em></p><p>If <em>softmax</em> doesn’t bother you, then you are good. But it does as it did to me, think <em>softargmax</em>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ee232ab4f6bd" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Managing your machine learning experiments and making them repeatable in TensorFlow, PyTorch…]]></title>
            <link>https://medium.com/@u39kun/managing-your-machine-learning-experiments-and-making-them-repeatable-in-tensorflow-pytorch-bc8043099dbd?source=rss-321747ff5910------2</link>
            <guid isPermaLink="false">https://medium.com/p/bc8043099dbd</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Yusaku Sako]]></dc:creator>
            <pubDate>Fri, 27 Apr 2018 03:44:55 GMT</pubDate>
            <atom:updated>2018-09-06T06:29:33.507Z</atom:updated>
            <content:encoded><![CDATA[<h3>Managing your machine learning experiments and making them repeatable in TensorFlow, PyTorch, Scikit-Learn, or any framework in Python</h3><p><a href="https://www.kaggle.com/u39kun">I participate in Kaggle (machine learning competitions) for fun</a> and I have literally run thousands of experiments over a number of competitions I’ve participated in.</p><p>Getting your models to perform really well, especially better than hundreds (or thousands!) of other teams that you are competing with is no easy task. The process is very iterative. You run a bunch of experiments, changing the underlying models, trying different optimizers and tuning hyperparameters, changing your sampling methods, changing the way you create splits, trying different libraries or different versions of libraries, changing your random seeds... You may also be running multiple experiments on multiple GPUs/multiple hosts to parallelize the process and each might have different hardware spec / OS.</p><p>If you don’t have a system for managing your experiments, it quickly gets out of control. Sometimes you run into situations where you cannot reproduce the great local CV or public LB score that you achieved since you don’t know what you changed since then! Arrrrggghhh!!! You forget why some experiments worked better than others. You forgot what worked and why, and equally important, what didn’t work and why.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wdtdcVQQRzc7xPNZzyCsUg.jpeg" /></figure><p>Hopefully you are using version control, such as Git, and that helps you keep track of code changes, but are you really disciplined enough to commit every single change you make? — a lot of times you want to just change your hardcoded hyperparameters and re-run the experiment. Or sometimes you parameterize arguments to your script so version control does not help you keep track of what was used at runtime, unless you explicitly captured them in some other ways. Other times, you use a different version of some library because you need new features/bug fixes. Version control does not help there either, unless you are disciplined about also checking in all the dependencies and their versions. You dump a whole bunch of useful information in the console output as the models are training/evaluating. Are you keeping track of all of that for every single run?</p><p>If you are running your machine learning experiments in Python, <em>there’s something that lets you do all of that very easily, with minimal changes to your code</em>. It is appropriately called… <a href="https://github.com/IDSIA/sacred">SACRED</a>!</p><p>Definitely check it out. It is super simple to integrate (just pip install add a few lines to your code). <a href="http://sacred.readthedocs.io/en/latest/">The documentation is awesome</a>. How does Sacred help you? Sacred dumps all your experiment results into MongoDB, which is a scalable JSON database. It literally dumps and saves everything… including:</p><ul><li>Metrics (training / validation loss, accuracy, or anything else you want to track)</li><li>Console output (stdout)</li><li>All the source code you executed (basically dumps a copy of your source code into MongoDB)</li><li>All the imported libraries and their versions used at runtime</li><li>All your config parameters that you expose through the command line</li><li>Hardware spec of your host / GPUs</li><li>Random seeds</li></ul><p>and much more. Everything you need to revisit and examine or repeat your experiments later as needed.</p><p>Also, it has nice integrations with Slack / Telegram, etc. I have it set up so that I get a Telegram instant message when my experiments complete / fail.</p><p>MongoDB does not come with Sacred, but <a href="https://hub.docker.com/_/mongo/">you can easily run MongoDB in Docker container while storing the data on the host</a>. To visualize the captured data from your web browser, you can use a Sacred viewer, such as <a href="https://github.com/chovanecm/sacredboard">sacredboard</a>. Or you can view / manipulate the stored data via <a href="https://github.com/mongo-express/mongo-express">mongo-express</a>.</p><p>Sacredboard works fine, but it has some major limitations as of this writing. For example, the columns that show up in the data table of your experiments are hardcoded and they cannot be modified (unless you change multiple files in the source code — for each Kaggle competition, I had been creating a custom version of sacredboard to expose the right hyperparameters that I wanted to see in the table.) Also it lacks the ability to add notes about your experiments. It also does not let you tag and filter your experiments. It does not let you see the best epoch, best validation loss, in the table unless you expand the row to view experiment details. Due to these limitations, a friend and I created our own version of Sacred viewer, which overcomes these limitations and more (basically built what we needed to meet our needs.)</p><p>UPDATE on September 5, 2018:<br>We are excited that we just released Omniboard, a new frontend for Sacred. It was inspired by Sacredboard with enhancements mentioned above. Special thanks to <a href="https://medium.com/u/6b1d1bc19723">Vivek Ratnavel</a> who built it from scratch using React + Express. Omniboard can be easily installed via “npm” or be run as a Docker container. Instructions (and code) are here: <a href="https://github.com/vivekratnavel/omniboard">https://github.com/vivekratnavel/omniboard</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bc8043099dbd" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Titan V vs 1080 Ti — Head-to-head battle of the best desktop GPUs on CNNs.]]></title>
            <link>https://medium.com/@u39kun/titan-v-vs-1080-ti-head-to-head-battle-of-the-best-desktop-gpus-on-cnns-d55a19866b7c?source=rss-321747ff5910------2</link>
            <guid isPermaLink="false">https://medium.com/p/d55a19866b7c</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[convolutional-network]]></category>
            <category><![CDATA[gpu]]></category>
            <category><![CDATA[pytorch]]></category>
            <dc:creator><![CDATA[Yusaku Sako]]></dc:creator>
            <pubDate>Thu, 21 Dec 2017 10:38:40 GMT</pubDate>
            <atom:updated>2018-01-12T07:12:43.788Z</atom:updated>
            <content:encoded><![CDATA[<h3>Titan V vs 1080 Ti — Head-to-head battle of the best desktop GPUs on CNNs. Is Titan V worth it? 110 TFLOPS! no brainer, right?</h3><p>NVIDIA’s Titan V is the latest “desktop” GPU built upon the Volta architecture boasting 110 “deep learning” TFLOPS in the spec sheet. That’s an incredible number. Compare that with 1080 Ti, the current king for the “desktop” GPU, that puts out 11 “normal” TFLOPS with 11GB of GDDR5 memory at a very reasonable $699 sticker price.</p><p>While the promise of Titan V sounds very exciting, the $2999 price point is a bit hard to swallow. But with 10x TFLOPS, should you be purchasing Titan V instead? Is that like getting 10 1080 Ti’s? Or are you better off buying 4 1080 Ti’s for the same money?</p><p>On a side note, the <a href="https://www.nvidia.com/en-us/data-center/dgx-station/">DGX Station is currently on sale for $49900</a> (normally priced at $69900). It comes with 4 watercooled server-grade Tesla V100’s which are slightly better spec’d than the desktop-grade Titan V (though the tower looks BOSS and I’m guessing it comes with support and SLA.) But most of us (heck, I bet a lot of university and AI research labs) don’t have that kind of money to throw around. Even if you did, not sure it if makes much economical sense. BTW, if you are operating a data center, <a href="https://www.theregister.co.uk/2018/01/03/nvidia_server_gpus/">NVIDIA has recently updated their EULA to prohibit “desktop” grade GeForce and Titan GPUs from being used in data centers.</a></p><p>So the question is, is Titan V worth it if you are looking to build your own GPU rig?</p><p>Let me start by saying that Titan V and its server-grade big brother Tesla V100 are pretty new. V100 came out in May 2017, and Titan V just came out this month (Dec 2017.) Most of deep learning frameworks have been rushing to add Volta support to make sure that all the potential performance can be exploited.</p><p>For example, PyTorch have only recently come out with the 0.3.0 release which adds support for CUDA 9 and Volta GPUs. <a href="https://medium.com/@u39kun/deep-learning-on-nvidia-titan-v-first-look-1d368d3443fb">I have been playing around with this setup </a>and the PyTorch community has been awesome (especially <a href="https://medium.com/u/45e1eae66802">Soumith Chintala</a> — thanks dude!) at helping me out. Now that I can run PyTorch on Titan V, I’ve written some benchmark code to demonstrate performance differences. So far, the results only include those of PyTorch 0.3.0 but I’m planning on adding results for other frameworks as well (EDIT: now <a href="https://github.com/u39kun/deep-learning-benchmark">the benchmark results contain the numbers from TensorFlow 1.4.0 and Caffe2 0.8.1 as well</a> — see the link for more details), all on CUDA 9.0.176 and CuDNN 7.0.0.5, to even out the playing field.</p><p><a href="https://github.com/u39kun/deep-learning-benchmark">u39kun/deep-learning-benchmark</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ve-8wjoqjzoZA6piYNM-UA.png" /><figcaption>Performance comparison of Titan V vs 1080 Ti</figcaption></figure><p>So let’s check out the results above.<br>Titan V and 1080 Ti were compared head-to-head against the same settings (in addition, these cards are both on the 16x PCIE slots on the same computer.)</p><p>The timing numbers shown above are for the forward pass through the CNN (“eval”) and forward and backward pass (“train”), measured in milliseconds. Those are average numbers computed over 20 passes after some warmup runs. I’ve run this many times, and these numbers are pretty stable.</p><p>Interesting points to note:</p><ul><li>Obviously, Titan V is faster than 1080 Ti. However, if you simply compare the 32-bit (“single” precision) runs, Titan V is only ~20% faster than 1080 Ti.</li><li>Titan V’s 16-bit (“half” precision) runs are non-trivially faster than those of the 32-bit counterpart. 1080 Ti benefited from going half-precision, but the performance gains are pretty modest compared with Titan V.</li></ul><p><em>These numbers do not really scream “OH MY GOODNESS TITAN V IS A NO BRAINER.”</em></p><p>Is NVIDIA lying about the 10x TFLOPS? I’m sure their marketing folks are good honest people, but there are multiple factors at play here. One, I’m sure there is still room for improvement from the software side to extract every bit of the Volta’s super fast Tensor cores. But even then, if most code paths do not conform to the conditions that allow for maximum theoretical performance (<a href="https://devblogs.nvidia.com/parallelforall/programming-tensor-cores-cuda-9/">see A FEW RULES in “Programming Tensor Cores in CUDA 9 by NVIDIA</a>), there’s only so much you can do. Time will tell how much improvements we will see at the framework/CUDA/CuDNN level to exploit the Volta GPU capabilities as much as they can, but the initial numbers that I have observed on popular CNN’s (I do computer vision stuff mostly hence my focus on CNN’s) don’t seem to justify getting a Titan V, especially if you were to get one now as software plays catch up.</p><p>Although it is fun to play around with the latest tech and be an early adopter, if you are focused on doing AI research or building products on your personal GPU rig <em>economically</em>, I would suggest buying 1080 Ti’s as of this writing (until NVIDIA comes out with an even better alternative in the near future; they tend to surprise us with new releases.) Also consider that a single Titan V is only 12GB of memory while 1080 Ti has marginally smaller memory at 11GB. But if you buy 4 of them for the same money (at least on GPUs… you also need a beefier power supply, motherboards that can support 4 GPUs, more RAM, and better cooling so on), you will end up with way more GPU memory (44GB vs 12GB.) Having said that I think a more practical thing to do, if you want to get more than one GPU, is to get 2x 1080 Ti’s. This way, the GPUs do not have to be stacked right on top of another (more airflow for cooling — when these cards thermal throttle, you could lose performance significantly… you can do water cooling but that adds significantly to the cost, effort, and some risk.) Also with only two cards, the power draw is 250Wx2 from the GPUs so you can get away with having a cheaper power supply. At any rate, I find it extremely valuable to have multiple GPUs and split experiments across them so that I can iterate quickly, so this is what I would recommend as a good compromise.<br>Another option for addressing the heat issue of stacked, air-cooled GPUs, if you must have 3–4 of them, don’t want to water cool, and don’t care about aesthetic or noise, is to <a href="http://blog.kaggle.com/2017/12/22/carvana-image-masking-first-place-interview/">get some PCIE extenders/risers and do what this winning Kaggler did</a>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/768/1*43kWtwSfU6Fg9ZX0Bc8h5Q.png" /><figcaption>A 4x 1080 Ti machine by <a href="https://medium.com/u/a5f99268704">Vladimir Iglovikov</a>, one of the <a href="http://blog.kaggle.com/2017/12/22/carvana-image-masking-first-place-interview/">winning team members of Carvana Image Masking Challenge</a></figcaption></figure><p>I hope you find this post helpful. Please let me know if there are any particular results that you want to see on Titan V or 1080 Ti. Thank you for reading!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d55a19866b7c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Deep Learning on NVIDIA Titan V — First Look]]></title>
            <link>https://medium.com/@u39kun/deep-learning-on-nvidia-titan-v-first-look-1d368d3443fb?source=rss-321747ff5910------2</link>
            <guid isPermaLink="false">https://medium.com/p/1d368d3443fb</guid>
            <category><![CDATA[gpu]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[pytorch]]></category>
            <category><![CDATA[titan-v]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Yusaku Sako]]></dc:creator>
            <pubDate>Tue, 19 Dec 2017 09:33:48 GMT</pubDate>
            <atom:updated>2017-12-21T10:43:43.494Z</atom:updated>
            <content:encoded><![CDATA[<p>On December 7, 2017 at 9PM PST, I got an email: “NVIDIA Titan V is here! Titan V — THE MOST POWERFUL PC GPU EVER CREATED. BUY NOW.”</p><p>I don’t usually get excited this much over what one might consider spam, but this got my heart racing. This came out of nowhere. This meant that the“consumer” version of the server-grade NVIDIA Tesla V100 GPU has become available for purchase. <a href="https://www.engadget.com/2017/05/10/nvidias-first-volta-powered-gpu-sits-in-a-149k-supercomputer/">NVIDIA sells a watercooled tower with four Tesla V100 GPUs for $69,000</a> (<a href="https://www.nvidia.com/en-us/data-center/dgx-station/?ncid=van-dgx-statio&amp;mkt_tok=eyJpIjoiWkdVNFpEaGhPRFk1TVRnMCIsInQiOiJPc0xKR25oNlA5Z2cwbU9oRXV4ekRCM2xlVDVKR3NhN1NEOTBqSHE5WEZtTU1NQTN4N3JpSSttS0ZIMDNkMmtra29aM1pCZTI5aExmNnZcLzRaZEpaTnVLUW9PdDBycVpaTks1aTg0dFhuazUxVitUVndmTE9hUDdEMnlNdDgrOHYifQ%3D%3D">on sale now for $49,900</a>) which is prohibitive for most enthusiasts/AI researchers where as a <a href="https://www.nvidia.com/en-us/titan/titan-v/">Titan V costs $2999</a> (which is still damn expensive but much more affordable.) Like the venerable V100, Titan V is built upon the same Volta architecture and boasts huge performance numbers. It is slightly “detuned” in terms of the spec (V100 is 16GB where as Titan V is 12GB, for example.)</p><p>Besides the $2999 price tag, the most eye-catching number on NVIDIA’s Titan V product page is “110 Deep Learning TeraFLOPs.” That is 110 trillion floating point operations per second! This is a HUGE number can turn your home PC into a bona fide supercomputer.</p><p>Throughout this year, I have been training deep learning models on several NVIDIA 1080 Ti’s that cost “only” $699 a piece for the NVIDIA’s stock version. These put out amazing performance with 11 TeraFLOPs with its 3584 CUDA Cores. So what is up with Titan V’s 110 TeraFLOPs? Can you swap your 1080 Ti with a Titan V and expect 10x speed up on training/evaluating your models?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DJ0BozFHiH24qrNtKJwfiQ.jpeg" /><figcaption>1080 Ti and Titan V ready to rock</figcaption></figure><p>The caveat is in the phrase “Deep Learning” TeraFLOPs. What this marketing jargon means is that “for certain operations used heavily in deep learning”, it can perform 110 trillion operations per second. That certain operation is “<a href="https://devblogs.nvidia.com/parallelforall/programming-tensor-cores-cuda-9/">matrix-multiply-accumulate</a>” and it can be performed extremely fast by Titan V’s “Tensor Cores.” Great! So existing models can just utilize these Tensor Cores and get an incredible speed up, right?</p><p>Well, there are more caveats to this. Each of these Tensor Cores can perform multiplications of a pair of 4x4 matrices of *half precision floating numbers* and add to a 4x4 matrix of either half precision or single precision floating numbers to create a resulting 4x4 matrix of either half precision or single precision floating numbers. Multiples of such Tensor Cores can be run in parallel to get massive gains in execution speed.</p><p>Typically, when models are trained, 32–bit “single-precision” floating point numbers (aka FP32) are used to store model weights, activations, gradients, etc. But these Tensor Cores require “half-precision” floating point numbers (aka FP16.) <em>So this means that your code must be modified to take advantage of these Tensor Cores.</em> We can just use FP16 instead of FP32 and be done with it? But if that’s the case why haven’t we been using FP16 to begin with, if FP16 is sufficient for training high performing models, rather than the standard FP32?</p><p>There’s an excellent paper <a href="https://arxiv.org/abs/1710.03740">“Mixed Precision Training” by Narang et al</a> that describes the implications of using FP16 to train deep learning models.<br>The TL;DR version is that 1) highly accurate models comparable to FP32-trained models can be trained utilizing FP16, 2) there still needs to be a “master copy” of the model weights kept in FP32, 3) FP16 can be used throughout forward and backward passes to represent a copy of weights, activations, and gradients, 4) when updating the weights using the computed gradients, the master weights are updated and stored in FP32, and 5) sometimes “loss scaling” is needed for certain models but not always. Basically, you need to make use of FP16 very carefully, or else your model may not converge or your model’s accuracy could suffer greatly. This is because you don’t have that many bits in FP16 so it could easily “underflow.” There could be several causes for this. When computing the delta for weight update, the gradient multiplied by the learning rate can be a very small non-zero number in FP32 whose FP16 representation is 0. Even if the representation in FP16 is non-zero, if the *scale* of the weights and the update delta is bigger than a certain threshold — more than a factor of 2048 — then the resulting sum is exactly the same as what it was before the sum, so essentially weights do not change. There are techniques available to workaround these. According to the experimental results shown in the paper, if we take care of these details, resulting models trained using FP16 can perform as well as if you had just used FP32 while significantly speeding up training.</p><p>Net net, utilizing Tensor Core takes work. What if we just swapped a 1080 Ti with a Titan V, you would still get some immediate speed up without any changes since Titan V has 5120 CUDA Cores (F32 14.9 TeraFLOPs) vs 1080 Ti‘s’ 3584 CUDA Cores (F32 11 TeraFLOPs), faster memory, and all that jazz?</p><p>I have been using PyTorch as my deep learning framework of choice. I had recently built a model based on the Multi-View Convolutional Network architecture (<a href="https://arxiv.org/abs/1505.00880">https://arxiv.org/abs/1505.00880</a>) to train networks that can automatically identify hidden threats from the 3D scans produced by the TSA scanners (those ones that you have to go thru at the airport security.) This was for the <a href="https://www.kaggle.com/c/passenger-screening-algorithm-challenge">$1.5-million-dollar-prize machine learning competition hosted by TSA on Kaggle</a> but that’s a topic for another time.) Since I had trained dozens of these for ensembling and it took a long time on 1080 Ti’s, I was really curious to see what kind of speed up I would get with Titan V. Would this have helped me iterate a little faster?<br>Here are the hoops that I had to jump through to make this work on Titan V:</p><ol><li>Update the NVIDIA driver to the latest version supporting Titan V. My machine had 384.90, and when I ran nvidia-smi after installing the Titan V card, it did not even show up as a device. Once I upgraded to 387.34, I could finally see the Titan V shown as a generic “Graphics Device.” This step was not that surprising, though. I got the Ubuntu 16.04 driver from <a href="http://www.nvidia.com/download/driverResults.aspx/128000/en-us">http://www.nvidia.com/download/driverResults.aspx/128000/en-us</a></li><li>Update PyTorch to the latest version 0.3.0 for CUDA 9 support. Titan V requires CUDA 9. After updating the NVIDIA driver and PyTorch, I ran some training epochs against a 1080 Ti to make sure that the training time is about the same as before as a baseline with PyTorch 0.2.0 with CUDA 8 support.</li><li>When I trained the same model against the Titan V, I was blown away by the performance of Titan V!!!… in a bad way. Training time was more than 40% slower! It took about 185s per epoch on 1080 Ti vs 264s per epoch on Titan V to train my model. I was a bit shocked as I was not expecting this performance degradation at all. So I whipped up a small piece of benchmark code and sought help from the PyTorch community: <a href="https://discuss.pytorch.org/t/solved-titan-v-on-pytorch-0-3-0-cuda-9-0-cudnn-7-0-is-much-slower-than-1080-ti/11320">https://discuss.pytorch.org/t/solved-titan-v-on-pytorch-0-3-0-cuda-9-0-cudnn-7-0-is-much-slower-than-1080-ti/11320</a><br>Soumith Chintala of PyTorch/Facebook Research was super responsive and helpful. He provided me with this “turbo button switch” (actually, an autotuner) option that I was not aware of: <a href="https://discuss.pytorch.org/t/what-does-torch-backends-cudnn-benchmark-do/5936">torch.backends.cudnn.benchmark = Tru</a>e <br>After adding that to my code, it got Titan V’s training epoch time down significantly to 146s (45% lower.) It also reduced the training epoch time for 1080 Ti down to 154s (27% lower.) While the Titan V is a little faster than the 1080 Ti (by ~5%), it is was not significant at least in this scenario (but I learned about this performance booster option and that was super valuable in itself!)</li></ol><p>That’s all for now on my Titan V adventure so far. I will follow up with a post on experimenting with the 110 TeraFLOPs Tensor Cores to see what kind of real-world boost we can expect to see in addition to gotchas/tips. Thank you for taking the time to read this. Until next time!</p><p>UPDATE: I have done more performance comparison of Titan V vs 1080 Ti against popular CNN’s, including use of half-precision compute to utilize Tensor Cores. <a href="https://medium.com/@u39kun/titan-v-vs-1080-ti-head-to-head-battle-of-the-best-desktop-gpus-on-cnns-d55a19866b7c">Check it out!</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1d368d3443fb" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>