<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Ali Abdalla on Medium]]></title>
        <description><![CDATA[Stories by Ali Abdalla on Medium]]></description>
        <link>https://medium.com/@si3luwa?source=rss-9ce508996fb8------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*raE2puBl6WHFSo9rpylx_Q.jpeg</url>
            <title>Stories by Ali Abdalla on Medium</title>
            <link>https://medium.com/@si3luwa?source=rss-9ce508996fb8------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 16:24:06 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@si3luwa/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Increase model performance by… removing data?]]></title>
            <link>https://medium.com/data-science/increase-model-performance-by-removing-data-a87d05183d7c?source=rss-9ce508996fb8------2</link>
            <guid isPermaLink="false">https://medium.com/p/a87d05183d7c</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data]]></category>
            <dc:creator><![CDATA[Ali Abdalla]]></dc:creator>
            <pubDate>Fri, 29 Nov 2019 18:23:21 GMT</pubDate>
            <atom:updated>2019-12-04T04:53:00.394Z</atom:updated>
            <content:encoded><![CDATA[<h4>Can less data be better for machine learning?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/882/1*8lff39a_jgsauALvGx6STA.png" /><figcaption>The effect of removing low value data on predictor performance in a breast cancer dataset. Image adapted from Data Shapley: Equitable Valuation of Data for Machine Learning <a href="https://arxiv.org/abs/1904.02868v2">arXiv:1904.02868v2</a> [Ghorbani and Zhou].</figcaption></figure><h3>Not all data is created equal</h3><p>In any given dataset, not every sample will contribute equally to training a machine learning model. This is obvious when you say it out loud. Some data will teach a model a ton. But some will be <strong>irrelevant</strong>, or <strong>redundant</strong>, and won’t really move the needle at all. And some yet may even <strong>harm</strong> your performance — like <strong>mislabeled</strong> data — which will teach the model the opposite of what it’s trying to learn. Below is an example of mislabeled data in the <a href="https://www.vision.ee.ethz.ch/datasets_extra/food-101/">Food-101 dataset</a> [1], a popular public dataset with 101,000 images of 101 food categories.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IQdaLIHGGe0H6el7lWDDKw.png" /><figcaption>The images shown here are from the <a href="https://www.vision.ee.ethz.ch/datasets_extra/food-101/">Food-101 dataset</a>. Images in the dataset are supposed to be of the food item itself, but the first image is of a keyboard, the second is of a beer, and the third is of a food truck.</figcaption></figure><p>But the common workflow in machine learning is still to acquire as much data as you can, label all of it, and feed everything into your model. More or less a blind operation. Everyone cares about the quality of the data, but there is no (easy) way to programmatically assess the relevance, or redundancy, of data, especially in a model-driven manner. <em>Or is there?</em></p><p>If there is, why is it surprising to think that removing data will actually push your performance up?</p><p>Because it absolutely does. You just have to<strong> remove the lowest value data</strong>. And to do that you need a way to value data.</p><h3>Data Valuation</h3><p>There’s been some interesting research on data valuation. The idea is that you look at a dataset, and rank data points based on their value with respect to a specific model or predictive task.</p><h4>Data Shapley</h4><p><a href="https://arxiv.org/abs/1904.02868v2">Equitable Valuation of Data for Machine Learning</a> [2] by Ghorbani and Zhou came out earlier this year. The authors took the <a href="https://en.wikipedia.org/wiki/Shapley_value">Shapley Value from game theory</a> [3] (which got Lloyd Shapley the 2012 Nobel Prize in Economics) and applied to data in machine learning. They proposed that an equitable valuation method should satisfy the following conditions:</p><ol><li>It should give a data point which does not change the performance of your ML system when added to <em>any</em> subset of the training data a value of <strong>zero</strong>.</li><li>It should give data points that have exactly equal contribution when added to <em>any</em> subset of the training data the <strong>same value</strong> by symmetry.</li><li>If the value of a machine learning system can be decomposed into a sum of subsystems, the value of the data should be <strong>additive </strong>in those subsystems.</li></ol><p>The only property that can satisfy this? The Shapley Value.</p><p>To compute the Shapley Value for a given data point, you train a model with every subset of the data with that data point, and then again without it. You take the difference in performance and the Shapley Value is the weighted average across all subsets.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*racPUHdaUoJAFLfmWE0ZTg.png" /></figure><p>But computing this value requires an exponentially large number of computations with respect to the number of training data sources. That’s why the paper proposed the following estimations which are all designed to save time and compute:</p><ol><li>Monte Carlo Estimation:</li></ol><p>We can estimate the <em>value of sample i</em> by only<strong> looking at <em>some</em> subsets of the data</strong>. These subsets are randomly sampled and can provably approximate the rest of the data.</p><p>2. Truncation:</p><p><strong>Ending the training early</strong> for many iterations if the marginal value of the ML model is not increasing (helpful for large datasets). With the above, this is TMC-Shapley.</p><p>3. Gradient Shapley:</p><p>Treating the value of the ML system as the performance gained by <strong>one epoch of training</strong> rather than through a full training of the model. This is G-Shapley.</p><p>4. Group Shapley:</p><p>Instead of valuing data at the sample level,<strong> group level valuation</strong> is valuing a batch of data together. You might want to rank your data sources by value, in which case you don’t care about sample level granularity.</p><p>So does any of this work?</p><p>The following are results from the paper showing the effects of removing training data on the prediction accuracy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KsQcxWJupkxVh2-nlpDxlg.png" /></figure><p>The data was valued in four ways, the highest value data was removed on the left, and the lowest value data on the right.</p><p>LOO is Leave-One-Out: a simpler valuation method that trains on the whole dataset, then everything but a single data point, and values that point based on the effect its exclusion had. Besides being too computationally heavy, this method also can’t really handle relationships between different data points.</p><p>Random values the data randomly as a baseline.</p><p>As you can see the effect of removing high value and low value data is dramatically greater than random. <strong><em>In the breast cancer example, prediction accuracy went up more than 16% with more than 40% of the data removed!</em></strong></p><h4>But what about overfitting?</h4><p>If I’m removing data and getting better, doesn’t that mean I’m removing data I perform worse on, and isn’t that bad?</p><p>Well, no. The idea is that you only remove the data from your training set. And performance is being measured on a hold-out test set, which is (hopefully) representative of the your data distribution. This is standard practice in ML.</p><p><strong><em>Valuation doesn’t punish outliers, it rewards them. </em></strong>If you’re confident in the quality and coverage of your test dataset, this type of valuation will rank higher any rare samples that appear in both the training set and the test set. There is no risk of overfitting by removing redundant, irrelevant, or mislabeled data.</p><h3>How can I try it?</h3><p>Shifting the data workflow to become more model-driven means acquiring, labeling, and generating exactly the data you need by ranking your data sources, annotators, and augmentation functions by value.</p><p>We started <a href="http://gradio.app/"><strong>Gradio</strong></a> around this exact idea of data valuation — because we think it will touch every part of the data pipeline. We’re developing a bunch of valuation methods that work well in different settings. If you’re deploying machine learning, at any capacity, we’d love to <a href="https://gradio.app/#lets-chat">talk to you</a>.</p><p>[1] L. Bossard, M. Guillaumin, and L. Van Gool, Food-101 — Mining Discriminative Components with Random Forests (2014), European Conference on Computer Vision</p><p>[2] A. Ghorbani, and J. Zhou, Data Shapley: Equitable Valuation of Data for Machine Learning (2019)</p><p>[3] L. S. Shapley, A Value for N-Person Games(1953), Contributions to Theory Games 2, 307–317</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a87d05183d7c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/increase-model-performance-by-removing-data-a87d05183d7c">Increase model performance by… removing data?</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>