<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Lightly - Medium]]></title>
        <description><![CDATA[Product updates, how to’s and stories directly from the Lightly team - Medium]]></description>
        <link>https://medium.com/whattolabel?source=rss----b2284519248a---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Lightly - Medium</title>
            <link>https://medium.com/whattolabel?source=rss----b2284519248a---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 09:23:06 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/whattolabel" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Navigating the Future of Edge AI: Observability and Intelligent Data Selection]]></title>
            <link>https://medium.com/whattolabel/navigating-the-future-of-edge-ai-observability-and-intelligent-data-selection-4665830bc911?source=rss----b2284519248a---4</link>
            <guid isPermaLink="false">https://medium.com/p/4665830bc911</guid>
            <dc:creator><![CDATA[Matthias Heller]]></dc:creator>
            <pubDate>Tue, 24 Oct 2023 13:01:02 GMT</pubDate>
            <atom:updated>2023-10-24T13:01:02.804Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PtgDH4hdSAPPay-wX9qYgw.png" /><figcaption>AI on the edge is a big challenge for many companies</figcaption></figure><p>The Lightly team recently had the pleasure of attending the 2023 AI Hardware &amp; Edge AI Summit, immersing ourselves in groundbreaking discussions and sharing insights with pioneers in the field of Edge AI. Through all the innovation and challenges, Pushpak Pujari’s statement from Verkada cut through the noise:</p><blockquote>“AI Observability on the edge is still unsolved.”</blockquote><h3>Back to the Cloud? Not for Everyone</h3><p>While Large Language Models (LLMs) are witnessing a significant shift back to the cloud, computer vision, a predominant application for running AI on the edge, treads a different path. The sheer volume of data involved is too extensive, rendering cloud-based inference impractical. Consequently, a growing number of companies are advancing towards edge-based solutions, addressing the inherent challenges in building Edge AI applications.</p><h3>The Practical Challenges</h3><p>Building Edge AI is hard. It’s not just about deploying models; it’s about managing them in the wild. Pushpak laid out the three big headaches during his presentation:</p><ol><li>Data</li><li>Deployment</li><li>Drift</li></ol><h3>‍Data: It’s About Reality</h3><p>For models to work on the edge, they need to be trained with datasets that mirror the real world to avoid unexpected breakdowns in production.</p><p><strong>‍</strong>The data should represent various conditions and scenarios, and bias should be minimized. It’s imperative that datasets contain diverse sites, angles, objects, scenes, lighting, and weather conditions, minimizing the risk of failure due to unanticipated variables. Data bias goes beyond mere representation and encompasses data redundancy that can hurt model performance. Organizations must either construct internal data curation tools or leverage external ones to mitigate these issues.</p><h3>Deployment: It’s a Balancing Act</h3><p>Deployment is critical and tricky. It needs to work on different hardware and requires regular updates. Managing a large number of devices is already challenging, and AI adds another layer of complexity to it.</p><p>Ideally, updates can be done over-the-air (OTA). Managing extensive fleets of devices is already a complex task, fortunately, mitigated by tools like Mender or Balena. However, introducing AI further complicates the situation, especially regarding MLOps and data pipeline management. Data must flow bidirectionally while maintaining stringent data privacy standards.</p><h3>Data Drift: Keep Up or Fall Behind</h3><p>Data drift is a constant challenge for companies deploying on the edge. Deployed models need to adapt to new environments and situations, requiring the integration of new data into the training sets to maintain their efficiency and accuracy.</p><p>Regular deployments in diverse locations necessitate the incorporation of environmental data into training sets, ensuring model resilience and generalization. This requirement introduces challenges related to data transfer volumes, labeling costs, and increased compute costs due to expanding training datasets. However, these tasks are crucial to maintain optimal model performance and prevent data or model drift in production environments.</p><h3>How Can Data Curation Help?</h3><p>Data curation using techniques like over- or undersampling and active learning can optimize datasets, improve model performance, and reduce training set sizes. A data curation tool on the edge can monitor live data streams and select valuable data in real-time.</p><p>Data curation plays a significant role in mitigating all the above-mentioned challenges. It enhances dataset quality and enables the identification and selection of valuable data, optimizing model performance in terms of generalization and robustness. Implementing a data curation solution on the edge serves as a powerful tool to monitor live data streams continuously for anomalies or out-of-distribution data.</p><h3>The Outcome?</h3><p>Implementing data selection and observability on the edge can lead to:</p><ul><li>Reduced compute costs due to optimized training set sizes</li><li>Lower labeling costs due to efficient data selection</li><li>Minimized transfer &amp; storage costs due to selective data on the edge</li><li>Better model performance due to high-quality training data</li><li>Early detection and rectification of data drift and model drift through edge observability and selection</li></ul><h3>Lightly’s Solution</h3><p>Lightly offers a data curation solution powered by active learning, providing observability and data selection on the edge to enhance AI. If you want to optimize your AI, connect with us.</p><p>‍<br>Matthias Heller<br>Co-founder Lightly.ai</p><p>Thanks Laura Schweiger, Malte Ebner, and Igor Susmelj for reviewing drafts of this blog.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4665830bc911" width="1" height="1" alt=""><hr><p><a href="https://medium.com/whattolabel/navigating-the-future-of-edge-ai-observability-and-intelligent-data-selection-4665830bc911">Navigating the Future of Edge AI: Observability and Intelligent Data Selection</a> was originally published in <a href="https://medium.com/whattolabel">Lightly</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Bias in Machine Learning]]></title>
            <link>https://medium.com/whattolabel/bias-in-machine-learning-d15ebee7db45?source=rss----b2284519248a---4</link>
            <guid isPermaLink="false">https://medium.com/p/d15ebee7db45</guid>
            <category><![CDATA[bias-in-ai]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[machine-learing]]></category>
            <category><![CDATA[data]]></category>
            <dc:creator><![CDATA[Clara]]></dc:creator>
            <pubDate>Wed, 16 Mar 2022 09:51:19 GMT</pubDate>
            <atom:updated>2022-03-16T09:51:19.502Z</atom:updated>
            <content:encoded><![CDATA[<p>This blog post will give a high-level overview on the topic of bias in machine learning, a significant issue that can often be traced back to the data used to train an algorithm. I will discuss the different types of bias in machine learning, and how to identify and analyze it using different tools.</p><h4>What is Machine Learning Bias?</h4><p>In popular culture, it is often assumed that computers or technology should be considered the highest source of decision-making: that they are perfectly neutral, supremely rational, and even omniscient. In fact, there tends to be a certain trustworthiness around every decision that is made with “smart”, data-driven technology like machine learning. This assumption could not be further from the truth- as the engineers working on such technologies know well.</p><p>It all boils down to one main idea: algorithms are only as knowledgeable or neutral as the humans that develop them, and the data they are trained on. In fact, a “neutral” ML model is an almost outlandish idea, or at the very least, a challenge. Algorithms amplify the biases present in their training data. The unfortunate truth is that most often, these biases can not be identified until after the model is deployed and the bias has already gone from a potential challenge to a real issue.</p><p>As we integrate machine learning and data science into more and more facets of our lives–business, healthcare, industry, education and more– the issue of ML bias has become all the more pressing.</p><p>The phrase machine learning bias is used to describe <em>what is happening when an over- or underrepresentation of certain data in a dataset produces a biased algorithm</em>, which puts obvious and often problematic limitations on the algorithm.</p><p>The machine learning process, like any meticulous and detailed process, is prone to errors. Due to the causal nature of the steps in the process (data collection, data curation, labeling, training, and deployment), an error in one step tends to echo throughout the entire pipeline, with the consequences eventually being realized in the deployed model. Issues like bias can most often be assumed to be sourced from the datasets used for training and deployment of a model’s core algorithms.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*aAJWomAPaurHTlTV.jpeg" /><figcaption><strong>Figure 1:</strong> An infographic showing the long term consequences of bias finding its way into our data in one example, the health sector (image from the British Medical Journal).</figcaption></figure><p>In figure 1, we see how uncareful data selection practices can have a profound domino effect throughout an entire system (for this example, the health sector). Without a properly representative dataset, and selection methods that account for diversity, the design and deployment of ML models have a high potential for showing biases. With ML being applied in some of the most important areas of our life, like in healthcare, this can create or support patterns of injustice within those systems.</p><p>The downstream consequences of ML bias can range anywhere from increased ML project overhead to deeper societal implications and systematic discrimination, like some of these well-known instances of biased ML in the news:</p><p>-Google Photos “gorilla” scandal: In 2015, Google Photos’ photo tagging algorithm <a href="https://www.bbc.com/news/technology-33347866">misclassified a user’s selfie</a> of him and his friend (both black) as being “Gorillas” (Figure 2).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/976/0*U9yqjSPIAigRdHVb.jpeg" /><figcaption><strong>Figure 2:</strong> A screenshot from Google Photos showing the photo tagging algorithm had classified two black humans as gorillas</figcaption></figure><p>-Google Translate sexism: Last year, a researcher from England found that Google Translate would <a href="https://www.dailymail.co.uk/news/article-9396937/University-lecturer-slams-sexist-Google-Translate.html">automatically gender sentences</a> translated from Hungarian (a gender neutral language) in a sexist way (see figure 3).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/634/0*UCa0yW3rEGuh9zGw.jpeg" /><figcaption><strong>Figure 3: </strong>Dora Vargha’s Twitter post showing how Google Translate automatically assigned certain actions to certain genders.</figcaption></figure><p>-COMPAS: This was a prediction algorithm developed by the company Equivant that was <a href="https://www.propublica.org/article/how-we-analyzed-the-compas-recidivism-algorithm">found to be racially biased</a>, consistently predicting black people to have a higher risk of recidivism than white people.</p><p>-The arrest of Robert Williams: In 2020, Detroit police wrongfully arrested black man Robert Williams after their facial recognition <a href="https://www.npr.org/2020/06/24/882683463/the-computer-got-it-wrong-how-facial-recognition-led-to-a-false-arrest-in-michig?t=1643816903921">algorithm determined that he was the crook </a>in security footage of a man committing a robbery. Numerous <a href="https://news.mit.edu/2018/study-finds-gender-skin-type-bias-artificial-intelligence-systems-0212">studies</a> have shown that many facial recognition algorithms misidentify people of color more often than white people.</p><h4>What are different types of bias in machine learning?</h4><p>That being said, there are many different types of bias that can occur in different scenarios and projects, and it’s important to understand where to look for each of them. Here are a few examples of some more prevalent biases that may find their way into your ML model.</p><p><strong><em>Selection bias</em></strong></p><p>This is a purely systematic issue. In this type of bias, the method used for data collection does not sample from the <em>full </em>population because, for example, there is a misconception about it. This occurs at the collection/cleaning stage of the ML pipeline. Models with selection bias can behave unpredictably when confronted with scenarios which are not included in their training data.</p><p><strong><em>Algorithmic bias</em></strong></p><p>This type of bias occurs at the algorithmic level, so it isn’t an issue with data. Instead, designing an algorithm that is incompatible with the application it is being developed for can lead to this issue. It is less common, however, for the core algorithm to be at fault than for the data to be at fault.</p><p><strong><em>Exclusion bias</em></strong></p><p>Exclusion bias results from exclusion of particular samples from your dataset that would otherwise have an important impact on the model. This exclusion can be systematic but it can also happen in cases of deleting value-adding data <em>thought </em>to be unimportant. For example, when randomly sampling videos, it can often happen that important or relevant frames are not included. Thus, those scenes are not represented in the model trained on those samples.</p><p><strong><em>Representation Bias</em></strong></p><p>A relatively unexplored area, representation bias is the problem posed by an algorithm leveraging certain representations without learning to solve the underlying task. It occurs when a model is trained on certain representations of a situation more than others. For example, autonomous driving data that represents more sunny days than rainy/dark days. This type of bias can be removed by oversampling scenarios which are underrepresented.</p><p>It’s obvious that bias should be promptly addressed if it comes up. But maybe we need to back up a little bit first. How can we detect and resolve it as early as possible in the pipeline?</p><h4>How to detect and analyze bias</h4><p>Detecting bias in your ML model can be done proactively or reactively, and depending on your specific case, either one is viable. Reactive bias detection happens organically, for example, if you notice that a certain group of inputs aren’t performing very well. An obvious conclusion from this is that you are potentially working with biased data.</p><p>Alternatively, you could build bias detection/analyzation steps directly into your model development workflow by using a tool, thus proactively searching for signs of bias and analyzing the result to gain a better understanding of them. Here are some common tools you can use for this purpose:</p><p><strong><em>Google What-If</em></strong></p><p>Google launched their What-If Tool in 2018 as a feature of the OS Tensorboard web app. The tool was built to allow engineers to visualize their dataset and manage it through the UI, as well as analyze the performance of their model, and assess algorithmic fairness across different subgroups. Supported data types are tabular, image, and text.</p><p><strong><em>IBM AI Fairness 360</em></strong></p><p>IBM’s open source AI Fairness 360 toolkit was built around the intention of examining and mitigating bias in machine learning models. Its 70 fairness metrics and 10 bias mitigation algorithms are designed and optimized to help ML teams not only detect bias but understand and remove it. It also gives users the ability to compare their model after applying bias mitigation algorithms to the original model, which is helpful for understanding common patterns of bias to be more prepared for them in the future.</p><p><strong><em>FairML</em></strong></p><p>FairML is a Python toolbox developed by Julius Adebayo for auditing black-box machine learning models for bias. It assesses the fairness or discriminatory extent of a model by quantifying the relative significance of a model’s input.</p><p><strong><em>Lightly</em></strong></p><p>Lightly is a data curation solution for machine learning. Its solution makes use of representation learning through self-supervised methods to understand raw data, enabling users to find and remove redundancy and bias introduced by the data collection process to reduce overfitting and improve ML model generalization. The principle behind Lightly is to help engineers understand their data more deeply, leading them to also understand their models more deeply (including the conclusions those models make). Supported data types are image and video.</p><h4>How to remove dataset bias</h4><p>For many ML teams, eliminating bias from a dataset often means collecting more data. With more data, the chances that more of the domain you’re attempting to represent will successfully be represented increases. While this is true to an extent, simply collecting more data ultimately creates more overhead, costs more money, and even <a href="https://www.lightly.ai/post/sustainable-ai-and-the-new-data-pipeline">produces more Co2</a>.</p><p>That’s why using a tool like <a href="https://docs.lightly.ai/index.html">Lightly</a> can be so valuable for engineers wanting to tackle bias in machine learning. It does two very important things regarding this issue:</p><ol><li>Eliminates the need to collect more data to improve your dataset</li><li>Helps to intelligently collect new, unbiased data</li></ol><p>Building off of that vision, a valuable feature of the solution is the ability to balance datasets based on predictions of models and metadata which should counteract dataset bias.</p><h4>Conclusion</h4><p>Because it is so difficult (sometimes impossible) to deeply analyze how machine learning models reach particular conclusions, avoiding model bias will remain a lingering problem in ML. Until black box models are removed from the picture entirely, there can always be room for doubt regarding how your model came to the conclusion it made about particular data.</p><p>That’s why tools that help to identify, analyze, and remove bias are so important to ensure the integrity of your ML models.</p><p>As always, if you find yourself wanting to start a conversation about data-centric AI or what Lightly’s solution can do for your ML team, feel free to reach out <a href="https://www.lightly.ai/contact">here</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d15ebee7db45" width="1" height="1" alt=""><hr><p><a href="https://medium.com/whattolabel/bias-in-machine-learning-d15ebee7db45">Bias in Machine Learning</a> was originally published in <a href="https://medium.com/whattolabel">Lightly</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Lightly’s Data Curation Approach]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/whattolabel/lightlys-data-curation-approach-db205f664239?source=rss----b2284519248a---4"><img src="https://cdn-images-1.medium.com/max/600/1*lcLDKY4e75FfTgxanp4bEw.gif" width="600"></a></p><p class="medium-feed-snippet">This overview is intended to help everyone understand how Lightly works behind the scenes and how it can be used for your data curation&#x2026;</p><p class="medium-feed-link"><a href="https://medium.com/whattolabel/lightlys-data-curation-approach-db205f664239?source=rss----b2284519248a---4">Continue reading on Lightly »</a></p></div>]]></description>
            <link>https://medium.com/whattolabel/lightlys-data-curation-approach-db205f664239?source=rss----b2284519248a---4</link>
            <guid isPermaLink="false">https://medium.com/p/db205f664239</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[data-curation]]></category>
            <dc:creator><![CDATA[Igor Susmelj]]></dc:creator>
            <pubDate>Tue, 15 Mar 2022 09:14:58 GMT</pubDate>
            <atom:updated>2022-03-15T09:14:58.745Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[The Best Data Curation Tools for Computer Vision in 2022]]></title>
            <link>https://medium.com/whattolabel/the-best-data-curation-tools-for-computer-vision-in-2022-43109304f709?source=rss----b2284519248a---4</link>
            <guid isPermaLink="false">https://medium.com/p/43109304f709</guid>
            <category><![CDATA[data-curation]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Matthias Heller]]></dc:creator>
            <pubDate>Mon, 07 Feb 2022 08:19:42 GMT</pubDate>
            <atom:updated>2022-02-07T08:19:44.296Z</atom:updated>
            <content:encoded><![CDATA[<p><strong><em>Integrating a quality data curation tool into your ML pipeline will have a direct impact on the quality and performance of your model. With so many solutions on the market, it can be difficult to get a clear understanding of which to choose. In this article, we describe the top data curation tools of 2022.</em></strong><em>‍</em></p><h4>What is data curation and why is it important?</h4><p>Data curation is a relatively new focus in the machine learning pipeline. Put broadly, it is the management of data throughout its lifecycle as it is used, evaluated, and reused. In practice, however, it involves using relevant tooling and filtering techniques to identify what data works and what data doesn’t. Additionally, data curation softwares typically allow teams to identify edge cases and measure key metrics from their dataset.</p><p>Before ML teams started to pay more attention to data curation, raw data was simply labeled and deployed within a model. But with inevitable redundancy, bias, and insufficient data, this has been determined to be an incomplete process. Since the performance of a machine learning model can be traced directly back to the data used to train it, the quality of the data should be seen as the top priority in any ML operation.</p><p>This is where data curation comes in.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/0*lTjw5yI9tumj2psD.png" /><figcaption><strong>Figure 1</strong>: The machine learning pipeline, highlighting the data curation step as the most important regarding overall model performance. Illustration by author.</figcaption></figure><p>2022 will not only be the year of data curation– it is bound to be a particularly innovative year for the machine learning community at large. With a significant shift in the model-development paradigm from model-centric to data-centric AI in 2021, the priorities in most ML projects have changed. Instead of pursuing how to change their <em>models </em>to improve performance, engineers are increasingly turning their focus to how to systematically change their <em>data</em>.</p><p>With all eyes on the data, data curation has become more crucial than ever. This means that the data curation tool you use has a direct impact on the quality and performance of your model. That’s why it’s important to understand all the data curation tool options available on the market today, to ensure you make the right choice for your particular use case.</p><p>The best tools for data curation empower ML teams to:</p><ul><li><strong>Select the right data: </strong>You should be able to choose the most interesting samples within your dataset for model training, query data based on metadata (e.g. location), and rebalance + manipulate distributions to curate an ideal training dataset.</li><li><strong>Easily integrate into data pipeline:</strong> A quality data curation tool should be able fit into your existing ML pipeline and storage.</li><li><strong>Perform data-based model debugging: </strong>Find false positives and negatives, label mistakes, and other model errors in your data.</li><li><strong>Visualize data: </strong>Be able to swiftly identify dataset bias on the distributions, outliers, and edge cases of your dataset.</li><li><strong>Manage dataset: </strong>Collaboration features are important to ensure your team can seamlessly work together.</li></ul><p>Keep reading to learn about the most commonly-used data curation tools on the market today and see if one could be a good fit for your computer vision project.</p><h4>Data Curation Tools</h4><ol><li>Scale Nucleus</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Wx-gPBgkdIZEh46E.png" /><figcaption><strong>Figure 2: </strong>Screen capture from Scale Nucleus</figcaption></figure><p>Scale Nucleus, launched in 2020 by Scale, allows teams to collaborate on the same platform and send data to be labeled by Scale or within the built-in editor in a few clicks. Currently, the platform supports image data only. Other valuable features include custom metadata searching, intuitive error debugging with error metrics, and integration with their API (but no support for on-premise deployability as of now).</p><p>Pros:</p><ul><li>Directly integrates with the Scale labeling workforce</li><li>Predictions and labels can be visualized to find model failures in the labeled dataset</li><li>Users can query for similar images in the dataset</li></ul><p>Cons:</p><ul><li>Limited functionality when working with large unlabeled datasets</li><li>Users are locked in to the Scale ecosystem</li><li>Data selection is done manually through the user interface</li></ul><p>‍</p><p>2. Aquarium Learning</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*tAcoTAFEflebcalq.png" /><figcaption><strong>Figure 3: </strong>Screen capture from Aquarium Learning</figcaption></figure><p>Aquarium Learning is a machine learning data management platform focused on improving training data. It allows for a variety of data to be curated, including image, 3D, audio and text data. It is accessible via Aquarium’s API or their cloud platform, but not through an on-premise solution. Aquarium places a great deal of importance on features that allow teams to identify mislabeled samples, corrupted data, and edge cases, playing to the solution’s strength in maintaining/curating <em>training</em> datasets.</p><p>Pros:</p><ul><li>Supports use of embeddings to find clusters of similar images</li><li>Integrates with various labeling providers</li><li>Can be used to create and manage “data issues”</li></ul><p>Cons:</p><ul><li>Does not work with videos</li><li>Limited functionality when working with large unlabeled datasets</li><li>Data selection is done manually through the user interface</li></ul><p>3. Labelbox</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*TCkGPlw5bOKr32Yn.png" /><figcaption><strong>Figure 4: </strong>Screen capture from LabelBox</figcaption></figure><p>‍</p><p>Labelbox’s solution is focused on the training data iteration loop part of the ML pipeline. The platform is designed to iterate around three main tasks: annotating data, diagnosing model performance, and prioritizing data based on those results. Collaboration features are also placed at the forefront of the platform to streamline teamwork on projects; these are particularly tailored to remote teams.</p><p>Pros:</p><ul><li>Directly integrates with the Labeling tool of Labelbox</li><li>Supports use of embeddings to find clusters of similar images</li></ul><p>Cons:</p><ul><li>Data selection is done manually through the user interface</li></ul><p>4. Lightly</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*J_iy_d1X5X3H2Yz9.gif" /><figcaption><strong>Figure 5: </strong>Screen capture from Lightly</figcaption></figure><p>Lightly is a data curation software for computer vision. Unlike other solutions, it scales to tens of millions of input images. It uses self-supervised learning to find clusters of similar data within a dataset. Lightly’s algorithms can then create a well-balanced and high-quality subset which the model can be trained and re-trained on. Ultimately, this results in the removal of overfitting, bias, and data patterns which lead to model failures.</p><p>Another Lightly feature growing in popularity among ML teams is the on-premise docker solution. Using Docker, the entire solution can be run from within your own infrastructure, and your data never leaves your servers. Depending on the use case, this more secure option can be advantageous.</p><p>Pros:</p><ul><li>Data selection can be done through active learning algorithms</li><li>Work with videos and scales to dataset of several millions of frames</li><li>On-prem version available</li><li>Doesn’t require any labels to work properly</li></ul><p>Cons:</p><ul><li>Focuses on data problems and not model problems</li></ul><p><strong>Want to learn more?</strong></p><p>Many issues within the ML pipeline can be traced back to the data that is used to train models. Crippling issues like biased or overfitted models and high labeling costs can be avoided using a quality data curation tool to ensure that your model is fed the most important and highest quality data.</p><p>At Lightly we know a great model boils down to great data. That’s why we want to expose more computer vision teams to our data-centric solution. If you’d like to learn more about what we are doing at Lightly and what’s next for our solution, don’t hesitate to reach out to us <a href="https://www.lightly.ai/contact">here</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=43109304f709" width="1" height="1" alt=""><hr><p><a href="https://medium.com/whattolabel/the-best-data-curation-tools-for-computer-vision-in-2022-43109304f709">The Best Data Curation Tools for Computer Vision in 2022</a> was originally published in <a href="https://medium.com/whattolabel">Lightly</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why are data pipelines important?]]></title>
            <link>https://medium.com/whattolabel/why-are-data-pipelines-important-594c9f20c071?source=rss----b2284519248a---4</link>
            <guid isPermaLink="false">https://medium.com/p/594c9f20c071</guid>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[data-pipeline]]></category>
            <dc:creator><![CDATA[Matthias Heller]]></dc:creator>
            <pubDate>Thu, 03 Feb 2022 14:21:06 GMT</pubDate>
            <atom:updated>2022-02-03T14:20:49.198Z</atom:updated>
            <content:encoded><![CDATA[<p><strong><em>Data pipelines are important because, without a professional setup, errors can occur which jeopardize dataset quality. At the same time, a lot of resources are wasted on manual work due to the inefficient setup (read more about what data pipelines are </em></strong><a href="https://www.lightly.ai/post/what-is-a-data-pipeline-in-computer-vision"><strong><em>here</em></strong></a><strong><em>, and how to build a data pipeline </em></strong><a href="https://www.lightly.ai/post/how-should-i-build-my-data-pipeline-for-computer-vision"><strong><em>here</em></strong></a><strong><em>). In some of the worst cases, we have seen millions of dollars of resources wasted on bad data which naturally resulted in disappointing outcomes from the whole machine learning project.</em></strong></p><p>I have seen many machine learning teams flop due to reasons that could have been avoided. The key mistake frequently observed is the neglect of a thorough consideration of the machine learning pipeline at the start of the project. This is often due to a lack of “real world” machine learning experience in the engineering teams. Thus, starting with a rather academic setup will most likely fail in a production setting. In Figure 1, you can find four of the most common failure reasons I have heard from ML teams and explanations as to why this is not a good idea:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/952/0*mSABIQoJ7oIgE5Nu.png" /><figcaption><strong>Figure 1:</strong> Machine learning projects failure reasons; own table by author</figcaption></figure><p>Above, the mentioned reasons in Figure 1 do not include wrong assumptions on the business case and bad project management. The common theme in all the named reasons is that the end result is not the focus and determining success is postponed until late in the project development process. In practice, this can mean that a poor machine learning pipeline is only exposed when the model is first trained and deployed in production. This could be avoided by setting up from the get-go a machine learning pipeline that closes the model-data feedback loop. For this, a data pipeline that enables fast end-to-end iterations through automation and scalability is required.</p><p>Automation and scalability are therefore important. Below the reasons why those two key points are so crucial for data pipelines are elaborated:</p><p>1) Automation: The model-data feedback loop can only be executed if iterations are fast (e.g., two weeks instead of one year in many companies). Therefore, spotlighting weaknesses much earlier in the development cycle before significant investments have been made and steering resources and development efforts into the right direction is crucial.</p><p>2) Scalability: Architecture matters. Most machine learning projects start with small datasets; however, dataset sizes and data storage can multiply quickly. Thus, building a pipeline that can scale from the get-go is crucial, to avoid having to redesign the entire pipeline from scratch later on.</p><p>All of these insights come from my own experience as a co-founder at Lightly but are also true in light of industry facts. Machine learning development underlies an S-curve for development (see Figure 2 below). On the Y-axis we have the AI model accuracy and on the X-axis we have the investment of resources into the development. Machine learning accuracy also behaves like an S-curve. In the beginning, with little resources, decent accuracy can be reached. However, the higher the targeted accuracy the higher the costs for each additional accuracy point.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/703/0*_NBYrnw-GYypmtMC.png" /><figcaption><strong>Figure 2</strong>: Machine Learning Economics S-Curve; Illustration by author</figcaption></figure><p>Therefore, companies with professional data pipelines have a higher success rate for their machine learning projects than companies without because they are able to more efficiently allocate their resources with respect to their end goal.</p><p>That’s why data pipelines are becoming more and more crucial in machine learning in general. But, in computer vision, there is a particular rise. There are two reasons for this.</p><p>The first reason is that machine learning technology is leaving the experimental and prototyping phase. Good accuracies are no longer good enough. ML-based products need to be reliable and work in the real world. This trend is accelerated by the fact that by today there is growing industry adoption in many areas such as autonomous driving, robotics, visual analytics, and inspection. The margin for error in an autonomous car is close to 0.</p><p>The second reason is that machine learning models are being deployed in many smart products meaning that they also continuously need to be updated. In a certain sense, machine learning is like software development. In software, we update products with code; in machine learning, we update products with data. This particularly applies to computer vision, where the visual world is changing constantly. There, data drift is an often occurring phenomenon that could lead to model failure (e.g., new product packaging in self-checkout systems or the Cybertruck of Tesla in autonomous driving). Thus, automated data pipelines become of crucial importance.</p><p>‍</p><p><strong>Conclusion</strong></p><p>Data pipelines are critical because they allow one to work directly with the model in production and iteratively improve it through fast re-training cycles. This is being enabled through (1) automation and (2) scalability.</p><p>‍</p><p>Matthias Heller,<br>Co-founder Lightly.ai</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=594c9f20c071" width="1" height="1" alt=""><hr><p><a href="https://medium.com/whattolabel/why-are-data-pipelines-important-594c9f20c071">Why are data pipelines important?</a> was originally published in <a href="https://medium.com/whattolabel">Lightly</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How should I build my data pipeline for computer vision?]]></title>
            <link>https://medium.com/whattolabel/how-should-i-build-my-data-pipeline-for-computer-vision-ce182493dc7c?source=rss----b2284519248a---4</link>
            <guid isPermaLink="false">https://medium.com/p/ce182493dc7c</guid>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[data-curation]]></category>
            <category><![CDATA[data-pipeline]]></category>
            <dc:creator><![CDATA[Matthias Heller]]></dc:creator>
            <pubDate>Mon, 31 Jan 2022 14:20:29 GMT</pubDate>
            <atom:updated>2022-01-31T14:19:13.596Z</atom:updated>
            <content:encoded><![CDATA[<p><strong><em>A 3-step guide for building data pipelines for computer vision: Setting up the data pipeline aims to automate data streams and transfers, data selection, and dataset management (read more about what a data pipeline is </em></strong><a href="https://www.lightly.ai/post/what-is-a-data-pipeline-in-computer-vision"><strong><em>here</em></strong></a><strong><em>). Thus, automated processes should be the primary focus when building a data pipeline. This article will elaborate on how to build a data pipeline for computer vision in 3 steps.</em></strong></p><p>With an automated data pipeline, data can be continuously collected. Information about the data can freely flow between the stages. Even if some tasks are outsourced or handled on different platforms, little friction occurs between the steps.</p><p>For this, feature information and storage need to be interconnected with tooling.</p><p>Thus, there are three crucial steps in order to set up your data pipeline: (1) data storage connection, (2) tooling connection, and (3) monitoring installation.</p><p>1. Data storage connection</p><p>In this first step, the idea is to automate the collected data flows from your private storage into the data pipeline. For example, if you have stored your data at a cloud provider such as Amazon AWS S3 you can connect it to your data pipeline using, for instance, Lightly, and such transfers will occur automatically. Then, as the data comes in, it will be streamed onto your browser for the Lightly UI to visualize and help manage the data. In addition, automatic processing can occur where only value-adding samples are added to the training dataset. In this setup, no data leaves the servers, guaranteeing data security.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/979/0*epdU_STDQZ2dWs-L.png" /><figcaption><strong>Figure 1</strong>: CLI command to <a href="https://docs.lightly.ai/docker/integration/s3fs-fuse.html?highlight=storage">connect S3 to Lightly</a>; screenshot by the author</figcaption></figure><p>2. Tooling connection</p><p>A common challenge in ML in terms of data pipelines is the overhead involved with getting a dataset curated and labeled. The overhead can come from a variety of tasks, e.g:</p><ul><li>frame extraction from videos</li><li>manual selection for representative samples</li><li>eliminating edge cases and corner cases</li><li>rebalancing of the dataset to avoid bias</li><li>manual annotation</li><li>finding wrongly labeled samples</li><li>training models and tracking experiments</li><li>model debugging and finding false positives</li><li>writing scripts for data transfer</li></ul><p>Indeed, as many engineering teams outsource this task, the data must be manually moved to a third party’s platform. With an automated data pipeline, the data transfer could happen with little effort. There are two options: writing scripts or using a tool such asLightly. If one decides to use the Lightly API it can be directly connected to a variety of data labeling tools &amp; services, MLops, and experiment tracking. As illustrated in Figure 3 of <a href="https://www.lightly.ai/post/what-is-a-data-pipeline-in-computer-vision">part 1</a> of this blog series, Lightly could be connected with Labelstudio, Weights &amp; Biases, and Grid.ai, and data could then be directly transferred from one platform to the other.</p><p>3. Installation and monitoring</p><p>In this step, we want to automate the workflow. Thus, we will decide which steps will be manual or automatic and set parameters for the different data operations (e.g., for which parameters new data should be added to the training set from the incoming data collection batch). If installed properly, the machine learning engineer can relax and see incoming data flowing automatically through the machine learning development steps. Finally, redundant data or data with little adding value is intelligently removed along the pipeline using Lightly to ensure that the model is exclusively trained on essential samples. This enables the engineer to focus on the model training and deployment while only occasionally being required to monitor and check the data flow, dataset quality, and flagged anomalies.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/979/0*5nze14qaexHsUbie.png" /><figcaption><strong>Figure 2: </strong>Automatic data flow management with Lightly; illustration by Lightly</figcaption></figure><p>‍</p><p><strong>Conclusion</strong></p><p>Data pipelines should be built in a way that enables automated processing. Additionally, it is vital to have a data pipeline design and architecture in mind that can scale to large amounts of data so that future refactoring can be avoided. The goal is to have no manual steps or at least as little as possible. Thus, having seamless integrations with every tool along the machine learning pipeline is crucial.</p><p>‍</p><p>Matthias Heller,<br>Co-founder Lightly.ai</p><p>‍</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ce182493dc7c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/whattolabel/how-should-i-build-my-data-pipeline-for-computer-vision-ce182493dc7c">How should I build my data pipeline for computer vision?</a> was originally published in <a href="https://medium.com/whattolabel">Lightly</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What is data redundancy in computer vision?]]></title>
            <link>https://medium.com/whattolabel/what-is-data-redundancy-in-computer-vision-a80c445f8dcc?source=rss----b2284519248a---4</link>
            <guid isPermaLink="false">https://medium.com/p/a80c445f8dcc</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[data-redundancy]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[computer-vision]]></category>
            <dc:creator><![CDATA[Sarah Meibom]]></dc:creator>
            <pubDate>Fri, 28 Jan 2022 10:48:22 GMT</pubDate>
            <atom:updated>2022-01-28T10:48:22.510Z</atom:updated>
            <content:encoded><![CDATA[<p>Also read this article and many others on <a href="https://www.lightly.ai/post/what-is-data-redundancy-in-computer-vision">https://www.lightly.ai/post/what-is-data-redundancy-in-computer-vision</a></p><p><strong><em>What is redundant data and why should you avoid it? This article will tackle these questions in the context of computer vision by providing concrete examples. Data redundancy is shown to have negative repercussions on model accuracy and to be wasteful of resources.‍</em></strong></p><h4>What is data redundancy?</h4><p>Data redundancy often refers to having two identical samples of data in two different places within your database. Such a scenario can be accidental or deliberate, and its effects therefore vary. (<a href="https://www.techopedia.com/definition/18707/data-redundancy">Technopedia</a>, 2020)</p><p>However, this article will focus on a slightly different understanding of data redundancy. <strong>We refer to redundant samples as nearby-duplicates within the same dataset.</strong> Indeed, this is data that is so similar it provides little value to the dataset. Further, data redundancy is assessed within the context of the other samples in the dataset (<a href="https://arxiv.org/pdf/1901.11409.pdf">The 10% you don’t need</a>). Thus the term “redundant” largely depends on the diversity of the dataset in terms of content, as well as the task the data is being used for.</p><p>One understanding of redundant data is semantic redundancy, which was described in a previous <a href="https://www.lightly.ai/post/the-data-you-dont-need">blog post</a>. In this case, when the samples are represented in a vectorial space, the distances that separate them are minimal. In other words, they are visually very similar, as illustrated below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*gI9medxaNjgAJNBO.png" /><figcaption>Semantic similarity</figcaption></figure><p>Nevertheless, other examples of data redundancies include scene similarities between samples, representations with similar weather conditions or representations of the same object. Below the two images show both scene similarity, as the windscreen wipers appear in both images, as well as weather condition similarity, as both images were taken on rainy days.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*PqevBXXZZqCLN61g.png" /><figcaption>Scene similarity and similar weather conditions</figcaption></figure><p>Data redundancy is a challenge often observed for engineers dealing with video data for machine learning tasks. Indeed, videos are broken down into individual frames which can then be treated as images. However, in contrast to individual pictures which are taken deliberately at specific moments, video cameras continue to record and capture moments with little variance. To illustrate this, one can think of a car recording video footage of streets to train an autonomous vehicle. When the car stops behind another vehicle at a red light or when it is driving alone on a straight highway, the frames captured are quasi-identical.</p><p>In the <a href="https://arxiv.org/pdf/1901.11409.pdf">The 10% you don’t need</a> paper it is shown that several public datasets such as CIFAR-10 or ImageNet have at least 10% of redundant samples. If the distribution of classes within the dataset is skewed or excessive proportions of datasets are redundant, imbalances can result which can in turn can cause bias. The next section of this blog post touches upon this, by presenting reasons to avoid data redundancy.</p><p>‍<strong>‍</strong></p><h4>Why should redundant data be reduced?‍</h4><p>Redundant images or video frames can have the following effects:</p><ul><li><strong>Negative repercussions on the model</strong></li></ul><p>Model performance is highly influenced by the data the model is trained on. If you feed a model with redundant data, the model will perform well in these specific situations but will lack experience for others. Indeed, this is because some types of data are over-represented while others are underrepresented and thus training a model on redundant samples negatively influences the model’s generalization and accuracy. Therefore, an optimized <a href="https://www.lightly.ai/post/data-selection-for-computer-vision-in-5-steps">data selection</a> is crucial to reduce redundancy as much as possible. In the figure below, the model’s average precision obtained with standard sampling strategies (labeled “other”) is compared with the average precision of Lightly’s diversity-based sampling algorithm which actively excludes redundant samples. The graph shows that removing similar data can improve average precision or model accuracy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/380/0*F6a0NgSKN469L6xG.png" /><figcaption>Average precision of a model using Lightly’s sampling strategy in comparison with another strategy</figcaption></figure><p>‍In another experiment made by Lightly, it was shown that the best test accuracy was found by training the model on 90% of the data selected by Lightly in comparison to a model trained on 100% of the dataset or 90% using random sampling techniques. Read more about this in a previous <a href="https://www.lightly.ai/post/the-data-you-dont-need">blog post</a>.</p><ul><li><strong>Wasted resources</strong></li></ul><p>Precious resources such as time and money are wasted on redundant data. First, a lot of time and financial resources are spent on labeling datasets. Thus, labeling unnecessary samples is inefficient and costly. Second, compute and processing resources can be wasted on redundant data. Generally, as raw data goes through the typical machine learning pipeline (pictured below), data related tasks become more time-consuming and expensive if datasets are larger due to redundant samples.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*RJazYu_BpY33o_Ck.png" /><figcaption>Machine learning pipeline</figcaption></figure><p>This blog has portrayed the challenge of data redundancy within the computer vision field, by (1) providing a definition of the concept and (2) arguing why it should be avoided. You can read more about how Lightly helped AI Retailer System remove redundant data of customers in stores <a href="https://www.lightly.ai/post/how-redundant-is-your-dataset">here</a>.</p><p>Author: Sarah Meibom</p><p>Thank you to the <a href="https://medium.com/whattolabel">Lightly</a> team for reading drafts of this blog post.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a80c445f8dcc" width="1" height="1" alt=""><hr><p><a href="https://medium.com/whattolabel/what-is-data-redundancy-in-computer-vision-a80c445f8dcc">What is data redundancy in computer vision?</a> was originally published in <a href="https://medium.com/whattolabel">Lightly</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What is a data pipeline in computer vision?]]></title>
            <link>https://medium.com/whattolabel/what-is-a-data-pipeline-in-computer-vision-64cf298956b3?source=rss----b2284519248a---4</link>
            <guid isPermaLink="false">https://medium.com/p/64cf298956b3</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[data-centric-ai]]></category>
            <category><![CDATA[data-pipeline]]></category>
            <dc:creator><![CDATA[Matthias Heller]]></dc:creator>
            <pubDate>Tue, 25 Jan 2022 09:53:56 GMT</pubDate>
            <atom:updated>2022-01-25T09:49:41.829Z</atom:updated>
            <content:encoded><![CDATA[<p><strong><em>A data pipeline in computer vision is the path the data flows through. From data collection to storage, being used for model training, and deployment. Ideally, it is a connected technical set-up where data storage is linked to various data preparation and MLops tools, which in turn are connected through an API to the machine learning model and the deployed product.</em></strong></p><p>Today, many people talk about data-centric AI. In this concept, engineers not only focus their efforts on the model but also on the data on which a model is trained. Data pipelines play a crucial role in doing this appropriately. The purpose of a data pipeline is to enable engineers to work more efficiently by streamlining and automating data flows and intelligently managing datasets and data lakes.</p><p>To better understand this concept the standard machine learning pipeline design is depicted in Figure 1.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*trFg4UBh6NwhrQOD.png" /></figure><p><strong>Figure 1: </strong>Machine Learning pipeline; illustration by author</p><p>Data collection is the first step in the pipeline. From there, the data is ingested into a data storage platform or infrastructure. This is where the typical data pipelines start via steps like data curation, data labeling, and model training until deployment. One goal of a data pipeline is that information can not only move forward but also backwards during the workflow. Such information can include:</p><ul><li>Metadata (e.g, weather, time, GPS location)</li><li>Labels/annotations</li><li>Model predictions</li><li>Embeddings</li><li>Images/videos</li><li>Experiment results (e.g., model accuracy on a given dataset)</li><li>Deployed system model confidences</li></ul><p>All the information and features need to be stored and managed accordingly while the data flows are automatized and monitored. Having a good data pipeline setup in place will enable one to efficiently conduct workflows like Active Learning, where model performance on specific samples directly impacts the subsequent data selection stage. Anomalies can also easily be detected and automatically flagged. At the same time, a good pipeline should provide analytics and insights about the state and health of the data in all stages of the pipeline.</p><p>Another goal of an automated data pipeline is to facilitate data operations within the ML pipeline. Not only does such a setup provide more information flow between the steps and thereby increasing the resulting efficiency and performance of products, but logistics for ML pipelines can be improved by reducing overhead resources related to them. Overhead resources include the efforts required to manage such pipelines, by transferring data to different tooling platforms, for example, or by manually monitoring each step and the data transfer between them.</p><p>Data curation occupies an essential role in building good data pipelines. Selecting the right data for labeling and model training is crucial. Thus, the aforementioned information will be the basis for making the selection of a batch of newly collected data to be labeled and subsequently fed to the model. Especially with large amounts of data, manual workflows for selecting the samples to be labeled become unfeasible. Instead, selection needs to be automated based on determined parameters, and data flows are monitored (see our article about <a href="https://www.lightly.ai/post/data-selection-for-computer-vision-in-5-steps">data selection</a> in computer vision). This is where tools like Lightly come in. Lightly enables you to automate the selection and the model-data feedback loop completely. This allows the data pipeline to scale and shorten re-training cycles.</p><p>The following paragraph elaborates and gives an example of how to build a data pipeline for computer vision.‍</p><p><strong>Example of a data pipeline for computer vision</strong></p><p>In Figure 2, several data pipeline set-ups with different tooling possibilities are pictured. One can either build their own scripts or use a tool like Lightly’s to connect the various pieces of the development chain.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*AwQXRFZyqfNByA08.jpeg" /></figure><p><strong>Figure 2:</strong> Automated data flows through AI data pipeline; illustration by author</p><p>Using Lightly, a data pipeline for computer vision could, for example, have the following tooling set-up:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Ds1P5o0RblBh8E3p.png" /></figure><p><strong>Figure 3</strong>: Machine Learning Pipeline Tools for Computer Vision; table by author</p><p><strong>Conclusion</strong></p><p>A data pipeline in machine learning in general and in computer vision, in particular, is the path data flows through. It aims to enable automated and scalable data processing for machine learning.</p><p>This blog post is part of a series about data pipelines. <a href="https://www.lightly.ai/post/how-should-i-build-my-data-pipeline-for-computer-vision">Part 2</a> elaborates on how to build a data pipeline for computer vision and part 3 argues why data pipelines are important.</p><p>‍</p><p>Matthias Heller, <br>Co-founder Lightly.ai</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=64cf298956b3" width="1" height="1" alt=""><hr><p><a href="https://medium.com/whattolabel/what-is-a-data-pipeline-in-computer-vision-64cf298956b3">What is a data pipeline in computer vision?</a> was originally published in <a href="https://medium.com/whattolabel">Lightly</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Selection for Computer Vision in 5 Steps]]></title>
            <link>https://medium.com/whattolabel/data-selection-for-computer-vision-in-5-steps-3b0dae5f0a95?source=rss----b2284519248a---4</link>
            <guid isPermaLink="false">https://medium.com/p/3b0dae5f0a95</guid>
            <category><![CDATA[data-selection]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[computer-vision]]></category>
            <dc:creator><![CDATA[Sarah Meibom]]></dc:creator>
            <pubDate>Thu, 16 Dec 2021 17:16:05 GMT</pubDate>
            <atom:updated>2021-12-16T17:16:05.421Z</atom:updated>
            <content:encoded><![CDATA[<p>In its many forms, data acts as the learning material for machine learning models. Indeed, the performance of machine learning models is dependent on the quality of the data it is trained on. Therefore, selecting the best training dataset is equally important than developing the model itself.</p><p>This blog post suggests five chronological steps to select data for computer vision tasks: (1) understanding collected data, (2) defining requirements for the training dataset, (3) sampling the best subset with diversity-based sampling and self-supervised learning, (4) improving the model iteratively with uncertainty-based sampling and active learning, and finally, (5) automating the data pipeline by continuously adding the right samples to the data repository.‍<strong>‍</strong></p><h4>1. Understanding collected data</h4><p>The first step towards having a high-quality training set is understanding the collected data at disposal to train computer vision models on. Although data attempts to represent the real world, the data collection process can contain biases, as data might be collected at certain times of the day, by a single person, or with a specific purpose in mind. Hence, the importance of uncovering such errors and gaining a better understanding of the large repository of images or video frames before selecting relevant samples for training. Below are a few examples of questions engineers could ask about the collected data:</p><ul><li>What does the dataset’s distribution look like?</li><li>Does the dataset represent the real world?</li><li>Is the data redundant?</li><li>Is the dataset biased?</li><li>Which data is there too little/too much of?</li><li>Are all edge cases/corner cases sufficiently represented?</li></ul><p>Such questions refer to common data challenges in the computer vision field including the long-tail distribution problem (where particular categories, classes, or situations are less represented with data) and the existence of false positives and false negatives (where model predictions or labels falsely relate to or fail to relate a sample to a certain object, class, or situation).</p><p>For computer vision tasks, data types include images or video footage. Such data is considered unstructured data, as they have no predetermined structure. Indeed, anything can appear anywhere on an image or video frame. This is why they are challenging to analyze, and a large number of samples give the impression of being a blackbox.</p><p>One solution to tackling raw and mysterious data is unsupervised learning. This method does not require any labels or further information about the images other than their mapping in a multi-dimensional vector space. <a href="https://www.lightly.ai/">Lightly</a>’s tool uses such techniques. Indeed, the tool identifies clusters of similar data by calculating the distance between their vectors. It allows for a simpler visualization and understanding of the dataset by reducing the dimensionality into 2D scatter plots (several methods can be used, such as UMAP, TSNE or PCA). This is illustrated in the image below with the Comma10k dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1001/0*p-ZzZNtQ6QFLZDrJ.png" /><figcaption>2D representation of the Comma10k dataset on the <a href="https://www.lightly.ai/">Lightly</a> platform</figcaption></figure><p>In order to tackle the long-tail problem, <a href="https://www.lightly.ai/">Lightly</a>’s similarity search function allows the manual identification of similar under-represented edge or corner cases. Below, similar images where windshield wipers block some of the image’s content have been spotted.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/987/0*Zg_26KCKS1JqzrVo.png" /><figcaption>Edge-case mining with <a href="https://www.lightly.ai/">Lightly</a>’s similarity search function on the Comma10k dataset</figcaption></figure><h4>2. Defining requirements for the training dataset</h4><p>Before sampling a subset of the collected data for training, the requirements for such subset should be set. One can define what the ideal training dataset looks like in terms of metadata distribution as well as size, for instance. Obviously, many other metrics can be relevant when defining the ideal training dataset such as data diversity or model certainty. These will be discussed in another section.</p><p>The dataset’s metadata distribution refers to parameters that describe the content of the image (such as luminance and color channels) or the conditions in which the image was captured (time, temperature, and location). The primary task is to define what the dataset should ideally represent according to these and in which proportions. Data querying based on metadata information is one sampling technique that can be employed for data selection. Other methods include diversity-based and uncertainty-based sampling, two approaches that will be discussed in the following steps. Below are some questions that could be asked in order to uncover the subset’s ideal distribution of metadata:</p><ul><li>What does the real world look like?</li><li>Should the dataset exactly represent the real world?</li><li>Which samples with certain metadata values are over-/under-represented?</li><li>Which proportions of samples should represent different metadata values?</li></ul><p>Setting target ratios for the metadata values is an approach to define an ideal subset. Below, histograms of certain parameters can be observed. For instance, one could aim for the different values of luminance to be more evenly represented in the dataset in terms of the number of samples. These histograms enable the querying of a subset based on metadata ratios. The dataset can also be adjusted according to these parameters by altering their minimum and maximum values.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*OELCi6BB_tXCEbJi.png" /><figcaption>Histograms of metadata on the <a href="https://www.lightly.ai/">Lightly</a> platform</figcaption></figure><p>Quality over quantity also applies to datasets for model training. Indeed, having a smaller representative dataset is more valuable than a large redundant one. The desired size of the selected dataset can depend on various parameters. Here are a few questions that are asked:</p><ul><li>Which resources do I have to label the data?</li><li>How complex is the model’s task?</li><li>How long does it take to process the data?</li></ul><p>As the subset is typically labeled before being sent to model training, the costs and time the labeling process incur play an essential role. The costs can be significant, hence the importance of reducing the amount of data and selecting the right data to be labeled. Moreover, the complexity of the task being taught to the computer can influence the required dataset size, as a complex task entails numerous types of scenarios that must be sufficiently represented within the dataset. For example, a task in an uncontrolled (e.g. outdoors) environment in comparison to the same task in a controlled (e.g. indoors) environment produces a variety of additional situations, most notably influenced by weather or lighting conditions. A further parameter can be processing time. As datasets become significant in terms of size, the time needed for model training increases as well. This, in turn, delays production. Thus, while the selection of a large dataset suggests that more cases and scenarios are covered, processing time is lengthened which can be more harmful and costly.</p><p>In sum, defining the requirements for a subset depends on a variety of parameters that are very specific to the task, industry, company size, models used, and resources. It is thus an individual yet important factor to consider for data selection.</p><h4>3. Sampling the best subset: diversity-based sampling with self-supervised learning</h4><p>Once one has gained insight into the similarity and metadata distribution of the raw collected data and defined the requirements for a training dataset, selecting the relevant samples becomes the next challenge. Listed below, are a few simple data selection methods commonly employed:</p><ul><li>Selection of frames based on metadata</li><li>Random sampling techniques</li><li>Manual sampling</li></ul><p>However, these selection methods are far from ideal. First, selection based on metadata such as time, where frames are sampled at a constant time interval, is a method that runs the risk of omitting interesting samples. Second, although random sampling seems like a reliable method, as statistically the distribution of the larger dataset should be reflected, it assumes that the collected data perfectly represents the real world, which is seldom the case. Finally, manual sampling methods cannot escape human error. Indeed, only machines are capable of making an accurate holistic selection.</p><p>Diversity-based sampling, a sampling technique based on the distance between the samples in an embedding space, is a more comprehensive and efficient approach to data selection. Its objective is to choose a diverse dataset which covers all possible scenarios, to ensure that the model is even trained on edge or corner cases. Self-supervised learning can offer a solution for this data selection task. As described previously, it has the powerful ability to identify similar images. Thereafter, samples from the different clusters can be chosen using a diversifying algorithm, thereby ensuring the representation of different types of images or video frames. <a href="https://www.lightly.ai/">Lightly</a> uses such techniques. Engineers can choose the quantity of samples to be selected and the algorithm will identify the most diverse samples according to the distances between the data’s vectors. To illustrate this, in the image below, the most diverse 750 images from the Comma10k dataset have been selected by <a href="https://www.lightly.ai/">Lightly</a>’s algorithm, appearing as the green points.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1001/0*83wV_ivWj3p2nwPo.png" /><figcaption>Data selection of the 750 most representative images from the Comma10k dataset on the <a href="https://www.lightly.ai/">Lightly</a> platform</figcaption></figure><p>As mentioned in the previous section, sampling can also be done based on metadata. In the first image below, a subset of bright samples was created with <a href="https://www.lightly.ai/">Lightly</a>’s metadata tool that allows for the minimum and maximum of luminance values to be altered. Here, higher values were aimed for, thus brighter images were selected. (Note how a few samples from the left cluster were still included in this selection. This is an interesting finding that could raise questions about their metadata values, labels, or ambiguous content .) In the second image below, a diversity-based sampling algorithm, such as the one previously described, was performed on these brighter samples. This resulted in a diverse subset of bright images.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/956/0*HKeuALQLan-eBYhs.png" /><figcaption>Step 1: subset of bright samples from the Comma10k dataset selected using <a href="https://www.lightly.ai/">Lightly</a>’s metadata tool</figcaption></figure><p>‍</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/953/0*WPhNHAKRE9Fi5bNo.png" /><figcaption>Step 2: data selection of the most representative bright images from the Comma10k dataset on the <a href="https://www.lightly.ai/">Lightly</a> platform</figcaption></figure><p>By selecting a diverse training subset, redundant samples, and over-represented data which can cause overfitting are removed, thereby improving model accuracy. The plot below shows the positive effects of <a href="https://www.lightly.ai/">Lightly</a>’s diversity-based sampling algorithms on model accuracy for the KITTI public dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/380/0*Z_bTNLNdT30piJeC.png" /><figcaption><a href="https://www.lightly.ai/">Lightly</a>’s sampling strategy performance in comparison to others with the KITTI dataset</figcaption></figure><p>‍</p><h4>4. Improving the model iteratively: uncertainty-based sampling with active learning</h4><p>It is usually observed that with some samples the model shows good results while with others it struggles. The following list describes some of the reasons for poor-performing data:</p><ul><li>Flawed requirements for the training subset</li><li>Unforeseen biases in the collected data</li><li>Edge cases/Corner cases insufficiently represented in the initial data repository</li></ul><p>The model can perform poorly on data if it makes false predictions about it. For instance, the concepts of false positives and false negatives refer to situations when the model falsely assumes that the image includes an object and when the model incorrectly assumes that the image does not include that object when in reality it does.</p><p>Identifying false negatives and false positives is a task that can also be undertaken using the embedding visualization tool. In the first image below, the samples have been color-coded according to the luminance metadata values. While most darker images appear on the right (with a lower luminance value), a few lighter samples still appear in this cluster (suggesting false positives). This can also be done with model predictions or labels by observing the positioning of certain labeled samples in comparison to others. In the second image below, the samples of a labeled class from another dataset appear as green dots. Although most green points are grouped together, some appear further away (suggesting false positives), and while most green clusters are filled with samples of the same class, one cluster contains a gray gap (suggesting false negatives). One can then investigate why samples with such metadata values appear in these places and question the attributed class labels or predictions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/964/0*Pu1VkoHe1bfZluyd.png" /><figcaption>False-positive and false-negative spotting with metadata distribution</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/954/0*ieNGNV8hwCoyCsvA.png" /><figcaption>False-positive and false-negative spotting with class labels</figcaption></figure><p>In reference to the metaphor in the introduction, the model should be taught with challenging learning material, i.e. the data it struggles with, from which it can improve. The concept of uncertainty-based sampling performs exactly this: algorithms ensure that the data where the model is uncertain is added to the training set. This creates a type of feedback loop, called active learning, where the output of the model, i.e. its performance, gives recommendations regarding its inputs, i.e. its training set.</p><p>Active learning is a feature that <a href="https://www.lightly.ai/">Lightly</a> has incorporated into its core data curation product. The collected active-learning scores can also be projected on the scatterplot to identify the uncertain samples. Below the average precision of the active learning algorithm is compared to a random selection algorithm for the KITTI public dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*HFXJi8W-oiJBtV08.png" /><figcaption><a href="https://www.lightly.ai/">Lightly</a>’s active learning algorithm in comparison to a random selection algorithm for the KITTI dataset</figcaption></figure><p>In combination, metadata-based, diversity-based, and uncertainty-based sampling allows for an optimal selection of data. It enables uncertain data to be included within the training set while ensuring that similar samples are left out and metadata targets are met. In consequence, model performance is considerably improved. You can read more about how Lightly implements this approach in another <a href="https://www.lightly.ai/post/active-learning-using-detectron2">blog post </a>about active learning. In comparison to manual similarity searches, also referred to as edge case mining, this process can be entirely automated, thereby making it scalable for larger datasets. This is covered in the next section.</p><h4>5. Automating the data pipeline by continuously adding the right samples to the data repository</h4><p>As computer vision tasks are performed over extended time periods, it is essential to maintain high model performance by updating the training dataset accordingly. Again, in reference to the metaphor, if models are taught with outdated learning material, their performance in the world today is inferior. This could happen in the case of computer vision models for the following reasons:</p><ul><li>The model is trained on outdated products, behaviors, environments, laws, etc. which no longer occur</li><li>New products, behaviors, environments, laws, etc. exist which provoke new situations that the model does not recognize</li></ul><p>Thus, collecting more “modern” data and intelligently updating the training set by adding the right samples becomes necessary. As new data collection processes are carried out, the need arises for an automated data pipeline, as pictured below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_5re7eZM-9XRegfe.png" /><figcaption><a href="https://www.lightly.ai/post/active-learning-using-detectron2">Lightly</a>’s automated data pipeline</figcaption></figure><p>‍<a href="https://www.lightly.ai/">Lightly</a> enables data to be automatically added to the data repository and selected based on metadata, diversity, and uncertainty. With more data being collected for computer vision tasks, the feedback loop created with the active learning feature is essential. This way, data selection becomes a continuous yet seamless background task. Moreover, <a href="https://www.lightly.ai/post/data-preparation-tools-for-computer-vision-2021">data preparation tools</a>, such as labeling or model training platforms can be connected through an API, allowing data to freely flow between the steps. You can read more about this in next week’s upcoming article. Automation is crucial for scaling machine learning pipelines. Indeed, manual workflows such as edge case mining work well for smaller datasets, but as more data is collected the need for an automated workflow becomes more important.</p><p>Five important steps to data selection for computer vision tasks were described in this blog post. In sum, the steps highlight that an understanding of the collected data, as well as the determination of requirements for the training dataset, should precede the intelligent data selection process of the best subset. Data selection techniques include metadata-based sampling, diversity-based sampling, and uncertainty-based sampling. Finally, the importance of an automated data pipeline was presented.</p><p>If you want to learn more about Lightly’s tool, feel free to contact the team directly at <a href="/service/mailto:sales@lightly.ai">sales@lightly.ai</a> or head over to the website and reach out through the contact form.</p><p>Author: Sarah Meibom</p><p>Thank you to <a href="https://medium.com/@matthiasheller">Matthias Heller</a> and the <a href="https://www.lightly.ai/">Lightly</a> team for reading drafts of this blog post.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3b0dae5f0a95" width="1" height="1" alt=""><hr><p><a href="https://medium.com/whattolabel/data-selection-for-computer-vision-in-5-steps-3b0dae5f0a95">Data Selection for Computer Vision in 5 Steps</a> was originally published in <a href="https://medium.com/whattolabel">Lightly</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[5 Answers to Data Problems in Computer Vision]]></title>
            <link>https://medium.com/whattolabel/5-answers-to-data-problems-in-computer-vision-cdf872048596?source=rss----b2284519248a---4</link>
            <guid isPermaLink="false">https://medium.com/p/cdf872048596</guid>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[solutions]]></category>
            <dc:creator><![CDATA[Sarah Meibom]]></dc:creator>
            <pubDate>Fri, 01 Oct 2021 15:11:39 GMT</pubDate>
            <atom:updated>2021-10-01T15:11:39.816Z</atom:updated>
            <content:encoded><![CDATA[<p><strong><em>Unstructured data is growing exponentially and creating new challenges. This article presents 5 ways to tackle data problems in the computer vision machine learning field. They include understanding data, curating data, building an efficient data pipeline, managing data, and sharing data.</em></strong></p><p>Two assumptions can be made about the machine learning field. First, it is no secret that obtaining data has become cheap and the amount of data is increasing, and fast. Indeed, in the next 3 years, more data will be created than in the last 30 years (<a href="https://www.idc.com/getdoc.jsp?containerId=prUS46286020">ICD</a>, 2020). While big data enables endless opportunities, its vastness generates new challenges: the handling of these huge amounts of data. Second, and intimately linked to rapidly growing data, research in the machine learning field is fast-moving. Papers are published at great speed and are often accompanied with a working codebase. In fact, websites exist exclusively for papers published with a GitHub repository, such as <a href="https://paperswithcode.com/">Papers With Code</a>. Along with the codebase, tutorials to implement the new code in various machine learning architectures are also available. Thus, despite the evolving nature of this complex field, the research can virtually be implemented instantly due to extensive practical indications. Indeed, any machine learning engineer would be able to newly train a model on a custom dataset.</p><p>As a result, the competitive advantage in the machine learning market does not lie in the rapidity of applying the most recent knowledge for the development of a new machine learning system, but rather in finding a solution to handle and curate the vast and large sets of almost unlimited raw data. This blog post outlines five answers to tackle this challenge with data and presents Lightly’s current solutions.</p><ol><li><strong>Understand your data</strong></li></ol><p>As its name indicates, machine learning is the process of teaching a machine to act in certain ways. A machine learns from the data it is being fed with, so machine learning models are highly dependent on data. Indeed, data has become a commodity for companies using machine learning and is being collected in great quantities. Once collected, the data is raw, in other words not processed, and of little use. Moreover, the processing steps, such as labeling, are very expensive and time-consuming, thereby usually only done for a small fraction of the dataset. Hence, understanding data in its raw form is essential to save time and costs.</p><p>Understanding raw data includes two processes: an assessment and an exploration of the data (<a href="https://towardsdatascience.com/data-understanding-for-machine-learning-assessment-exploration-aca1aadc1cb6">towardsdatascience</a>, 2019). The assessment includes describing high-level characteristics of the dataset, such as size and format. This process is enabled by uploading datasets to the Lightly platform. There, such information (datatype, number of samples, and sample size) is automatically shown, as exemplified below for a public dataset of road images, Comma10k.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/708/0*oBnUNUrFk_xKQimf" /><figcaption>Screenshot of the characteristics of the Comma10k dataset on the Lightly platform</figcaption></figure><p>Once the data has been assessed, it must be explored. Indeed, the size and format of the Comma10k dataset provide little insight into the actual content and distribution of these images. Exploring data is one of Lightly’s unique features. Its visualization tool allows teams to visually perceive clusters of similar images in a scatterplot. This way, categories of data are formed. Additionally, outliers or missing points can be identified in this view. Exploring data is crucial in every industry. Automotive companies developing autonomous vehicles, for instance, have huge datasets of lengthy hours of video footage. Without an understanding of the collected data, these companies have no precise indication of the type of images they have collected. The plot obtained with Lightly is a way of obtaining visual insights about the data collected by observing groups of similar data.</p><p>Below, the Comma10k dataset is shown through the visualization tool on the Lightly platform. A cluster of night images on the left and another of day images on the right can be observed. Additionally, an outlier is shown, placed in between the day and night clusters. This image seems to have been taken during dawn, hence its in-between nature. This is a significant insight, as it exposes a scenario with little collected data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/759/0*Ol1DI-3y7hHWv8FR" /><figcaption>Screenshot of Comma10k’s visualization on the Lightly platform: image from night cluster</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/752/0*DIzFgYXgt3Wtw7f7" /><figcaption>Screenshot of Comma10k’s visualization on the Lightly platform: image from day cluster</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/754/0*u_T06UTwmn8sz7_6" /><figcaption>Screenshot of Comma10k’s visualization on the Lightly platform: outlier image in between clusters</figcaption></figure><p>The distribution of the data can also be observed for various parameters, such as file size, width, height, signal to noise ratio, luminance, contrast or color channels. Engineering teams can select the data that interests them by setting certain levels to these parameters. Below the signal to noise ratio is explored. For images, this refers to the image quality. In the examples below, images of different qualities have been selected. It can be noted that darker images, oftentimes taken during the night, are of lower quality. Thus, the parameters are an additional way of understanding the dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/510/1*OgcfA_X1S7HIaelofRHstA.png" /><figcaption>Screenshots of low signal to noise ratio images of the Comma10k dataset</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/513/1*LW0d5IiraRSejB2aS3yvaw.png" /><figcaption>Screenshots of high signal to noise ratio images of the Comma10k dataset</figcaption></figure><p><strong>2. Curate your data</strong></p><p>No matter how much machine learning engineers attempt to perfect their models and increase their performance, a model is only as good as the data it is trained on. Think of children and education: children can be very smart but the books they read and their access to education ultimately determine their intellectual knowledge. Thus, once the raw data has been properly understood, the selection of training data is crucial. This data must be of the highest quality and relevance, according to the task to be executed.</p><p>The curation process involves selecting and filtering data to ensure the model is well-adjusted to perform well. To make a machine learning model as performant as possible, the training set must be representative of different scenarios. With Lightly’s algorithms, redundant data is eliminated, while outliers and special scenarios are retained. For a dataset of street images, this would result in removing similar images captured at the same streetlight, while keeping interesting scenarios of people crossing the streets, for instance, and making sure images of the different clusters, such as different types of road signs, vehicles or sidewalk distractions, are sufficiently represented. Lightly intelligently does this with its algorithms, thereby creating a diverse and representative dataset. This automated data curation process is, in conclusion, more efficient, faster, and cheaper.</p><p>The diversification of the subset is enabled by a sampling strategy named CORESET. The new dataset can be customized in terms of size, i.e. how many images it should contain, and minimum distance between two dataset points, i.e. the amount of visual difference between two images. This gives engineers the opportunity to have full control over the training set and to gain an understanding of it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/590/0*vb_DfPZO16O911Oa" /><figcaption>Screenshot of creating a subset of the Comma10k dataset on the Lightly platform</figcaption></figure><p>The subset is then processed, and a view of the new subset of images can then be visualized. This visualization tool gives full transparency and eliminates the black-box nature of most large datasets.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*q9o02MzyGGevW4No" /><figcaption>Screenshot of the Comma10k subset created on the Lightly platform</figcaption></figure><p><strong>3. Build an efficient data pipeline</strong></p><p>The continuously evolving industry brings along new obstacles in the data pipeline, as every step of the process becomes more complex and market players specialize. Hence, managing data through this crowded pipeline becomes a challenge.</p><p>While Lightly’s main tasks are centered around data selection, also referred to as data curation, their solutions facilitate the entire machine learning pipeline. First, collection processes are optimized as a result of a thorough understanding of the collected data. The insights highlight scenarios that are sufficiently covered as well as underrepresented edge cases which must be collected further. Second, data selection occurs twice as fast with Lightly’s algorithms. Data selection for model training is no longer done manually or randomly. Indeed, training sets are intelligently diversifiable according to preferences in a few clicks. Third, the data labeling process is optimized. Usually, as labeling is expensive and the datasets are large, only a very small fraction of random data is labeled. With annotation costs amounting up to $10 per image, a well-curated dataset for autonomous driving can easily cost millions. With Lightly’s smart data selection process, the same accuracy can be reached with half the amount of data and saving up to millions in labelling costs. Fourth, model training is improved, as the curated training sets entail better generalization and increased accuracy. Lastly, as all steps gain efficiency, the development process is accelerated, significantly reducing the time to market of a product.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/718/0*e2JmrVUE7Vrs_--T" /><figcaption>Lightly’s contributions to the machine learning pipeline (own illustration)</figcaption></figure><p>Furthermore, Lightly’s application programming interface (API) allows companies to easily integrate the solution into existing data pipelines. We refer to this as Lightly Mission Control. Data can be managed in one place, and easily transferred from step to step.</p><p>Lightly also offers active learning. This is the process of choosing samples the model struggles with, labeling them, and adding them back to be trained on again. These difficult samples mostly help the model in improving accuracy when labelled. It is similar to teaching children: once they can solve simple problems, it no longer makes sense to teach them more simple problems. Instead, they should be given more challenging tasks to further improve.</p><p>In addition, the platform can integrate labeling partners from across the world in order to facilitate the entire machine learning pipeline. Thus, engineering teams can simply upload the collected dataset and the rest of the process will be carried out on the same platform, thereby increasing the data pipeline’s efficiency.</p><p><strong>4. Manage your data</strong></p><p>Managing data is an obvious challenge of big data. The large quantities of data collected over time require to be administered carefully in order to be efficiently exploited.</p><p>Data management, defined as “the process of ingesting, storing, organizing and maintaining the data created and collected by an organization” (<a href="https://searchdatamanagement.techtarget.com/definition/data-management">Tech Target</a>, 2019), entails different practices. First, data can be managed in time. Versioning allows to continuously update and improve datasets over time for different projects, for example. A past dataset could become relevant for new projects. Therefore, its storage with a precise indication of its content is essential. Tracking data over time is related to versioning. Yet, the emphasis is on the ability to go back in time and observe a dataset’s evolution or check for current relevance. Second, data can also be managed in space. Splitting or merging datasets are examples of such management. Indeed, some models are trained to specialize in specific tasks with specific data, while others require a vast and diverse dataset to train a general model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/766/0*5eZwsYFrMXnMwPyI" /><figcaption>Lightly’s Data Management Vision (own illustration)</figcaption></figure><p>Data management is relevant in almost every industry for its own purposes. On the one hand, autonomous driving companies split data by creating subsets for the different countries they operate in, as road signalization differs. On the other hand, night and day footage as well as frames of unexpected animals or passengers crossing roads are significant for most cases and can be added to or merged with several training subsets. New versions of datasets for these different use cases are then created and tracked over time.</p><p>Currently, with Lightly, subsets can easily be generated by manually selecting clusters with the visualization tool. They can be named and utilized separately. Below, a cluster of 63 similar images of bridges has been manually selected. A separate subset could be created if further diversification or manipulations are desired within this subset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/903/1*oeDtBHqcjeQCxABsShkK5Q.png" /><figcaption>Screenshot of a subset of the Comma10k dataset created with the visualization tool on the Lightly platform</figcaption></figure><p><strong>4. Share your data internally</strong></p><p>As previously mentioned, data has become a commodity. It is valuable to many players of an organization for various projects and goals. Having a central platform to share data is crucial for coordination and optimization within engineering teams, as teams can work on projects in one place. Along with engineers, other stakeholders should be granted access to the datasets in order to facilitate coordination and communication amongst various functions and divisions of the company. This way, processes become more transparent to others, who are otherwise disconnected from the data pipeline. Nevertheless, while data sharing is important, access management should be implemented as well due to data privacy concerns.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/399/0*HUgzQHf1Ig7dVW8l" /><figcaption>The Lightly Platform (own illustration)</figcaption></figure><p>In addition, data sharing allows for reproducibility. In other words, the same results can be obtained with different datasets of various teams for other projects. Indeed, with the work and processes of colleagues perceivable to others, internal processes for working on data can be standardized across the organization.</p><p>As of right now, Lightly users sign in with individual accounts where they have access to their personal datasets. Ideally, this system would be upgraded to allow datasets to be shared amongst users of the same organization if desired.</p><p>Author: Sarah Meibom</p><p><strong>Thanks </strong>to Malte Ebner, Jeremy Prescott, and <a href="https://medium.com/@isusmelj">Igor Susmelj</a> for reading the drafts of this.‍</p><h4>Contact Us</h4><p>If this blog post caught your attention and you’re eager to learn more, follow us on Twitter and Medium! If you’d like to hear more about what we’re doing at Lightly, reach out to us at info@lightly.ai. If you’re interested in joining a fun, rockstar engineering crew to help make data preparation easy, reach out to us at jobs@lightly.ai!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cdf872048596" width="1" height="1" alt=""><hr><p><a href="https://medium.com/whattolabel/5-answers-to-data-problems-in-computer-vision-cdf872048596">5 Answers to Data Problems in Computer Vision</a> was originally published in <a href="https://medium.com/whattolabel">Lightly</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>