<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Hamza Tahir on Medium]]></title>
        <description><![CDATA[Stories by Hamza Tahir on Medium]]></description>
        <link>https://medium.com/@htahir1?source=rss-7804b7e02c3------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*ZYUkAI2_Hhect6lQH7SP-A.jpeg</url>
            <title>Stories by Hamza Tahir on Medium</title>
            <link>https://medium.com/@htahir1?source=rss-7804b7e02c3------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 12:21:48 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@htahir1/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Unconventional splitting techniques for time-series datasets]]></title>
            <link>https://htahir1.medium.com/splitting-data-in-time-series-d0284038dff0?source=rss-7804b7e02c3------2</link>
            <guid isPermaLink="false">https://medium.com/p/d0284038dff0</guid>
            <category><![CDATA[training]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[time-series-analysis]]></category>
            <category><![CDATA[mlops]]></category>
            <dc:creator><![CDATA[Hamza Tahir]]></dc:creator>
            <pubDate>Wed, 14 Sep 2022 09:32:06 GMT</pubDate>
            <atom:updated>2022-09-14T09:32:36.987Z</atom:updated>
            <content:encoded><![CDATA[<h4>Be careful when time is a dimension in your dataset</h4><p>One attempt to ensure that ML models generalize in unknown settings is splitting data. This can be done in many ways, from 3-way (train, test, eval) splits to k-splits with cross-validation. The underlying reasoning is that by training a ML model on a subset of the data, and evaluating on unknown data, one can reason much better if the model has underfit or overfit in training.</p><p>For me, splitting data is the most under-rated task in all of data science. Whilst talking to users of <a href="https://github.com/zenml-io/zenml">ZenML</a>, an opensource MLOps framework to create production-ready ML pipelines, I have consistently seen users make the splitting as one of the first parts of their ML workflows.</p><p>It is understandable that for most jobs, a simple 3-way split suffices. I have stumbled across many problems where there is a need for more complicated splits to ensure generalization. These splits are more complex because they are derived from the actual data, rather than the structure of the data which the hitherto mentioned split methods are based on. This post attempts to break down some of the more unconventional ways to split data in ML development, and the reasoning behind them.</p><h3>Lets start with a dataset</h3><p>In order to illustrate the split mechanisms, it helps to start with a sample dataset to do the splits on. To make things easy, lets use a simple multi-variate, timeseries dataset represented in tabular format. This data consists of<br>3 numerical features, 1 categorical feature and 1 timestamp feature. Below this is visualized:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*lLKAOO1zUoSEGvFIUFqwVQ.png" /><figcaption>Image by author</figcaption></figure><p>This type of dataset is common across many use-cases and industries in machine learning. A concrete example can be multiple timestreams transmitted from different machines with multiple sensors on a factory floor. The categorical variable would then be the ID of the machine, the numerical features would be what the sensors are recording (e.g. pressure, temperature etc.), and the timestamp would be when the data was transmitted and recorded in the database.</p><h3>Doing the splits</h3><p>Imagine you receive this dataset as a csv file from your data engineering department and are tasked with writing a classification or a regression model. The label in such a case could be any of the features or an additional column. Regardless, the first thing to do would be to try to split up the data into sets that are meaningful.</p><p>To make things easy, you decide to go make a simple split with train and eval. You know immediately that a naive random split with shuffling won’t fly here - the data does have multiple sensor streams that are indexed by time after all. So how do you split the data so that order is maintained and subsequent models are sufficiently generalizable?</p><h3>Another view of the data</h3><p>The most straightforward transformation we can do is to represent the data per categorical class (in our running example, visualize the data per machine). This would yield the following result:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*T44uQPj2wa6lekAcAL76MA.png" /><figcaption>Image by author</figcaption></figure><h3>The Horizontal Split</h3><p>The grouping together suddenly makes the issue of splitting a bit simpler, and largely dependent on your hypothesis. If the machines are running under similar conditions, one question you might ask is: How would a ML model trained on one group generalize to other groups. That is, if trained on class_1, class_2 and class_3 timestreams how would the model fair on class_4 and class_5 timestreams. Here is a visualization of that split:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*y7yMDunqCVdzFctsJcWGMg.png" /><figcaption>Image by author</figcaption></figure><p>I call this the Horizontal split due to nature of the cut line in the above visualization. This split can be easily achieved in most ML libraries by simply grouping by the categorical feature and partitioning along it. A successful training with this split would show evidence that the model has picked up signals that generalize across previously unseen groups. However, it would not showcase that it is able to predict future behavior of one group.</p><p>Its important to note that the the split decision did NOT account for time as a basis of the split itself. One can assume however that you would also sort by time per timestream to maintain that relationship in your data. Which brings us to the next split..</p><h3>The Vertical Split</h3><p>But what if you want to split across time itself? For most time-series modeling, a common way to split the data is past and future. That is, to take in the training set historical data relative to the data in the eval set. The hypothesis in this case would be: How would a ML model trained on historical data per group generalize to future data for each group?. This question might be answered by the so called Vertical split:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*CWXSi43lHA5xrHw3UFrQ4g.png" /><figcaption>Image by author</figcaption></figure><p>A successful training with this split would showcase that the model is able to pick up patterns across timestreams it has already seen, and make accurate predictions of behavior in the future. However, this itself would not show that this model will generalize well to other timestreams from different groups.</p><p>Of course, your multiple timestreams have to be sorted now individually, so we still need to group. However, this time, rather than cutting across groups, we take a sample of the past of each group and put it in train and the future of each group in eval. In this idealized example, all the timestreams are of the same length, i.e., each timestream has exactly the same number of data points. However, in the real world, this maybe not be the case - so you would require a system to build an index across each group to make this split.</p><h3>The Hybrid Split</h3><p>An inquisitive ML researcher might at this point wonder if they could produce a model that would generalize under both constraints of the Horizontal and the Vertical split. The hypothesis in that case would be: How would a model trained on historical data for SOME groups generalize to future data of these groups AND all data from other groups?. A visualization of this Hybrid split would look like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*5a5Fdoh5T9ImToCS_F2hGw.png" /><figcaption>Image by author</figcaption></figure><p>Naturally, if model training is successful, this model would surely be more robust than the others in a real world setting. It would have displayed evidence to not only learning patterns of some of the groups it has already seen, but also evidence of the fact that it has picked up signals that generalize across groups. This might be useful if we are to add more similar machines to the factory in the future.</p><h3>Multi-dimensional splits</h3><p>The notion of the horizontal and vertical splits can be generalized to many dimensions. For example, one might want to do group by two categorical features rather than one to even further isolate sub-groups in the data, and sort them per sub-group. There might also be complex logic in the middle to filter groups that have a lower number of samples, and other business-level logic pertaining to the domain.</p><h3>Conclusion</h3><p>The hypothetical example is used to illustrate the endless possibilities of various kinds of machine learning splits that can be created by an astute data scientist. Just like it is important to <a href="https://developers.google.com/machine-learning/fairness-overview">ensure ML fairness</a> whilst evaluating your models, it is equally important to spend sufficient time thinking about splitting a dataset and its consequences to bias the model downstream.</p><p>Thank you and happy splitting!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d0284038dff0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[‘It’s the data, silly!’ How data-centric AI is driving MLOps]]></title>
            <link>https://htahir1.medium.com/its-the-data-silly-how-data-centric-ai-is-driving-mlops-740cdb6ca09e?source=rss-7804b7e02c3------2</link>
            <guid isPermaLink="false">https://medium.com/p/740cdb6ca09e</guid>
            <category><![CDATA[data-centric]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[ml-so-good]]></category>
            <category><![CDATA[pipeline]]></category>
            <dc:creator><![CDATA[Hamza Tahir]]></dc:creator>
            <pubDate>Thu, 07 Apr 2022 13:54:15 GMT</pubDate>
            <atom:updated>2022-04-08T06:29:44.796Z</atom:updated>
            <content:encoded><![CDATA[<h4>The hidden link between data-centric machine learning and MLOps</h4><p>You start your day with jupyter lab, and look at the latest state of affairs from yesterday. The notebook springs up on port 8888 and you see your matplotlib graph from yesterday, annoyingly stuck at 80% accuracy. After a few hours of hard work, you finally find a hyperparameter that works. You set it, go through the cells, and suddenly you’re at 95% accuracy. You re-check it over and over, and after making sure you did everything right, you export your model into a file and hand over the code and model to engineering, a smile on your face, and with the assurance that <strong>you</strong> did your job well. Now you can sit back and see the bottom line growing. Right?</p><p>Wrong. Your engineer glares back at you and asks for your requirements.txt file. He cannot reproduce your results — there is an error on their machine. They ask what data you trained on, which operating system you developed on, and just generally about following “MLOps” principles. You go back and try to answer these questions but now you’re in a painful debugging loop.</p><p>Does any of this sound familiar? As classically-trained data scientists, we have been tuned to stay within a notebook environment and iterate on the code and model. Doing this can lead to unintended consequences and optimizing for the wrong result. Perhaps you would have faired better if you would have taken a slightly different approach to your model development, rooting yourself from the start in these MLOps principles engineers keep talking about.</p><p>There is an ever-increasing plethora of resources around MLOps (see the end of this article) and an increasing amount regarding the shift from model-centric to data-centric machine learning. However, few speak about the link between data-centric machine learning and how it is driving MLOps practices today. In this article, I go deeper into how a data-centric, pipeline-based approach to machine learning is one of the best ways to follow MLOps principles, and how you as a data scientist would benefit greatly from understanding why that is the case.</p><h3>MLOps is not just about deploying models</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*zinvVCp1yEjAxBy_.png" /><figcaption>Twitter can be helpful, sometimes.</figcaption></figure><p>When speaking about MLOps, developers often confuse it with the simple act of deployment. However, conversations such as the above do not simply refer to deploying models. Machine learning engineering tackles a broader set of challenges that span more than merely wrapping up a model in a server application and deploying it.</p><p>ML development may be broken down into the following (relatively) simple processes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*tgQ5X9cf0xbMpK1b.png" /><figcaption>ML in production: What we think it looks like</figcaption></figure><p>Taken as silos, these processes don’t sound too hard:</p><ul><li>Feature engineering is getting easier with feature stores such as <a href="https://feast.dev/">Feast</a>.</li><li>The training loop is made easier by thousands of tools that help in the iterative process, from experiment tracking tools like <a href="https://mlflow.org/">MLflow</a> and <a href="https://wandb.ai/site">Weights &amp; Biases</a> to advanced training frameworks like <a href="https://www.pytorchlightning.ai/">PyTorch Lightning</a>.</li><li>Deploying models are also getting easier with the advent of advanced tooling such as <a href="https://github.com/SeldonIO/seldon-core">Seldon Core</a>, or managed services offered by all the major cloud providers.</li></ul><p>However, the reality is that the process looks more like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*uBGQP0_A3GeMKGD6.png" /><figcaption>Its hard to feedback loops reliable in machine learning.</figcaption></figure><p>More than code, in machine learning <strong>data</strong> affects the output of the system directly. There are feedback loops that happen implicitly and often explicitly within the lifetime of a model that is deployed in production. While in classical software development one could simply test and vet code as it passes through various stages to production, it is more complex and difficult to this in a system affected by data.</p><p>It is in these feedback loops where MLOps ultimately lives. It is not enough to do this process once: A successful ML team needs to execute this process over and over again, and in a manner such that the system can be trusted.</p><p>Simply stated, MLOps is a set of practices that aims to deploy and maintain machine learning models in production reliably and efficiently. This is including and beyond getting these models deployed into production.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*PzIP9DgB-LBwn5ki.png" /><figcaption>MLOps live in the feedback loops of machine learning development.</figcaption></figure><h3>Post-deployment Woes</h3><p>When looked at from this perspective, it is more intuitive to understand that the trouble starts after the first deployment. Here are just a few examples:</p><p><strong>Latency problems:</strong></p><p>If latency is not accounted for whilst developing ML models, it can have a huge impact on a business. You could lose <a href="https://www.thinkwithgoogle.com/consumer-insights/consumer-trends/mobile-site-load-time-statistics/">half your traffic</a> with a slow load-up of your application. This means that when employing models in production, one needs to be cognisant of latency requirements in production.</p><p><strong>Maintaining fairness and avoiding bias</strong></p><p>If left unchecked, bias can creep into a ML system very easily. <a href="https://www.nytimes.com/2016/03/25/technology/microsoft-created-a-twitter-bot-to-learn-from-users-it-quickly-became-a-racist-jerk.html">Microsoft’s rogue racist Twitter bot</a> is an example of not setting up systems to maintain fairness in ML development. If a model is not inspected closely (e.g. in terms of slicing metrics), then a practitioner risks that they might unintentionally model bias into the system. This is where examining data closely is critical to ensure a robust and fair system.</p><p><strong>Lack of explainability and auditability</strong></p><p>If fairness is not maintained in a system, then legislators will be in their full right to come after ML practitioners. The <a href="https://ec.europa.eu/commission/presscorner/detail/en/IP_21_1682">European Commission</a> is already rolling out new laws and checks, and we can only expect this to grow over time. Practitioners should be able to answer questions such as why a certain prediction was made, how a certain model was trained, and on which slice of data. These audit trails are all parts of the MLOps workflow.</p><p><strong>Painfully slow development cycles</strong></p><p>Getting a model into production can take <a href="https://algorithmia.com/state-of-ml">up to a year for many companies</a>. That means going through the above process only once can cost hundreds of thousands of dollars, let alone having to do it again and again. Teams need to automate most of the tedious stuff away if they are to have any sort of argument for a legitimate return-on-investment for machine learning being applied in a business.</p><p><strong>Model, concepts, and data drifts</strong></p><p>The real world is not static. Training models on data that do not change are willingly ignoring the fact that this does not happen in the real world. The recent <a href="https://www.geekwire.com/2021/zillow-shutter-home-buying-business-lay-off-2k-employees-big-real-estate-bet-falters/">disaster at Zillow</a> is illustrative of how drift can cost a business dearly. As the world changes, MLOps systems need to be robust to these changes and deal with them as they arise.</p><h3>Model-centric vs Data-centric Machine Learning</h3><p>Andrew Ng recently popularized the term data-centric machine learning with his excellent talk in 2020. Watch the full video below:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F06-AZXmwHjo%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D06-AZXmwHjo&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F06-AZXmwHjo%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/9af3778599f8f156edf1fb1ce5b80495/href">https://medium.com/media/9af3778599f8f156edf1fb1ce5b80495/href</a></iframe><p>The essence of the talk is as follows: You can get a lot of bang for your buck from your data by being data-centric rather than model-centric. This means that rather than iterating on the model/code and holding the data static in ML development, it would pay more dividends if you were to hold the code/model static (or even start with a simple model) and try to simulate real-world behavior with the data. This of course is in stark contrast to how ML is taught in crash courses and universities, where the process usually starts in a notebook with reading a static, well-prepared dataset, and training a model on it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*YiIDnnvAlm9ug7ql.png" /><figcaption>Model-centric vs data-centric deployment</figcaption></figure><p>When taking a look at the challenges facing ML in production today, it is clear that a shift towards being data-centric is simply the natural mindset shift that is required. Latency problems can be solved by exposing data scientists to real-world data ingestion patterns. Fairness and bias can be avoided by inspecting the data at the moment of model training. Auditability trails can be kept if data is versioned and tracked as models are developed. The development cycle can also be accelerated vastly by creating data-centric workflows that can adapt to changing data. Finally, drift and data quality can be accounted for early on in the development process.</p><p>Here is an example of a model-centric decision vs a data-centric decision, that also showcases its link with MLOps. Let’s say a data scientist has a dataset for the last year and is tasked with developing a model. The model-centric way of approaching such a task would be to use the entire data to train the model with a little bit leftover as a test set to verify the model metrics. Perhaps hyperparameter tuning is applied to squeeze out the maximum accuracy from the model and data.</p><p>On the other hand, a data-centric decision, and a decision that would help ultimately in production, would be to partition the data in a way that a portion of it (let’s say the first three quarters of the year) is used for the training process, and the last quarter is used as a separate dataset to see how the model drifts over time. This would perhaps incur a slight loss in accuracy for the model, but give key information about the behavior of the model by simulating it being deployed out in the real world.</p><p>In the end, while being model-centric has its benefits, adding data-centric decisions into the mix is ultimately what is the best path forward when applying ML in the real world. With that, there is a natural synergy between this and the adoption of MLOps.</p><h3>Towards data-centric ML(Ops): From scripting to pipelines</h3><p>A concrete shift to data-centric machine learning often involves an ML team shifting focus from script-based development to pipeline-based development. Machine learning lends itself very nicely to developing in terms of pipelines because most development does consist of a sequence of steps carried out in order.</p><p>Here it is important to make a distinction between data-driven pipelines vs task-driven pipelines. This means that serious teams develop ML code as chunks of steps, using some form of tooling to isolate the orchestration of execution of steps from each other. This has the following benefits:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*jovgu0xlrEZZ-E7m14NBzQ.png" /><figcaption>Developing in pipelines lends itself naturally to data-centric machine learning.</figcaption></figure><p>Often this means stepping out of a notebook environment or finding some way of transporting notebook code to such a paradigm.</p><h3>Takeaways</h3><p>So, in short, here is the link between MLOps and data-centric machine learning:</p><ul><li>ML in production is different from ML in research and has a different set of challenges.</li><li>These challenges are growing in relevance as the adoption of ML increases.</li><li>MLOps helps solve these problems.</li><li>MLOps is rooted in being more data-centric than model-centric.</li><li>Developing pipelines help in being more data-centric.</li></ul><p>I hope that helps to clarify the link between these two popular terms and gives beginner MLOps practitioners an indication of where to take their efforts as they develop internal ML tooling for their organizations.</p><p>Shameless plug: If you’d like to start the shift towards data-centric machine learning by developing ML pipelines, then you might want to take a look at <a href="https://github.com/zenml-io/zenml">ZenML</a>. It is designed with the following goals:</p><ul><li>Be simple and intuitive to give a simple path toward data-centric machine learning.</li><li>Be infrastructure and tooling agnostic across the MLOps stack.</li><li>Start writing your pipeline in a notebook and carry it easily into the cloud with minimum effort.</li></ul><h3>Resources</h3><p>If you’d like to get more into MLOPs, I would recommend the following excellent resources to get started:</p><ul><li>MLOps Course <a href="https://madewithml.com/">MadeWithML</a></li><li><a href="https://mlsys.stanford.edu/">CS 329S: Machine Learning Systems Design</a></li><li><a href="https://fullstackdeeplearning.com/">Full Stack Deep Learning</a></li><li><a href="https://github.com/zenml-io/zenbytes">ZenBytes — learn MLOps through ZenML</a></li></ul><h3>References</h3><p>Some images inspired by Andrew Ng’s “From Model-centric to Data-centric AI” on YouTube: <a href="https://www.youtube.com/watch?v=06-AZXmwHjo">https://www.youtube.com/watch?v=06-AZXmwHjo</a></p><p><a href="https://medium.com/mlearning-ai/mlearning-ai-submission-suggestions-b51e2b130bfb">Mlearning.ai Submission Suggestions</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=740cdb6ca09e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Taking on the ML pipeline challenge]]></title>
            <link>https://medium.com/data-science/taking-on-the-ml-pipeline-challenge-3dfeb7b24cc6?source=rss-7804b7e02c3------2</link>
            <guid isPermaLink="false">https://medium.com/p/3dfeb7b24cc6</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-pipeline]]></category>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[editors-pick]]></category>
            <category><![CDATA[notes-from-industry]]></category>
            <dc:creator><![CDATA[Hamza Tahir]]></dc:creator>
            <pubDate>Wed, 27 Oct 2021 08:04:53 GMT</pubDate>
            <atom:updated>2021-10-27T20:50:51.284Z</atom:updated>
            <content:encoded><![CDATA[<h4><a href="https://towardsdatascience.com/tagged/notes-from-industry">Notes from Industry</a></h4><h4>Why data scientists need to own their ML workflows in production</h4><p>This article discusses the benefits of giving data scientists ownership of their workflows in a production environment. We also discuss ZenML, an open-source framework that is built specifically to facilitate this ownership.</p><h3>Why do we need ML pipelines?</h3><p>Let’s start with the most obvious question: Why do we need ML pipelines in the first place? Here are some solid reasons:</p><ul><li>ML pipelines are the best way to <strong>automate ML workflows </strong>in a repeatable, robust and <strong>reproducible</strong> manner.</li><li>Organizations can centralize, collaborate, and manage workflows with a <strong>standardized</strong> interface.</li><li>ML pipelines can be seen as a means of <strong>coordination</strong> between different departments.</li></ul><p>However, awesome as ML pipelines are, they do come with some inherent questions which might prove problematic.</p><h3>The Ownership Dilemma</h3><figure><img alt="A data scientist who is confused how to get ML in production" src="https://cdn-images-1.medium.com/max/704/1*qWJBz5cI6X6asHL9oSldjQ.png" /><figcaption>ML In Production is confusing — Image by Author</figcaption></figure><p>One question that organizations developing machine learning need to answer is <strong>who owns ML pipelines in production</strong>? Is it the data scientist who creates the model? Is it the data engineer who deploys it in production? Is it someone else altogether?</p><blockquote>Note: For an overview of the roles involved in the ML process, check out this part of <a href="https://fall2019.fullstackdeeplearning.com/course-content/ml-teams/roles">UC Berkeley’s Full Stack Deep Learning Course</a>.</blockquote><p>The data dimension of ML makes it <a href="https://research.google/pubs/pub46555/">significantly more complex</a> than other deployments in production. More often than not, data scientists do not have the skill set to reliably guide a model into production. Therefore, it is only natural for teams to organize in such a way that there is a<em> shift of ownership</em> from the training phase of development to the deployment phase in the direction of the engineering department (<a href="https://wiki.c2.com/?ThrownOverTheWall">thrown over the wall</a>).</p><p>Organizing this way makes it easy enough to get a model in production <strong>once</strong>. However, the ownership question really comes into play when things (inevitably) <strong>go wrong</strong>. This could be as the model/concept drifts, or something times out, or the data format changes, or a million other things that the engineering department would know nothing about as they <strong>didn&#39;t produce the model</strong>.</p><p>Also, the farther away you push the data scientist from a production setting, the harder it is to establish a <a href="https://blog.modyo.com/posts/data-flywheel-scaling-a-world-class-data-strategy"><strong>data-flywheel effect</strong></a> of consistent improvement of your models in production. The data scientist needs a wider context in order to be able to make the right calls when training the model. E.g. Should we sacrifice the AUROC a few percent, if that means the model might perform faster in production and yield higher revenue? There is no chance one can even ask that question if just looking at model metrics like AUROC.</p><p>Indeed, the further we push the data scientist from production, the <a href="https://multithreaded.stitchfix.com/blog/2019/03/11/FullStackDS-Generalists/#back-1">less productive the overall ML process will get, and the higher the coordination costs and wait times will get</a>. That’s a no-go for ML, which is supposed to be fast and iterative process.</p><h3>Solving the Ownership Dilemma</h3><p>Recently, there has been a lot of talk about <a href="https://huyenchip.com/2021/09/13/data-science-infrastructure.html">how data scientists shouldn’t need to know Kubernetes</a>, or the underlying production infrastructure needs to be necessarily abstracted from them to perform at a high level. I would not only agree with this, but also posit that we have to go one step further. We not only need to abstract the infrastructure from data scientists, but help them take <strong>ownership</strong> of their models all the way to production.</p><p>If a team leaves writing ML pipelines for later, they will <a href="https://towardsdatascience.com/avoiding-technical-debt-with-ml-pipelines-3e5b6e0c1c93?source=your_stories_page-------------------------------------&amp;gi=9118ab490b18">quickly accrue technical debt</a>. With the right abstractions, an organization can incentivize their data scientists to start <a href="https://towardsdatascience.com/why-ml-should-be-written-as-pipelines-from-the-get-go-b2d95003f998">writing end-to-end ML pipelines early in the development process</a>. Once data scientists start writing their training/development workflows with easily ‘transferable’ ML pipelines, they would find themselves in a familiar environment once these same pipelines go into production. They would then have the tools necessary to also go into production systems and fix problems as they occur, or make improvements as time goes on.</p><p>Ideally, the pipeline becomes a mechanism through which the producers of the models (i.e. the data scientists) can take ownership of their models all the way to production.</p><p>Note that this does <strong>NOT</strong> mean that data scientists should now know every toolkit in the engineering toolbox (like complex deployments of Kubernetes clusters). Rather, the argument is that we need to set data scientists up so that they can take their code to production themselves, with the help required from necessary tooling.</p><h3>Enter ZenML: A framework designed for modern MLOps</h3><p><a href="https://github.com/zenml-io/zenml">ZenML</a> is an open-source <em>MLOps Pipeline Framework </em>built specifically to address the problems above. Let’s break it down what a MLOps Pipeline Framework means:</p><ul><li><strong>MLOps</strong>: It operates in the domain of operationalizing machine learning, i.e., putting ML workflows in production.</li><li><strong>Pipeline</strong>: It does this by helping create pipelines, i.e., a sequence of steps performed in this case, specifically for a ML setting.</li><li><strong>Framework: </strong>Finally, it creates these pipelines software with abstractions providing generic functionality, which can be selectively changed by additional user-written code.</li></ul><p>So its a tool that let’s you define pipelines but how is it different from the others? Here is what sets it apart:</p><h4>Accommodating the Exploding ML Tooling Landscape</h4><p>Everybody knows that we are now <a href="https://huyenchip.com/2020/06/22/mlops.html">in the midst of an explosion in the ML/MLOps tooling landscape</a>. ZenML is explicitly designed to have <strong>no opinions</strong> about the underlying infrastructure/tooling that you would like to use. Rather it exposes higher level concepts like Metadata Stores , Artifact Stores, and Orchestrators that have common interfaces. A ML team can then swap out individual components of their pipelines backends and it will ‘just work’.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kK1J8ni2p4YKEDhtG0cFcw.jpeg" /><figcaption>Look closer: This isn’t the <a href="https://proceedings.neurips.cc/paper/2015/file/86df7dcfd896fcaf2674f757a2463eba-Paper.pdf">Hidden Technical Debt Diagram</a> ;-) — Image by Author</figcaption></figure><p>So, if you even want to use <a href="https://mlflow.org/">MLFlow</a> to track your experiments, run the pipeline on <a href="https://airflow.apache.org/">Airflow</a>, and then deploy a model to a <a href="https://neptune.ai/">Neptune</a> Model Registry, ZenML will facilitate this MLOps Stack for you. This decision can be made jointly by the data scientists and engineers. As ZenML is a framework, custom pieces of the puzzle can also be added here to accommodate legacy infrastructure.</p><h4>Focus on Machine Learning Workflows (at every phase)</h4><p>There are many tools that let you define workflows as pipelines but few that focus <strong>explicitly on machine learning use-cases</strong>.</p><pre>@trainer<br>def trainer(dataset: torch.Dataset) -&gt; torch.nn.Module:<br>   ... <br>   return model</pre><p>As an example, in the above code, ZenML will understand that this is not just a step in the pipeline, but a trainer step. It can use that information to aid in ML-specific use-cases like storing the result in a model registry, deploying this step on GPUs, hyper-parameter tuning etc.</p><p>Also notice that ZenML fully supports objects from common ML frameworks like torch.Dataset and torch.nn.Module . These objects can be passed between steps and results cached to enable faster experimentation.</p><h4>Incentivizing the data scientist to write these pipelines</h4><p>ZenML understands that pipelines are going to change over time. Therefore, it encourages running these pipelines locally to begin with and experimenting with the results as they are produced. You can query pipelines in a local Jupyter notebook, and materialize it with different pre-made visualizations like statistics visualizations and schema anomalies.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/685/1*B1xIuK6BY6WgpECvl7e8EQ.png" /><figcaption>After running pipelines, one can fetch them and see results easily, no matter if run locally or not — Image By Author</figcaption></figure><p>This is a different approach to pipeline development, and is more representative of how a data scientist would like to work in the earlier phases of a project → i.e. with fast, quick iterations and visualizations that help them to make informed decisions about experiments. We call this approach Pipelines As Experiments (PaE) .</p><p>In short, ZenML allows you to create automated ML workflows with simple, extensible abstractions that lift the common pattern burden off of you. In doing so, it does not take opinions on the underlying infrastructure and aims to be cloud- and tooling-agnostic.</p><p>By helping the target audience, i.e. the data scientist, to write their code in ZenML pipelines <strong>early in the development lifecycle</strong>, the transition from the experimentation phase to the production phase is made much easier. The goal is that by the time the experimentation phase of the ML lifecycle is over, the data scientists can flip a switch to the production ML stack and get their pipelines running in production. At this moment, they would have complete ownership of these pipelines and can manage, update, and debug them as they please.</p><h3>The Story So Far</h3><p>To date, ZenML has received:</p><ul><li>1336 <a href="https://github.com/zenml-io/zenml">GitHub Stars</a>.</li><li>10 <a href="https://github.com/zenml-io/zenml/graphs/contributors">Contributors</a>.</li><li>~200 <a href="https://zenml.io/slack-invite">Slack</a> Members.</li></ul><p>It has been fantastic to see people’s interest in what was initially just a simple idea. We’re now building <a href="https://zenml.io/newsletter">ZenML out in the open (no stealth here)</a>, and just <a href="https://github.com/zenml-io/zenml/releases">made a major release with a complete refactor of the codebase</a>. So, if any of the above appealed to you, it would be lovely if you gave ZenML a spin with an <a href="https://docs.zenml.io/guides/low-level-api">end-to-end example of deploying a pipeline in production</a>. Feedback and contributions are welcome!</p><p>My sincerest thanks to <a href="https://medium.com/u/7dfb6d58faaa">Alex Strick</a> and <a href="https://medium.com/u/6f18d7b671e6">Adam Probst</a> for helping edit this article.</p><h4>References and Appreciation</h4><p>Some useful links referenced in this post:</p><ul><li><a href="https://towardsdatascience.com/why-ml-should-be-written-as-pipelines-from-the-get-go-b2d95003f998">Why ML should be written as pipelines from the get-go</a></li><li><a href="https://fall2019.fullstackdeeplearning.com/course-content/ml-teams/roles">Roles</a></li><li><a href="https://towardsdatascience.com/avoiding-technical-debt-with-ml-pipelines-3e5b6e0c1c93">Avoiding technical debt with ML pipelines</a></li><li><a href="https://huyenchip.com/2021/09/13/data-science-infrastructure.html">Why data scientists shouldn&#39;t need to know Kubernetes</a></li><li><a href="https://eugeneyan.com/writing/end-to-end-data-science/#from-start-identify-the-problem-to-finish-solve-it">Unpopular Opinion - Data Scientists Should be More End-to-End</a></li><li><a href="https://multithreaded.stitchfix.com/blog/2019/03/11/FullStackDS-Generalists/#back-1">Beware the data science pin factory: The power of the full-stack data science generalist and the perils of division of labor through function | Stitch Fix Technology - Multithreaded</a></li><li><a href="https://docs.zenml.io/">Introduction | ZenML Documentation</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3dfeb7b24cc6" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/taking-on-the-ml-pipeline-challenge-3dfeb7b24cc6">Taking on the ML pipeline challenge</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why ML should be written as pipelines from the get-go]]></title>
            <link>https://medium.com/data-science/why-ml-should-be-written-as-pipelines-from-the-get-go-b2d95003f998?source=rss-7804b7e02c3------2</link>
            <guid isPermaLink="false">https://medium.com/p/b2d95003f998</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[workflow]]></category>
            <category><![CDATA[pipeline]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[mlops]]></category>
            <dc:creator><![CDATA[Hamza Tahir]]></dc:creator>
            <pubDate>Wed, 31 Mar 2021 13:09:34 GMT</pubDate>
            <atom:updated>2021-03-31T20:42:50.952Z</atom:updated>
            <content:encoded><![CDATA[<h4>MLOps</h4><h4>Eliminate technical debt with iterative, reproducible pipelines</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*rwwyYm2y5W16LFC5" /><figcaption>Photo by <a href="https://unsplash.com/@xoforoct?utm_source=medium&amp;utm_medium=referral">EJ Strat</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Today, Machine Learning powers the top 1% of the most valuable organizations in the world (FB, ALPH, AMZ, N etc). However, 99% of enterprises struggle to productionalize ML, even with the possession of hyper-specific datasets and exceptional data science departments.</p><p>Going one layer further into how ML propagates through an organization reveals the problem in more depth. The graphic below shows an admittedly simplified representation of a typical setup for machine learning:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*0lWL-rCF5yQnwsWJrlWgGQ.png" /><figcaption>Figure 1: Why it’s hard to reproduce ML models</figcaption></figure><p>There are three stages to the above process:</p><h4>Experimenting &amp; PoCs:</h4><ul><li><strong>Technologies</strong>: Jupyter notebooks, Python scripts, experiment tracking tools, data exploration tools</li><li><strong>Persona</strong>: Data scientists</li><li><strong>Description</strong>: Quick and scientific experiments define this phase. The team wants to increase their understanding of the data and machine learning objective as rapidly as possible.</li></ul><h4>Conversion:</h4><ul><li><strong>Technologies</strong>: ETL pipelining tools such as Airflow</li><li><strong>Persona</strong>: Data Engineers</li><li><strong>Description</strong>: Converting finalized experiments into automated, repeatable processes is the aim of this code. Sometimes this starts before the next phase, some times after, but the essence is the same — take the code from the data scientists and try to put them in sort form of an automated framework.</li></ul><h4>Productionalization &amp; Maintenance:</h4><ul><li><strong>Technologies</strong>: Flask/FastAPI, Kubernetes, Docker, <a href="http://cortex.dev/">Cortex</a>, <a href="https://www.seldon.io/">Seldon</a></li><li><strong>Persona</strong>: ML Engineers / Ops</li><li><strong>Description</strong>: This is the phase that starts at the deployment of the model, and spans monitoring, retraining, and maintenance. The core focus of this phase is to keep the model healthy and serving at any scale, all the while accounting for drift.</li></ul><p>Each of these stages requires different skills, tooling, and organization. Therefore, it is only natural that there are many potholes that an organization can run into along the way. Inevitably things that are important downstream are not accounted for in the earlier stages. E.g. If training happens in isolation from the deployment strategy, that is never going to translate well in production scenarios — leading to inconsistencies, silent failures, and eventually failed model deployments.</p><h3>The Solution</h3><p>Looking at the above multi-phase process in Figure 1, it seems like a no-brainer to simply reduce the steps involved and therefore eliminate the friction that exists between them. However, given the different requirements + skillsets for each step, this is easier said than done. Data scientists are not trained or equipped to be diligent to care about production concepts such as reproducibility — they <strong>are trained to iterate and experiment.</strong> They don’t really care about code quality and it is probably not in the best interest of the company at an early point to be super diligent in enforcing these standards, given the trade-off between speed and overhead.</p><p>Therefore, what is required is an implementation of a framework that is <strong>flexible but enforces production standards </strong>from the get-go. A very natural way of implementing this is via some form of pipeline framework that exposes an automated, standardized way to run ML experiments in a controlled environment. ML is inherently a process that can be broken down into individual, concrete steps (e.g. preprocessing, training, evaluating, etc), so a pipeline is a good solution here. Critically, by standardizing the development of these pipelines at the early stages, organizations can lose the cycle of destruction/recreation of ML models through multiple toolings and steps, and hasten the speed of research to deployment.</p><p>If an organization can incentivize their data scientists to buy into such a framework, <strong>then they have won half the battle of productionalization</strong>. However, the devil is really in the details — how do you give data scientists the flexibility they need for experimentation in a framework that is robust enough to be taken all the way to production?</p><h3>An exercise in finding the right abstractions</h3><p>Having motivated writing in pipelines from the get-go, it is only fair that I give more concrete examples of frameworks on how to achieve this. However, in my opinion, currently, the tooling landscape is too split into frameworks that are ML tools for ML people, or Ops tools for Ops people, not really satisfying all the boxes I mentioned in the last section. What is missing is an Ops (read pipelines) tool for ML people, with <strong>higher-order abstractions at the right level for a data scientist.</strong></p><p>In order to understand why this is important, we can cast an eye towards how web development has matured from raw PHP/JQuery-based scripts (the Jupyter notebooks of web development) with the LAMP stack to the powerful React/Angular/Vue-based modern web development stacks of today. Looking at these modern frameworks, their success has been dictated by providing higher-order abstractions that are easier to consume and digest for a larger audience. They did not change the fundamentals of how the underlying web technology worked. They simply re-purposed it in a way that is understandable and accessible to a larger audience. Specifically, by providing <em>components</em> as first-class citizens, these frameworks have ushered in a new mechanism of breaking down, utilizing, and resharing the HTML and Javascript that powers the modern web. However, ML(Ops) does not have an equivalent movement to figure out the right order of abstraction to have a similar effect.</p><p>To showcase a more concrete example of my more abstract thoughts above, I’ll use <a href="https://github.com/maiot-io/zenml">ZenML</a>, an open-source MLOps framework to create iterative, reproducible pipelines.</p><blockquote>Disclaimer: I am one of the core maintainers of ZenML.</blockquote><p><a href="https://github.com/maiot-io/zenml">ZenML</a> is an exercise in finding the right layer of abstraction for ML. Here, we treat <em>pipelines as first-class citizens. </em>This means that data scientists are exposed to pipelines directly in the framework, but not in the same manner as the data pipelines from the ETL space (<a href="https://www.prefect.io/">Prefect</a>, <a href="https://airflow.apache.org/">Airflow</a> et al.). Pipelines are treated as <em>experiments — </em>meaning they can be compared and analyzed directly. Only when it is time to flip over to productionalization, can they be converted to classical data pipelines.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*NGiwUYBznCQXQw1g" /><figcaption>Figure 2: ZenML abstract pipelines with familiar language to increase ownership of model deployments.</figcaption></figure><p>Within pipelines are steps, that are abstracted in <em>familiar ML language towards the data scientist. </em>E.g. There is a TokenizerStep , TrainerStep , EvaluatorStep and so on. Paradigms that are way more understandable than plugging scripts into some form of orchestrator wrapper.</p><p>Each pipeline run tracks the metadata, parameters and can be compared to other runs. The data for each pipeline is automatically versioned and tracked as it flows through. Each run is linked to git commits and compiled into an easy-to-read YAML file, which can be optionally compiled to other DSL’s such as on Airflow or Kubeflow Pipelines. This is necessary to satisfy other stakeholders such as the data engineers and ML engineers in the value chain.</p><p>Additionally, the interfaces exposed for individual steps are mostly set up in a way to be easy to extend in an idempotent, and therefore a distributed, manner. The data scientist can therefore scale-out with different processing backends (like Dataflow/Spark) when they are dealing with larger datasets.</p><p>All in all, ZenML is trying to get to the following scenario:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*WfjO5QzcLLyD2g4HifhkBg.png" /><figcaption>Figure 3: ZenML unifies the ML process.</figcaption></figure><p>Of course, ZenML is not the only mechanism to achieve the above — Many companies build their own home-grown abstraction frameworks to solve their specific needs. Often-times these are built on top of some of the other tools I have mentioned above. Regardless of how to get there, the goal should be clear: Get the data scientists <strong>as close to production as possible</strong> with as little friction as possible, incentivizing them to increase their ownership of the models after deployment.</p><p>This is a win-win-win for every persona involved, and ultimately a big win for any organization that aims to make it to the top 1% using ML as a core driver for their business growth.</p><h3>Plug</h3><p>If you like the thoughts here, we’d love to hear your feedback on ZenML. It is <a href="https://github.com/maiot-io/zenml">open-source</a> and we are looking for early adopters and <a href="https://github.com/maiot-io/zenml">contributors</a>! And if you find it is the right order of abstraction for you/your data scientists, then let us know as well via our <a href="http://zenml.io/slack-invite">Slack</a> — looking forward to hearing from you!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b2d95003f998" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/why-ml-should-be-written-as-pipelines-from-the-get-go-b2d95003f998">Why ML should be written as pipelines from the get-go</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[“Spot” the difference in ML costs]]></title>
            <link>https://medium.com/data-science/spot-the-difference-in-ml-costs-358202e60266?source=rss-7804b7e02c3------2</link>
            <guid isPermaLink="false">https://medium.com/p/358202e60266</guid>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[data-sceince]]></category>
            <dc:creator><![CDATA[Hamza Tahir]]></dc:creator>
            <pubDate>Thu, 28 Jan 2021 00:02:00 GMT</pubDate>
            <atom:updated>2021-01-29T14:34:50.968Z</atom:updated>
            <content:encoded><![CDATA[<h4>Optimize ML training costs on the cloud using spot instances</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*9wpNDN2Hr-rq_lGH" /><figcaption>Save more than 70% costs using spot instances on the cloud. [<em>Source: </em><a href="https://unsplash.com/photos/sD1VpeflHBY"><em>Unsplash</em></a><em>]</em></figcaption></figure><p>Every organization at any scale understands that leveraging the public cloud is a trade-off between convenience and cost. While cloud providers like Google, Amazon, and Microsoft have immensely reduced the barrier of entry for machine learning, GPU costs are still at a premium.</p><p>There is an <a href="https://venturebeat.com/2020/06/01/ai-machine-learning-openai-gpt-3-size-isnt-everything/">increasing fear in the machine learning community</a> that the true power of machine learning is still within the hands of the few. The flagship example of this is OpenAI’s massive GPT-3 model containing 175 billion parameters, a memory footprint of 350GB and reportedly costing at least <a href="https://lambdalabs.com/blog/demystifying-gpt-3/">$4.6 million</a> to train. The trend also looks set to continue: Rumours consistently circulate regarding the next generation GPT-4’s size, with some estimates ranging in the order of <a href="https://www.metaculus.com/questions/4852/how-many-parameters-will-gpt-4-have-if-it-is-released-in-billions-of-parameters/"><strong>trillions</strong> of parameters</a>. Even with more efficient training techniques, these models will still cost in the order of millions to train.</p><p>For the rest of the ML community, there is now an increasing reliance on their secret weapon: Transfer learning. Just recently, the excellent <a href="https://huggingface.co/">HuggingFace</a> library <a href="https://twitter.com/huggingface/status/1351560093658198022">announced a simple method</a> to fine-tune large-scale parameter models on a single cloud GPU. This gives hope to ML practitioners that even if they are unable to train models from scratch, utilizing the immense power of modern-day machine learning models is still within reach.</p><h3>The need for cloud</h3><p>Whether training from scratch or fine-tuning, it is still clear that public cloud providers offer the most convenient path to provision and utilize compute resources for most ML practitioners out there. However, even for fine-tuning tasks or smaller models, these costs can quickly grow and become unmanageable. For example, here is a simple breakdown of much it costs to train a machine learning model on a relatively mid-tier configuration suitable for many machine learning tasks:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b410c79301d38aeab48b3d8ec7267d6f/href">https://medium.com/media/b410c79301d38aeab48b3d8ec7267d6f/href</a></iframe><p>Bear in mind that the above costs are just for one training run. Most machine learning projects go through much more experimentation phases, and these numbers add up quickly. Therefore, most ML teams who do not have vast budgets on their hands usually resort to sampling their datasets and portioning out big training runs for when they are sure. This can be slow and tedious, not to mention hard to coordinate. It can also lead to teams converging on the wrong results. If the smaller, sampled datasets are not representative of the larger dataset, it can lead to frustrating and diverging results as the models develop.</p><h3>Spot instances: A perfect fit for ML experimentation</h3><p>Would it not be easy if ML practitioners had the luxury to launch experiments without having to fret so much about the costs exploding over time? There might just be a solution, offered by all major cloud providers, and severely underutilized by the machine learning community: <strong>Preemptible/Spot instances.</strong></p><p>The word preemptible instance is largely a Google Cloud Platform term, while spot instance is used by AWS/Azure. Whatever you call it, the concept is the same: These instances cost a fraction of the cost of normal instances, and the only catch is that there is no guarantee that the instance stays up all the time. Usually, this means that within 24 hours the instance is shut down by the provider.</p><p>These sorts of instances are a mechanism for the cloud providers to maximize the utilization of all their resources at any given time. They are intended for batched, non-critical workloads. Most training jobs for the majority of the use-cases out there take less than 24 hours to complete. And even if the job is interrupted before that happens, they can almost always be restarted from checkpoints.</p><h3>Cost comparison: 80% cost reduction</h3><p>Therefore, machine learning training fits the intended use of spot instances perfectly. By using these instances, practitioners stand to gain massive cost reductions. We have conducted a rough analysis across the three major cloud providers to showcase the cost benefits. The raw data can be found <a href="https://docs.google.com/spreadsheets/d/1wErQviA3sI22fh3BscO4CMJyg6w1Qqi468O1bCxUFhc/edit?usp=sharing">here</a>. Feel free to share the doc and leave a comment if you find something to add. Here is a snapshot, with the same configurations as before:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/dd10002f79fbd94e3242420cee07bbe9/href">https://medium.com/media/dd10002f79fbd94e3242420cee07bbe9/href</a></iframe><p>As can be seen, depending on the configuration, there is up to <strong>82%</strong> cost reduction by using spot instances, with the average cost savings across multiple cloud and configurations being roughly <strong>74%</strong>. This can equate to hundreds of dollars worth of savings. Especially for hobbyists, smaller companies, or smaller departments experimenting with Machine Learning, this may mean the difference between getting a model deployed vs crashing and burning before lift-off.</p><p>Using this technique is not new: Way back in 2018, the FastAI team <a href="https://www.fast.ai/2018/08/10/fastai-diu-imagenet/">trained ImageNet in 18 minutes with 16 AWS spot instances</a>. This cost $40 at the time and was the most public display of the insane cost benefits of spot instances in the community.</p><p>However, given the trend of increasingly big models, and the increasing adoption of AI worldwide, I can only see the need for spot instance training increasing over time. Given the dramatic difference in costs, it is almost a no-brainer to use spot instance training as a primary mechanism for training, at least in the experimentation phase.</p><h3>ZenML: A simple tool to orchestrate spot instance training</h3><p>If you’re looking for a head start for spot instance training, check out <a href="https://github.com/maiot-io/zenml">ZenML</a>, an open-source MLOps framework for reproducible machine learning. Running spot pipeline in ZenML, is as easy as :</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/aed2e75b98d0c18448ab47e72a1f58b1/href">https://medium.com/media/aed2e75b98d0c18448ab47e72a1f58b1/href</a></iframe><p>ZenML not only zips your code up to the instance, but it also makes sure the right CUDA-drivers are enabled to take advantage of the accelerator of your choice. It provisions the instance and spins it down when the pipeline is done. Not to mention the other benefits of experiment tracking, versioning, and metadata management which are provided anyway by the framework. Give it a spin yourself: A full code example can be found <a href="https://github.com/maiot-io/zenml/tree/main/examples">here</a>.</p><p>AWS and Azure support is on the horizon, and we’d love your feedback on the current setup. If you like what you see, leave us a <a href="https://github.com/maiot-io/zenml">star at the GitHub repo</a>!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=358202e60266" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/spot-the-difference-in-ml-costs-358202e60266">“Spot” the difference in ML costs</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Avoiding technical debt with ML pipelines]]></title>
            <link>https://medium.com/data-science/avoiding-technical-debt-with-ml-pipelines-3e5b6e0c1c93?source=rss-7804b7e02c3------2</link>
            <guid isPermaLink="false">https://medium.com/p/3e5b6e0c1c93</guid>
            <category><![CDATA[technical-debt]]></category>
            <category><![CDATA[deployment-pipelines]]></category>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Hamza Tahir]]></dc:creator>
            <pubDate>Fri, 22 Jan 2021 18:57:31 GMT</pubDate>
            <atom:updated>2022-09-06T09:59:19.746Z</atom:updated>
            <content:encoded><![CDATA[<h3>Avoiding Technical Debt with ML Pipelines</h3><h4>Strike a balance between rapid results and high-quality coding.</h4><figure><img alt="A wall which says “Until debt tear us apart”" src="https://cdn-images-1.medium.com/max/1000/0*IVn7GaX4rCEwPrlG" /><figcaption>Technical Debt can be hard to manage in machine learning. Source: <a href="https://unsplash.com/photos/Olki5QpHxts">Unsplash</a></figcaption></figure><p>Okay, let&#39;s make it clear at the start: This post is NOT intended for people who are doing one-off, siloed projects like participating in Kaggle competitions or doing hobby projects on Jupyter notebooks to learn the trade. The value of throw-away, quick, dirty script code is obvious there — and has its place. Rather, it is intended for ML practitioners working in a <em>production</em> setting. So if you’re working in an ML team that is struggling to manage technical debt while pumping out ML models, this one’s for you.</p><h3>A typical workflow</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*WCWM6M2eu7A7MtSU.jpg" /><figcaption><em>Image Source: </em><a href="https://www.flickr.com/photos/michael_mayer/8701850930"><em>Michael Mayer on Flickr</em></a><em> | Attribution Licence found </em><a href="https://creativecommons.org/licenses/by/2.0/"><em>here</em></a></figcaption></figure><p>Here is the premise: You’re a ML/DL/AI engineer/analyst/scientist working in a startup/SME/enterprise. It’s your job to take a bunch of random data from all over the place, and produce value. What do you do? You sit down, somehow get the data on to your local machine, and inevitably do something along the lines of:</p><pre>jupyter notebook</pre><p>Or you go to a colab if you’re fancy and your teams privacy rules allow it.</p><p>Following is a story I’ve seen so many times before — pseudo-code of course.</p><pre>import pandas <strong>as</strong> pd<br>import xyzlibraryforml</pre><pre><strong><em># CELL 1: Read</em></strong><br>df = pd.read_*(&quot;/path/to/file.*&quot;)<br>df.describe()</pre><pre><em># INSERT HERE: a 100 more cells deleted and updated to explore data.</em></pre><pre><strong><em># CELL 2: Split</em></strong><br>train, eval = split_the_data()  <em># basic</em></pre><pre><em># INSERT HERE: trying to figure out if the split worked</em></pre><pre><strong><em># CELL 3: Preprocess</em></strong><br><em># nice, oh lets normalize</em><br>preprocess(train)<br>preprocess(eval)</pre><pre><em># exploring preprocessed data, same drill as before</em></pre><pre><strong><em># CELL 4: Train</em></strong><br>model = some_obfuscated_library.fit(train, eval)  <em># not being petty </em></pre><pre><em># if youre lucky here, just look at accuracy. otherwise:</em></pre><pre><strong><em># CELL 5: Evaluate</em></strong><br>complicated_evaluation_methods(model)</pre><pre><em># INSERT HERE: do this a 1000 times</em></pre><pre><strong><em># CELL 6: Export (i.e. pickle it)</em></strong><br>export_model()</pre><p>So you’re done, right? That&#39;s it — boom. Test set results are great. Let&#39;s give it to the ops guy to deploy in production. Lunch break and Reddit for the rest of the day!</p><p>Okay, am I grossly exaggerating? <strong>Yes</strong>. Is this scarily close to the truth for some businesses? <strong>Also yes</strong>.</p><h3>So, what&#39;s the problem?</h3><p>The problem is that this notebook above is a ball of technical debt that will keep growing, if not culled early enough. Let&#39;s break down what was wrong with it:</p><h3>Not moving towards a bigger picture</h3><p>When you put generalizable logic into non-versioned, throw-away notebook blocks, it is taking away the ability of your team to take advantage of it. For example, the logic of loading/extracting your data from your static datasource. Sure, a pd.read_json is easy right now, but what happens if the format changes? Worse, what happens if the data grows and is split into multiple files? Even worse, what happens if it won&#39;t fit into memory anymore? What happens if your colleague runs into the same problem—she’ll probably go through the same loops as you did not even knowing it’s an already solved problem. Sure, there are solutions to all these problems, but are you going to keep solving them in your local notebook?</p><p>The answer is probably no (<a href="https://netflixtechblog.com/open-sourcing-polynote-an-ide-inspired-polyglot-notebook-7f929d3f447">unless you’re Netflix or something</a>). Usually, the logical thing to do is to extract the loading into a logically separate service. This way, you can abstract the actual extraction of data away from yourself and into a maintainable layer transparent to everyone. For example, this could be some form of feature store that collects all the various data streams in your organizations into one point, which can then be consumed in defined APIs by everyone in your team.</p><p>The same applies to the pre-processing, training, and evaluation part of the script above.</p><h3>Building logic locally (and therefore again and again)</h3><p>As in the above example, when you write code to explore data, you generate tons of great stuff — visualizations, statistical dataframes, cleaned and preprocessed data, etc. The random, arbitrary order of execution of Jupyter notebooks ensures that the path taken to realize these artifacts is forever lost in overridden local variables and spontaneous kernel restarts. Even worse, the logic has complex, easily overridden configurations embedded deep inside the code itself — making it much harder to recreate artifacts.</p><p>Look, I understand — I do it all the time. Data wrangling is a random, grinding process, and it&#39;s going to be messy. But setting up some framework to keep track of your data exploration pipelines is going to pay off big time. Similar to commenting on your code, the biggest beneficiary of keeping track is going to be yourself. Also, your team will be faster and avoid redundant work if these artifacts and mini-answers to questions were made transparent and clear to everyone automatically.</p><h3>Not owning the deployment</h3><p>The last part of this notebook is probably the most frustrating. I really do not believe that the job of the person writing that notebook ends with exporting the model for ops. It does not make any sense.</p><p>First of all, the preprocessing_fn() function has to go with it, otherwise training-serve skew is going to crash your model from the get-go. Secondly, how on earth is the ops person supposed to know what assumptions you took while building the model? Are you going to write extensive documentation of what data goes in, how it should be preprocessed, and which shape it should be in when deploying to an endpoint? I mean there are now automated ways of doing this — so own the deployment!</p><p>One aspect lacking in the above script is a disregard for measuring the performance of a model. Most data scientists I know do not care how big the model is, how much memory it consumes for predictions, and how fast/efficient it is in deployment. The model does not produce value if it does not fulfill the performance criteria of the end application. Again, the person developing the models should have ownership of its end deployment.</p><h3>Suggestions</h3><p>The easiest way to fix the above is to develop a framework where an ML team can balance throw-away exploratory code development with maintainable, easy-to-share code. If you are to do this, you might want to keep the following in mind:</p><h3>Create well-defined interfaces (i.e decompose into pipelines)</h3><p>The split, transform, train, evaluate, deploy components of your workflow are logically independent entities/services. Architect a setup where individual components of your ML workflow are abstracted away from concrete implementations. This could be by defining actual interfaces object-oriented style, or simply ensuring that your repo has some form of structure that is easy for everyone to contribute to and extend. This does not have to be rocket-science at the start, but it will help enormously.</p><p>This is where the notion of ML pipelines come into play: Pipelines are abstract representations that define a series of data processing tasks. Thinking in pipelines would help your team separate out logical entities in their workflow, and have data flow through it independently. This would inevitably yield a more robust, maintainable codebase. Also, defining ML pipelines like this ensures you can automate continuous training of your stale models on new data as it comes in. However, you need to also track your data metadata for that (See below).</p><h3>Make a plan for your ML Metadata</h3><p>Every experiment you run yields ML metadata: who ran it, when it was run, what data went in, where the results are stored etc. Make sure you map these out and provide a convenient way to add to this store. Important to note: I am not just talking about experiment tracking either. There are many wonderful libraries out there that will help in tracking the model-centric metadata, i.e., metrics, etc. However, what is often neglected is the data-centric metadata — especially if the data keeps changing. Stuff like data versioning, statistics, visualizations, what seed was used when random splitting — stuff like that. There should be an easy way to trace the various routes your data takes in your development.</p><h3>Ensure that your workloads can run in any environment</h3><p>Running a workload on a single machine is always going to be easier than making the same code run on any arbitrary environment. I know Docker is inapproachable and hard for many ML people, but at least make a requirements.txt and add a __init__.py! Ideally, containerize your code and run experiments on some form of orchestration framework. Doing this one step now will save a lot of pain when you scale and automate this whole thing to work on bigger data.</p><h3>Do not separate deployment from training</h3><p>This is perhaps the most no-brainiest of all no-brainer suggestions so far. End-to-end ownership led to the whole DevOps revolutions 20 years ago, and that has not gone away in ML development either. Provide a smooth mechanism to transfer a trained model to an endpoint, and make sure the Ops people are sitting in the same room (not always literally) as your ML developers. Put in place processes where everyone understands the end goal in production. Automate things when possible.</p><h3>Do not compromise on repeatability and traceability</h3><p>You know when people start coding in Python and then move to C++ or Java and they do not understand concepts like pointers and static typing? They think: “What is the use of giving a variable a type, <strong>I</strong> know what it is so why am I <strong>forced</strong> to do this?” Well, sorry to break it to you, but pointers and static typing have a purpose — knowing them protects your code from your own failings and ensures high-quality robust output. Ultimate flexibility can be a bad thing, especially with developers who tend to err towards laziness (like me).</p><p>Something very similar happens in Jupyter notebooks — the freedom of running any arbitrary code in any order gives freedom but makes you lose the very important notion of repeatability and traceability, two cornerstones of any robust, production-ready engineering discipline. Please, at least ensure that your notebooks are executable top-down and in a repeatable manner. Cluttered and randomly ordered Jupyter notebooks should be punished with long rants like this one on Code Review meetings.</p><p>One way of ensuring both traits is to extract the settings of your code from the implementation. Which brings me to my next point…</p><h3>Separate configuration from implementation</h3><p>Separating configuration from actual code implementation is definitely a pain. Yet, this is another one of those ‘pays off in the long run’ sort of things. <a href="https://blog.zenml.io/declarative_configs_for_mlops/">We’ve written about it before</a> but to summarize: Separating your configuration allows you to automate repetitive tasks, increases predictability of results, and ensures reproducibility. Ideally, the configuration should be treated as code, versioned, and maintained. A good way of managing this is to write code as pipelines — something I’ve <a href="https://blog.zenml.io/technical_debt/">written about extensively</a> and is motivated greatly in this <a href="https://neptune.ai/blog/reducing-pipeline-debt-great-expectations">Neptune.ai blog article</a>.</p><h3>Conclusion</h3><p>ML practitioners in many organizations are heavily incentivized to make quick wins to produce early results. However, this leads to accumulated technical debt which eventually slows down the team over time. The solution is to follow proper software engineering principles from the get-go and lean on guidelines to strike a balance between rapid results and high-quality software development.</p><p>The thoughts above are very personal lessons I have learned over the last 4 years deploying models in production. They are by no means an exhaustive list, and you’re welcome to use it as a boilerplate to benchmark your production architecture, or as a blueprint to design your own. If you’re looking for a practical tutorial on how to</p><p>We used these factors as the guiding principles for <a href="https://github.com/maiot-io/zenml">ZenML</a>, our open-source MLOps framework. So before you start from scratch, check out ZenML on GitHub: <a href="https://github.com/maiot-io/zenml">https://github.com/zenml-io/zenml</a>, and don’t forget to give us a star if you like what you see!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3e5b6e0c1c93" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/avoiding-technical-debt-with-ml-pipelines-3e5b6e0c1c93">Avoiding technical debt with ML pipelines</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Is your Machine Learning Reproducible?]]></title>
            <link>https://medium.com/data-for-ai/is-your-machine-learning-reproducible-644330c77fe6?source=rss-7804b7e02c3------2</link>
            <guid isPermaLink="false">https://medium.com/p/644330c77fe6</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Hamza Tahir]]></dc:creator>
            <pubDate>Wed, 20 Jan 2021 06:33:39 GMT</pubDate>
            <atom:updated>2021-01-27T15:25:41.278Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*LkFXpl-q1Bxb4qBW" /><figcaption>It can be frustrating to reproduce machine learning [Source: <a href="https://unsplash.com/photos/-2vD8lIhdnw">Unsplash</a>]</figcaption></figure><p>It is now widely agreed that <a href="https://blog.ml.cmu.edu/2020/08/31/5-reproducibility/">reproducibility is an important aspect of any scientific endeavor</a>. With Machine Learning being a scientific discipline, as well as an engineering one, reproducibility is equally important here.</p><p>There is widespread fear in the ML community that we are living through a <a href="https://www.wired.com/story/artificial-intelligence-confronts-reproducibility-crisis">reproducibility crisis</a>. Efforts like the <a href="https://paperswithcode.com/rc2020">Papers with Code Reproducibility Challenge</a>, signaled a clear call-to-action for practitioners, after a 2016 Nature survey revealed that <a href="https://www.nature.com/news/1-500-scientists-lift-the-lid-on-reproducibility-1.19970">70% of results are non-reproducible</a>.</p><p>While a lot of the talk amongst the community has centered on <a href="https://www.sciencedirect.com/science/article/pii/S2666389920300933">reproducing machine learning results in research</a>, there has been less focus on the production side of things. Therefore, today let’s focus more on the topic of reproducible ML in production and create a larger conversation around it.</p><h3>Why is reproducibility important?</h3><blockquote>“If you can’t repeat it, you can’t trust it” — All Ops Teams</blockquote><p>A good question to start with is why exactly reproducibility is important, for machine learning in particular. Here are list of benefits one gains by ensuring reproducibility:</p><ul><li>Increases <strong>trust</strong></li><li>Promotes <strong>explainability</strong> of ML results</li><li>Increases <strong>reliability</strong></li><li>Fulfils <strong>ethical</strong>, <strong>legal</strong>, and <strong>regulatory</strong> requirements</li></ul><p>Concretely, ML models tend to go through a lifecycle of being destroyed, forged anew and re-created as <a href="https://blog.maiot.io/technical_debt/">development evolves from rudimentary notebook snippets to a testable, productionized codebase</a>. Therefore, we better make sure that every time a model is (re-) trained, the results are what we expect them to be.</p><h3>What’s the big deal?</h3><p>One would think that reproducibility in production ML should be easy. After all, most machine learning is scripting. How hard can it be to simply execute a bunch of scripts again at a later stage and come to the exact same result, right?</p><p><strong>wrong</strong></p><p>Reproducibility of machine learning is hard because it spans many different disciplines, from understanding non-deterministic algorithmic behaviours to software engineering best practices. Leaving aside the fact that most machine learning code quality tends to err towards the low side (due to the experimental nature of the work), there is an inherent complexity to ML which makes things even harder.</p><p>E.g. Just training a model on the same data with the same configuration does not mean the same model is produced. Perhaps one could achieve a similar <em>overall</em> accuracy (or whatever other metric), but even a slight change in parameters might skew metrics for slices of your data — leading to <a href="https://www.theverge.com/2018/1/12/16882408/google-racist-gorillas-photo-recognition-algorithm-ai">sometimes very unpleasant results</a></p><p>So, how can we ensure that stuff like does not happen? In my opinion, one can break down reproducibility in the following aspects:</p><ul><li>The code</li><li>The configuration</li><li>The environment</li><li>The data</li></ul><p>Let’s look at each of these in turn.</p><h3>Code</h3><p>Checking code into a version control system like Git ensures a clean trace of how code evolves, and the ability to rollback to any point in history. However, Git alone is not a fix for reproducibility, but only for one aspect of it.</p><p>In reality, reproducibility in production is solved by version control, testing of code as well as integrations, and idempotent deployment automation. This is hard to apply in practice. E.g. The main tool for ML is Jupyter notebooks, which are notoriously difficult to check into version control. Even worse, most notebook code is not sequential in its execution, and can have an arbitrary, impossible to reproduce, sequence of execution.</p><p>But even if ML practitioners follow a pattern of refactoring their code into separate modules, simply checking modules into source control is still not enough to ensure reproducibility. One needs to link the commit history to model training runs and models. This can be achieved e.g. by enforcing a standard in your team that pins a git sha to experiment runs. That way there is a global unique ID that ties the code and configuration (see below) to the results it produced.</p><h3>Configuration</h3><p>Software Engineering preaches separation of application code and application configuration to allow for predictable and deterministic software behavior across environments. This actually translates well in a machine learning code: E.g. one can separate the model definition and training loop code, from the associated hyper-parameters which define the configuration.</p><p>The first step to unlock reproducibility is to actually separate configuration from code in the first place. For me this means, the code itself should NOT define:</p><ul><li>Features</li><li>Labels</li><li>Split parameters (e.g. 80–20 split)</li><li>Preprocessing parameters (e.g. the fact that data was normalized)</li><li>Training hyper-parameters (including pre-processing parameters)</li><li>Evaluation criteria, .e.g, metrics</li></ul><p>Ideally all these are tracked separately in a <a href="https://blog.maiot.io/declarative_configs_for_mlops/">declarative config</a> that is human readable.</p><h3>Environment</h3><p>If a ML result is produced on a dev local machine, there is a high chance it is not going to be reproducible. Why? Because developers, especially relatively inexperienced ones, are not super diligent in creating and maintaining proper virtual environments.</p><p>The obvious solution for this one is containerizing applications, with let’s say, <a href="https://docker.com/">Docker</a>. However, here is another example of when skills of ML practitioners begin diverging from conventional software engineers. Most data scientists are not trained in these matters, and require proper organizational support to help and encourage them to produce containerized applications.</p><h3>Data</h3><p>And finally, we arrive at the data. Data versioning has become one of the most discussed topics in the production machine learning community. Unlike code, you can’t simply check data into version control easily (although tools like <a href="https://dvc.org/">DVC</a> are attempting just that).</p><p>In the same way as code, achieving basic versioning of data does not necessarily ensure reproducibility. There is a whole bunch of metadata associated with how data is utilized in machine learning development, all of which is necessary to persist to ensure trainings are reproducible.</p><p>Here is a simple, but common, example that illustrates this point. If you have ever worked with machine learning, have you ever created a folder/storage bucket somewhere that has random files in varying preprocessing states? Something like, normalized_1.json or perhaps even timestamped 12_02_19.csv? Technically, a timestamped file is versioned data, but that does not mean associated runs with it are reproducible: One would have to know how, when and where (i.e. the aforementioned metadata) these versioned files are used to ensure reproducibility.</p><h3>Concrete Example</h3><p>While it may fall outside the scope of this blog, the open-source MLOps framework ZenML showcases a clear example of putting these principles in action <a href="https://docs.zenml.io/tutorials/ensuring-ml-reproducibility.html">here</a></p><h3>Conclusion</h3><p>Reproducibility in machine learning is not trivial, and ensuring it in production is even harder. ML teams need to be fully aware of the precise aspects to track in their processes to unlock reproducibility.</p><p>If you’re looking for a head start to enable reproducibility: check out <a href="https://github.com/maiot-io/zenml">ZenML</a>, an open-source MLOps framework for reproducible machine learning — and leave a star while you’re there!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=644330c77fe6" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-for-ai/is-your-machine-learning-reproducible-644330c77fe6">Is your Machine Learning Reproducible?</a> was originally published in <a href="https://medium.com/data-for-ai">Feature Stores for ML</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why ML in production is (still) broken — [#MLOps2020]]]></title>
            <link>https://htahir1.medium.com/why-ml-in-production-is-still-broken-mlops2020-bdb57f91a876?source=rss-7804b7e02c3------2</link>
            <guid isPermaLink="false">https://medium.com/p/bdb57f91a876</guid>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Hamza Tahir]]></dc:creator>
            <pubDate>Fri, 26 Jun 2020 10:20:00 GMT</pubDate>
            <atom:updated>2021-12-07T12:06:06.722Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*G1AeMR4tA9-XCxIOgni7RQ.png" /></figure><h3>Why ML in production is (still) broken — [#MLOps2020]</h3><p>Just a few days ago, I was able to share my thoughts on the state of Machine Learning in production, and why it’s (still) broken, on the MLOps World 2020. Read on for a writeup of my presentation, or checkout the recording of the talk on Youtube.</p><h3>From research to production</h3><p>By now, chances are you’ve read the famous paper about <a href="https://papers.nips.cc/paper/5656-hidden-technical-debt-in-machine-learning-systems.pdf">hidden technical debt by Sculley et al. from 2015</a>. As a field, we have accepted that the actual share of Machine Learning is only a fraction of the work going into successful ML projects. The resulting complexity, especially in the transition to “live” environments, lead to large amounts of failed ML projects never reaching production.</p><p>Figures on the scope of the problem vary. <a href="https://www.forbes.com/sites/enriquedans/2019/07/21/stop-experimenting-with-machine-learning-and-start-actually-usingit/#:~:text=Stop%20Experimenting%20With%20Machine%20Learning%20And%20Start%20Actually%20Using%20It,-Enrique%20DansSenior&amp;text=It%20turns%20out%20there&#39;s%20a,never%20make%20it%20into%20production.">Forbes</a> and <a href="https://venturebeat.com/2019/07/19/why-do-87-of-data-science-projects-never-make-it-into-production/">Venturebeat</a> talk about 87%, <a href="https://www.gartner.com/en/newsroom/press-releases/2018-02-13-gartner-says-nearly-half-of-cios-are-planning-to-deploy-artificial-intelligence">Gartner</a> claims 85%. Less important than a precise figure is the fundamental fact behind them: Getting ML projects from research to production is hard.</p><p>A crucial factor for ML models crashing and burning in production is that ML systems are inherently more complex than more traditional applications. For the latter, code is the only thing that affects the behavior of a system. For the former, code and data combined affects how the system behaves. Unfortunately, data is notoriously hard to wrangle with, and can change in unexpected ways — leading to ML teams naturally accumulating dangerous technical debt as they solve short-term problems.</p><p>Managing technical debt is a well-understood paradigm of “traditional” software engineering. We manage dependencies and code through explicit versioning to ensure their impact on a system stays within a defined scope. Following established practises like writing tests and abstractions, continuous refactoring and documentation will always have an immediate positive impact on ML projects.</p><p>This is, in large, due to technical debt. However, it won’t solve the source of a large portion of technical debt of ML: Data directly affects the behavior of the system. When data changes, your models AND your architecture need to adapt.</p><h3>Accumulation of technical debt</h3><p>On the path from first explorations to production systems projects accumulate technical debt. Let’s take a look at an exemplary progression. Each step of your maturing project incentivizes you to solve a different problem.</p><ol><li>You start out explorative in jupyter notebooks, and eventually end up with a first well-performing model.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iqXIHkfGcsiyIxkxvtG5_Q.png" /></figure><p>2. That model needs to be deployed, right? To preserve speed and keep complexity low you wrap it in a Flask-based API container, pass on the API endpoint, done.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vhwwii0bVekyLTUzPEfJbw.png" /></figure><p>3. But wait, actually your data will change over time. You’re not just doing a one-of batch inference, so your deployment needs to be connected to the newly incoming data. New data might change in its structure, so you’re also adding monitoring for input and output distributions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4IzgEBKK1aJtMloPhEzhsQ.png" /></figure><p>4. The team, on top of that, will also start to refactor its codebase, because your needs have by far outgrown what you can do in Jupyter Notebooks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wgqv7jzrPr1Mkb-7zYaHNg.png" /></figure><p>5. By now you’re also facing the challenge to orchestrate all that preprocessing and training across resources. That beefy VM you were running your code on simply doesn’t cut it anymore.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*en-9nfVRKf2Zdxt6u5zpWw.png" /></figure><p>As you progress you’re further dividing your codebase in more granular functions — splitting and preprocessing is separated, eval becomes semi-automated, your ETLs become standardized, and metadata plays a decisive role for your architecture.</p><p>The resulting architecture is grown into a state of complexity that can’t be deployed by new team members. Changes impact your entire codebase. Your deploys get less frequent by the day. Eventually, you’re facing either costly rewrites or disproportionate efforts of maintenance.</p><p>Worst of it all? You did everything right at each step of the way. Ad-hoc necessities and spontaneous fires dictated the incentives for your engineering decisions. Unfortunately, these incentives are misaligned with the actual incentive of building a production-ready system.</p><h3>There is hope</h3><p>The picture is not as bleak as it was a few years ago. There are solutions popping up everywhere. The FAANGs (plus Uber) of the world are sharing their experiences openly.</p><p>The industry is quickly converging on agreed upon components that are integral to a production ML system. A state-of-the-art high level architectural diagram for a production ML system can be pictured now clearly below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kC-bG099gQpYIRqxvVzzSA.png" /></figure><p>To avoid train-serve drift, ML companies increasingly centralize their data in feature stores. Next to classical database systems (MySQL, Postgres, DynamoDB) and key-value stores ( <a href="https://github.com/hivedb">HiveDB</a>) there are also a number of proprietary systems are entering the market with powerful feature level automations. The people behind <a href="https://eng.uber.com/michelangelo-machine-learning-platform/">Uber’s Michaelangelo</a> now raised $20 million for <a href="https://techcrunch.com/2020/04/28/tecton-ai-emerges-from-stealth-with-20m-series-a-to-build-operational-machine-learning-platform/">Tecton</a>, and <a href="https://www.logicalclocks.com/feature-store-lp">Logical Clocks</a> is doubling down on their feature store.</p><p>In the middle layers there are more tools than ever before. Some range from solving individual steps like distributed data manipulation ( <a href="https://beam.apache.org/">Apache Beam</a>, <a href="https://spark.apache.org/mllib/">Spark</a>), training (see <a href="https://spacy.io/">spaCy</a>, <a href="https://github.com/ray-project/ray">Ray</a>) and monitoring (see <a href="https://prometheus.io/">Prometheus</a>).</p><p>At the orchestration layer, the industry is now looking to solutions such as <a href="https://airflow.apache.org/">Airflow</a> and <a href="https://www.kubeflow.org/">Kubeflow</a>, which gain steam every year. Individual pain points like complex model deployments now have solutions such as <a href="https://www.seldon.io/">Seldon</a> and <a href="https://www.cortex.dev/">Cortex</a>. Finally, there are many great metadata tracking tools from easy-to-use ones like <a href="https://www.wandb.com/">Weights&amp;Biases</a> or <a href="https://www.comet.ml/site/">Comet ML</a> to more complex ones like the <a href="https://www.tensorflow.org/tfx/guide/mlmd">TensorFlow ML Metadata</a> project.</p><h3>ZenML</h3><p>Despite the ever growing MLOps landscape, there is still plenty of room for improvement. In particular, it is still challenging to comprehend, interconnect and maintain relevant tooling in a reliable production environment.</p><p>This is where <a href="https://zenml.io/">ZenML</a> comes in. Built as an end-to-end MLOps platform, it provides developers easy interfaces to production-ready data manipulation, training, serving, experiment tracking and orchestration layers. In addition, we put a focus on standardization, automation and comparability of ML pipelines.</p><p>To read more about ZenML head over to our <a href="https://zenml.io/">website</a>for more details. If you want to start using ZenML for your own ML production environment, <a href="/service/mailto:support@zenml.io">contact us</a>!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bdb57f91a876" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Can you do the splits?]]></title>
            <link>https://htahir1.medium.com/can-you-do-the-splits-1265009e1c2d?source=rss-7804b7e02c3------2</link>
            <guid isPermaLink="false">https://medium.com/p/1265009e1c2d</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[reproducibility]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Hamza Tahir]]></dc:creator>
            <pubDate>Thu, 11 Jun 2020 10:20:00 GMT</pubDate>
            <atom:updated>2021-12-14T12:11:05.614Z</atom:updated>
            <content:encoded><![CDATA[<p>One attempt to ensure that ML models generalize in unknown settings is splitting data. This can be done in many ways, from 3-way (train, test, eval) splits to k-splits with cross-validation. The underlying reasoning is that by training a ML model on a subset of the data, and evaluating on unknown data, one can reason much better if the model has underfit or overfit in training.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/910/0*263Aj1i8iK-a5-86.jpg" /><figcaption><em>Courtesy: </em><a href="https://www.pexels.com/photo/man-doing-yoga-1701207/"><em>Yogendra Singh on Pexels</em></a></figcaption></figure><p>For me, splitting data is the most under-rated task in all of data science. It is understandable that for most jobs, a simple 3-way split suffices. However, I have stumbled across many problems where there is a need for more complicated splits to ensure generalization. These splits are more complex because they are derived from the actual data, rather than the structure of the data which the hitherto mentioned split methods are based on. This post attempts to break down some of the more unconventional ways to split data in ML development, and the reasoning behind them.</p><p>In order to illustrate the split mechanisms, it helps to start with a sample dataset to do the splits on. To make things easy, lets use a simple multi-variate, timeseries dataset represented in tabular format. This data consists of<br> 3 numerical features, 1 categorical feature and 1 timestamp feature. Below this is visualized:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*VO4_OzX19wDZ3Osh.png" /></figure><p>This type of dataset is common across many use-cases and industries in machine learning. A concrete example can be multiple timestreams transmitted from different machines with multiple sensors on a factory floor. The categorical variable would then be the ID of the machine, the numerical features would be what the sensors are recording (e.g. pressure, temperature etc.), and the timestamp would be when the data was transmitted and recorded in the database.</p><p>Imagine you receive this dataset as a csv file from your data engineering department and are tasked with writing a classification or a regression model. The label in such a case could be any of the features or an additional column. Regardless, the first thing to do would be to try to split up the data into sets that are meaningful.</p><p>To make things easy, you decide to go make a simple split with train and eval. You know immediately that a naive random split with shuffling won&#39;t fly here - the data does have multple sensor streams that are indexed by time after all. So how do you split the data so that order is maintained and subsequent models are sufficiently generalizable?</p><h3>Another view of the data</h3><p>The most straightforward transformation we can do is to represent the data per categorical class (in our running example, visualize the data per machine). This would yield the following result:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*6oub2n3Cqnctt7U8.png" /></figure><h3>The Horizontal Split</h3><p>The grouping together suddenly makes the issue of splitting a bit simpler, and largely dependant on your hypothesis. If the machines are running under similar conditions, one question you might ask is: How would a ML model trained on one group generalize to other groups. That is, if trained on class_1, class_2 and class_3 timestreams how would the model fair on class_4 and class_5 timestreams. Here is a visualization of that split:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*1dPzzcCpENPQigHG.png" /></figure><p>I call this the Horizontal split due to nature of the cut line in the above visualization. This split can be easily achieved in most ML libraries by simply grouping by the categorical feature and partitioning along it. A successful training with this split would show evidence that the model has picked up signals that generalize across previously unseen groups. However, it would not showcase that it is able to predict future behavior of one group.</p><p>Its important to note that the the split decision did NOT account for time as a basis of the split itself. One can assume however that you would also sort by time per timestream to maintain that relationship in your data. Which brings us to the next split..</p><h3>The Vertical Split</h3><p>But what if you want to split across time itself? For most time-series modelling, a common way to split the data is past and future. That is, to take in the training set historical data relative to the data in the eval set. The hypothesis in this case would be: How would a ML model trained on historical data per group generalize to future data for each group?. This question might be answered by the so called Vertical split:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*9yfmGsc-K0oWDzX8.png" /></figure><p>A successful training with this split would showcase that the model is able to pick up patterns across timestreams it has already seen, and make accurate predictions of behavior in the future. However, this itself would not show that this model will generalize well to other timestreams from different groups.</p><p>Of course, your multiple timestreams have to be sorted now individually, so we still need to group. However, this time, rather than cutting across groups, we take a sample of the past of each group and put it in train and the future of each group in eval. In this idealized example, all the timestreams are of the same length, i.e., each timestream has exactly the same number of data points. However, in the real world, this maybe not be the case - so you would require a system to build an index across each group to make this split.</p><h3>The Hybrid Split</h3><p>An inquisitive ML researcher might at this point wonder if they could produce a model that would generalize under both constraints of the Horizontal and the Vertical split. The hypothesis in that case would be: How would a model trained on historical data for SOME groups generalize to future data of these groups AND all data from other groups?. A visualization of this Hybrid split would look like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*Gulphn0H44AeqLHP.png" /></figure><p>Naturally, if model training is successful, this model would surely be more robust than the others in a real world setting. It would have displayed evidence to not only learning patterns of some of the groups it has already seen, but also evidence of the fact that it has picked up signals that generalize across groups. This might be useful if we are to add more similar machines to the factory in the future.</p><h3>Multi-dimensional splits</h3><p>The notion of the horizontal and vertical splits can be generalized to many dimensions. For example, one might want to do group by two categorical features rather than one to even further isolate sub-groups in the data, and sort them per sub-group. There might also be complex logic in the middle to filter groups that have a lower number of samples, and other business-level logic pertaining to the domain.</p><p>The hypothetical example is used to illustrate the endless possibilities of various kinds of machine learning splits that can be created by an astute data scientist. Just like it is important to <a href="https://developers.google.com/machine-learning/fairness-overview">ensure ML fairness</a> whilst evaluating your models, it is equally important to spend sufficient time thinking about splitting a dataset and its consequences to bias the model downstream.</p><p>One easy way to do the Horizontal, Vertical and the Hybrid split by writing just a <a href="https://docs.zenml.io/docs/developer_guide/pipelines_config_yaml#main-key-split">few lines of YAML is via ZenML</a>. ZenML is a <a href="https://zenml.io/">MLOps framework</a> developed while we deployed models to production, for datasets with similar characteristics as the example above. If you are interested in the content above, and would like to try ZenML, please feel free to reach out to me at <a href="/service/mailto:hamza@zenml.io">hamza@zenml.io</a>. Head over to <a href="https://docs.zenml.io/">our docs</a> to understand more how it works in more detail.</p><p>Thank you and happy splitting!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1265009e1c2d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why deep learning development in production is (still) broken]]></title>
            <link>https://htahir1.medium.com/why-deep-learning-development-in-production-is-still-broken-b6f0602b040b?source=rss-7804b7e02c3------2</link>
            <guid isPermaLink="false">https://medium.com/p/b6f0602b040b</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Hamza Tahir]]></dc:creator>
            <pubDate>Sun, 01 Mar 2020 10:20:00 GMT</pubDate>
            <atom:updated>2021-12-28T12:18:21.939Z</atom:updated>
            <content:encoded><![CDATA[<p>Around 87% of machine learning projects do not survive to make it to production. There is a disconnect between machine learning being done in Jupyter notebooks on local machines and actually being served to end-users to provide some actual value.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*IHmR4haGuayKkLc6.png" /></figure><p>Source: Hidden Technical Debt in Machine Learning Systems (Sculley et al.)</p><p>The oft-quoted Hidden Technical Debt paper, whose diagram can be seen here, has been in circulation since 2017, yet still, deep learning in production has a ways to go to catch up to the quality standards attained by more conventional software development. Here is one take on what is broken:</p><h3>Data is not treated as a first-class citizen</h3><p>In traditional software development, code is (rightly-so) **a first-class citizen. In ML development, there is a further need for data to be a first-class citizen as well. Therefore, data has to be treated with the same care that most developers give to the code they write.</p><p>Right now in most organizations, data is spread everywhere and inaccessible. This is not just about raw data either-even if an organization spends a lot of money into centralizing their data into lakes, critical data is spread across the organization in colabs, notebooks, scripts and pre-processed flat files. This causes, amongst other things:</p><ul><li>Wasted compute on redundant transformations of data</li><li>No transparency and accountability of what data trains what models</li><li>Inability to transfer important training phase to the serving phase (see below)</li></ul><h3>Different requirements in training and serving</h3><p>Teams often find it surprising when a well-trained model starts to give spurious results in the real world. The transition from training a model to serving it is far from trivial.</p><p>For examples, there is a skew in training and production data, that needs to be taken into account. Secondly, one has to be very careful in making sure that production data goes through the same preprocessing steps in production as in training. Lastly, while training involves running experiments and quickly iterating, serving has even further requirements on the application level, e.g. inference time and costs at scale. All these need to be taken into account to avoid unnecessary surprises when the transition from training to serving happens.</p><h3>No gold standard yet for ML Ops</h3><p>Applying DevOps principles for ML development (or MLOps) is all the rage right now. However, there is yet no gold standard for it. The field in its infancy needs to tackle:</p><ul><li>Resources (compute, GPU etc) are scattered and not being used efficiently across teams</li><li>No proper CI/CD pipelines</li><li>No proper monitoring in production (change in data quality etc.)</li><li>Scaling is hard — in training and in serving</li><li>Machine learning compute works in spikes, so systems need to be equipped to deal with that</li></ul><h3>Collaboration is hard</h3><p>In conventional software development, we use workflows that integrate tickets and version control to make collaboration as seamless and transparent as possible. Unfortunately, ML development still lags behind on this front. This is largely due to the fact that ML developers tend to create silos which include glue-code scripts, preprocessed data pickles, and jupyter notebooks. While all these are useful for research and experimentation, they do not translate well into a robust, long-running, production environment.</p><p>In short, in the ML world, there is largely:</p><ul><li>Non-transparency coupled with individual experimentation</li><li>Notebook Hell with glue-code scripts</li><li>No versioning, in data, code or configuration</li></ul><h3>Conclusion</h3><p>Most of the problems highlighted above can be solved by proper attention being paid to machine learning development in production, from the first training onwards. The field is catching up, slowly but surely, but it is inevitable that machine learning will catch up with traditional software engineering quickly. Will we see new, even improving, and exciting ML products in our lives at that point? Lets hope so!</p><p>Our attempt to solve these problems is ZenML, an extensible, open-source MLOps framework. We recently launched and are now looking for practitioners to solve their problems in production use-cases! So, head over to <a href="https://github.com/zenml-io/zenml">GitHub</a>, and don’t forget to leave us a star if you like what you see!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b6f0602b040b" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>