<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by yaron haviv on Medium]]></title>
        <description><![CDATA[Stories by yaron haviv on Medium]]></description>
        <link>https://medium.com/@yaronhaviv?source=rss-bdd9246d060d------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*_fwAL0KixP690ySt.jpg</url>
            <title>Stories by yaron haviv on Medium</title>
            <link>https://medium.com/@yaronhaviv?source=rss-bdd9246d060d------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 09:11:10 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@yaronhaviv/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[MLOps Beyond Training: Simplifying & Automating the Operational Pipeline]]></title>
            <link>https://medium.com/data-science/mlops-beyond-training-simplifying-automating-the-operational-pipeline-e028a21a269e?source=rss-bdd9246d060d------2</link>
            <guid isPermaLink="false">https://medium.com/p/e028a21a269e</guid>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[yaron haviv]]></dc:creator>
            <pubDate>Wed, 15 Dec 2021 17:55:08 GMT</pubDate>
            <atom:updated>2021-12-15T17:55:08.701Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/940/1*lsmhyXY2SFOZBXNP2iqZWQ.png" /><figcaption>source Shutterstock (under license)</figcaption></figure><h3>MLOps Beyond Training: Simplifying and Automating the Operational Pipeline</h3><h3>The Evolving Meaning of ‘MLOps’</h3><p>When you say ‘MLOps’, what do you mean? As the technology ecosystem around ML evolves, ‘MLOps’ now seems to have (at least) two very different meanings:</p><ol><li>One common usage of ‘MLOps’ refers to <strong>the cycle of training an AI model</strong>: preparing the data, evaluating, and training the model. This iterative or interactive model often includes AutoML capabilities, and what happens outside the scope of the trained model is not included in this definition.</li><li>My preferred definition of MLOps refers to <strong>the entire data science process</strong>, from ingestion of the data to the actual live application that runs in a business environment and makes an impact at the business level. In this article, I’ll explain the difference between these two approaches and why it matters to your data science team, and to the success of your organization as a whole.</li></ol><h3>The Research-First Approach to MLOps</h3><p>The typical journey of an organization with a data science use case and a small team is to start from what they perceive to be the logical beginning: building AI models. A business idea based on data science is selected, and budget is allocated for the data scientists to start the work of building and training machine or deep learning models. They get access to data extractions, search for patterns, and build models that work in the lab.</p><p>For veterans of this space, it’s remarkable to observe how the industry has changed. Years ago, ML models were used to generate static or interactive reports for business analysts, and data science was handled as a silo, doing batch predictions on historical data and returning the results for someone else to incorporate manually into applications. In those conditions, there was very little demand for resiliency, scale, real-time access, or continuous integration and deployment (CI/CD), but we also gained limited value from our models.</p><p>As a vestige of this old mindset, most data science solutions and platforms today still start with a research workflow and fail to deliver when it comes time to turn the generated models into real-world AI applications. Even the concept of a CI/CD pipeline is often used to refer to just the training loop, and not extended to include the entire operational pipeline. This approach forces the ML team to re-engineer the entire flow to fit production environments and methodologies. This way of duplicative working consumes far too much resources and time, and often results in inaccuracies.</p><h3>Research-First Approach: How it Works and Why it’s Incomplete</h3><p>The common way to tackle AI projects is to start by developing a model. The data scientist receive data which is extracted manually from a variety of sources, he or she explores and prepares data in an interactive way (the most common tool for this stage is Notebooks), the training and experiments are run while tracking all the results, a model is generated and tested/validated until satisfying results are obtained. With the resulting model, different teams of ML or data engineers will figure out how to build it into the business application, how to handle API integrations, build data pipelines, apply monitoring, and so on. In many cases, the original data science will be set aside and re-implemented in a robust and scalable way which fits production, but which may not be what the data scientist originally intended.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*z7YJOPmgjkvbLtY68wV_qw.png" /><figcaption>Figure 1: Interactive model development flow (picture by author)</figcaption></figure><p>Many tools offer ways to version data and track experiments or models during the research and development phase, and some also allow automation for the model development pipeline and can generate an endpoint which serves the model. Yet they stop after the model development flow, and they don’t contribute much to the production pipeline which starts from automated data collection and preparation, automated training and evaluation pipelines, real-time application pipelines, data quality and model monitoring, feedback loops, etc.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FbuGFxXXkm9CebqjNmUQ0Q.png" /><figcaption>Figure 2: Full production deployment (picture by author)</figcaption></figure><p>As you can see from the two diagrams above, there isn’t much overlap between the model development phase and the production environment.</p><p>The production deployment:</p><ul><li>Involves many more components which need to work and be versioned together</li><li>Significant effort is placed on data processing, application integration, and monitoring tasks</li><li>Interactive development is replaced with managed micro-services which can handle scale and automated (re)deployment</li><li>Focuses on resiliency, observability, security, and continuous operations.</li></ul><p>The common approach of developing models in a data science silo leads to waste of valuable resources and far longer time to production.</p><h3>Training is done! Now what?</h3><p>Modern applications in which AI models provide real-time recommendations, prevent fraud, predict failures and guide self-driving cars bring far more value, but also require significant engineering efforts and a new approach to make it all feasible. Business needs have forced data science components to be robust, performant, highly scaleable, and aligned with agile software and DevOps practices. For every hour spent on developing a model there are a dozen more spent on engineering and deployment.</p><p>Often, the broader team only realizes the challenges of operationalizing AI that lie before them once they’ve built the model in the lab. At this late stage, building the actual AI application, deploying it, and maintaining it in production become acutely painful, and sometimes nonviable. Then comes the realization that to create a process that is repeatable and reproducible, so that more AI applications can be built and deployed on an ongoing basis, is a whole other ballgame.</p><h3>Adopting a Production-First Mindset</h3><p>It happens all too often that operationalizing machine learning — in the sense of considering all the requirements of the business, such as federated data sources, the need for scale, the critical implications of real-time data ingestion or transformation / online feature engineering, handling upgrades, monitoring etc. — comes as an afterthought, making it all the more difficult to create real business value with AI.</p><p>That’s why I advocate for a mindset shift. <strong>Begin with the end in mind</strong>: that is, take a production-first approach to designing a continuous operational pipeline, and then make sure the various components and practices map into it. Automate as many components as possible, and make the process repeatable so that you can scale along with the organization’s needs.</p><h3>The Production-First Approach to MLOps &amp; Its Benefits</h3><p>Instead of this siloed, complex and manual process, start by designing the production pipeline using a modular strategy, where the different parts provide a continuous, automated, and far simpler way to move from research and development to scalable production pipelines, without the need to refactor code, add glue logic, and spend significant efforts on data and ML engineering.</p><p>There are four key components that should be considered for every production pipeline:</p><ol><li><strong>Feature Store</strong>: collects, prepares, catalogs, and serves data features for development (offline) and real-time (online) usage</li><li><strong>ML CI/CD pipeline</strong>: automatically trains, tests, optimizes, and deploys or updates models using a snapshot of the production data (generated by the feature store) and code from the source control (Git)</li><li><strong>Real-time/event-driven application pipeline</strong>: includes the API handling, data preparation/enrichment, model serving, ensembles, driving and measuring actions, etc.</li><li><strong>Real-time data and model monitoring</strong>: monitors data, models, and production components and provides a feedback loop for exploring production data, identifying drift, alerting on anomalies or data quality issues, triggering re-training jobs, measuring business impact, etc.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/978/1*iY2_qd6GElC5OoYIvJLL1g.png" /><figcaption>Figure 3: Integrated Approach to Production ML Pipelines (picture by author)</figcaption></figure><p>While each of those steps may be independent, they still require tight integration.</p><p>For example:</p><ul><li>The training jobs need to obtain features from the feature store and update the feature store with metadata, which will be used in the serving or monitoring.</li><li>The real-time pipeline needs to enrich incoming events with features stored in the feature store, and may use feature metadata (policies, statistics, schema, etc.) to impute missing data or validate data quality.</li><li>The monitoring layer must collect real-time inputs and outputs from the real-time pipeline and compare it with features data/metadata from the feature store or model metadata generated by the training layer, and it needs to write all the fresh production data back to the feature store so it can be used for various tasks such as data analysis, model re-training (on fresh data), model improvements.</li></ul><h3>MLOps Automation &amp; Orchestration</h3><p>When we update one of the components detailed above, it immediately impacts the feature generation, the model serving pipeline, and the monitoring, so we need to apply versioning to each component as well as versioning and rolling upgrades across components.</p><p>Because these four components are so tightly connected, they cannot be managed in silos. This is where MLOps orchestration comes in. ML teams need a way to collaborate using the same set of tools, practices, APIs, metadata, and version control. This collaboration can happen on a custom-built platform, built with individual components that must be glued together and maintained by large internal teams, or using an off the shelf automated solution like the <a href="https://www.iguazio.com/">Iguazio MLOps platform</a> or the open source MLOps orchestration framework, <a href="https://mlrun.org/">MLRun</a>.</p><h3>MLOps Should Provide Real Business Value!</h3><p>Did your data science team prevent fraud for your organization, reducing fraudulent transactions to save costs for your organization, enable extension of credit lines to new customers and reduce legal hassles?</p><p>Was your organization able to predict which machines are about to malfunction, and proactively mitigate the issue to save costs on repairing broken machines or purchasing new ones?</p><p>Were your customer success teams able to predict customer churn and produce the exact benefit needed to retain your customers, directly impacting your organization’s bottom line?</p><p>If the answer to these is no, and software / infrastructure inefficiencies are holding your organization back from seeing the real value your data science team could deliver, now is the time to look into solutions that make the data science process automated, orchestrated, accelerated and reproducible. With a solid MLOps foundation, you will be able to quickly and continuously deliver new AI services for the business, even at scale and in real-time. You will enable your data science, data engineering and DevOps teams to collaborate more efficiently and effectively. Most importantly, the entire organization will be able to benefit from your team’s innovative solutions, and see them directly impacting the organization’s goals and bottom line.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e028a21a269e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/mlops-beyond-training-simplifying-automating-the-operational-pipeline-e028a21a269e">MLOps Beyond Training: Simplifying &amp; Automating the Operational Pipeline</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[6 Tips for MLOps Acceleration & Simplification]]></title>
            <link>https://medium.com/data-science/6-tips-for-mlops-acceleration-simplification-36539adab29b?source=rss-bdd9246d060d------2</link>
            <guid isPermaLink="false">https://medium.com/p/36539adab29b</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[yaron haviv]]></dc:creator>
            <pubDate>Tue, 18 May 2021 13:30:49 GMT</pubDate>
            <atom:updated>2021-05-18T23:18:14.723Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Y9Ugj61K1qxw20mWzBii4g.png" /><figcaption>(image by author)</figcaption></figure><p>MLOps (machine learning operations) is key for improving business productivity by getting data science to production. Therefore, it’s essential for any company that wants to gain a competitive edge with AI. In this post, I’ll cover six tips and best practices to accelerate and simplify your path to production, based on my experience with enterprises worldwide. It is an overview of a talk I gave at NVIDIA’s latest GTC conference, which you can view <a href="https://www.youtube.com/watch?v=o6wGnT0dU6s">here</a>. I hope you find it useful!</p><h3>Challenge #1: A long and difficult path to production</h3><p>Developing the model in research is just the beginning. There are still many steps to cover, to ensure the model works online and generates value in real business applications. These include:</p><ul><li>Packaging code, scripts, dependencies and configuration into containers</li><li>Collecting and preparing data from operational and online sources at scale</li><li>Training the models with large scale data, various algorithms and params</li><li>Incorporating the models into real-time/interactive pipelines, which feed from live data</li><li>Monitoring model accuracy and creating automated re-training workflows</li><li>Adding instrumentation, tracking, versioning, and security to every step</li><li>Refactoring code for performance in order to meet application SLA</li><li>Creating continuous development and delivery (CI/CD) pipelines</li></ul><p>It can take months to complete these processes, and they are resource intensive.</p><h3>The Solution: Incorporate automation and adopt a serverless approach</h3><p>Automating the production process can shorten it from months to weeks. This can be done with serverless frameworks that automatically transform simple code or Notebook to managed and elastic microservices. Open source tools can help, like <a href="https://www.iguazio.com/open-source/mlrun/">MLRun</a>, an <a href="https://github.com/mlrun/mlrun">open source </a>MLOps orchestration framework. (Full disclosure: My team and I maintain it). MLRun automatically converts code into powerful serverless functions and allows simple composition of multiple functions into batch, CI/CD or real-time pipelines.</p><p><a href="https://docs.mlrun.org/en/latest/runtimes/functions.html">MLRun functions</a> are fully managed with automated tracking, monitoring, logging, versioning, security, and horizontal scalability, which help to eliminate much of the development and DevOps efforts. They also accelerate time to production without compromising on performance or scalability.</p><p>In MLRun, we can use our own functions or the marketplace functions for data preparation, training, model testing, serving, etc. Then, we can compose complex ML pipelines in minutes, and run them at scale with full experiment tracking and artifact versioning.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lVKz9nCzKmyqTQTUCCwGIA.png" /><figcaption>MLRun development flow (picture by author)</figcaption></figure><p>MLRun supports multiple serverless runtime engines (job, dask, spark, MPI/Horovod, Nuclio ..) to address various workloads at scale (data analytics, machine learning, deep learning, streaming, APIs, serving …). It supports multiple pipeline/CI engines for running the ML workflows including Kubeflow Pipelines, Github Actions, GitLab CI/CD, etc.</p><p>Functions can also be used to compose real-time production pipelines, handle stream processing, data enrichment, model serving, ensembles, monitoring, and more. MLRun real-time pipelines are built on top of the popular high-performance open source serverless framework <a href="https://nuclio.io/">Nuclio</a> (which my team and I maintain as well).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/601/1*ry-RAX_ixg7LP7auD-IjLA.png" /><figcaption>MLRun Serving Graph Example (picture by author)</figcaption></figure><p>Check out the example notebooks for <a href="https://docs.mlrun.org/en/latest/serving/graph-example.html">serving graph</a> and <a href="https://docs.mlrun.org/en/latest/serving/distributed-graph.html">distributed NLP pipeline</a> to see how easy it is to deploy real-time production pipelines with MLRun.</p><h3>Challenge #2: Inefficiency due to effort duplication &amp; siloed work</h3><p>Machine learning work is divided up between different teams: data science, engineering, and MLOps. Each of these teams works with different toolsets, making it very difficult to collaborate, and forcing resource intensive code and pipeline refactoring. In addition, in the current practice, teams end up building three separate pipeline architectures to address the research, production, and governance needs, which leads to resource waist and code refactoring or reimplementation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/892/1*xEdCGiE7gZmRnkbR5qZBJA.png" /><figcaption>Data Science Pipelines (picture by author)</figcaption></figure><p>We can see from the diagram that the data transformation logic is not the same in the research and production pipelines, leading to inaccurate features and model results. Not to mention the effort duplication.</p><h3>The Solution: Consolidate platforms and processes for data science, data engineering, and DevOps</h3><p>Consolidating all the work through a unified production-minded workflow and technology that enables collaboration, will enable teams to work together. It will also lead to improved accuracy and rapid, ongoing deployment of AI for the organization. This serves the needs of all stakeholders.</p><p>For example, data scientists will be able to build features without needing to constantly ask for data from data engineers, MLOps teams will be able to redeploy without reengineering, and so on. In addition, utilizing a feature store at the heart of the platform allows unifying the data collection, transformation, and cataloging process for all three use cases (research, production and governance), saving time and effort in one of the most labor-intensive parts of the MLOps process.</p><p>Having one end-to-end platform that consolidates it all: the feature store, ML training pipelines, model serving and monitoring, is important for enabling acceleration and automation. Such a platform saves the time, effort and frustration of having to stitch together different components to get a full enterprise-grade working solution.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/795/1*tEcXmuBSUul0Cgcx1g_aIw.png" /><figcaption>MLOps Platform Architecture (picture by author)</figcaption></figure><h3>Challenge #3: Under-utilized resources and limited scaling</h3><p>When every developer or team has their own dedicated hardware, VMs, or GPUs, the result is infrastructure silos. These silos lead to a waste of resources and to management complexity. They also limit our ability to process data at scale or shorten execution time, compared to if we clustered all those resources together.</p><p>Enterprises want to make sure they capitalize on the investment they make on AI Infrastructure, to ensure their compute, data and GPU resources are fully utilized.</p><h3>The Solution: Use a shared and elastic pool of resources</h3><p>Pooling resources together into an elastic cluster that can dynamically shrink or grow will enable sharing resources, smarter allocation, and resource-saving. Find a solution that accelerates time to production and makes the most out of your AI infrastructure.</p><p>Here is one such example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_S8-GDa9rOA-OEIAf34IfQ.png" /><figcaption>(picture by author)</figcaption></figure><p>By taking servers and VMs, running Kubernetes on top of the cluster, adding a data layer for moving data across microservices and scheduling workloads on top, you should be able to gain better scale and performance. Include a suite of tools, services for different aspects, development environments and tracking aspects, in your stack., This will accelerate TTP and ensure less resources are consumed, that the pool is shared, and that larger scale workloads can run as needed.</p><h3>Challenge #4: Handling data &amp; features takes 80% of your team’s time</h3><p>Handling data requires a lot of work. While AutoML tools have become more widespread and simplify the model building part, you still constantly need to use, prepare, store, and ingest application-specific data. Data handling is the most time consuming and resource-intensive tasks in ML.</p><h3>The Solution: Use feature stores to simplify feature engineering</h3><p>Feature engineering enables transforming our raw data into something meaningful that feeds ML workloads. For example, adding activity times, aggregating data and calculating statistics, joining data from different sources, running sentiment analysis, picture rotation, and more. These features can then be used for training offline data, in the online production pipeline and for monitoring and governance.</p><p>The solution to this challenge is a <a href="https://www.iguazio.com/feature-store/">feature store.</a> A feature store is a central place for building a pipeline for automatically creating features for training and production. Tech giants like Uber, Twitter and Spotify built their own feature stores, and there are few open-source and commercial tools for feature stores. When you select yours, make sure to check that it suits your needs. For, example if you are planning to ingest streaming data, check whether the feature store can run calculations on the fly and perform joins, aggregations etc., and how well it can be integrated with the other MLOps components.</p><h3>Challenge #5: Inaccurate models and limited visibility</h3><p>Models will only stay accurate if we keep having the same data, assumptions, and behavior. But that is rarely the case, since these changes all the time and lead to model drift and inaccuracy.</p><p>For example, Covid-19 changed consumer behavior significantly, from buying more food to less flights. If your models were not modified accordingly, they would have produced invalid and inaccurate results.</p><p>We must constantly monitor our data and models and alert when they deviate. If these challenges aren’t visible and identified on time, the models will not be accurate, which can lead to negative business outcomes or legal exposure.</p><h3>The Solution: Make sure model monitoring and re-training is built into your MLOps pipeline</h3><p>Include a feedback and drift-aware system in your <a href="#_msocom_1">[1]</a> <a href="https://www.iguazio.com/mlops/">MLOps pipeline</a> to measure the difference between predicted and actual results. The systems should trigger alerts so you can act on the change, for example retrain the model, switch it, etc.</p><p>The models of this feedback system should track everything in a real-live stream, with functions grabbing statistical information from the feature store and comparing it with real-time behavior. If a misbehavior is identified, an alert is triggered to a microservice. The microservice will then trigger the alert to you by email, retrain or switch a model, or run any other custom operation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*a7LOtJua_3K1VBNeLAzzlg.png" /><figcaption>Production Pipeline With Monitoring (picture by author)</figcaption></figure><p>When using MLRun, model activities are tracked in databases and visualized using Grafana or MLRun dashboards. Here are some dashboard snapshots:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6ofQXcFpvgi-V6DZPvfNZQ.png" /><figcaption>Model Monitoring in MLRun &amp; Grafana (picture by author)</figcaption></figure><h3>Challenge #6: A constantly evolving toolset &amp; steep learning curve</h3><p>MLOps and data science have a lot of technology behind them, and there is a tremendous and growing number of tools out there. It can be overwhelming to understand what you need to get started, or to support a specific use case. Considerations like data locality, security, collaborations etc. all need to be considered. Often, as soon as you’ve made the decision on your AI infrastructure software stack, something new comes along and alters the equation…</p><h3>The Solution: Adopt an ‘open yet managed’ state of mind</h3><p>You want your stack to be future proof. You want to be able to harness the latest updates from the growing data science community. I agree with that approach and will always recommend using open-source tools as much as possible. In addition to their quality, the open-source community provides a lot of answers and support. However, depending on your needs, it might get overwhelming to integrate, maintain, secure, and support them all yourself. That’s where a pre-baked stack of the leading open-source tools along with a managed service experience might be helpful and help you focus your efforts on building your AI applications.</p><p>MLOps can be daunting, but sharing from our collective experience will make us all smarter, and faster. I hope you found my article helpful, and if you like you can watch my full talk <a href="https://www.youtube.com/watch?v=o6wGnT0dU6s">here</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=36539adab29b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/6-tips-for-mlops-acceleration-simplification-36539adab29b">6 Tips for MLOps Acceleration &amp; Simplification</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Kubeflow: Simplified, Extended and Operationalized]]></title>
            <link>https://medium.com/data-science/kubeflow-simplified-extended-and-operationalized-fcbcc34f79e5?source=rss-bdd9246d060d------2</link>
            <guid isPermaLink="false">https://medium.com/p/fcbcc34f79e5</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[yaron haviv]]></dc:creator>
            <pubDate>Thu, 12 Nov 2020 21:12:28 GMT</pubDate>
            <atom:updated>2020-11-12T21:12:28.445Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uUv3JzqPFVQuO8d1cYOjVw.png" /><figcaption>source: shutterstock</figcaption></figure><p>The success and growth of companies can be deeply intertwined with the technologies they use in their tech stack. Nowhere is this more apparent than in the case of developing ML pipelines.</p><p>The nature of delivering robust ML models and data pipelines to production is a complex business. Every data science team is faced with the same limitations and constraints, regardless of the scope of a given project:</p><ul><li>Feature analysis and preparation at scale is hard</li><li>Running many ML training tasks, each with different parameters or algorithms, takes too much time and requires expensive resources</li><li>Tracking and reproducing experiments, along with the right versions of code, data and results, is complex</li><li>Transferring data science notebook code to production pipelines takes a lot of effort</li><li>Production pipelines need to handle many operational tasks for scalability, availability, automated model drift detection, integration with live data, security, etc.</li></ul><p>Building a model is far from the end of the journey. To deploy AI enabled applications to production you’ll need an army of developers, data engineers, DevOps practitioners and data scientists, ideally collaborating and working together on a single platform.</p><p>There aren’t many solutions in this emerging field that cover this particular space, and <a href="https://www.kubeflow.org/">Kubeflow </a>has gained momentum as the open-source leader in the past few years. <a href="https://www.kubeflow.org/">Kubeflow </a>is a wonderful tool for orchestrating complicated workflows running on <a href="https://kubernetes.io/">Kubernetes</a>, but it also poses challenges, especially for data scientists who aren’t accustomed to working with these types of solutions. Many ML practitioners find it complex to use, encounter various usability issues and become frustrated with functionality gaps.</p><p>Luckily, it’s possible to extend the capabilities of Kubeflow and transform it into a comprehensive, easy-to-use<a href="https://www.iguazio.com/mlops/"> MLOps</a> platform. At<a href="https://www.iguazio.com/"> Iguazio</a>, we’ve done just that. We’ve embraced Kubeflow. Our approach has been to add it to our managed services catalog and bridge the functionality gaps that exist, wrapping and extending its capabilities with new open-source frameworks we’ve developed.</p><h3>Why Do We Need an MLOps Solution?</h3><h4>The Life of a Data Scientist: Tool Wrangler</h4><p>Data scientists generally go through the following process when developing models:</p><ol><li>Data collection from CSV or dense formats (e.g.<a href="https://parquet.apache.org/"> Parquet</a>) which require manual extraction from external sources. Tools used: ETL and data exporting tools.</li><li>Data labelling, exploration and enrichment to identify potential patterns and features. Tools used: Interactive tools for data labelling, exploration and model development; analytics tools/databases to aggregate, join and pivot data at scale.</li><li>Model training and validation. Tools used: Parallel model training frameworks.</li><li>Model evaluation using automation and/or visualization tools. Tools used: CI/CD frameworks, visualization libraries.</li><li>Go back to step 1 and repeat until the desired outcomes have been achieved.</li></ol><p>This results in a lot of time-consuming and complexity-snowballing legwork. It’s imperative that data science teams introduce some method into this madness if they have any intention of ever getting a model out of the lab.</p><h3>It’s Just the Beginning</h3><p>But this long process is just the first step! You’ll still need to operationalize the model, tie it to applications, and automate your feature engineering process. This can take a <em>year or more</em> and requires an army of ML practitioners attacking the problem with brute force.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/292/1*b025cYS3ecCkk0tULn8iFw.png" /><figcaption>source: Twitter</figcaption></figure><p>Once we have a model, we need to go over all of the many steps in order to deliver AI/ML based applications to production. This includes: packaging, scaling, performance tuning, instrumentation and automation.</p><h3>Kubeflow is the ML Toolkit for Kubernetes</h3><p><a href="https://github.com/kubeflow/kubeflow">Kubeflow</a> is an open-source project originated by Google that groups leading ML frameworks over<a href="https://kubernetes.io/"> Kubernetes</a>. It provides common installation tools and cross-validation, but its tools can also be deployed as a standalone. It acts as the brick and mortar for model development, focusing on automation, scaling and tracking of ML model development pipelines over Kubernetes.</p><h3>Notable Kubeflow Components:</h3><h4>Containerized Jupyter Notebooks</h4><p>Kubeflow provides a<a href="https://jupyter.org/"> Jupyter</a> Notebook service over Kubernetes so users can create a Jupyter server with given resource requirements and get provisioned containers.</p><h4>Kubeflow Pipelines (KFP)</h4><p>What is<a href="https://github.com/kubeflow/pipelines"> Kubeflow Pipelines</a>? Simply put, it’s a description of an ML workflow with all of its components. A component can be responsible for different steps in the ML process, like data processing, transformation, model training, validation, etc.</p><p>KFP enables you to describe a workflow with individual steps (where each step is a container microservice or serverless function) through a Python SDK, execute that workflow on the cluster, and track the progress and experiment results.</p><p>KFP enables users to automate ML pipelines instead of running individual jobs manually.</p><h4>Scalable Training Operators</h4><p>Application lifecycle and scaling in Kubernetes can be managed through an<a href="https://kubernetes.io/docs/concepts/extend-kubernetes/operator/"> operator</a>. The Kubeflow project hosts various operators to run scalable ML training tasks using different ML frameworks (<a href="https://www.tensorflow.org/">TensorFlow</a>,<a href="https://pytorch.org/"> PyTourch</a>,<a href="https://eng.uber.com/horovod/"> Horovod</a>,<a href="https://mxnet.apache.org/"> MXNet</a>,<a href="https://chainer.org/"> Chainer</a>).</p><h4>Model Serving Solutions</h4><p>The Kubeflow project hosts a basic model serving framework (KFServing) and supports external serving frameworks (Seldon, Triton and<a href="https://github.com/mlrun/mlrun/tree/development/mlrun/serving"> Nuclio-serving</a>). Iguazio’s leading open-source project, <a href="https://github.com/nuclio/nuclio">Nuclio</a>, a serverless engine for real-time APIs, ML and data engineering tasks has a <a href="https://github.com/mlrun/mlrun/tree/development/mlrun/serving">model serving</a> flavor with fast performance and vast set of features (automated model deployment, canary, model monitoring, drift and outlier detection, ensembles and real-time pipelines, etc.) .</p><h3>Operationalizing Kubeflow: Key Challenges</h3><p><a href="https://github.com/kubeflow/kubeflow">Kubeflow</a> is itself a component of a broader system of ML operationalization. Kubeflow contains some powerful components that make it a frontrunner in its space — it’s Kubernetes native, highly customizable, open-source and loosely coupled. But it has some key limitations that need to be addressed by an ML team.</p><h3>Challenge 1: Complexity</h3><p>Kubeflow is an ecosystem of tools rather than a holistic or integrated solution, and it relies on many underlying platform services and tools (Kubernetes, user management, data services, data versioning, monitoring, logging, API gateways, etc.). Without a managed solution, companies need a team of developers, DevOps and ML engineers who will work on integrating the various Kubeflow and add-on components into a complete service with unified portals and security, managing the deployment, supporting internal users, upgrading each of the sub-services, building tools and expertise for troubleshooting, and following the community work in an ongoing basis. This may make sense if you already have a large DevOps organization and inexpensive resources, but won’t make sense for many leaner organizations who want to focus on their business applications.</p><p>Let’s assume you want to run hyperparameter jobs using some tool like Katib, with Kubeflow Pipelines, a distributed ML operator like MPIJob, and track or compare the results. You would need to stitch all of those tools together manually, even though they are all child projects under Kubeflow.</p><p>Installing Kubeflow properly means running and managing more than a dozen individual services, and some are not production grade. ML practitioners who like the concept of a single AI/ML framework over Kubernetes find themselves struggling to deploy, manage and secure it. Once they do, they notice their internal users reluctant to use it, due to usability and integration issues.</p><h3>Challenge 2: Kubeflow Speaks the Language of Engineers, Not Scientists</h3><p>The core focus of Kubeflow is advanced ML engineering using Kubernetes. It was originally designed by Kubernetes community members for DevOps and MLOps engineers, rather than for data scientists.</p><p>Like every other Kubernetes functionality, it all starts with Docker containers, YAML files and scripts. Data scientists are the main consumer for a data science platform. They don’t want to manage YAML files — they’d rather use a simple UI and data engineering, or ML logic in Python, and have something run it for them on a cluster and then collect the results, hopefully in a scalable and managed way.</p><p>While running ML tasks in distributed pipelines is powerful, data scientists usually start off using a local IDE like Jupyter, PyCharm or VSCode. They want to develop, debug, run and track the work locally without creating Kubernetes resources or complex pipelines, and only after they get the individual runs to work successfully they (or an ML engineer from the team) want to transition to running on a scalable cluster or use multi-stage pipelines. This is unfortunately not possible in Kubeflow, as every tracked execution requires building a full pipeline and Kubernetes resources.</p><p>Data science teams would like to organize and manage their work based on logical ML projects, with clear project level membership, project specific resources and billing, ability to isolate resources and security credentials between projects, etc. In Kubeflow users can create jobs with elevated privileges, there is no isolation, monitoring or tracking at the user or project level.</p><p>And again, many tasks are complex and manual. For example, if you want to view the results of a pipeline run, you must write code to store and visualize the run artifacts, most commercial tools or managed just do that for you.</p><p>The data science team should be able to use familiar tools and simple UI portals, and have resources be provisioned and monitored for them under the hood.</p><h3>Challenge 3: Kubeflow Is A Partial MLOps Solution</h3><p>The Kubeflow toolkit applies mainly to the model development stage, meaning teams need several other mandatory services, adding further overhead. To get to business value fast (or at all), teams need to extend Kubeflow functionality to the rest of the data science pipeline and manually integrate with external data and identity management systems.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/566/1*2NIljh3czqLaKncS7LbOgg.png" /><figcaption>The MLOps Stack and Kubeflow (image by author)</figcaption></figure><p>Kubeflow focuses on the model development pipeline for running and tracking ML experiments, however most users need additional services as outlined in the picture above, services for scalable data and feature engineering, model management, production/real-time pipelines, versioned and managed data repositories, managed runtimes, authentication/security, etc.</p><p>Iguazio developed open-source frameworks (like<a href="https://github.com/nuclio/nuclio"> Nuclio</a>,<a href="https://github.com/mlrun/mlrun"> MLRun</a>) and additional data services and management layers which complement and extend Kubeflow into a fully functional, managed data science platform. This is further described in the section below.</p><h3>Delivering an End-to-End Data Science Platform, Kubeflow Inside</h3><p>Much of the complexity in delivering data science based products to production is the integration between different organizational and technology silos, taking individual “best of breed” components and integrating them manually require significant amount of resources, maintain the organizational silos, and leads to a huge technical debt since you maintain all the integration and glue layers.</p><p>We believe that organizations should deploy a data science solution which spans the data engineering, data science, ML engineering and DevOps space. We need to abstract away much of the complexity while enabling a high-performing, scalable and secure platform which can be deployed in any cloud or on-prem.</p><p>Here’s a list of the additional functionality found on the Iguazio platform (much of it is open source):</p><h4>Integrated Solution with User-Friendly Portal, APIs and Automation</h4><p>Usability is one of the main challenges in Kubeflow, compounded by the fact that Kubeflow only addresses parts of the solution (no offline development, data engineering or production deployments). Users find themselves working against multiple portals and UIs, and adding a lot of glue code.</p><p>Assume that all the data science elements and metadata are managed by one service, you can create a feature-set object using the feature store, pass this as an object to training, validation or model serving logic, use it to validate your model behavior and concept drift, and update it and the corresponding labeled data directly from the production pipeline. Using glue logic and scripts to achieve that is almost impossible and leads to significant extra development and ML engineering effort.</p><p>The open-source framework called<a href="https://github.com/mlrun/mlrun"> MLRun</a> controls the entire functionality from data ingestion to production pipelines, it delivers a comprehensive Web UI, CLI, API/SDK, and it is tightly integrated with the underline Kubeflow components (Operators, Pipelines, etc.)</p><p>With<a href="https://github.com/mlrun/mlrun"> MLRun</a>, users can work through the SDK and IDE plug-ins from anywhere using their native IDE (Jupyter, PyCharm, VSCode, etc.), there’s no need to run on the Kubernetes cluster, and you can run individual tasks or complete batch and real-time pipelines with a few simple commands.</p><p>Execution, experiment, data, model tracking and automated deployment is done automatically through<a href="https://github.com/mlrun/mlrun"> MLRun</a> serverless runtime engines. MLRun maintains a project hierarchy with strict membership and cross team collaboration (rather than using a flat namespace like in Kubeflow).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lW2IfveqE34SVeJUjznUOA.png" /><figcaption>Example: MLRun project dashboard screen (image by author)</figcaption></figure><p>You can deploy the various components as described in<a href="https://mlrun.readthedocs.io/en/latest/"> MLRun documentation</a>, or alternatively, you can use Iguazio’s managed offering which adds end-to-end security and fully managed service experience on top of MLRun and Kubeflow plus additional set of data and management services.</p><h4>Feature Stores</h4><p>Feature engineering can be one of the most time intensive tasks in building ML pipelines. Without a shared catalog of all features, teams risk wasted time on redevelopment and duplication of work across teams and projects. The features need to be computed regularly using data engineering frameworks (like Spark, Presto, Dask, etc.) and stream processing engines (Flink, Nuclio, etc.) and be delivered using offline/batch APIs for training and real-time APIs and databases for serving. One key benefit of feature stores is the delivery of a consistent feature set for the training and serving layer, ensuring that trained models maintain their performance in production.</p><p>Furthermore, feature stores are a critical element for storing versioned feature metadata and statistics, which are essential to conducting model monitoring, validation, concept drift detection and continuous retraining.</p><p>Building your own feature store is a luxury usually reserved to tier-one web companies due to the complex stack it entails. The alternative of not having a feature store means the team will invest a significant amount of manual data engineering work to build and maintain features.</p><h4>Real-Time Pipelines, Model Management and Monitoring</h4><p>Once models are ready, they need to turn into running microservices which accept feature vectors through APIs or data streams and produce prediction results, which need to be used in production business applications. In reality, production pipelines are far more complex than a simple serving end point and they may include steps for feature retrieval, transformations, validation, outlier detection, serving of multiple models under an ensemble or more complex graph, alerting, etc.</p><p><a href="https://github.com/nuclio/nuclio">Nuclio</a> and<a href="https://github.com/mlrun/mlrun/tree/development/mlrun/serving"> MLRun-serving</a> open-source frameworks enable teams to build complex real-time data processing and model serving pipelines from the training results and data from the real-time feature store in a matter of minutes, and deploy or upgrade to production with a single API/UI command.</p><p>Getting a model to production is only the beginning of its lifecycle. A model’s predictions and behavior must be logged, they must be monitored for concept drift, and sometimes retrained on fresh data under shifting circumstances. Data requires governance and auditing. Without a centralized system in place, model maintenance can quickly become unmanageable and overly expensive.</p><h4>Managed Services and Serverless Runtimes</h4><p>Significant effort goes into creating and managing various software packages and turning them into production services with auto scaling, configuration management, upgrades and security. A typical data science stack can consist of dozens of services like data analysis tools, ML/AI frameworks, development tools, dashboards, security and auth services, and logging. You could hire a large team to build a stack manually, or you could use a platform with managed services that allow users to choose a service, specify params and click deploy.</p><p>And further, turning code into production services is a time consuming project involving many team members (see figure below). You need to package the code into containers, reengineer it for resiliency, scale, performance, hardened security, add instrumentation, etc. Many data scientists aren’t proficient in software development and DevOps, so these tasks can be even more challenging for them. The use of serverless technologies enables a team to automate all of these tasks.</p><p>MLRun runtimes and Nuclio can take a piece of code and automatically turn it into an elastic and fully managed service, and this can be done from a UI or from within your development environment (like Jupyter) with a single line of code.</p><p>The serverless functions can run as standalone or can be plugged in as steps in a larger Kubeflow pipeline without any additional development. MLRun projects enable automated CI/CD pipelines. A workflow can be triggered through a Git hook every time code, data or configuration changes, and members of the project can review the results and approve automated deployment into test or production systems.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aCCCPx1lT_q44LTqiezuUg.png" /><figcaption>Effort of migrating from development to production (image by author)</figcaption></figure><p><strong>Function Marketplace and AutoML</strong></p><p>You can develop your own functions or use a set of prebaked functions from a marketplace and plug them into your pipeline with zero development effort.<a href="https://github.com/mlrun/functions"> The MLRun function marketplace</a> provides a set of AutoML functions for feature analysis, detection and automated ML training. The functions are designed for scale, performance and are ready for production with instrumentation and visual reports.</p><p>Teams can implement a local function marketplace/repository which maximize collaboration and reuse. One member can build a function that can be used in another project, or improved by another team member and returned to the market (using a different version tag).</p><h3>Production-Ready Managed Data Services</h3><p>Every data science solution works on data, and data can come in many forms (files, tables, streams, time-series, etc.), which require integration with several underlying data services. Data access should be controlled where every job or user is confined to specific datasets and with specific access permissions, furthermore data must be versioned for reproducibility.</p><p>You can obtain and integrate managed data services from the different cloud providers. Iguazio has an industry-leading high-performance multi-model database and low latency distributed file system, with unique technology that uses flash/SSD storage but performs like in-memory DB at a fraction of the cost, and with much higher scale. These ingredients enable faster time to deployment, faster performance, larger scale and reduced costs.</p><h3>End-to-End Security</h3><p>Data science platforms can contain very sensitive data, aggregated from multiple users. A multi-layered security approach must be implemented, including:</p><ol><li>Authentication and identity management: Determine which user or service is accessing a piece of data or another service, using industry standard protocols such as LDAP, OAuth2 and OIDC.</li><li>Role Based Access Control (RBAC): Enforce the access rights of every user, task or role, which resource they can access and what are the operations they are allowed to perform.</li><li>Data security: Control the access and privacy of data assets.</li><li>Secrets management: Protect secrets and access credentials, avoid storing those secrets inside the code or the data assets, but make them available to specific users and jobs at execution time.</li></ol><p>MLRun supports integration with security services. You can deploy and integrate them by yourself or use Iguazio managed platform with all of that solidified and fully managed for you.</p><h3>Summary</h3><p>At <a href="https://www.iguazio.com/">Iguazio</a>, we’re big believers in the potential of Kubeflow, which is why we’ve chosen to wrap it and extend it into a complete end-to-end MLOps solution. We also believe in breaking the silos between data engineering, data science and DevOps practitioners, so that complexity is abstracted away and organizations aren’t left with heavy technical debts. With some additional services and functionalities, Kubeflow can be the right solution to the challenges of operationalizing machine learning — giving data scientists a high-performing, scalable and secure technology which can be deployed in any cloud, on-prem or in hybrid environments.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fcbcc34f79e5" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/kubeflow-simplified-extended-and-operationalized-fcbcc34f79e5">Kubeflow: Simplified, Extended and Operationalized</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How To Productize ML Faster With MLOps Automation]]></title>
            <link>https://medium.com/data-science/how-to-productize-ml-faster-with-mlops-automation-d3f25caf38c4?source=rss-bdd9246d060d------2</link>
            <guid isPermaLink="false">https://medium.com/p/d3f25caf38c4</guid>
            <category><![CDATA[serverless]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[kubeflow]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[yaron haviv]]></dc:creator>
            <pubDate>Tue, 30 Jun 2020 21:05:07 GMT</pubDate>
            <atom:updated>2020-06-30T21:05:07.562Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DpwN3kdobSpPdg1hGeRQGQ.jpeg" /><figcaption>source <a href="https://www.shutterstock.com/image-photo/concept-devops-illustrates-software-delivery-automation-1418438333">shutterstock</a></figcaption></figure><p>In a recent survey we ran during our bi-weekly <a href="http://www.iguazio.com/mlops#live">MLOps Live webinar series</a>, the number one challenge data science teams are struggling with was confirmed by hundreds of attendees — bringing machine learning to production. This is validated by Gartner research, which consistently pinpoints productizing ML to be one of the biggest challenges in AI practices today. Many AI projects, more than 80% <a href="https://venturebeat.com/2019/07/19/why-do-87-of-data-science-projects-never-make-it-into-production/">according to research</a>, get stuck in the lab, produce partial success, or consume far more resources and time than initially planned. It is essential to find faster and simpler ways to productize machine learning projects so that they can make an impact on the business.</p><p>Setting aside political and organizational challenges, the main obstacles are the fact that the data science team often works in a silo, cut off from the engineering and DevOps teams, and use manual development processes (a.k.a “jumbo” <a href="https://jupyter.org/">Jupyter</a> playgrounds), which then need to be converted manually to production ready ML pipelines. This requires separate teams of ML engineers, data engineers, DevOps and developers to invest additional time and resources, often much more than initially anticipated. Every time there is a change in the data preparation or model training logic, this whole cycle is repeated.</p><p>AI services and applications are starting to become an essential part of any business. With this trend comes liability which drives further complexity. There is a need to add data, code and experiment tracking, monitor models to detect concept drift, improve model accuracy through the use of AutoML techniques and Ensembles, and so on.</p><p>The currently dominant research-oriented data science approach can no longer prevail in this environment. Data science MUST adopt agile software development practices with micro-services, <a href="https://en.wikipedia.org/wiki/Continuous_integration">continuous integration</a> (CI), <a href="https://en.wikipedia.org/wiki/Continuous_deployment">continuous deployment</a> (CD), code versioning (Git), and configuration/metadata versioning (GitOps).</p><p>When I speak to data science team leaders, they all agree with this approach but struggle with the implementation. After all, there are reasons why data scientists are not software developers or ML engineers and the other way around. We need ways to automate the process and bridge the gaps between software and data science practices. This is where some new open source tools can come in handy as I will demonstrate.</p><p>In this post, I will suggest and show how to accelerate ML operationalization and reduce friction.</p><h3>Divide and Conquer</h3><p>Have you ever tried to turn data-science notebooks or a <a href="https://www.kaggle.com/">Kaggle</a> project into something which can run in production? Good luck. The typical project involves an XXL notebook which begins with data gathering, data preparation, feature analysis, charting, training, validation, etc.</p><p>You would usually find the same boiler plates and helper functions duplicated in multiple notebooks, code is not organized in functions, exceptions are rarely handled, etc. If you try to version control the notebooks it is very hard to find the difference between versions, not to mention trying to review and comment on a pull request.</p><p>The first step we need to take is to break our project into functions (for example data exploration, training, etc.) with well-defined and documented APIs. If those functions are shared we should place them in a separate Python or notebook file. It is easier to debug, trace and understand code when we define APIs, and we can test individual functions or use them as part of a larger flow. Refactoring individual functions or steps to micro-services is much simpler than refactoring the entire project notebook. I will later show how even this can be automated.</p><h3>From Experiments to Production without the Hassle</h3><p>Now that we broke out the data science code and workflow to a set of individual functions, we need to find a way to run these functions with production or scale in mind and address the following requirements:</p><ul><li>Make it deployable (build a docker image with required packages, Kubernetes runtime spec, etc.)</li><li>Add robustness and fault handling</li><li>Track executions with parameters, results, data, and metadata</li><li>Optimize for performance, support scaling for demanding workload or larger datasets</li><li>Operationalize: support logging, monitoring, hardened security, etc.</li></ul><p>This is usually where teams of developers, data and ML engineers refactor the code and make it production ready, which means adding far more resources and significantly delaying the process. In organizations where teams are siloed this is extremely difficult and leads to an abandonment of many data science initiatives mid-process.</p><p>An alternative approach is to collaborate across teams and minimize the manual work by automating the migration from code to production and the way we track our executions and data.</p><p>We want to take our data-science functions, build and deploy them automatically, scale them based on the actual usage, and monitor all their inputs and outputs with minimal work. This is already done today with “Serverless” technologies, we just need to adapt it to unique data-science problems and development practices.</p><p>A new open-source framework called <a href="https://github.com/mlrun/mlrun">MLRun</a> (which was initiated by my company, <a href="https://www.iguazio.com/">Iguazio</a>) works in combination with other tools like <a href="https://nuclio.io/">Nuclio</a> (serverless engine) and <a href="https://github.com/kubeflow/pipelines">Kubeflow pipelines</a> (pipeline orchestration) to automate the <a href="https://en.wikipedia.org/wiki/MLOps">MLOps</a> process and bring CI/CD + Git practices to data science. Early adopters of this technology reported that they cut overall delivery times from many months to weeks, significantly reducing computation resources and labor, and now they finally have full visibility into all their experiments and data.</p><p><a href="https://github.com/mlrun/mlrun">MLRun</a> provides an SDK and a Kubernetes service. The SDK can be used anywhere (in your code IDE, Jupyter notebook, managed notebook service like <a href="https://colab.research.google.com/">Google Colab</a> or AWS <a href="https://aws.amazon.com/sagemaker/">Sagemaker</a>, in production containers, etc.). It can auto track all your function activity, inputs and outputs. It handles parameters and data passing between various functions, and it can turn your local code into a distributed function or a complete ML pipeline using a single command. (<a href="https://mlrun.readthedocs.io/en/latest/">see mlrun docs for details</a>)</p><p><a href="https://github.com/mlrun/mlrun">MLRun</a> service records the tracking information from all the clients, auto provision “serverless” functions based on the provided code + metadata, manage workflows end to end by orchestrating Kubeflow pipelines, and provide web based UI for management end to end visibility into running jobs/experiments, functions, and artifacts.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_F6f00IBBGg6lzW7DUe1qQ.png" /><figcaption>from code to reusable micro-service (image by author)</figcaption></figure><p>You can check <a href="https://github.com/mlrun/mlrun/blob/development/examples/mlrun_export_import.ipynb">this notebook</a> to see how a function is written and documented in Jupyter and how with a single command it is converted to a serverless function (micro-service) object which can be published to Git or be cataloged in a reusable function marketplace.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/838/1*DbljDNgYyermwFmGXW2NbQ.png" /><figcaption>code to function (image by author)</figcaption></figure><p>That notebook also demonstrates how functions can be loaded from the repository/marketplace and used to execute tasks in one of three ways: 1. As a local executable, 2. As an auto-scaling containerized micro-service, 3. As an in-memory module. The executions are auto-tracked along with their metadata, inputs, and outputs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*w2oiaehKomBl-QsZiabi3g.png" /><figcaption>MLRun Experiment Tracking (image by author)</figcaption></figure><p>In other examples you can see how to scale-out computations with one of the supported distributed runtime engines (Nuclio, Dask, Spark, Horovod over MPI), manage models, manage complete projects, and provide on demand resource or GPU allocation (covered in details in my GPU as a service blog).</p><p>The functions are an atomic (code + all the runtime definitions), versioned, and documented which makes them highly reusable. Developers can take functions extend/enhance them or use them as is. This can greatly accelerate development time and you can view and reuse a list of useful pre-baked functions for different data science stages and algorithms.</p><p>Check my video showing how we automatically turn notebook code to scalable, auto tracked micro-services:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F_3mxz3zMPpw%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D_3mxz3zMPpw&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F_3mxz3zMPpw%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/b16848be0919dafc887d5ceac91b9936/href">https://medium.com/media/b16848be0919dafc887d5ceac91b9936/href</a></iframe><h3>Automated ML Pipelines</h3><p>Our data science pipeline requires more than a single function. We need to ingest and prepare data, select dominant features, train our model, validate it, deploy it, etc. We need to assemble a pipeline (workflow) — we use Kubeflow pipelines for that — and let MLRun compose, execute, and track our pipelines.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rpicfAa4tdm97tAWcZPQuA.png" /><figcaption>Building automated workflow from ML functions (image by author)</figcaption></figure><p>You can <a href="https://github.com/mlrun/demo-github-actions">see this example</a>, which demonstrates a complete ML pipeline composed of both notebook and marketplace functions. The pipeline ingests data, runs feature analysis training with AutoML, validation, model deployment, and production testing. The pipeline can be triggered manually via the command line or SDK, or it can be triggered by Git events such as a pull request comment. Git-based MLOps automation will be covered in more detail in my next blog post.</p><p>If we want to track our pipeline results, we can use the MLRun UI, or simply push the result summary into our slack channel or to our Git pull request as a comment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/653/1*o4UCLEJvxtAEqwQPyD0dNw.png" /><figcaption>Getting Slack Notifications (image by author)</figcaption></figure><h3>Summary</h3><p>Data science and MLOps tasks can be automated in a way which saves significant time and resources by adopting agile, CI/CD and serverless practices. The MLRun project is new, but is the first holistic MLOps automation solution aiming to accelerate productivity and time to production. The MLRun development team welcomes new users who want to try it out and new contributors, if you are not a fan of installations and devops you can consume it as a manage service by <a href="https://www.iguazio.com/">Iguazio</a>.</p><p>If you want to discuss this blog or other MLOps related challenges and best practices, I invite you to <a href="https://join.slack.com/t/mlopslive/shared_invite/zt-er5oclrl-BK_xTJYAt5M0pRNFaKDpWw">join the MLOps Live Slack community</a> and chat with me and other MLOps fans/experts.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d3f25caf38c4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/how-to-productize-ml-faster-with-mlops-automation-d3f25caf38c4">How To Productize ML Faster With MLOps Automation</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[GPU-as-a-Service on KubeFlow: Fast, Scalable and Efficient ML]]></title>
            <link>https://medium.com/data-science/gpu-as-a-service-on-kubeflow-fast-scalable-and-efficient-ml-c5783b95d192?source=rss-bdd9246d060d------2</link>
            <guid isPermaLink="false">https://medium.com/p/c5783b95d192</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[serverless]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[yaron haviv]]></dc:creator>
            <pubDate>Mon, 02 Mar 2020 17:45:04 GMT</pubDate>
            <atom:updated>2020-03-02T17:45:04.790Z</atom:updated>
            <content:encoded><![CDATA[<p>Machine Learning (ML) and Deep Learning (DL) involve compute and data intensive tasks. In order to maximize our model accuracy, we want to train on larger datasets, evaluate a variety of algorithms, and try out different parameters for each algorithm (hyper-parameter tuning).</p><p>As our datasets and model complexity grow, so does the time we need to wait for our jobs to complete, leading to inefficient use of our time. We end up running fewer iterations and tests or working on smaller datasets as a result.</p><p>NVIDIA GPUs are a great tool to accelerate our data science work. They are the obvious choice when it comes to Deep Learning workloads and deliver better ROI than CPUs. With new developments like <a href="https://developer.nvidia.com/rapids">RAPIDS</a>, NVIDIA is tackling data analytics and machine learning workloads (like XGBoost) efficiently (read the details in my previous post:<a href="https://towardsdatascience.com/python-pandas-at-extreme-performance-912912b1047c"> Python Pandas at Extreme Performance</a>). For example, an analytic task of reading Json data, aggregating its metrics and writing back to a compressed (Parquet) file runs 1.4 sec on GPU versus 43.4 sec on CPU (that’s 30x faster!).</p><h3>The Challenge: Sharing GPUs</h3><p>CPUs have long supported technologies such as virtualization (hypervisors), virtual memory, and IO management. We can run many different workloads, virtual machines and containers on the same CPUs while ensuring maximum isolation. We can use a variety of clustering technologies to scale computation across multiple CPUs and systems and use schedulers to dynamically allocate computation to tasks.</p><p>GPUs, on the other hand, must be assigned to specific VMs or containers. This leads to inefficiency. When a GPU intensive task finishes, the GPU will stay idle. If we associate a notebook sever with a GPU (for example), we waste GPU resources whenever a task is not running — for example when we are writing or debugging code, or when we go to lunch. This is a problem, especially when GPU equipped servers are more expensive.</p><p>Some solutions exist which partition a single GPU to smaller virtual GPUs, but this doesn’t solve the problem, as the (mostly idle) fragment we get is too small or has too little memory to run our task, and given that it’s hard to isolate memory between tasks on the same GPU we can run into many potential glitches.</p><h3>Solution: Dynamic GPU Allocation and Scale-Out</h3><p>The solution users are looking for is one that can harness multiple GPUs for a single task (so it can complete faster) and allocate GPUs just for the duration of the task. This can be made possible by combining containers with orchestration, clustering and a shared data layer.</p><p>Let’s assume we write some code in our Jupyter notebook or IDE (e.g. PyCharm). We can execute it locally, but when we need scale, we turn on a knob and it runs 10–100x faster on a distributed cluster. Wouldn’t that be nice? Can we implement such a dream? Yes, we can, and I will show you a demo a little further into this article.</p><p>To achieve this, we need to be able to package and clone our code and libraries to multiple dynamically scheduled containers at run time. We need all those containers to share the same data and to implement task distribution/parallelism mechanism, as illustrated in the following diagram.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/722/1*vic9F51ZhM16D_cGFGZ8aA.png" /><figcaption>Dynamic GPU/CPU Allocation (image by author)</figcaption></figure><p>A new open-source ML Orchestration framework called<a href="https://github.com/mlrun/mlrun"> MLRun</a> allows us to define “serverless” ML functions which consist of code, configuration, packages and infrastructure dependencies (such as CPUs, GPUs, memory, storage, etc.). Those “serverless” functions can run locally in our notebook or run over one or more containers which are created dynamically for the duration of the task (or stay longer if needed), the client/notebook and containers can share the same code and data through a low-latency shared data plane (i.e. virtualized as one logical system).</p><p>MLRun builds on top of<a href="https://kubernetes.io/"> Kubernetes</a> and<a href="https://www.kubeflow.org/"> KubeFlow</a>, it uses the Kubernetes API to create and manage resources. It leverages<a href="https://www.kubeflow.org/"> KubeFlow</a> custom resources (CRDs) for seamlessly running horizontal scaling workloads (such as<a href="https://github.com/nuclio/nuclio"> Nuclio functions</a>, Spark, Dask, Horovod …), KubeFlow SDK for attaching tasks to resources like storage volumes and secrets, and KubeFlow Pipelines to create multi-step execution graphs (DAG).</p><p>Every local or remote task which is executed through<a href="https://github.com/mlrun/mlrun"> MLRun</a> is tracked by the MLRun service controller, all inputs, outputs, logs, and artifacts are stored in a versioned database and can be browsed using a simple UI, SDK or REST API calls, i.e. a built-in job and artifact management. MLRun functions can be chained to form a pipeline, they support hyper-parameters and AutoML tasks, Git integration, Projects packing, but those are topics for different posts,<a href="https://github.com/mlrun/mlrun"> read some more here</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/985/1*wnHqA38_pb30-g8xTE6Sqw.png" /><figcaption>GPU-as-a-Service Stack (image by author)</figcaption></figure><h3>Example: Distributed Image Classification Using Keras and TensorFlow</h3><p>In our example we have a 4-step pipeline based on the famous Cats &amp; Dogs TensorFlow use-case:</p><ol><li>Data ingestion function — loading an image archive from AWS S3</li><li>Data labeling function — labeling images as Dogs or Cats</li><li>Distributed training function — Use TensorFlow/Keras with Horovod to train our model</li><li>Deploy an interactive model serving function</li></ol><p>You can see the full MLRun project and notebooks<a href="https://github.com/mlrun/demos/tree/master/image_classification"> </a><a href="https://github.com/mlrun/demo-image-classification">source here</a>, we will focus on the 3rd and 4th steps. For this to work, you need to use Kubernetes with few open-source services running over it (Jupyter, KubeFlow Pipelines, KubeFlow MpiJob, Nuclio, MLRun) and shared file system access, or you can ask for an<a href="https://www.iguazio.com/"> Iguazio</a> cloud trial with those pre-integrated.</p><p>Our code can run locally (see the 1st and 2nd steps in the notebook), to run the distributed training on a cluster with GPUs we simply define our function as an MPIJob kind (one which uses MPI and Horovod to distribute our TensorFlow logic), we specify a link to the code, a container image (alternatively MLRun can build the image for us), the required number of containers, number of GPUs (per container), and attach it to a file mount (we apply<a href="https://www.iguazio.com/"> iguazio</a> low-latency v3io fabric mount, but other K8s shared file volume drivers or object storage solutions work just as well).</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/39b7a9258e0ed619482768655aacadbe/href">https://medium.com/media/39b7a9258e0ed619482768655aacadbe/href</a></iframe><p>once we defined the function object all we need to do is run it with a bunch of parameters, inputs (datasets or files), and specify the default location for output artifacts (e.g. trained model files).</p><pre>mprun = trainer.run(name=&#39;train&#39;, params=params, artifact_path=&#39;/User/mlrun/data&#39;, inputs=inputs, watch=<strong>True</strong>)</pre><p>Note that in this example we didn’t need to move code or data, the fact that we used the same low-latency shared file system mounts across our notebook and worker containers means that we can just modify the code in Jupyter and re-run our job (all the job containers will see the new Py file changes), and all the job results will be instantly viewed in the Jupyter file browser or MLRun UI.</p><p>In addition to viewing the job progress interactively (watch=True) the run object (mprun) holds all the information on the run including pointers to the output artifacts, logs, status, etc. We can use MLRun web-based UI to track our job progress, compare experiment results or access versioned artifacts.</p><p>We use “.save()” to serialize and version our function object into a database, we can retrieve this function object later in a different notebook or CI/CD pipelines (no need to copy code and config between notebooks and teams).</p><p>If we want to deploy the generated model as an interactive serverless function all we need is to feed the “model” and “category_map” outputs into a “serving” function and deploy it to our test cluster.</p><p>MLRun orchestrates auto-scaling<a href="https://github.com/nuclio/nuclio"> Nuclio functions</a> which are super-fast and can be stateful (support GPU attachment, shared file mounts, state caching, streaming, etc.), the functions will auto scale to fit the workload and will scale to zero if requests will not arrive for a few minutes (consume zero resources). In this example we use “nuclio-serving” functions (Nuclio functions which host standard <a href="https://github.com/kubeflow/kfserving">KFServing </a>model classes), as you can see below, it only takes one command (deploy) to make it run as a live serverless function.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/845/1*vao52YzSp7-yomIbmjYDsA.png" /></figure><p>Now we have a running inference function and we can test the end point using simple HTTP request with a url or even a binary image in the payload.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/816/1*lSIOZH-npV47YdC3STnrjA.png" /></figure><h3>End to End Workflow with KubeFlow Pipelines</h3><p>Now that we tested each step of our pipeline manually, we may want to automate the process and potentially run it on a given schedule or be triggered by an event (e.g. a Git push). The next step is to define a<a href="https://www.kubeflow.org/docs/pipelines/overview/pipelines-overview/"> KubeFlow Pipeline</a> graph (DAG) which chains the 4 steps into a sequence and run the graph.</p><p><a href="https://github.com/mlrun/mlrun">MLRun</a> functions can be converted into<a href="https://www.kubeflow.org/docs/pipelines/overview/pipelines-overview/"> KubeFlow Pipeline</a> steps using a simple method (.as_step()), and specifying how step outputs are fed into other step inputs, check the full notebook <a href="https://github.com/mlrun/demo-image-classification/blob/master/load_project.ipynb">example here</a>, the following code demonstrates the graph DSL.</p><p>MLRun projects can have multiple workflows, and they can be launched with a single command or can be triggered by various events such as a Git push or HTTP REST call.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/38d546e9ce3a466b983cfdec642266c5/href">https://medium.com/media/38d546e9ce3a466b983cfdec642266c5/href</a></iframe><p>Once we run our pipeline we can track its progress using KubeFlow, MLRun will automatically register metrics, inputs, outputs and artifacts in KubeFlow UI without writing a single extra line of code (I guess you should try doing it first without MLRun to appreciate it 😊).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/870/1*fKBGHxJTq7Nik8C-TvcdGg.png" /><figcaption>KubeFlow Output (image by author)</figcaption></figure><p>For a more basic project example you can see the MLRun <a href="https://github.com/mlrun/demo-xgb-project">Iris XGBoost Project</a> , other demos can be found in <a href="https://github.com/mlrun/demos">MLRun Demos repository</a>, and you can check<a href="https://github.com/mlrun/mlrun/blob/master/README.md"> MLRun readme</a> and examples for tutorials and simple examples.</p><h3>Summary</h3><p>This article demonstrates how computational resources can be used efficiently to run data science jobs at scale, but more importantly, I demonstrated how data science development can be simplified and automated, allowing far greater productivity and faster time to market. Ping me on KubeFlow Slack if you have additional questions.</p><p>Check out my GPU As A Service presentation and live demo from KubeCon NA in November.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FjSuecrUr2LA%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DjSuecrUr2LA&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FjSuecrUr2LA%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/f43ce26097efd18ee2c0a062125f4d98/href">https://medium.com/media/f43ce26097efd18ee2c0a062125f4d98/href</a></iframe><p>Yaron</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c5783b95d192" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/gpu-as-a-service-on-kubeflow-fast-scalable-and-efficient-ml-c5783b95d192">GPU-as-a-Service on KubeFlow: Fast, Scalable and Efficient ML</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ML Ops Challenges, Solutions and Future Trends]]></title>
            <link>https://medium.com/data-science/ml-ops-challenges-solutions-and-future-trends-d2e59b74dc6b?source=rss-bdd9246d060d------2</link>
            <guid isPermaLink="false">https://medium.com/p/d2e59b74dc6b</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[serverless]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[analytics]]></category>
            <dc:creator><![CDATA[yaron haviv]]></dc:creator>
            <pubDate>Tue, 07 Jan 2020 14:29:00 GMT</pubDate>
            <atom:updated>2020-01-07T14:29:00.869Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*viQFL6lQYFHBye6hmDAmeg.jpeg" /></figure><h4>Summary of my MLOps NYC talk, major AI/ML &amp; Data challenges and how they will be solved with emerging open source technologies</h4><p>AI and ML practices are no longer the luxury of research institutes or technology giants, they are becoming an integral part of any modern business application. According to analysts, most organizations fail to successfully deliver AI-based applications and are stuck in the process of turning data-science models, which were tested on sample or historical data, into interactive applications which work with real-world and large-scale data.</p><p>A new engineering practice called MLOps has emerged to address the challenges. As the name indicates, it combines AI/ML practices with DevOps practices, and its goal is to create continuous development and delivery (CI/CD) of data and ML intensive applications.</p><p>Recently <a href="https://www.iguazio.com/">Iguazio</a>, the provider of an end-to-end data science platform and developer of open-source MLOps technologies, together with the major cloud providers, leading enterprises and technology giants, held the <a href="https://www.mlopsnyc.com/">MLOps NYC</a> event. The agenda was to discuss different approaches, best practices, and start creating collaboration and standardization for MLOps.</p><p>My session summarized the current challenges of MLOps and the trends we will be seeing in the near future (see the 12 minute <a href="https://www.youtube.com/watch?v=kQto7JYq0qg">video</a>).</p><h3>Challenges</h3><p>According to different surveys, data-science teams don’t do data science, they spend most of their time on data wrangling, data preparation, managing software packages and frameworks, configuring infrastructure, and integrating various components. Those can be generalized as feature management tasks and MLOps tasks (i.e. DevOps for ML).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/838/1*wGQ57P2WgcS_NU81QrWzuw.png" /></figure><h3>The MLOps Challenge</h3><p>Data-science originated in research organizations and was later used to produce reports and detect anomalies within mountains of data. The emerging trend to incorporate data-science in every business application, in order to intelligently react to events and data as they occur, is creating fundamental changes in machine learning practices.</p><p>Unlike research or postmortem data analysis, business applications need to be able to handle real-time and constantly changing data, they must be on 24/7, they must meet decent response times, support a large number of users, etc. What once was the goal — producing an ML model — today is just the first step in a very long process of bringing data science to production (as illustrated in the following diagram). Many organizations underestimate the amount of effort it takes to incorporate machine learning into production applications. This leads to entire projects being abandoning halfway (75% of data science projects never make it to production) or to the consumption of far more resources and time than first anticipated.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/921/1*LuSoEIs95nMTur67SNORQw.png" /></figure><h3>The Data Challenge</h3><p>Data scientists start with sample data, they work on Jupyter notebooks or use AutoML tools to identify patterns and train models. At a certain point, they need to train the models on larger data sets. This is when things start to become difficult. They might find that most of the tools which work off CSV files and load data into memory just can’t work at scale, and that they need to re-architect everything to fit distributed platforms and structured databases.</p><p>Much time is spent creating features from raw data, and in many cases the same feature extraction work is duplicated for multiple projects, or by different teams. The overhead is further amplified every time datasets change, the derived data and models changes, or experiments need to be repeated to get the required accuracy.</p><p>When the data science team tries to deploy models into production, they find that the real-world data is different, and that they can’t use the same data preparation methodologies on data that is always changing. Latency or computation power constraints require a fundamentally different data pipeline, one which depends on stream processing, fast key/value and time-series databases to deliver real-time feature vectors.</p><p>At MLOps NYC, Uber, Twitter, and Netflix discussed and shared their experience of building online and offline feature stores which are a fundamental component in their data-science platforms.</p><h3>Solutions and Future Trends</h3><p>In my session I outline the industry’s vision to overcoming the challenges outlined above. The way to solve these challenges is through:</p><ol><li>Use of scalable and production ready data-science platforms and practices from day-one.</li><li>Adoption of automation and higher-level abstractions where possible.</li><li>Design for collaboration and re-use.</li></ol><h3>Serverless ML Functions</h3><p>The way we eliminate ML pipelines complexity is by adopting the concept of serverless “ML Functions”. Serverless technologies allow you to write code and specification which automagically translates itself to auto-scaling production workloads. Until recently, these were limited to stateless and event driver workloads, but now with the new <a href="https://github.com/mlrun/mlrun">open-source technologies we demonstrated (MLRun+Nuclio+KubeFlow)</a>, Serverless functions can take on larger challenges of real-time, extreme scale data-analytics and machine learning.</p><p>The steps of packaging, scaling, tuning, instrumentation, and continuous delivery are fully automated, addressing the two main challenges of every organization:</p><ol><li>Significantly reduce time to market</li><li>Minimize the amount of resources and skill level needed to complete the project</li></ol><p>ML Functions can easily be chained to produce ML pipelines (using <a href="https://www.kubeflow.org/">KubeFlow</a>). They can generate data and features which will be used by subsequent stages. The following diagram demonstrates the pipeline used to create a real-time recommendation engine application:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/918/1*BRSFuilQRtzdrnn1gTslkQ.png" /></figure><p>Transitioning to micro-services and functional programming models enables collaboration and code re-use. Users can gradually extend and tune functions without breaking their pipelines, while only consuming the right amount of CPU, GPU, and memory resources. <a href="https://kubernetes.io/">Kubernetes</a> and <a href="https://www.kubeflow.org/">KubeFlow</a> play a central role in this architecture, scheduling the right resources, scaling out the workloads, and managing pipelines.</p><p>Orit Nissan-Messing, <a href="https://www.iguazio.com/">Iguazio’s</a> Chief Architect &amp; Co-Founder, presented a session describing the Nuclio ML Functions and MLRun architecture with a live demo at KubeCon + CloudNativeCon 2019 in San Diego (watch her <a href="https://www.youtube.com/watch?v=GmCpdRRFmxw">video</a> to learn more about this challenge and ways to overcome it).</p><h3>Built-in Feature Stores</h3><p>The second challenge is the complexity in building, managing, and consuming offline and online features. Digital giants like Uber, Netflix and others have all built “Feature Stores” internally to overcome this. Most organizations can’t afford to or don’t have the skills internally to build a feature store from scratch and need it to be an integral part of the data platform they use.</p><p>We can build Feature Stores using ML Functions which are connected to a shared online + offline data repository and wrap it up with metadata management and automation as shown in the following diagram.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/584/1*zGkBIpg_JHpeicL5OVLOBw.png" /></figure><p>Summary</p><p>Organizations which plan to incorporate ML and AI in their applications must start with the end in mind and build for production by adopting MLOps, i.e. continuous integration and deployment (CI/CD) and DevOps practices for their data-science activities, this way they gain agility and can serve real-world online application.</p><p>MLOps and DataOps can be a resource drain and can lead to significant delays without proper abstractions and automation, those challenges will lead to a rise in data-science optimized “Serverless”, SaaS offering, ML function marketplaces, and managed feature stores.</p><p>It is important to bet on the right technology, one which is open and uses Kubernetes and its vast eco-system vs. point or cloud specific solutions.</p><p>For my 12 minute video:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FkQto7JYq0qg%3Ffeature%3Doembed&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DkQto7JYq0qg&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FkQto7JYq0qg%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/815cf661580220a7660d4430a5c8e6ae/href">https://medium.com/media/815cf661580220a7660d4430a5c8e6ae/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d2e59b74dc6b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/ml-ops-challenges-solutions-and-future-trends-d2e59b74dc6b">ML Ops Challenges, Solutions and Future Trends</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Python Pandas at Extreme Performance]]></title>
            <link>https://medium.com/data-science/python-pandas-at-extreme-performance-912912b1047c?source=rss-bdd9246d060d------2</link>
            <guid isPermaLink="false">https://medium.com/p/912912b1047c</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[serverless]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[yaron haviv]]></dc:creator>
            <pubDate>Thu, 08 Aug 2019 17:36:01 GMT</pubDate>
            <atom:updated>2019-08-09T03:43:46.501Z</atom:updated>
            <content:encoded><![CDATA[<p>Today we all choose between the simplicity of <a href="https://www.python.org/">Python</a> tools (pandas, <a href="https://scikit-learn.org/">Scikit-learn</a>), the scalability of <a href="https://spark.apache.org/">Spark</a> and <a href="https://hadoop.apache.org/">Hadoop</a>, and the operation readiness of <a href="https://kubernetes.io/">Kubernetes</a>. We end up using them all. We keep separate teams of Python-oriented data scientists, Java and Scala Spark masters, and an army of devops to manage those siloed solutions.</p><p>Data scientists explore with <a href="https://pandas.pydata.org/">pandas</a>. Then other teams of data engineers re-code the same logic and make it work at scale, or make it work with live streams using Spark. We go through that iteration again and again when a data scientist needs to change the logic or use a different data set for his/her model.</p><p>In addition to taking care of the business logic, we build clusters on Hadoop or Kubernetes or even both and manage them manually along with an entire CI/CD pipeline. The bottom line is that we’re all working hard, without enough business impact to show for it…</p><p>What if you could write simple code in Python and run it faster than using Spark, without requiring any re-coding, and without devops overhead to address deployment, scaling, and monitoring?</p><p>I guess you may “say I’m a dreamer” 😊. Well I am a dreamer, but I’m not the only one! This post will prove it is possible today using by <a href="https://nuclio.io/">Nuclio</a> and <a href="https://rapids.ai/">RAPIDS</a>, a free, open- source data science acceleration platform incubated by NVIDIA.</p><p>We’ve been collaborating closely with NVIDIA in the last few months to integrate <a href="https://rapids.ai/">RAPIDS</a> with the <a href="https://nuclio.io/">Nuclio</a> open source serverless project and <a href="https://www.iguazio.com/">Iguazio’s</a> PaaS. It’s now possible to deliver magnitudes faster data processing speeds and scalability with the SAME Python code, and with minimal operational overhead due to the serverless approach.</p><p>I will demonstrate the same widely popular use case of crunching live data comprised of Json-based logs. I’ll perform some analytics tasks on it and dump the aggregated results to a compressed Parquet format for further queries or ML training. We will look at batch and real-time streaming (yes, real-time streaming with simple Python code). But first, an overview.</p><h3>What Makes Python Slow and Not Scalable?</h3><p>Performance is decent when working with pandas on a small dataset, but that’s if the entire dataset fits in memory and processing is done with optimized C code under the pandas and NumPy layer. Processing lots of data involves intensive IO operation, data transformations, data copies, etc. which are slow. Python is synchronous by nature due to the infamous GIL and is very inefficient when dealing with complex tasks, asynchronous Python is better but complicate development and doesn’t solve the inherent locking issues.</p><p>Frameworks like <a href="https://spark.apache.org/">Spark</a> have the advantage of asynchronous engines (<a href="https://akka.io/">Akka</a>) and memory optimized data layouts. They can distribute work to multiple workers in different machines, leading to better performance and scalability, making them the de-facto standard.</p><h3>RAPIDS Puts Python on Steroids</h3><p>Our friends at NVIDIA came up with a brilliant idea: keeping Python-facing APIs of popular frameworks like pandas, Scikit-learn and <a href="https://xgboost.readthedocs.io/en/latest/">XGBoost</a> but processing the data in high-performance C code in the GPU. They adopted the memory friendly Apache <a href="https://arrow.apache.org/">Arrow data format</a> in order to speed up data transfer and manipulation.</p><p><a href="https://developer.nvidia.com/rapids">RAPIDS</a> supports data IO (cuIO), data analytics (<a href="https://github.com/rapidsai/cudf">cuDF</a>) and machine learning (<a href="https://github.com/rapidsai/cuml">cuML</a>). These different components share the same memory structures, so a pipeline of data ingest, analytics, and ML essentially runs without copying data back and forth into the CPU.</p><p>The following example demonstrates a use case of reading a large Json file (1.2GB), aggregating the data with pandas API. We can see how the SAME code runs <strong>30 times faster</strong> with RAPIDS (<a href="https://github.com/nuclio/rapids/blob/master/demo/benchmark_cudf_vs_pd.ipynb">see the full notebook here</a>), if we compare the computation without IO it is 100 times faster, meaning we have room to do much more complex computation on the data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/792/1*aVklEIGhdxxU-1buEymCmw.png" /></figure><p>We used a single GPU (<a href="https://www.nvidia.com/en-us/data-center/tesla-t4/">NVIDIA T4</a>) which adds about 30% to the price of the server and got 30X faster performance. We just processed 1 gigabyte of complex data per second using a few lines of Python code. WOW!!</p><p>If we pack this code inside a serverless function it can run every time user request or periodically, and read or write to dynamically attached data volumes.</p><h3>Is Real-Time Streaming Possible with Python?</h3><p>Did you try performing real-time streaming with Python? Well, we did. The following code taken from the Kafka best practice guides reads from a stream and does minimal processing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/598/1*kdQSX_JQSozYpr7PivcB7w.png" /></figure><p>The problem is that Python is synchronous by nature and rather inefficient when it comes to real-time or complex data manipulation. This program only generates a throughput of few thousand messages per second and that’s without doing any interesting work. When we add the json and pandas processing used in our previous example (<a href="https://github.com/nuclio/rapids/blob/master/demo/python-agg.ipynb">see notebook</a>) the performance is further degraded and we only process 18MB/s. So, do we need to go back to Spark for our stream processing?</p><p>No, wait.</p><p>The fastest serverless framework is Nuclio, which is now also part of <a href="https://www.kubeflow.org/">Kubeflow</a> (Kubernetes ML framework). Nuclio runs various code languages wrapped by a real-time and highly concurrent execution engine. Nuclio runs multiple instances of code in parallel (with efficient micro-threads) without extra coding. Nuclio handles auto-scaling within the process and across multiple processes/containers (<a href="https://theburningmonk.com/2019/04/comparing-nuclio-and-aws-lambda/">see this tech coverage blog</a>).</p><p>Nuclio handles stream processing and data access in highly optimized binary code and invokes functions through a simple function handler. It supports 14 different triggering or streaming protocols (including HTTP, <a href="https://kafka.apache.org/">Kafka</a>, <a href="https://aws.amazon.com/kinesis/data-analytics/">Kinesis</a>, Cron, batch) which are specified through configuration (without changing the code), and it support fast access to external data volumes. A single Nuclio function can process hundreds of thousands of messages per second and more than a gigabyte/sec throughput.</p><p>Most importantly though, Nuclio is the only serverless framework today with optimized NVIDIA GPU support. It knows how to make sure GPU utilization is maximized and scales out to more processes and GPUs if needed.</p><p>I’m obviously biased here, but it’s also all true.</p><h3>30x Faster Stream Processing Without Devops</h3><p>Let’s combine Nuclio + RAPIDS to get the full nirvana of GPU-accelerated, Python-based stream processing. The following code is not so different than the batch processing case, we just placed it in a function handler and collected incoming messages into larger batches to make fewer GPU calls (<a href="https://github.com/nuclio/rapids/blob/master/demo/nuclio-cudf-agg.ipynb">see the full notebook</a>).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*j-TJOzPo4-CIZMzwpomBpQ.png" /></figure><p>We can test the same function with an HTTP or Kafka trigger : in both cases Nuclio will handle the parallelism and divide the stream partitions to multiple workers without any extra development work on our side. The setup we tested is using a 3 node Kafka cluster and a <strong>single</strong> Nuclio function process (on a dual socket Intel server with one <a href="https://www.nvidia.com/en-us/data-center/tesla-t4/">NVIDIA T4</a>). We managed to process 638 MB/s, that is 30 times faster than writing your own Python Kafka client, and it auto-scales to handle any amounts of traffic. And it used small and trivial Python code!!</p><p>In our test we noticed the GPU was underutilized, meaning we can make more complex computation on the data (joins, ML predictions, transformations, etc.) while maintaining the same performance levels.</p><p>Ok, so we got faster performance with less development, but the true benefit of serverless solutions is in the fact that they’re “serverless” (<a href="https://towardsdatascience.com/serverless-can-it-simplify-data-science-projects-8821369bf3bd">read more in my post</a>). Take the same code, develop it in a notebook (<a href="https://github.com/nuclio/rapids/blob/master/demo/nuclio-cudf-agg.ipynb">see this notebook example</a>) or your favorite IDE, and in one command it is built, containerized and shipped to a <a href="https://kubernetes.io/">Kubernetes</a> cluster with full instrumentation (logs, monitoring, auto-scaling, …) and security hardening.</p><p>Nuclio integrates with Kubeflow Pipelines. Build multi-stage data or ML pipelines, with minimal effort you automate your data science workflow, collect execution and artifact metadata allowing to easily reproduce experiment results.</p><p>Download Nuclio <a href="https://nuclio.io/">here</a> and deploy it on your Kubernetes (<a href="https://github.com/nuclio/rapids">see RAPIDS examples</a>).</p><p>Check out this related article <a href="https://towardsdatascience.com/life-after-hadoop-getting-data-science-to-work-for-your-business-c9ab6605733f">Life After Hadoop: Getting Data Science to Work for Your Business</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=912912b1047c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/python-pandas-at-extreme-performance-912912b1047c">Python Pandas at Extreme Performance</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why is it So Hard to Integrate Machine Learning into Real Business Applications?]]></title>
            <link>https://medium.com/data-science/why-is-it-so-hard-to-integrate-machine-learning-into-real-business-applications-69603402116a?source=rss-bdd9246d060d------2</link>
            <guid isPermaLink="false">https://medium.com/p/69603402116a</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[serverless]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[yaron haviv]]></dc:creator>
            <pubDate>Mon, 08 Jul 2019 20:45:27 GMT</pubDate>
            <atom:updated>2019-07-08T20:45:27.105Z</atom:updated>
            <content:encoded><![CDATA[<p>You’ve played around with machine learning, learned about the mysteries of neural networks, almost won a <a href="https://www.kaggle.com/">Kaggle </a>competition and now you feel ready to bring all this to real world impact. It’s time to build some real AI-based applications.</p><p>But time and again you face setbacks and you’re not alone. It takes time and effort to move from a decent machine learning model to the next level of incorporating it into a live business application. Why?</p><p>Having a trained machine learning model is just the starting point. There are many other considerations and components that needs to be built, tested and deployed for a functioning application.</p><p>In the following post I will present a real AI-based application (based on a real customer use case), explain the challenges and suggest ways to simplify development and deployment.</p><h3>Use Case: Online Product Recommendations</h3><p>Targeted product recommendations is one of the most common methods to increase revenue, computers make suggestions based on users’ historical preferences, product to product correlations and other factors like location (e.g. proximity to a store), weather and more.</p><p>Building such solutions requires analyzing historical transactions and creating a model. Then when applying it to production you’ll want to incorporate fresh data such as the last transactions the customer made and re-train the model for accurate results.</p><p>Machine learning models are rarely trained over raw data. Data preparation is required to form feature vectors which aggregate and combine various data sources into more meaningful datasets and identify a clear pattern. Once the data is prepared, we use one or more machine learning algorithms, conduct training and create models or new datasets which incorporate the learnings.</p><p>For recommendation engines, it is best to incorporate both deep learning (e.g. <a href="https://www.tensorflow.org/">TensorFlow</a>) to identify which products are bought “together”, and machine learning (e.g. <a href="https://xgboost.readthedocs.io/en/latest/">XGboost</a>) to identify the relations between users and products based on their historical behavior. The results from both models are then combined into a single model serving application.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/909/1*fzvFA7C6Frd0ZO8o062j3w.png" /><figcaption>Example pipeline: Real-time product recommendations</figcaption></figure><p>The serving application accepts a user’s ID, brings additional context from feature and user tables, feeds it into a model and returns a set of product recommendations.</p><p>Note that serving must be done in real-time while the user is still browsing in the application, so its always better to cache data and models. On the other hand, recent product purchases or locations may have significant impact on future customer product choices and you need to constantly monitor activities and update feature tables and models.</p><p>An online business requires automation and a CI/CD process applied into machine learning operations, enabling continuous applications. It is important to support auto-scaling and meet demand fluctuations, sustain failures and provide data security, not to mention to take regulatory constraints into consideration.</p><h3>The Machine Learning Operational Flow</h3><p>In a typical development flow, developing code or models is just the first step. The biggest effort goes on making each element, including data collection, preparation, training, and serving production-ready, enabling them to run repeatedly with minimal user intervention.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/921/1*mmDPLBSnSO9AYksUOJT_7A.png" /><figcaption>What it takes to turn code or algorithms into real application</figcaption></figure><p>The data science and engineering team is required to package the code, address scalability, tune for performance, instrument and automate. These tasks take months today. Serverless helps reduce effort significantly by automating many of the above steps, as explained in my previous post Serverless: <a href="https://www.iguazio.com/serverless-can-it-simplify-data-science-projects/">Can is Simplify Data Science Projects?</a>. Other important tools to keep in mind are <a href="https://kubernetes.io/">Kubernetes</a> and <a href="https://www.kubeflow.org/">KubeFlow</a>, which bring CI/CD and openness to the machine learning world. Read more about them in my post <a href="https://www.iguazio.com/kubernetes-the-open-scalable-approach-to-ml-pipelines/">Kubernetes: The Open and Scalable Approach to ML Pipelines</a>.</p><h3>Machine Learning Code Portability and Reproducibility</h3><p>A key challenge is that the same code may run in different environments, including notebooks for experimentation, IDEs (e.g. <a href="https://www.jetbrains.com/pycharm/">PyCharm</a>) and containers for running on a cluster or as part of an automated ML workflow engine. In each environment you might have different configurations and use different parameters, inputs or output datasets. A lot of work is spent on moving and changing code, sometimes by different people.</p><p>Once you run your work, you want to be able to quickly visualize results, compare them with past results and understand which data was used to produce each model. There are vendor specific solutions for these needs, but you can’t use them if you want to achieve portability across environments.</p><p><a href="https://www.iguazio.com/kubernetes-the-open-scalable-approach-to-ml-pipelines/">Iguazio</a> works with leading companies to form a cross platform standard and open implementation for machine learning environments, metadata and artifacts. This allows greater simplicity, automation and portability.</p><p>Check out this video to learn how you can move from running/testing code in a local IDE to a production grade automated machine learning pipeline in less than a minute (based on KubeFlow).</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FPiD7_MVjpsA%3Ffeature%3Doembed&amp;url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DPiD7_MVjpsA&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FPiD7_MVjpsA%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/8c7afb72107895b27fb9ba49498a332e/href">https://medium.com/media/8c7afb72107895b27fb9ba49498a332e/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=69603402116a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/why-is-it-so-hard-to-integrate-machine-learning-into-real-business-applications-69603402116a">Why is it So Hard to Integrate Machine Learning into Real Business Applications?</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Kubernetes, The Open and Scalable Approach to ML Pipelines]]></title>
            <link>https://medium.com/data-science/kubernetes-the-open-and-scalable-approach-to-ml-pipelines-1e972a965f0d?source=rss-bdd9246d060d------2</link>
            <guid isPermaLink="false">https://medium.com/p/1e972a965f0d</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[serverless]]></category>
            <dc:creator><![CDATA[yaron haviv]]></dc:creator>
            <pubDate>Wed, 15 May 2019 01:52:06 GMT</pubDate>
            <atom:updated>2019-05-15T01:52:06.606Z</atom:updated>
            <content:encoded><![CDATA[<p>Still waiting for ML training to be over? Tired of running experiments manually? Not sure how to reproduce results? Wasting too much of your time on devops and data wrangling?</p><p>It’s okay if you’re a hobbyist, but data science models are meant to be incorporated into real business applications. Businesses won’t invest in data science if they don’t see a positive ROI. This calls for the adoption of an “engineered” approach — otherwise it is no more than a glorified science project with data.</p><p>Engineers use microservices and automated CI/CD (<a href="https://www.infoworld.com/article/3271126/what-is-cicd-continuous-integration-and-continuous-delivery-explained.html">continuous integration and deployment</a>) in modern agile development. You write code, push and it gets tested automatically on some cluster at scale. It then goes into some type of beta/<a href="https://www.quora.com/What-is-Canary-testing">Canary</a> testing phase if it passes the test, and onto production from there. <a href="https://kubernetes.io/">Kubernetes</a>, a cloud-native cluster orchestrator, is the tool now widely used by developers and devops to build an agile application delivery environment.</p><p>Leading ML engineers and AI/ML-driven companies are already using Kubernetes, which comes with pre-integrated tools and scalable frameworks for data processing, machine learning and model deployment. Some Kubernetes frameworks also enable horizontal scaling and efficient use of GPUs, which further cut down overall wait times and costs.</p><p>Don’t settle for proprietary SaaS/cloud solutions or legacy architectures like <a href="https://hadoop.apache.org/">Hadoop</a>. Get a Kubernetes-based solution for a scalable and forward-looking data science platform. The following post reviews best practices for the process and Kubernetes frameworks used to scale and operationalize data science endeavors and provides detailed information about the <a href="https://www.kubeflow.org/docs/started/">KubeFlow</a> ML toolkit for Kubernetes (K8s).</p><h3>The Data Science Pipeline</h3><p>Data scientists mostly go through the following steps during a development phase:</p><ol><li>Data collection from CSV or dense formats (e.g. <a href="https://parquet.apache.org/">Parquet</a>) which entail manual extraction from external sources.</li><li>Manual data labeling, exploration and enrichment to identify potential patterns and features.</li><li>Model training and validation.</li><li>Model testing with real/fresh data, or as part of a bigger application.</li><li>Rinse and repeat when unhappy with the outcome.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lIUt-5wQoasxmTLCLChkBw.png" /><figcaption><em>Data Science Pipeline and Output Artifacts</em></figcaption></figure><p>Once done with development, against all odds, models are deployed into a production environment where fancy dashboards are on display to identify anomalies or predict behavior. Those of us who are more advanced in our AI journey also create AI-driven user interactions/apps or trigger alerts and actions.</p><p>But then data changes… model accuracy drifts… the users ask for update in apps and dashboard, etc… Now we find ourselves running much of the pipeline again to produce new models. This is where we start Googling a continuous development and integration approach, but more on that later.</p><p>The tools required to build our pipeline can be divided into the following six categories:</p><p>A. Automated data collectors from offline or online sources.</p><p>B. Interactive tools for data labeling, exploration and model development.</p><p>C. Analytics tools/databases to aggregate, join and pivot data at scale.</p><p>D. Parallel model training frameworks.</p><p>E. Model and API serving frameworks.</p><p>F. Management, monitoring and dashboards.</p><p><strong>The only sane way to integrate all these tools under one platform is with Kubernetes!</strong></p><h3>Kubernetes, The ML Platform of Choice</h3><p>Kubernetes hosts several packaged and pre-integrated data and data science frameworks on the same cluster. These are usually scalable or they auto-scale, and they’re defined/managed with a declarative approach: specify what your requirements are and the service will continuously seek to satisfy them, which provides resiliency and minimizes manual intervention.</p><p><a href="https://www.kubeflow.org/">KubeFlow</a> is an open source project that groups leading relevant K8 frameworks. KubeFlow components include <a href="https://jupyter.org/">Jupyter</a> notebooks, KubeFlow <a href="https://www.kubeflow.org/docs/pipelines/pipelines-quickstart/">Pipeline</a> (workflow and experiment management), scalable training services (for <a href="https://www.tensorflow.org/">TensorFlow</a>, <a href="https://pytorch.org/">PyTourch</a>, <a href="https://eng.uber.com/horovod/">Horovod</a>, <a href="https://mxnet.apache.org/">MXNet</a>, <a href="https://chainer.org/">Chainer</a>), model serving solutions, and Iguazio’s serverless framework <a href="https://nuclio.io/">(Nuclio</a>. KubeFlow also offers <a href="https://www.kubeflow.org/docs/examples/resources/">examples</a> and pre-integrated/tested components.</p><p>In addition to typical data science tools, Kubernetes can host data analytics tools such as <a href="https://spark.apache.org/">Spark</a> or <a href="https://prestosql.io/">Presto</a>, various databases and monitoring/logging solutions such as <a href="https://prometheus.io/">Prometheus</a>, <a href="https://grafana.com/">Grafana</a>, <a href="https://www.elastic.co/">Elastic Search</a>, etc. It also enables the use of serverless functions (i.e. auto built/deployed/scaled code like AWS <a href="https://aws.amazon.com/lambda/">Lambda</a>) for a variety of data-related tasks or APIs/model serving, read more on that in my post (<a href="https://towardsdatascience.com/serverless-can-it-simplify-data-science-projects-8821369bf3bd">Serverless: Can It Simplify Data Science Projects?</a>).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KIaURlja8EiFi7g2MYFzXg.png" /><figcaption>Using Kubernetes for Data-Science and Analytics</figcaption></figure><p>The key advantage of Kubernetes vs proprietary cloud or SaaS solutions is that its tools are regularly added and upgraded, Google’s search and Stack overflow are often the fastest path to help and the solution can be deployed everywhere (any cloud or even on-prem or on a laptop). A community project also forces associated components/services to conform to a set of standards/abstractions which simplify interoperability, security, monitoring, etc.</p><p>You can build and manage a Kubernetes data science platform yourself (<a href="https://www.kubeflow.org/docs/started/">follow the doc</a>), or use managed and pre-integrated solutions from cloud providers or companies like Iguazio.</p><h3>Building an Efficient Data Science Pipeline</h3><p>Unfortunately, just betting on a credible platform is not enough. Believe me, I speak from experience: Life gets easier once you adopt three guiding rules:</p><ol><li><strong>Make it functional</strong> — create reusable abstract functions/steps which can accept parameters.</li><li><strong>Make it scalable</strong> — apply parallelism to every step (or as often as possible, within reason).</li><li><strong>Automate </strong>— avoid manual and repetitive tasks by using declarative semantics and workflows.</li></ol><p>The current trend in data science is to build “ML factories,” i.e. much like in agile software development, build automated pipelines which take data, pre-process it, run training, generate, deploy and monitor the models. The declarative and automated deployment and scaling approach offered by Kubernetes is a great baseline, but it’s missing a way to manage such pipelines on top of that.</p><p>A relatively new tool which is part of the KubeFlow project is <a href="https://www.kubeflow.org/docs/pipelines/pipelines-overview/">Pipelines</a>, a set of services and UI aimed at creating and managing ML pipelines. We can write our own code or build from a large set of pre-defined components and algorithms contributed by companies like Google, Amazon, Microsoft, IBM, NVIDIA, <a href="https://www.iguazio.com/">Iguazio</a>, etc.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/536/1*JydT3LRDJqOg-HYeDMA02Q.png" /><figcaption>Kubeflow Piplines UI</figcaption></figure><p>Once we have a workflow, we can run it once, at scheduled intervals, or trigger it automatically. The pipelines, experiments and runs are managed, and their results are stored and versioned. Pipelines solve the major problem of reproducing and explaining our ML models. It also means we can visually compare between runs and store versioned input and output artifacts in various object/file repositories.</p><p>A major challenge is always running experiments and data processing at scale. Pipelines orchestrate various horizontal-scaling and GPU-accelerated data and ML frameworks. A single logical pipeline step may run on a dozen parallel instances of TensorFlow, Spark, or <a href="https://github.com/nuclio/nuclio">Nuclio functions</a>. Pipelines also have components which map to existing cloud services, so that we can submit a logical task which may run on a managed <a href="https://cloud.google.com/products/ai/">Google AI and data service</a>, or on Amazon’s <a href="https://docs.aws.amazon.com/sagemaker/index.html">SageMaker</a> or <a href="https://aws.amazon.com/emr/">EMR</a>.</p><p>KubeFlow and its Pipelines, like most tools in this category, are still evolving, but it has a large and vibrant multi-vendor community behind it. This guarantees a viable and open framework. Much like the first days of Kubernetes, cloud providers and software vendors had their proprietary solutions for managing containers, and over time they’ve all given way to the open source standard demanded by the community.</p><p>If you happen to be around at KubeCon in Barcelona next week, visit the KubeFlow sessions and demos various members are presenting, including myself. I’ll be at the Serverless Practitioners Summit May 20th (Fira Gran Via Hall 8.0 Room D4 + CC8.17–20) and with Google’s Maulin Patel on May 23rd in the afternoon <a href="https://kccnceu19.sched.com/event/MPaf/economics-and-best-practices-of-running-aiml-workloads-on-kubernetes-maulin-patel-google-yaron-haviv-iguazio">https://kccnceu19.sched.com/event/MPaf/economics-and-best-practices-of-running-aiml-workloads-on-kubernetes-maulin-patel-google-yaron-haviv-iguazio</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1e972a965f0d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/kubernetes-the-open-and-scalable-approach-to-ml-pipelines-1e972a965f0d">Kubernetes, The Open and Scalable Approach to ML Pipelines</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Serverless: Can It Simplify Data Science Projects?]]></title>
            <link>https://medium.com/data-science/serverless-can-it-simplify-data-science-projects-8821369bf3bd?source=rss-bdd9246d060d------2</link>
            <guid isPermaLink="false">https://medium.com/p/8821369bf3bd</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[aws-lambda]]></category>
            <category><![CDATA[serverless]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[yaron haviv]]></dc:creator>
            <pubDate>Wed, 24 Apr 2019 23:25:17 GMT</pubDate>
            <atom:updated>2019-04-24T23:25:17.296Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XuFiQx_VwYkOitS25494LQ.png" /></figure><p>How much time do you think data scientists spend on developing and testing models? Only 10%, according to this <a href="https://medium.com/thelaunchpad/the-ml-surprise-f54706361a6c">post</a> by Google’s Josh Cogan. Most of their time is spent on data collection, building infrastructure, devops and integration. When you finally build a model, how long and complex is the process of delivering it into production (assuming you managed to get that far)? And when you finally incorporate the models into some useful business applications, how do you reproduce or explain its results? Do you monitor its accuracy? And what about continuous application and model upgrades?</p><p>One way to simplify data science development and accelerate time to production is to adopt a serverless architecture for data collection, exploration, model training and serving.</p><p>This post will explain serverless and its limitations, and provide a hands-on example of using serverless to solve data science challenges.</p><h3>Serverless Overview</h3><p>The term “serverless” was coined a few years ago by Amazon to describe its <a href="https://aws.amazon.com/lambda/">Lambda functions</a>, a service where developers write some code and specification and click “deploy.” Lambda automatically builds a containerized application, deploys it on a production cluster, and provides automated monitoring, logging, scaling and rolling upgrades. Other benefits of serverless are a lower cost pay-per-use model and its native integration with platform resources and events.</p><p>Overall, serverless addresses three main data science challenges:</p><p>1. It reduces the overhead and complexity of developing, deploying and monitoring code. Serverless leads to a faster time to production and allows both data scientists and developers to focus on business logic instead of instrumentation and infrastructure-related tasks.</p><p>2. It provides a well-defined and versioned runtime which incorporates the code, software package dependencies, ML models, environment variables, and data sources and run-time configuration. This can guarantee consistent and reproduceable results.</p><p>3. Serverelss allows mobilizing, scaling and duplicating functions to multiple enforcement points. A good example we see at Iguazio relates to developing and testing functions in the cloud and then running them in edge gateways or multiple regions, or just scaling them to meet growing traffic.</p><p>Serverless has gotten a lot of buzz and wide adoption among app and front-end developers. We see it being used opportunistically in some data science pockets, but it has yet to reach wide adoption among data scientists.</p><p>Serverless can address many of the data science challenges outlined above, so I’m surprised that Amazon, the inventor of “serverless,” is not using Lambda in SageMaker, Amazon’s own data science platform. Have you wondered why?</p><p>The reasons are simple! The first wave of serverless functions, such as Amazon Lambda, aren’t a good fit for data science because:</p><ol><li>They are stateless and data science is usually about lots of data (state).</li><li>They are slow and single threaded, while data processing requires parallelism.</li><li>They limit the function image size and ML models tend to be large (larger than Lambda’s maximum image size limit).</li><li>They don’t support GPUs and GPUs can be quite efficient in AI/ML applications.</li><li>There is no native integration between data science tools/frameworks and serverless.</li></ol><p>But all the above limitations are associated with cloud vendor specific serverless implementations, not serverless in general. Other implementations do provide serverless benefits while addressing data science needs.</p><h3>Nuclio, A Serverless Framework Optimized for Data Science</h3><p><a href="https://nuclio.io/">Nuclio</a> was designed from inception as an open source project that addresses data science applications and eliminates the complexity of development and operations. Nuclio provides extreme performance, supports stateful and data intensive workloads, supports and maximizes the efficiency of GPUs, and doesn’t come with any “baggage” holding it back (see: <a href="https://medium.com/theburningmonk-com/comparing-nuclio-and-aws-lambda-66b76d2b631d">Nuclio vs Lambda comparison</a> for details).</p><p>Nuclio runs over the widely adopted <a href="https://kubernetes.io/">Kubernetes</a> orchestration layer and is integrated with various data-science and analytics frameworks.</p><p>Nuclio uses a high-performance concurrent execution engine (supporting up to 400,000 events per process). It is currently the only serverless framework with GPU support and fast file access, which enables accelerating the performance of machine learning code and maximizing the utilization of CPUs and GPUs (using a resource pooling model) so that code runs faster on Nuclio.</p><p>One of Nuclio’s coolest features is its <a href="https://github.com/nuclio/nuclio-jupyter">integration with Jupyter</a>, the most widely used data science workbench. This allows developers to automatically turn data science code and notebooks into scalable real-time functions, which provide data integration, versioning, logging, monitoring, security and out-of-the-box scalability without requiring any additional coding.</p><p>Nuclio is used in wide variety of data science tasks:</p><ol><li>Data collectors, ETL, stream processing</li><li>Data preparation and analysis</li><li>Hyper parameter model training</li><li>Real-time model serving</li><li>Feature vector assembly (real-time data preparation)</li></ol><p>The following example demonstrate how to develop, test, and deploy functions from a notebook. The source, more detailed documentation and variety of examples are available in the <a href="https://github.com/nuclio/nuclio-jupyter">Nuclio-Jupyter git</a>.</p><h3>Example: Implementing a Natural Language Processing Function</h3><p>See the full notebook in <a href="https://github.com/nuclio/nuclio-jupyter/blob/master/docs/nlp-example.ipynb">this Github link</a>.</p><p>The first step is to deploy Nuclio on Docker or Kubernetes. <a href="https://github.com/nuclio/nuclio">See instructions</a> on how, or just use the hosted version.</p><p>Now install the python and Jupyter SDK in your notebook using:</p><pre>!pip install nuclio-jupyter</pre><p>We write and test our code inside a notebook like any other data science code when working with <a href="https://github.com/nuclio/nuclio-jupyter">Nuclio-Jupyter</a>. We add some “magical” commands to describe additional configurations such as which packages to install, how the code will get triggered (http, cron, stream), environment variables, additional files we want to bundle (e.g. ML model, libraries), versioning, etc.</p><p>We use the deploy command or API to automatically convert the notebook to code, and then deploy it in a containerized function. Code cells we want to exclude from the function are marked with a “# nuclio: ignore” comment.</p><p>The following section demonstrates the import of a Nuclio library (in an “ignored” cell), followed by the specification of the packages to install, setting of environment variables and building configuration (e.g. indicating the base Docker image).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/999/1*SKMMwukITqsO4OBJwq-TpQ.png" /></figure><p>The next section is implementing our code which accepts some text from the trigger (e.g. HTTP POST request) and uses the “TextBlob” library to correct, analyze sentiments and respond with the text translated to another language (language is specified by the environment variable).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/997/1*_EC05oDpe8hWM9HdpA1SjA.png" /></figure><p>Once we implemented our code, we test it inside the notebook using a mocked context and event objects. Note that our code was instrumented with logs which are also displayed while debugging.</p><p>The next step is to deploy the function onto our cluster and serve real user requests — using a single “deploy” command. Once the deploy process is complete, it will also provide a URL allocation for the function, which we can then use to test our fresh function.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1001/1*o3k5FpXOIX4uIqxfMAeVeQ.png" /></figure><p>The example demonstrates how to create code in a familiar data science environment. We used a few magic decorations to specify our production requirements (package dependencies, environment and configuration) and we deployed it to a production cluster without thinking about the underlying infrastructure.</p><p>This was a simple example. More advanced function examples can be found <a href="https://github.com/v3io/tutorials/tree/master/demos">in the following repository</a>, including implementations of Twitter and stock data collection and analysis, image recognition and predictive infrastructure monitoring.</p><p>Nuclio is not limited to Python code. It supports 8 coding languages and allows us to choose the right and most familiar tool for the job. <a href="https://github.com/nuclio/nuclio-jupyter/blob/master/docs/nuclio_bash.ipynb">See this example</a> for how to turn a bash script into an interactive serverless function with a few trivial steps and without any knowledge of Docker.</p><h3>Summary</h3><p>Serverless allows us to focus on data science logic! It makes our code production ready. Functions are instrumented with structured logging. They are hardened for security and they provide automated scale-out and scale-up to maximize performance and resource efficiency. Given that functions are containerized artifacts, we can develop and test them in one place and later deploy them to one or more other locations (e.g. on edge devices or regional clusters). More on that in another post.</p><p>Try it out and go to Nuclio’s <a href="https://lit-oasis-83353.herokuapp.com/">community Slack channel</a> if you stumble into any issues.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8821369bf3bd" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/serverless-can-it-simplify-data-science-projects-8821369bf3bd">Serverless: Can It Simplify Data Science Projects?</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>