<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Ro Kommalapati on Medium]]></title>
        <description><![CDATA[Stories by Ro Kommalapati on Medium]]></description>
        <link>https://medium.com/@__ro__?source=rss-06f77cd28371------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*tfKgdzPjEuXmQwjf_3-NfQ.jpeg</url>
            <title>Stories by Ro Kommalapati on Medium</title>
            <link>https://medium.com/@__ro__?source=rss-06f77cd28371------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 19:31:06 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@__ro__/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Streamlining Success with AI Pipelines]]></title>
            <link>https://medium.com/ai-assimilating-intelligence/streamlining-success-with-ai-pipelines-68e6fb51380a?source=rss-06f77cd28371------2</link>
            <guid isPermaLink="false">https://medium.com/p/68e6fb51380a</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[ai-training]]></category>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Ro Kommalapati]]></dc:creator>
            <pubDate>Sat, 09 Mar 2024 23:30:33 GMT</pubDate>
            <atom:updated>2024-03-10T02:35:44.028Z</atom:updated>
            <content:encoded><![CDATA[<h4><a href="https://medium.com/ai-assimilating-intelligence/lucid-learners/home">[Lucid-Learners] A Decoding AI Series</a></h4><h4>Part 1: ML Training Pipelines</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JxvEtdB0O7ETDjX6IGNzig.png" /></figure><p>When starting to work with AI and machine learning, there’s a lot of complex terminology and concepts to learn and unpack. The theory, research, and intuition needed to develop new AI and ML approaches is deep and difficult to learn. But when working on building ML solutions using existing approaches, it can be surprisingly easy to get started and quickly deliver solutions. In practice, most ML systems are straightforward to build and deploy when approached from a pipeline perspective. This series details how to deploy ML systems as 3 simple pipelines to <strong>train</strong>, <strong>serve</strong>, and<em> </em><strong>generate predictions</strong> from models built using your data. When done correctly, we can open the door to fully automated, check-pointed ML pipelines that are robust and continuously deliver value for users (<a href="https://medium.com/ai-assimilating-intelligence/elevating-ml-productivity-the-compounding-returns-of-developer-ergonomics-ab6aa5897119">this post explores why that’s important</a>). The first article goes into detail on the <strong>training pipeline.</strong></p><h4>Requirements</h4><p>Let’s start off with assuming that we have some data stored in a database. The specific kind of database isn’t too important, only that it stores the data that we want to use to train our model on. One common approach is to use a data lake such as Snowflake as the data store.</p><p>In addition to the data, we’ll also assume that we have access to some compute to run the pipeline. The specific compute (CPU vs GPU) is an optimization detail, and something that can be iterated on. We can start with something very easy to set up, like the basic EC2 cloud compute offered on AWS.</p><p>As a rule of thumb, the cost of the data and compute we’re using to train our model should be cheaper than the expected profits generated by our monetized model predictions. Cost, like model performance or training speed, is a metric that we can track and optimize as we iterate. Most of these metrics often trade-off against each other, so it’s useful to measure and optimize them jointly to select the best outcomes for our business use case. Once we have both a data store and compute, then we’re all set to start creating our first pipeline.</p><h4>The Training Pipeline</h4><p>The job of the training pipeline is to generate an effective model from our training data. The training pipeline consists of 4 basic stages, with the following inputs and outputs for the code implementing each stage:</p><ul><li>Feature Fetching: data primary keys -&gt; dataset</li><li>Vectorizing: dataset -&gt; train set, test set</li><li>Training: train set -&gt; model</li><li>Evaluation: test set, model -&gt; performance results</li></ul><h4>Feature Fetching</h4><p>The first stage (feature fetching) is the step where we take a collection of primary keys, each of which uniquely identifies a data point in our data store, and use it to fetch the additional fields we want to use in our dataset. In order to do so, we’ll want to implement a data store client, and authorize it with the credentials needed to fetch the dataset features. Once we’ve done so, we can run the stage as new data is added to the data store to periodically refresh the dataset with more data.</p><p>In the case that some features are missing for data points in the data set, we’ll want to replace (impute) them with meaningful values. When building the dataset, we should also pay close attention to the label data fields, which are needed for supervised learning. When there’s a lack of label fields for data points, there’s typically a downstream process of annotation, where we generate the labels from human input by providing them the existing data point features. We can also generate the label programmatically using the data itself, a common approach called weak supervision.</p><h4>Vectorizing</h4><p>Once we’ve constructed a dataset, the second stage (vectorizing) is where we transform the dataset into a matrix of numbers. Since AI and ML algorithms work numerically, the vectorizing code is needed to turn the raw features into semantically encoded vectors. This step is very important, since many ML algorithms assume the inputs are encoded in a certain way, and it can have a very large impact on model performance. In modern approaches, this encoding is usually done with the use of embedding models, which are pre-trained models that convert data into vectors.</p><p>The resulting matrix is then split into train and test sets. There are a number of algorithms for the splitting process, usually revolving around clever tricks to conserve more data for training without lowering confidence in the downstream evaluation results. It’s important to keep the data between the splits separated to prevent leakage, which can taint the evaluation metrics downstream. In the case of the downstream stages including hyper-parameter tuning, the splitting algorithm should create an additional tuning set for the same reason.</p><h4>Training</h4><p>Once we have the train set, the third stage (training) allows us to use our training algorithm to generate a model. The training algorithm uses the train set to adjust the model parameters (weights and biases in neural networks) during the process of learning. Optionally, there can also be a search over the model hyper-parameters, which are not stored in the model but can impact its performance, using the tuning set. There’s typically a trade-off here between simpler models which have less variance, and larger models which have less bias, and which one to use can depend on the complexity and size of the train set. There’s a lot of existing material covering this step in detail, since it’s usually the focus of AI and ML experiments.</p><h4>Evaluation</h4><p>Finally, we can now run the fourth stage (evaluation), where we use the test set from the second stage to measure the performance of the newly generated model. This step is crucial, since it allows us to know the efficiency of the training algorithm used to build the model. The confidence level of the measured performance results can vary based on the amount and distribution of the data in the test set, which should be calibrated as needed to model real user traffic.</p><p>There are a number of model metrics that can be used to evaluate the model, including some common ones such as accuracy, precision, recall, and F1 score. These metrics can also trade-off against each other, and which one is more important depends on the nature of the problem the model is solving for our users.</p><h4>Conclusion</h4><p>After completing these 4 stages in the training pipeline, we should now have a working model built from our data! The power and flexibility of a training pipeline allows model devs to iterate quickly on experiments, and release incremental improvements on their AI models that grow with their data collection. In the next article, we’ll look at releasing the trained models to users through the <strong>serving pipeline</strong>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=68e6fb51380a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ai-assimilating-intelligence/streamlining-success-with-ai-pipelines-68e6fb51380a">Streamlining Success with AI Pipelines</a> was originally published in <a href="https://medium.com/ai-assimilating-intelligence">AI: Assimilating Intelligence</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>