<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Pavan Pothukuchi on Medium]]></title>
        <description><![CDATA[Stories by Pavan Pothukuchi on Medium]]></description>
        <link>https://medium.com/@pavan.pothukuchi?source=rss-eea17b8308d4------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*xnrJzhdNcTCEwZKZMyOhIQ.jpeg</url>
            <title>Stories by Pavan Pothukuchi on Medium</title>
            <link>https://medium.com/@pavan.pothukuchi?source=rss-eea17b8308d4------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 09:35:38 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@pavan.pothukuchi/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Small ML Models, the nimble sedans of AI]]></title>
            <link>https://medium.com/snowflake/small-ml-models-the-nimble-sedans-of-ai-9b4017be6e70?source=rss-eea17b8308d4------2</link>
            <guid isPermaLink="false">https://medium.com/p/9b4017be6e70</guid>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Pavan Pothukuchi]]></dc:creator>
            <pubDate>Tue, 24 Oct 2023 17:02:59 GMT</pubDate>
            <atom:updated>2023-10-24T21:30:22.562Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Authors</strong>: <a href="https://www.linkedin.com/in/flexwang/"><strong>Flex Wang</strong></a><strong>,</strong> <a href="https://www.linkedin.com/in/micheladar/"><strong>Michel Adar</strong>,</a> <a href="https://www.linkedin.com/in/mona-attariyan-038b6115a/"><strong>Mona Attariyan</strong></a>, <a href="https://www.linkedin.com/in/pavanpothukuchi/"><strong>Pavan Pothukuchi</strong></a></p><figure><img alt="spotlight on a sedan, flanked by two giant robots" src="https://cdn-images-1.medium.com/max/1024/0*zfBWf1TcgmrVI7aU" /><figcaption><em>Source: OpenAI Dall-E 3</em></figcaption></figure><p>Imagine AI as a fleet of cars with Large Language Models (LLMs) being the giant trucks of the bunch. Just like a big, powerful, and expensive truck can carry a heavy load and navigate various terrains, LLMs can handle a wide array of tasks — from summarizing lengthy documents to generating marketing collateral. These trucks are loaded with a vast range of knowledge, ready to roll at your command. All you need to do is give them a “prompt,” like telling a truck driver where to go, and off they go to complete the task. On the other hand, smaller language and ML models are like quick and nimble sedans, designed for specific journeys like detecting sentiment in customer reviews. It’s astounding what these automotive wonders of the AI world can achieve, and just like truck technology keeps advancing, the possibilities for what these models can do are ever-expanding!</p><p>What kind of transportation should you use for real-world business use-cases? In Snowflake’s AI team, we have a unique perspective on this question: we see every model under the sun, and we handle a lot of data. As of April 30, 2023, we handle 3.3 billion queries a day on customer datasets that sometimes exceed 50 trillion rows![1] At that scale, customers must think hard about cost, latency, and ease-of-use tradeoffs between models. It’s like deciding whether you need a heavy-duty truck that can do it all but comes with higher costs and takes a bit longer, or a streamlined sedan that’s faster but limited in its capabilities.</p><p>The purpose of this post is to discuss a few factors to take into consideration when deciding between using a small model vs a large language model. Note that a variety of service providers such as Open AI, Anthropic, and AI21 offer access to their proprietary LLMs of various sizes via APIs. There are also several open source models such as Llama, Falcon, and Mistral that can be securely hosted in <a href="https://www.snowflake.com/blog/container-services-llama2-snowpark-ml/">Snowpark Container Services</a> (currently in private preview). While an exhaustive comparison of each of these options is out of the scope of this post, we picked specific examples in some cases for directional comparison.</p><p><strong>Type of data</strong>: (1) Suppose you’re forecasting sales, predicting customer churn based on product usage patterns, or targeting customers for marketing campaigns based on website clicks. For these use cases related to tabular data (think rows and columns in a datastore), traditional ML models are still your best bet. Generative AI models and LLMs are optimized for extracting insights from predominantly unstructured data like text and images. While recent research shows that LLMs could achieve promising results in certain areas in this space [2] [3], they’re still not performing as well as traditional models, and their inference cost is also higher. Customers with tabular data already in Snowflake can easily train and run machine learning models right in Snowflake using <a href="https://docs.snowflake.com/en/developer-guide/snowpark-ml/index">Snowpark ML</a>.</p><p>(2) For use cases related to analyzing unstructured data (e.g. text, images, audio, videos), you can either use traditional ML models or LLMs depending on cost, performance, and accuracy trade-offs discussed in subsequent sections below. There are use cases like building chatbots, summarizing documents, creating advertising materials, etc., where LLMs offer significant accuracy advantages with very little to no up front work. While there are smaller language models (e.g. <a href="https://huggingface.co/t5-small">T5 small</a> with 60M parameters) for some of these scenarios, it may take a <a href="https://huggingface.co/docs/transformers/tasks/summarization">lot of work</a> to arrive at the desired level of quality. That being said, there are other use cases like sentiment analysis where smaller models offer cost and latency advantages, and offer great performance.</p><p><strong>Latency and Throughput</strong>: Let’s say your use case is to detect the sentiment of a customer service call in real-time and alert a supervisor to step in if there is heated discussion from the customer. In order to address this use case, you may want to define clear end-to-end latency requirements. For instance, you might define that transcribing speech segments, detecting negative sentiment, and generating alerts need to take less than one second. You should then compare your requirements with the latency profile of custom-built smaller models vs. out of the box LLMs to decide what works best to address your needs.</p><p>The larger the model, the more computationally intensive and hence more time it takes to generate prediction results. While there is a lot of R&amp;D going into inference optimization for LLMs and the latency profile will continue to get better, smaller models have a latency advantage. For example, let’s compare the inference latency for <a href="https://arxiv.org/abs/1907.11692">RoBERTa base model</a> with various Llama flavors. RoBERTa is a transformers-based model with 124 million parameters that is popular for fine-tuning on downstream natural language tasks. The inference latency for RoBERTa for an input of 512 tokens on one Nvidia A100 GPU is about 250 milliseconds. In comparison, the latency of a complete function on Llama 7b on the same input in a Hugging Face container is more than 14 seconds, on a p4d.24xlarge instance with 8 Nvidia A100 GPUs. At Snowflake, we have lowered it to about 3 seconds with various inference optimization techniques, but it’s still an order of magnitude higher than the smaller RoBERTa model with a lot more GPU horsepower.</p><p>In addition to latency, understanding your throughput requirements is also important. If you’re running the LLM yourself in a container, you can batch requests together and get higher throughput with the same latency. For example, on a p4d.24xlarge instance, you can batch 360 requests with input size of 512 tokens, and receive the output of similar size in around 15 seconds. If you’re using an LLM API provider, however, you have less control over this. Many of the LLM service providers have limits around the number of prediction requests you can submit per minute and also how many tokens you can submit as part of your prompts per minute. If your application requires more throughout you will need to work with the service providers to increase these limits. For any of the small models you build or deploy on your own, you have more control over the underlying inference infrastructure and you can scale it according to your needs.</p><p><strong>Cost</strong>: Let’s now take a concrete scenario where you want to use ML to <a href="https://medium.com/snowflake/ai-for-ai-by-streamlit-part-i-how-streamlit-empowers-data-augmentation-7b5ab3ee3c0f">categorize incoming email support requests into specific categories</a> (e.g. account related, billing, technical support etc.) to streamline routing and faster processing. There are generally two high level cost considerations for ML workflows. First one is the cost of training — this includes procuring and preparing data, labeling (for supervised learning or fine tuning scenarios), hardware to run training, evaluation, and ongoing updates. Second is the cost of inference — this includes hardware to deploy the model and monitoring. If you are using a pre-trained model that is available as an open source distribution or as an API, the cost for training can range from zero to whatever is needed to fine tune the model, if applicable.</p><p>As LLMs require massive amounts of capital to train, vendors providing APIs recoup this investment as part of their pricing for inference and fine-tuning. For traditional ML models, unless they are pre-trained and available as open source distributions, cost of training can be non-trivial but much lower than what is needed for LLMs. These models can be served through CPUs or general purpose GPUs (vs. specialized GPUs like Nvidia A100s needed for LLMs). For example, a g5.xlarge instance is roughly 1/98th the cost of a p5.48xlarge on AWS.</p><p>Turning back to our email classification example, this falls into the second category of use cases described in the “Type of data” section above given the unstructured nature of the data. Let’s look at the costs for inference using an LLM API. Assuming you get 10K emails a day with an average word count of 500 (roughly 400 tokens), you will need to process 4M tokens/day. That costs you $131/day (or $47K per year) using Anthropic’s Claude 2 and $120/day (or $44K per year) using Open AI GPT-4. If you train a text classification model from scratch using the data you have (emails and labels for classification) and host it on a g5.xlarge instance on AWS, it will cost you about $8K per year (or &lt;$4K per year with a savings plan). You can also host multiple models on the same instance and amortize costs further. Of course, the training, annotation, and ML expertise you need to build such models incurs additional costs, but depending on the number of models you need to build, economics might look very different between these options.</p><p><strong>Accuracy: </strong>For tasks related to tabular data, accuracy of smaller models trained with your own data will likely outperform LLMs. For tasks related to unstructured data, smaller models trained for specific tasks can outperform LLMs depending on the granularity and the uniqueness of the underlying task, and whether additional tabular data is available to train the model. For example, the custom text classification model for categorizing incoming customer support email above can also include additional data like the products the customer purchased, their tenure, recent browsing history, etc. These data points can significantly improve the accuracy of the model. As another example, in a <a href="https://arxiv.org/pdf/2305.15005.pdf">research study</a> related to sentiment analysis tasks, it was found that LLMs demonstrated satisfactory performance in simpler tasks but lagged behind purpose-built smaller models in more complex tasks requiring deeper understanding or structured sentiment information.</p><p>For tasks related to generating text, LLMs will most likely outperform smaller models as they are trained with large amounts of data and easily understand context embedded in textual data. That said, it’s no secret that LLMs can make stuff up. It is also a non-trivial problem to figure out when and how that happens. While techniques like Retrieval Augmented Generation (RAG) can reduce the risk, it still is an issue and it is important to have mitigations (e.g. using LLMs in a copilot setting with human evaluation).</p><p><strong>Privacy and Security: </strong>An important consideration as you choose between building your own models or using open source models vs. using LLMs from specific service providers is your privacy and security needs. You want to ensure that your enterprise data has adequate access control, audit, and encryption policies and service providers you choose are able to apply those policies through the lifecycle of ML interactions.</p><p>In summary, choosing the right vehicle for your ML journey depends on a mix of factors like speed, cost, accuracy, and security. Just as you wouldn’t use a giant truck to deliver a pizza when a fast, nimble sedan would do, your specific use case helps dictate whether you need the heavy-lifting capabilities of an LLM or the specialized efficiency of a smaller model. At Snowflake, we encounter a variety of these use cases and are thrilled to see an increasing number of crossover efforts that blend the best of both worlds. Truly, we’re living in remarkable times, and we couldn’t be more excited to be in the driver’s seat, steering the future of innovation in this incredible field.</p><p>[1] These figures are pulled from Snowflake’s internal data.</p><p>[2] <a href="https://proceedings.mlr.press/v206/hegselmann23a/hegselmann23a.pdf">TabLLM: Few-shot Classification of Tabular Data with Large Language Models</a></p><p>[3] <a href="https://arxiv.org/abs/2207.01848">[2207.01848] TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9b4017be6e70" width="1" height="1" alt=""><hr><p><a href="https://medium.com/snowflake/small-ml-models-the-nimble-sedans-of-ai-9b4017be6e70">Small ML Models, the nimble sedans of AI</a> was originally published in <a href="https://medium.com/snowflake">Snowflake Builders Blog: Data Engineers, App Developers, AI, &amp; Data Science</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>