<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Crystal Liu on Medium]]></title>
        <description><![CDATA[Stories by Crystal Liu on Medium]]></description>
        <link>https://medium.com/@liu-gendary?source=rss-47ff7651992------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*34LUlJ4b2VJhTB9O</url>
            <title>Stories by Crystal Liu on Medium</title>
            <link>https://medium.com/@liu-gendary?source=rss-47ff7651992------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 06:59:01 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@liu-gendary/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[AI Pricing: Introducing Output-Based, Per Task Pricing Approach]]></title>
            <link>https://liu-gendary.medium.com/ai-pricing-introducing-output-based-per-task-pricing-approach-efa5ec322eda?source=rss-47ff7651992------2</link>
            <guid isPermaLink="false">https://medium.com/p/efa5ec322eda</guid>
            <dc:creator><![CDATA[Crystal Liu]]></dc:creator>
            <pubDate>Wed, 01 Nov 2023 01:40:56 GMT</pubDate>
            <atom:updated>2023-11-01T01:40:56.886Z</atom:updated>
            <content:encoded><![CDATA[<p>How to price AI products and applications has been an open question since Generative AI crossed the threshold of technological viability. Not only do companies want to design a pricing strategy that’s sustainable and reflective of the value of their products, but they want to make a profit despite Generative AI’s high compute costs. This is a common struggle for all AI-related companies, even for the hyperscaler backing OpenAI. According to the <a href="https://www.wsj.com/tech/ai/ais-costly-buildup-could-make-early-products-a-hard-sell-bdd29b9f?mod=followamazon">Wall Street Journal</a>, Microsoft is reportedly losing $20 per user per month on average for the GitHub Copilot product. So how should companies leveraging AI think about pricing? And how should they get it right the first time, to avoid painful pricing changes in the long-term?</p><p>I’d like to propose a new way of thinking about pricing AI products.</p><h3>First, why is it hard to price AI products?</h3><h4>1. Variable costs are high.</h4><p>Unlike SaaS products, where the cost of serving one more customer is close to zero, the variable cost for AI products can be quite high. For every customer interaction, the AI companies would incur an inference cost that scales in an almost linear fashion with the size of the model employed. And the bill could get even higher given current GPU supply constraints. Moreover, this is not a volume game for the application players — the more users you have does not necessarily mean a flatter cost curve for you. Counterintuitively, more active users are often less profitable. All this makes it challenging for AI startups to get predictable gross margins. They’d love to charge high enough to recoup infrastructure costs, but don’t want to limit their early traction with steep prices.</p><p>I believe this will be an issue until the models or infrastructure is optimized enough to bring down inference costs for application players. While the tech development is still early, it’s challenging for startups building on top to reliably forecast their margins, which creates headaches for builders and investors alike.</p><h4>2. Adoption is still very early.</h4><p>Although AI is the absolute 2023 buzz word, consumers and enterprises are still early in their adoption journey. <a href="https://www.mckinsey.com/capabilities/quantumblack/our-insights/the-state-of-ai-in-2023-generative-ais-breakout-year">McKinsey</a> indicates only one-third of the enterprises have adopted some form of AI in their orgs, and anecdotal evidence indicates only a fraction of those enterprises have deployed AI into production. I believe there are several reasons for it. For one, the availability of many dev tools and foundation models makes it very easy to start an AI company these days. There are lots of products targeting the same use case. So products are not that different from one another. Enterprises are simply trying them out and figuring out their own AI strategies. So consumption patterns are constantly changing. Second, it is not yet clear the actual value AI brings to the table. So users themselves don’t know and probably can’t justify the ROI for adopting AI tools for now. How can they put a willingness-to-pay for the AI tools?</p><h4>3. AI solutions are “work” based.</h4><p>Call me optimistic, but I’ve always seen AI products as the “replacements” of human efforts and that their true competitors are not another AI product but us humans. It should not be another level of productivity improvement as what SaaS brought to us, but rather a replacement of human work to some degree. Think about the early players for example. <a href="https://www.harvey.ai/">Harvey</a> does legal search, analyzes legal contracts, and generates insights from legal documents, and that replaces a lot of the work that paralegals would do. <a href="https://tome.app/">Tome</a> generates story-telling presentations based on a few prompts, replacing the work of junior sales &amp; marketing people, creators, freelancers, and even startup founders.</p><p>But just because these AI startups sell creative work, there is no standard or unified pricing terms. It’s almost like a consulting service model. Each customer values the “work” differently, and customers from different verticals have different use cases even out of the same AI products. So the usual seat-based subscription plans probably do not work well here.</p><p>Just to summarize, startups leveraging AI usually have the following pricing considerations:</p><ul><li>Be able to recoup or at least break even from the high compute and inference costs;</li><li>Want to capture all the value their products deliver without leaving money on the table;</li><li>Want flexible pricing terms for different customer profiles and leverage pricing to improve customer relationships.</li></ul><h3>How to price given the challenges?</h3><p>One of the first things I learnt at Wharton was that to capture value, one has to create value. And value is derived by growing the pie — increasing willingness-to-pay, and decreasing costs to suppliers. With this framework in mind, the first thing companies should think about is to select metrics that best represent the value delivered to customers, and tie those to pricing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_p0SHjHU-YflSTTiMAT7ng.png" /></figure><h3>Introducing Output-Based Per Task Pricing Method</h3><h4>Tying Pricing to ROI</h4><p>As argued above, I believe AI products deliver “actual work,” so pricing should tie to how much customers are valuing the “work” they receive. And the true competitors of AI products are not other AI tools, but human ourselves.</p><p>Let’s do an example. I recently called LG customer support because my fridge was not cooling. Let’s say that instead of hiring reps to take calls, LG now deploys AI chatbots. Say the hourly rate for reps was $20, and they can take 5 calls per hour, so the cost to LG was $4/call. Now with the AI chatbots, LG does not need to hire as many reps as before but can achieve a similar number of calls taken. The AI chatbot helps LG save a big amount of salaries it used to pay out to reps. Assuming the quality of the calls AI chatbots conduct is the same as those by the reps, the highest willingness-to-pay from LG is maxed at $4/call. Of course the goal of using AI chatbots for LG in this case is cost-saving, and LG probably will still have to employ some number of reps, so in reality, LG would pay the AI chatbots for a fraction of the $4/call. And the startup building these AI chatbots would also want to price the products above costs incurred for every interaction.</p><p>In this case of using AI products to boost up revenues, the same pricing method applies. In the case of Harvey AI, ideally it would help law firms generate more revenue by taking up more cases with the same or even less number of employees. So pricing such a product should use the headcount costs of employing extra paralegals to generate what Harvey could achieve as a benchmark, factoring the quality and the complexity of the work delivered, and the willingness-to-pay of the law firms. Ideally, the pricing would be some fraction of (the extra revenue created and the employee salary saved).</p><p>This output-based pricing approach — using the value captured by end users (whether cost-saving, revenue gains, or a combination of both) as a benchmark to price AI products — is both reflective of the values provided to users, and easier for startups to gain visibility on profit margin.</p><p>I’ve seen startups leveraging AI capabilities begin to charge on a per “completion” basis. For example, the AI chatbot <a href="https://www.intercom.com/fin">Fin</a> by Intercomm is measured in resolutions. It charges $0.99 for every customer service request it resolves — only after customers “exited” a conversation with the AI chatbot. Another example is <a href="https://cresta.com/">Cresta</a>, another AI-based contact center solutions, actually shifted from per-seat pricing plans to “work”-based as well, charging on the conversations it helps contact center agents complete.</p><h4>Pricing Structure</h4><p>In reality, there is much more to consider — heavy users might want a discount based on volume, customers from different verticals might value the same AI products differently, customers of various sizes might have different consumption behaviors, etc. I would advise startups leveraging AI to not fall into the trap that there is one single method for pricing. I’d actually encourage them to explore the different willingness-to-pay from customers of different sizes, in different sectors, using different use cases. It might even make sense to charge them differently. Moreover, it’s okay to use customized pricing. Just like in the world of consulting, AI companies are also selling work as I had argued.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fiw_cCS0MCcVH9Q0Sq9KeA.png" /></figure><h4>For existing players that are integrating AI solutions…</h4><p>Unlike AI-native companies who can design a brand new pricing strategy for their products, existing players who are offering AI capabilities as add-on features might not have the pricing flexibility. They’d need to consider their existing pricing strategy, making the upsell easy for existing customers without disrupting their current pricing model.</p><p>Take Notion as an example. <a href="https://www.notion.so/pricing">Notion AI</a> is offered as an optional add-on, priced at $8/member/month if billed annually or $10/member/month if billed monthly. Based on what I’ve talked about above, you’d probably wonder if Notion is able to recoup its inference costs with this seat-based pricing. And that’s what I believe Notion has been doing right — Notion AI rolled out a “<a href="https://www.notion.so/blog/notion-ai-is-here-for-everyone#:~:text=To%20ensure%20optimal%20performance%20and,slower%20performance%20for%20that%20period.">fair use limit</a>” such that if a user generates 30+ AI requests within 24 hours, the user will get slower responses for that period. To me, it seems like Notion is switching that user to a lower-grade LLM (GPT 3.5 vs. GPT 4), which has lower per token inference cost. This effectively protects Notion from heavy users taking advantage of its seat-based pricing model. With that being said, I’m not 100% sure if $8-$10/member/month pricing plan is a great way to reflect the value Notion delivers to end users. Would some kind of tiered pricing plans help, to separate out those that use up lots of tokens? But there are also many competitors in the writing copilot space, what if they are charging much lower? Is the current pricing strategy sustainable for the long-term? …</p><h3>Parting Words</h3><p>I personally think pricing is a very complicated subject when it comes to AI solutions, and it will remain challenging for some time, until compute infra costs come down or more efficient inference methods emerge. Analyzing AI pricing also makes it very clear that it is still very, very early for meaningful AI adoption. High compute costs restrain developments and app integration, so until AI use cases and their associated economic value become more clear to customers, pricing strategy will remain clouded. Under current conditions, it is likely that only companies with superior model optimization capabilities can have some control over pricing and profit margin.</p><p><strong>In conclusion</strong>: 1. constantly review your pricing strategy and talk to your customers to understand their willingness-to-pay; 2. don’t wait too long to monetize if you are offering a freemium model now; and 3. tie pricing to the metrics that matter to maximize the value you create and capture.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=efa5ec322eda" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Foundation Model: It’s all about PMF]]></title>
            <link>https://liu-gendary.medium.com/foundation-model-its-all-about-pmf-165d05f36f05?source=rss-47ff7651992------2</link>
            <guid isPermaLink="false">https://medium.com/p/165d05f36f05</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[openai]]></category>
            <category><![CDATA[llama-2]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Crystal Liu]]></dc:creator>
            <pubDate>Wed, 06 Sep 2023 04:33:34 GMT</pubDate>
            <atom:updated>2023-09-06T04:33:34.952Z</atom:updated>
            <content:encoded><![CDATA[<p>Ever since LLaMA-2 became commercially free to use in July, the debate of open-source vs. closed-source LLMs has only got more heated. You could really argue from both sides. I agree that open-source models are quickly catching up in quality and making AI more accessible. But I also think open-source LLMs are somewhat overhyped as I saw many startups still chose to build on top of OpenAI APIs for easier set-ups. In my opinion, it’s not a “winner takes all” situation; it’s all about choosing the right customer segments to serve and finding the product-market fit, and both can exist and win.</p><h3>How can you access or consume foundation models nowadays?</h3><p>Now that OpenAI also made <a href="https://platform.openai.com/docs/guides/fine-tuning">fine-tuning API </a>and <a href="https://openai.com/blog/introducing-chatgpt-enterprise">ChatGPT Enterprise</a> available, it becomes more accessible for general public to develop and deploy fine-tuned or domain-specific models, and build applications and even agents on top. Besides building base models from scratch, below are the different ways of consuming foundation models.</p><ol><li><strong>Closed-source:</strong></li></ol><ul><li>Access closed-source LLMs that are pre-trained and fine-tuned by third parties such as OpenAI, Cohere, and Anthropic directly via API;</li><li>Fine-tune on closed-source OpenAI models (available by OpenAI on Aug 22);</li><li>Customize with own data / private deployment services provided by OpenAI or Cohere (i.e., have them fine-tune for you with your own data).</li></ul><p>2. <strong>Fine-tune on top of open-source base models:</strong></p><ul><li>Fine-tune using third-party tools such as Weights &amp; Biases;</li><li>Fine-tune internally (might need to hire ML engineers).</li></ul><p>** Here, one need to take into consideration where you are hosting the model — on cloud or on-premise, which could significantly impact the costs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*E803rCNLAy_m_NFb_zA6uA.png" /><figcaption>Different Ways of Consuming LLMs</figcaption></figure><h3>Who are the buyers?</h3><ol><li><strong>Hyperscalers</strong> build their own LLMs, but the goal is not to sell those AI capabilities as standalone, but more for integrating into their products, and charging a premium for better productivity and user experience.</li><li><strong>Large enterprises</strong> want to differentiate themselves by finetuning with lots of private customer data, so they would either fine-tune on their own, or have 3P model developers fine-tune for them. Companies in financial services or healthcare industries that value privacy and have sensitive customer data would consider private development.</li><li><strong>AI-native startups</strong> tend to develop domain specific models once they have enough customer data. To date, most AI-native startups I came across still build their products on top of 3P models first.</li><li><strong>Individual researchers and SMBs</strong> could also use LLMs for research or product prototyping.</li></ol><h3>Despite the hype around open-source LLMs, the majority of the companies still choose OpenAI API. And here’s why:</h3><ol><li><strong>Open-sourced LLMs are not as cheap as you’d think:</strong></li></ol><p>When people say how accessible open-source models are from a cost perspective, they might have missed the fact that you would need to host your models while doing fine-tuning on your own, which can get very expensive.</p><p>From a cost perspective, here is the by category breakdown you would need to consider:</p><ul><li><strong>Inference costs</strong>: this is essentially the by usage API costs you pay to OpenAI or Anthropic. Usually charged by # of tokens inputted into and generated out of fine-tuned LLMs.</li><li><strong>Hosting</strong>: hosting training models for fine-tuning is what open-source model users need to pay attention to. The benefit of using closed-source models is that you don’t have to worry about the hosting infrastructure, since the models are trained and fine-tuned using their GPUs and other infrastructure. It can get challenging for companies building on top of open-source models, especially those that build on-premise. Not only is securing GPUs extremely challenging these days, but the hosting costs are hard to justify until your product scales to a very large number of users.</li><li><strong>Fine-tuning</strong>: for those building on top of open-source models, they need to pay for using third-party fine-tuning tools or frameworks, and/or hiring ML engineers. Won’t be cheap either.</li></ul><p>Let’s do the calculation. Say we are building an app to generate meeting summary. Assume we are already at scale with 10,000 daily users. Assume on average, the meetings all of these users are in have 50M tokens as input (~37.5M words) and 10M tokens as output (~7.5M words) in total every day. Assume 1B tokens used for fine-tuning, and assume <a href="https://console.cloud.google.com/marketplace/product/nvidia-ngc-public/nvidia-gpu-optimized-vmi?pli=1">8 A100 GPUs</a> used for self-hosting LLaMA-2 70B.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5fNmIGWn-kzsYcjHJEaOJw.png" /><figcaption>Cost Comparisons of Closed-Source vs. Open-Source LLMs</figcaption></figure><p>Of course this is an extreme case with lots of customer inquiries. So for starters, i.e., companies with users below 10K monthly, it makes sense to use OpenAI APIs to play around the ideas and test out the products, where you don’t have to worry about hiring the engineers and buying GPUs. Closed-source models would be much cheaper in this case as you are essentially renting only a small portion of the services and infrastructure support from the proprietary vendors.</p><p>However, for heavy usage as shown above, the upfront costs for open-source models can be justified if your startups scale up with more than 10K MAUs.</p><p>My thesis of going open-source once you have enough customer traction to justify the upfront costs, is that you don’t have to pay a premium to every stack of the solution. For accessing proprietary models, you are essentially paying a premium for GPUs, cloud hostings, pre-training, fine-tuning, and inference. So taking it in-house would definitely save you some development money.</p><p>2. <strong>It is so easy to call an API vs. build your fine-tuning infra and team.</strong></p><p>Especially for startup builders who want to test out their ideas and build quick prototypes, calling the OpenAI API is much easier than building on top of open-source LLMs. The latter requires infrastructure resources management, from selecting the right environment for hosting and deployment, to scaling the resources as the products take off.</p><p>Additionally, finding the right AI talents with both skills for fine-tuning and understanding of the domain knowledge is time-consuming and costly. According to <a href="https://www.zdnet.com/article/more-skills-needed-to-help-ai-plug-skills-gaps/">ZDNet</a>, in a survey of 1,420 IT leaders, 67% faced the challenge of AI talent shortage. <a href="https://www.cnbc.com/2023/07/13/us-companies-are-on-a-hiring-spree-for-ai-jobs-that-pay-an-average-of-146000.html">CNBC</a> noted that there were 169,045 jobs in the U.S. that required AI skills in June 2023. And simply having the words “AI skills” in the job description would yield a 45% salary increase. Moreover, top AI engineers are even more expensive to have — OpenAI engineers have a median total compensation of <a href="https://www.levels.fyi/companies/openai/salaries/software-engineer">$930K</a>.</p><p>3. <strong>Accuracy still matters!</strong></p><p>Although open-source LLMs are catching up with closed-source LLMs in quality, to date, LLaMA-2 lacks GPT-3.5 and GPT-4 in <a href="https://scontent-sjc3-1.xx.fbcdn.net/v/t39.2365-6/10000000_662098952474184_2584067087619170692_n.pdf?_nc_cat=105&amp;ccb=1-7&amp;_nc_sid=3c67a6&amp;_nc_ohc=bGkSwJy8Xa4AX9mXq1Q&amp;_nc_ht=scontent-sjc3-1.xx&amp;oh=00_AfDRAb3a59yn_yhVerLwkEeWhgE0K8sHj78jDTcTsqe4mQ&amp;oe=64FBB4BF">performance from the chart below</a>. One could argue that fine-tuning could close some of those performance gaps, but I’d say that the difference matters to many enterprises. Think about how customer-facing enterprises today still use AI tools for internal use cases mostly, would they trade quality for free open-source models? I’d imagine that they’d care for customer experience more. And the cost-saving from using open-source models might not be worth it if customer experience had to suffer. Maybe one day when open-source models actually have better performance than closed-source models, enterprises could then invest more in open-source models.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/956/0*ovo29HKWJPjCsa4r" /><figcaption>Performance benchmark of LLaMA-2 vs. other closed-source LLMs. Source: <a href="https://scontent-sjc3-1.xx.fbcdn.net/v/t39.2365-6/10000000_662098952474184_2584067087619170692_n.pdf?_nc_cat=105&amp;ccb=1-7&amp;_nc_sid=3c67a6&amp;_nc_ohc=bGkSwJy8Xa4AX9mXq1Q&amp;_nc_ht=scontent-sjc3-1.xx&amp;oh=00_AfDRAb3a59yn_yhVerLwkEeWhgE0K8sHj78jDTcTsqe4mQ&amp;oe=64FBB4BF">https://scontent-sjc3-1.xx.fbcdn.net/v/t39.2365-6/10000000_662098952474184_2584067087619170692_n.pdf?_nc_cat=105&amp;ccb=1-7&amp;_nc_sid=3c67a6&amp;_nc_ohc=bGkSwJy8Xa4AX9mXq1Q&amp;_nc_ht=scontent-sjc3-1.xx&amp;oh=00_AfDRAb3a59yn_yhVerLwkEeWhgE0K8sHj78jDTcTsqe4mQ&amp;oe=64FBB4BF</a></figcaption></figure><h3>Open vs. Closed — It’s about matching product-market fit.</h3><p>I listed below the main criteria companies could have when selecting between building on top of open-source models or using closed-source models directly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fxGuQT0MKeVM1nHB06tzWA.png" /><figcaption>Pros and Cons of Open-Source vs. Closed-Source LLMs</figcaption></figure><p>From the chart above we can see that self-hosting and proprietary models have their own pros and cons. So my thesis is that they should each target different customer profiles for different needs and use cases. From a customer perspective, below is what they might prefer to use:</p><ol><li><strong>Closed-source models:</strong></li></ol><ul><li><em>Enterprises with high accuracy requirements</em>: those that want to deploy AI products externally for customer use, or those that are in verticals such as legal or healthcare, would want higher quality models. I’ve also seen many agents startups building on top of OpenAI API because of the same reason.</li><li><em>Enterprises with horizontal applications</em>: companies with generic productivity tools that sell into multiple sectors might benefit a lot from generic-purpose closed-source models, as they have higher quality.</li><li><em>Enterprises without AI talents and technical capabilities</em> are probably more suitable for easy-to-start solutions such as ChatGPT Enterprise or Fine-tuning API.</li><li><em>Emerging AI-native startups</em>: these companies usually don’t have existing private customer data to start with. They can build on top of closed-source models to gain tractions and collect customer data before starting their own domain specific models.</li></ul><p>2.<strong> Open-source models:</strong></p><ul><li><em>Enterprises with vertical applications and lots of customer data</em> are more suitable for fine-tuning on top of open-source models into specialized tasks. This is because their business units likely have lots of domain expertise and data to start with.</li><li><em>Enterprises with technical talents and resources</em>: it would be much easier for these companies to set up the hosting environment. With their internal engineer talents that know the ins and outs of the companies, they can fine-tune the models that fit best for their respective companies.</li><li><em>Enterprises with privacy concerns</em> are suitable for either fine-tuning with private dataset or private deployments. There are service companies like Sambanova that help enterprises with privacy concerns (such as government agencies or financial institutions) build their own AI models on top of LLaMA-2.</li><li><em>AI-native startups with enough tractions</em>: I’ve seen startups pivoting from OpenAI API to LLaMA-2 recently. Now that they’ve collected enough customer data to fine-tune their domain specific models, they’d like to have more control of and better visibility into the models.</li></ul><p>While from a model-developer perspective, are the above categorized customers really the ones they should target? OpenAI has made it increasingly easy for starters to access high-quality models, but is this monetization strategy sustainable? In today’s market where lots of AI use cases are still unclear, not many startups can scale well. So their use of OpenAI API might stop later on when they have trouble finding customers. Or if they actually manage to scale, would they still stick with OpenAI where inference costs would also scale with more customers? Those companies might shift to open-source models to build their domain specific models once they’ve accrued enough customer data.</p><p>More importantly, would proprietary model developers be able to recoup the upfront investments and high development costs by targeting entry-level customers? It might not be OpenAI’s concern, but I do find many closed-source model developers have that product-market fit issue. At the end of day, investors are not going to pay for their huge amounts of model training and development fees. So they need to figure out the right customers to target, and whether they should sell access to models (e.g., Cohere), or AI-driven end products (e.g., AI21labs). In my opinion, I see service model as a way out for closed-source model developers — helping enterprises build and deploy customized models. At least in today’s market where enterprises lack the technical talents and resources, they might prefer to have a full-stack solution. And model developers can also charge more with enterprise sales schemes.</p><h3>Limitations / Concerns:</h3><ol><li><strong>How to scale?</strong> For proprietary model developers that are on service business models, the ability to scale efficiently would be a concern. Because each customer has different needs, their services cannot be replicated easily. And the enterprise sales teams need to build an extensive customer pipeline, and build relationships with each of them. The sales cycle can also get long here.</li><li><strong>Maintenance</strong>. For those self-hosting their models, continuous development and maintenance of the models can get tricky. Companies would need full-stack observability into each stack. Identifying issues and bugs can be challenging.</li><li><strong>Will proprietary models maintain the high quality?</strong> Not only are open-source models catching up, but it’s also getting challenging for OpenAI to collect more data, as more than 15% of the 100-most-popular websites are <a href="https://www.businessinsider.com/chatgpt-openai-gptbot-crawler-major-companies-media-outlets-blocking-2023-8">blocking OpenAI’s web crawler GPTBot</a>. With researchers and developers contributing to the open-source community, open-source models can potentially have better quality models in the near future.</li></ol><h3>Future opportunities:</h3><ol><li><strong>Model hosting, deployment, and orchestration platforms</strong>. Challenges such as deploying models at scale and serving models are still unsolved today. I’m looking for startups that are AWS SageMaker equivalent — those that can serve models at lower costs and more securely, and handle complexities well are of interest. As models are getting smaller in size, and MoE (mixture of experts) models are being developed, I’m also interested in orchestration platform and frameworks.</li><li><strong>Fine-tuning tools</strong>. As open-source models become increasingly available, I’m also interested in startups that offer fine-tuning capabilities. I’m interested in those that offer collaboration capabilities with internal functional teams, and that work with expert models.</li><li><strong>Model evaluation and testing</strong>. There are already many models now, and there will be more in the future. Companies may use different models for different use cases. And there are so many factors to consider (as listed in the above section) when selecting a model. So I’m excited about evaluation and testing tools that help enterprises and developers make better LLM choices.</li></ol><h3>Closing Thoughts:</h3><p>Although I do believe that open-source models will be widely adopted in the future, we are not there yet. In an ideal world where infrastructure and model-hosting are not that expensive, where AI talents are not that hard to find, where time-to-market is not that long, and when open-source models have better quality, I’d imagine open-source models to have very bright futures. But as companies are still trying to figure out the right use cases for LLM adoption and as developers are playing around different kinds of developer tools, it will take some time until open-source models become mainstream.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=165d05f36f05" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[LLM Explained: The LLM Training Landscape]]></title>
            <link>https://liu-gendary.medium.com/llm-explained-the-llm-training-landscape-82c803495caa?source=rss-47ff7651992------2</link>
            <guid isPermaLink="false">https://medium.com/p/82c803495caa</guid>
            <category><![CDATA[openai]]></category>
            <category><![CDATA[gpt]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Crystal Liu]]></dc:creator>
            <pubDate>Sun, 06 Aug 2023 22:52:24 GMT</pubDate>
            <atom:updated>2023-08-08T06:27:10.644Z</atom:updated>
            <content:encoded><![CDATA[<p>Since the introduction of <a href="https://arxiv.org/abs/1706.03762">Transformer model in 2017</a>, large language models (LLMs) have evolved significantly. ChatGPT saw <a href="https://www.similarweb.com/website/chat.openai.com/#traffic">1.6B visits</a> in May 2023. Meta also released three versions of <a href="https://ai.meta.com/llama/">LLaMA-2</a> (7B, 13B, 70B) free for commercial use in July.</p><p>In this post, I will explain how a LLM like ChatGPT or LLaMA came to life, from pre-train, to fine-tune and reinforcement learning from human feedback (the “make AI” process). I will also talk about some common approaches of model observability, guardrail, and governance. In the next series, I will explain the “use AI” — inference process.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vZK250i8PIWid6BiaZ1QCA.png" /></figure><h3>Pre-training:</h3><p>The training process starts with building a pre-training model. The goal here is to teach the base model to generate more text data and complete documents by reasonably predicting the next words. In order to do this, researchers and developers expose the transformer model to a large amount of training dataset, usually texts with trillions of words, yet low-quality, unsupervised, and unstructured. The texts are then tokenized into a long list of integers. The transfer model will find optimal patterns and likelihoods between the text words, and the patterns will be stored as “weights” — consider them as the coefficients of a polynomial. The more parameters the models has, the more flexibility the model has to capture details and relationships of the trained text.</p><p>Think of training as of a continuous refining process. Through thousands of iterations, making not so good predictions, and receiving feedback, the transformer model learns to make less bad predictions and gradually gets closer to the correct predictions — updating and optimizing weights.</p><p>Models like <a href="https://openai.com/gpt-4">GPT</a>, <a href="https://ai.meta.com/blog/large-language-model-llama-meta-ai/">LLaMa</a>, and <a href="https://ai.googleblog.com/2022/04/pathways-language-model-palm-scaling-to.html">PaLM</a> are all pre-trained base models in general language domains. While <a href="https://academic.oup.com/bib/article/23/6/bbac409/6713511?guestAccessKey=a66d9b5d-4f83-4017-bb52-405815c907b9&amp;login=false">BioGPT</a>, <a href="https://hpc.nih.gov/apps/BioBERT.html#:~:text=BioBERT%20is%20a%20biomedical%20language,extraction%2C%20question%20answering%2C%20etc.">BioBERT</a>, and <a href="https://pubmed.ncbi.nlm.nih.gov/31501885/">PubMedBERT</a> are domain-specific pre-trained models, which were exposed to biomedical texts during pre-training.</p><h4>Challenges and Requirements:</h4><p>The pre-training process is capital-, compute-, time-, and memory-intensive.</p><p>So, how does compute hardware support AI training workflows? In order to find the optimal weights, sparse matrix multiplications are distributed to tens of thousands of GPUs. The GPUs would go through a compute, exchange, and reduce cycle — the GPUs each compute intensively themselves, then synchronize with other GPUs to average and reduce their computations together, and then another cycle starts.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*AfuqO394pQ7pv2Nr" /><figcaption><em>Compute-Exchange-Reduce Cycle. </em><a href="http://www.arista.com./"><em>www.Arista.com.</em></a></figcaption></figure><p>Ultimately, if we want to train the model as fast and as precisely as possible, we would need more GPUs and other servers. However, GPU utilization does not scale as we add more of them. The more GPUs are in use, the more GPU-to-GPU communication and synchronization there is. As data is being transmitted across more GPUs, GPU utilization starts to drop due to idle time amid data transmission, making it more expensive to train the models.</p><ul><li><em>Computational Power</em>: From the compute-exchange-reduce cycle shown above, we can see that there are intensive matrix multiplication operations for each iteration, which grow as the model gets larger and needs higher precision. Taking GPT-3 as an example, the 175B version requires 3e23 FLOP (Floating Point Operation) of computation for training. If we are using the most available state-of-the-art GPU, Nvidia A100, which can conduct around <a href="https://www.nvidia.com/content/dam/en-zz/Solutions/Data-Center/a100/pdf/nvidia-a100-datasheet-us-nvidia-1758950-r4-web.pdf">312 TeraFLOP per second </a>(TFLOPs), a single GPU needs 30 years to train GPT-3 without factoring in actual utilization.</li><li><em>Training Time</em>: It took OpenAI <a href="https://medium.com/codex/gpt-4-will-be-500x-smaller-than-people-think-here-is-why-3556816f8ff2#:~:text=Training%20100%20Trillion%20Parameters,GPUs%2C%20would%20take%2053%20Years.">34</a> days to train GPT-3, and Meta <a href="https://www.reddit.com/r/MachineLearning/comments/11nhl03/d_is_it_possible_to_train_llama/">21</a> days to train LLaMa. As we have done the calculation above, thousands of GPUs (e.g.,<a href="https://arxiv.org/pdf/2302.13971.pdf"> 2,048 A100 used to train LLaMa</a>) are easily needed for training these LLMs in order to achieve a reasonable training time. Due to GPU shortage and communication loss for using too many GPUs, the largest LLM usually takes ~1 month to train for a good value for cost.</li><li><em>Memory Constraints</em>: For all these LLMs, the weight tensor usually has a size of a few hundred GBs. The newest AMD MI300x (which has<a href="https://www.hpcwire.com/2023/06/13/amd-has-a-gpu-to-rival-nvidias-h100/"> 2.4x memory of Nvidia’s H100</a>) still can’t store the entire weight on a single chip, so larger on-chip memory would give the AI chip more flexibility when dividing the training workload and avoid communication. From the below graph, it is clear that on-chip memory size was growing at a slower pace compared to the model size (directly contributes to weight size). In the future, even though the model size might not grow as fast as the previous years, we might see more GPT-4-like multi-modal models, which could contain image, voice, or video capability, resulting in more complex memory requirements.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*U-7GJqBZ2tY1W5Iu" /><figcaption>Growth of Compute Memory vs. Transformer Size. <a href="https://medium.com/riselab/ai-and-memory-wall-2cb4265cb0b8"><em>https://medium.com/riselab/ai-and-memory-wall-2cb4265cb0b8</em></a></figcaption></figure><h3>Finetuning:</h3><p>Now that a base model is ready, finetuning is performed so that the model can achieve certain desired performance, or to be used for a particular task. During finetuning, new dataset that is specific to a task will be used. Unlike pre-training, here, the dataset is smaller in size and more specialized. The weights will continue to be updated here based on the new and more specific data used.</p><p>Finetuning an open-source model brings many benefits such as cost- and time-saving, while using well-performed models without having to train one from scratch. One can use as little as only one GPU for finetuning (vs. thousands of GPUs used in pre-training), and finetuning can be done in just one day. For example, the training of <a href="https://lmsys.org/blog/2023-03-30-vicuna/">Vicuna-13B</a>, finetuned on LLaMA, was only $300. <a href="https://docs.databricks.com/machine-learning/train-model/huggingface/fine-tune-model.html">Here</a>, Databricks also laid out the instructions for finetuning Hugging Face models on a single GPU.</p><p>I’ve also seen closed-source model developers partnering with finetuning startups to help their customers better refine the base model. For example, <a href="https://wandb.ai/wandb_fc/use-cases/reports/How-Cohere-Trains-Business-Critical-LLMs-with-the-Help-of-W-B--VmlldzozNTEyNzU1">Cohere partnered with Weights &amp; Biases</a> to experiment customized models for customers’ specific use cases. When training the first version of its Command model, Cohere leveraged the annotation workforces and high-quality datasets at <a href="https://scale.com/customers/cohere">Scale</a> to make it happen.</p><p>However, I’d note that there are use cases where finetuning might not work so well, and that training from scratch might be needed. For example, finetuning the pre-trained models for completely irrelevant tasks, or finetuning from pre-trained models that are not very generic in the first place (i.e., finetuning BioGPT for legal use cases is probably not a good idea).</p><p>What are some ways of finetuning?</p><ul><li><em>Repurposing</em> a pre-trained text-generating LLM for a different type of application, such as sentiment analysis, could require changing the architecture of the LLM (e.g., freezing the attention layers of the transformer model while training the task-specific layer on top of the pre-trained model).</li><li><em>Unsupervised Finetuning</em> uses unstructured dataset and updates the knowledge of LLM to a specific domain (e.g., medical, legal, etc.), essentially adjusting the model from source domain to the target domain.</li><li><em>Supervised Finetuning (</em><a href="https://github.com/microsoft/DeepSpeedExamples/blob/master/applications/DeepSpeed-Chat/training/step1_supervised_finetuning/README.md"><em>SFT</em></a><em>)</em> uses high-quality question-answer pairs to finetune the model for human-preferred generation. The prompt-response paired data can either be curated by human manually or generated from other LLMs. In the case of ChatGPT, human-written, low quantity, and high quality pair data was used for it to follow user instructions to perform specific tasks. <em>Instruction-tuning</em> is a subset of SFT, guiding the LLMs to follow the chain-of-thought similar to human’s.</li><li><em>Reward Model (RM) Finetuning:</em> similar to SFT, it also uses prompt-response pairs. Yet here, each batch of data consists of <strong>two</strong> prompt-response pairs — the same prompt but one with higher-score response and another with lower-score response. Here, the LLM learns to predict consistent with the preference, i.e., higher-score response.</li></ul><h4>Challenges and Requirements (compared to pre-training):</h4><ul><li><em>Computational Power</em>: Usually the finetuning compute is as intensive as pre-training, because it is conducted on the same model. However, the amount of data and number of iterations required to run finetuning is way less. As a result, the total amount of compute power needed (FLOP) is in a more reasonable range for more companies to get involved.</li><li><em>Training Time</em>: Since the total compute required is reduced by many order of magnitude, the total training time would usually fall into the range of hours to days, using less number of compute devices as well (GPU).</li><li><em>Memory Constraints</em>: Due to the weight size being the same from pre-training to finetuning, the memory requirements are the same, needing multiple GPUs at least to be able to train.</li></ul><h3>RLHF:</h3><p>Reinforcement Learning from Human Feedback (RLHF) is another way to optimize model performance. Models like ChatGPT and Claude are all LLMs that have gone through RLHF.</p><p>Think of RLHF as the interaction of a human agent with the environment. The first step in RLHF is to create a Reward Model (RM) by having human annotators rank and rate different model outputs, the reward signals. Those reward labels are then used to guide LLMs to adapt to human preferences, by aligning the behaviors of RM to the preferences of human annotators, thus giving it “human feedback.” These steps are then reiterated with the LLMs continuously “improving” from the reward signals. RLHF is particularly useful when evaluating behavior is easier than generating it.</p><p>To date, RLHF is largely a research territory. The result of RLHF depends on the quality of the inputs from human annotators. Additionally, bad behaviors could still happen even with RLHF. But with the emergence of open source LLMs, base models are becoming more “accessible.” It would be more cost-efficient for people to use state-of-the-art models. As such, I expect more developments around the applications of finetuning and RLHF, for people and companies to directly adapt models into specific use cases.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*z41DCdhDpf-ANcs1" /><figcaption><em>“InstructGPT models (PPO-ptx) and its variant trained without pretraining mix (PPO) significantly outperform the GPT-3 baselines (GPT, GPT prompted).” OpenAI, </em><a href="https://openreview.net/pdf?id=TG8KACxEON"><em>https://openreview.net/pdf?id=TG8KACxEON</em></a></figcaption></figure><h3>Deploy and model hosting:</h3><p>Now that the LLMs are trained and fine-tuned, we will need to put LLMs into an environment for production (inference) and to be used by end users — the so-called model deployment and hosting process.</p><p>Here, people choose how and where to deploy the model, whether on premise or cloud. Factors that influence the decision include latency (e.g., the delay customers might experience to get a response from the chatbot), hosting costs (which depend on the computational power and memory required based on model size), and even safe deployment (to reduce harm and biases).</p><p>Usually, those fine-tuning startups will also handle the deployment process. Users go to all-in-one platforms like <a href="https://www.cerebrium.ai/">Cerebrium</a>, either with base models of their choice, or to explore pre-built models on the platform. These platforms often create a serverless environment for the users, where they don’t need to worry about choosing the infrastructure. Users could finetune models on these platforms, and then once ready, they ship productions directly on the platform. Some even allow users to monitor their models post-deployment. Other “deployment-only” startups like <a href="https://www.bentoml.com/">BentoML</a> take trained models and then help users schedule inference productions.</p><h3>Model evaluation and monitoring:</h3><p>Continuous evaluation and monitoring is needed throughout the model development process, so customers can learn how to further refine them, and what kinds of additional data are needed. There is also growing attention to more responsible model development and releases, which need to be monitored throughout the process. Model evaluation is also important, as there are many LLMs to choose from for customers to build products.</p><p>Below are some factors that people would consider when evaluating LLMs. Other key factors include explainability, reliability, limitation, and etc. For models that are used commercially, a very important metric is how well it aligns with customer needs and expectations, as what <a href="https://www.fiddler.ai/blog/top-5-questions-on-llmops-from-our-generative-ai-meets-responsible-ai-summit">Jasper’s Director of AI</a> described — “form follows function.”</p><ul><li><em>Safety / fairness</em>: Biases are unavoidable for LLMs. Datasets are biased inherently. And biases may also come from human annotators when curating the dataset. All those biases could even be reinforced when the models pick them up. As such, we will try our best to avoid the unfairness. Startups like <a href="http://fiddler.ai/fairness">Fiddler</a> detect biases in datasets and LLMs in real time. The figure below shows how Fiddler monitors metrics such as disparate impact, demographic parity, and equal opportunity in the dataset. While <a href="https://arize.com/explainability/">Arize AI</a> helps customers trace biases, and compare fairness metrics among different model versions.</li><li><em>Accuracy </em>measures how consistent the model outputs are with the expected results. Companies like <a href="https://www.arthur.ai/accuracy">Arthur AI</a> proactively detects and reacts to model drifts. While <a href="https://truera.com/#">TruEra</a> helps evaluate how much of model drift can be explained by feature change. Below is a list of some commonly used evaluation metrics.</li></ul><ol><li><a href="https://aclanthology.org/P02-1040.pdf?ref=blog.paperspace.com">BLEU</a> (Bilingual Evaluation Understudy) score is a metric commonly used to assess the quality of machine-generated translations vs. reference translations.</li><li><a href="https://aclanthology.org/W04-1013/?ref=blog.paperspace.com">ROUGE</a> (Recall-Oriented Understudy for Gisting Evaluation) score is often used to assess the quality of machine-generated text summaries vs. reference summaries, and used in text summarization tasks. It calculates the syntactic overlap between candidate and reference summaries.</li><li><a href="https://arxiv.org/abs/1904.09675?ref=blog.paperspace.com">BERTScore</a> assesses the quality of text generated by capturing the semantic meaning, generated by the BERT model.</li></ol><ul><li><em>Robustness </em>evaluates if the model can maintain model performance under adversarial attacks. Fiddler has an <a href="https://www.fiddler.ai/blog/introducing-fiddler-auditor-evaluate-the-robustness-of-llms-and-nlp-models">Auditor</a> product that analyzes how models will respond to subtle variations in prompts, and compares outputs vs. expected outcomes. It also leverages LLMs to generate variations in prompts. I’ll talk more about adversarial attacks in the below “Security” section.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*cO0jjAO2tU8ps5v9" /><figcaption><em>Fiddler AI. </em><a href="https://www.fiddler.ai/fairness"><em>https://www.fiddler.ai/fairness</em></a></figcaption></figure><h3>Model governance and security:</h3><p>Models are actually very vulnerable. Open-source models might be more vulnerable as access to them is unconstrained, so it would be harder to control the misuse.</p><p>First of all, the data that the LLMs are trained on could possess unsafe or dangerous information. So the training process might augment the potential harm.</p><p>Additionally, models are subject to adversarial attacks such as <strong>prompt injection</strong>. By carefully or maliciously crafting the prompts, one can “lure” the models to give unintended responses and influence their behaviors. <a href="https://arxiv.org/pdf/2307.15043.pdf">Researchers at Carnegie Mellon University actually successfully attacked ChatGPT</a>. If you simply ask ChatGPT “how to build a bomb,” the machine knows it cannot give you the answer because there could be harmful impact. But if you ask it to write “step-by-step” guides on how to conduct certain criminal activities, it would actually give you the details… See example below. Quite scary.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*fkqUasPJfoFDASp0" /><figcaption><a href="https://www.cmu.edu/news/stories/archives/2023/july/researchers-discover-new-vulnerability-in-large-language-models"><em>https://www.cmu.edu/news/stories/archives/2023/july/researchers-discover-new-vulnerability-in-large-language-models</em></a></figcaption></figure><p>Also, because models have limited short-term working memory (they are not database themselves), they are not able to cite where they have got the answers, i.e., they just don’t remember. As such, <strong>hallucination</strong> could happen, when the model generates answers that are not real, or that do not make any sense. In this case, the responses the models provide are probably not related to the training data, but based on the prompts themselves.</p><p>To mitigate such risks and avoid these attacks, one potential approach is to also expose the LLMs to unsafe or harmful content when training — adversarial training, such that the LLMs will learn to identify what is safe and what is not. We can then guide the models to generate safe responses even when they are given unwanted prompts. Another approach could be filtering the data input before the data is used in model development and production. Companies like <a href="https://www.robustintelligence.com/">Robust Intelligence</a> would detect anomalies from both the inputs and outputs in real time, as well as screen the new datasets that would be exposed to the models.</p><p>The good thing is that the model-developers are working together to build a responsible future. Just last week, <a href="https://www.whitehouse.gov/briefing-room/statements-releases/2023/07/21/fact-sheet-biden-harris-administration-secures-voluntary-commitments-from-leading-artificial-intelligence-companies-to-manage-the-risks-posed-by-ai/">seven of the model-makers</a>, both open-source and closed-source, both cloud service providers and startups, including Google and OpenAI, voluntarily committed to the safe, transparent, and secure development of AI technology, to the Biden-Harris Administration. Look forward to both building and using AI more securely.</p><h3>Conclusion:</h3><p>Here is a ChatGPT-generated conclusion:</p><p>“In conclusion, the emergence of Large Language Models (LLMs) has brought about a revolution in the field of artificial intelligence and natural language processing. These powerful models have demonstrated extraordinary abilities to comprehend and generate human-like text, revolutionizing various applications, from virtual assistants and chatbots to content creation and translation. As their impact on society grows, it becomes vital to address ethical considerations and potential risks to ensure their responsible use.</p><p>Delving deeper into their training process and exploring innovative applications will provide valuable insights into their potential and limitations. Let us embark on this exciting journey together, exploring the boundless possibilities of LLMs while being mindful of their implications. Together, we can ensure that LLMs become a force for good, transforming our world and enabling a brighter, more connected future.”</p><h3>References:</h3><p><a href="https://karpathy.ai/stateofgpt.pdf">https://karpathy.ai/stateofgpt.pdf</a></p><p><a href="https://www.leewayhertz.com/fine-tuning-pre-trained-models/">https://www.leewayhertz.com/fine-tuning-pre-trained-models/</a></p><p><a href="https://research.aimultiple.com/llm-fine-tuning/">https://research.aimultiple.com/llm-fine-tuning/</a></p><p><a href="https://huggingface.co/blog/rlhf">https://huggingface.co/blog/rlhf</a></p><p><a href="https://bdtechtalks.com/2023/07/10/llm-fine-tuning/">https://bdtechtalks.com/2023/07/10/llm-fine-tuning/</a></p><p><a href="https://wandb.ai/ayush-thakur/RLHF/reports/Understanding-Reinforcement-Learning-from-Human-Feedback-RLHF-Part-1--VmlldzoyODk5MTIx">https://wandb.ai/ayush-thakur/RLHF/reports/Understanding-Reinforcement-Learning-from-Human-Feedback-RLHF-Part-1--VmlldzoyODk5MTIx</a></p><p><a href="https://labelbox.com/blog/using-reinforcement-learning-from-human-feedback-to-fine-tune-large-language-models/">https://labelbox.com/blog/using-reinforcement-learning-from-human-feedback-to-fine-tune-large-language-models/</a></p><p><a href="https://arize.com/blog-course/large-language-model-llm-deployment/">https://arize.com/blog-course/large-language-model-llm-deployment/</a></p><p><a href="https://deepchecks.com/glossary/llm-evaluation/">https://deepchecks.com/glossary/llm-evaluation/</a></p><p><a href="https://medium.com/@sthanikamsanthosh1994/understanding-bleu-and-rouge-score-for-nlp-evaluation-1ab334ecadcb">https://medium.com/@sthanikamsanthosh1994/understanding-bleu-and-rouge-score-for-nlp-evaluation-1ab334ecadcb</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=82c803495caa" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>