<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Anurag Bhagat on Medium]]></title>
        <description><![CDATA[Stories by Anurag Bhagat on Medium]]></description>
        <link>https://medium.com/@bhagatanurag03?source=rss-122376ea00fe------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*9Hc7B09FnwAqfMasKTgnLA.png</url>
            <title>Stories by Anurag Bhagat on Medium</title>
            <link>https://medium.com/@bhagatanurag03?source=rss-122376ea00fe------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 09:10:05 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@bhagatanurag03/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[The Build vs. Buy Dilemma for GenAI Applications]]></title>
            <link>https://medium.com/data-science/the-build-vs-buy-dilemma-for-genai-applications-a6828f99c922?source=rss-122376ea00fe------2</link>
            <guid isPermaLink="false">https://medium.com/p/a6828f99c922</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[generative-ai-use-cases]]></category>
            <category><![CDATA[cto]]></category>
            <category><![CDATA[strategy]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Anurag Bhagat]]></dc:creator>
            <pubDate>Tue, 07 Jan 2025 14:01:38 GMT</pubDate>
            <atom:updated>2025-01-07T14:01:38.621Z</atom:updated>
            <content:encoded><![CDATA[<h4>A strategic guide to build vs. buy for GenAI</h4><p>Generative AI has been transformational to the world already, and it’s just getting started. It has fast been adopted in multiple industries, from retail to healthcare and banking, offering multiple capabilities ranging from information retrieval, expert help, and new content creation. With the growing interest in GenAI from a majority of boardrooms, there is a big question that CTOs/CIOs are facing now: <em>Should you build your GenAI application in house, or buy a pre-built solution?</em></p><p>This article provides a framework to help product managers, business-leaders and technology leaders navigate this decision. Please note that many of these fundamental arguments hold true for all decisions of build vs. buy, but we bring forward some nuances that are unique to the current landscape in GenAI.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*uxadMW-3o4OlAonR" /><figcaption>Source: Generated with the help of AI</figcaption></figure><h3><strong>The core of the decision</strong></h3><p>The build vs. buy (vs. modify, which I consider under the umbrella of buy) decision depends on multiple factors. What makes the decision even more difficult is that the AI landscape is fast-evolving, with new models and products launched every single week. What may be a gap in the market offerings today might see a new product in the next few weeks.</p><p>The key factors that impact this decision are:</p><ol><li>Market availability (now and in the near term) and business’s needed speed to market</li><li>Strategic importance of the application to the business</li><li>ROI for the business</li><li>Risk and Compliance factors</li><li>Ability to maintain and evolve</li><li>Integration complexities</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/0*rk8J_2qky4yTFJqZ" /><figcaption>Source: Generated with the help of AI</figcaption></figure><p>At the core, my personal advice is to reframe the build vs. buy question as <em>why do you need to build?</em> There are hundreds of amazing organizations pouring billions into the development of GenAI applications, so unless you are one of them, you should really try to understand what is out in the market that does not suit your needs.</p><h3><strong>The case for building GenAI applications</strong></h3><ol><li><strong>Unique business requirements: </strong>If your needs are unique such that off-the-shelf applications in the market do not fit your needs, and you believe that no such applications will be in the market in the near to medium term horizon. Given the fast-paced nature of development in GenAI, I have personally seen instances where organizations started to build a functionality, only to see it being available as a commodity in the market within a few months. One example of this is evaluations, which saw a lot of launches from many key players in 2024, including <a href="https://aws.amazon.com/bedrock/evaluations/">AWS </a>and <a href="https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/evaluations?tabs=question-eval-input">Azure</a>.</li><li><strong>Competitive Advantage: </strong>If the application is of strategic importance to your business and is critical for you to maintain your IP and market differentiation. These would be very unique circumstances in general and should have strong leadership alignment for this. One well-known example here is that of LLMs. Most organizations do not need to build their own LLM models; they can either use what is available in the market with well-engineered prompts, or fine-tune them to their own context. Bloomberg made a call to build their <a href="https://www.bloomberg.com/company/press/bloomberggpt-50-billion-parameter-llm-tuned-finance/">own model</a>, which was a strategic move to enable the use of their proprietary data with a finance-specific lexicon while solidifying their position as a leader in financial innovation.</li><li><strong>Long-term cost efficiency: </strong>While upfront costs for development are higher, in-house solutions may be most cost-effective in the long run if your usage is large scale. One common pitfall here is not baking in the long-term maintenance overhead to the cost when building the business case. One must also note that even though many GenAI applications may be expensive now, the<a href="https://opusresearch.net/2024/07/29/costs-of-generative-ai-continue-to-drop-unlocking-new-possibilities/"> costs are falling rapidly</a> as we speak, so what may look like an expensive buy now, may end up being cheap in a few months.</li><li><strong>Data Privacy and Security</strong>: Sensitive industries like <a href="https://www.cio.com/article/3480467/how-to-build-a-safe-path-to-ai-in-healthcare.html">healthcare </a>and finance often have to abide by stringent data privacy regulations and concerns. In-house solutions provide additional control over data handling and compliance.</li></ol><p>If you do end up deciding to build in-house, there are a few key challenges that emerge:</p><ul><li>You will need a skilled team of AI experts, substantial time, and significant upfront investment.</li><li>Maintenance and updates, including abiding by the changing regulatory landscape become your responsibility.</li><li>Even with the right team of experts, you may not be able to keep up with the speed of innovation that there is currently in GenAI.</li></ul><h3><strong>The case for buying GenAI applications</strong></h3><p>Pre-built GenAI solutions, available as APIs or SaaS platforms, offer rapid deployment and lower upfront costs. Here’s when buying might be the better choice:</p><ol><li><strong>Speed to market</strong>: if you are looking to deploy quickly, even if an existing solution may not be a 100% fit to your needs right now. With new developments and releases, new features may address more of your needs.</li><li><strong>Predictable costs: </strong>Subscription-based pricing models provide absolute clarity in terms of expense, avoiding cost overruns. On top of that, for GenAI, we have seen frequent price drops and expect that to happen in the near term. A recent example was that of <a href="https://aws.amazon.com/about-aws/whats-new/2024/12/amazon-bedrock-guardrails-reduces-pricing-85-percent/">Amazon Bedrock reducing pricing by 85%.</a></li><li><strong>Focus on core priorities: </strong>Buying allows your team to focus on business-specific tasks rather than the complexities of building AI. This is especially true of the solutions that are available as commoditized solutions and provide no competitive advantage.</li></ol><p>If you do end up deciding to buy, there are a few key challenges that emerge:</p><ol><li>You may be limited by how much you can customize. There may be features you need, which remain on the vendor backlog for longer than you may want.</li><li>Potential concerns around vendor lock-in and data privacy.</li></ol><h3>Key Questions to Guide Your Decision</h3><p>The decision for build vs. buy for each GenAI application has to take into account the overall enterprise GenAI strategy for building vs buying. The decision cannot be made in isolation, as you need a critical mass of applications to justify having a team to build these. The following questions, however should be asked to help guide the answer for individual applications:</p><ol><li>Will this application enable a distinct competitive advantage?</li><li>What solutions exist in the market already?</li><li>What is your timeline for deployment?</li><li>What capabilities are needed to build and maintain the GenAI application? Do you have that expertise, either internally or with partners?</li><li>What are your data privacy and compliance requirements?</li><li>How does the business case differ for build vs. buy?</li></ol><h3>Conclusion</h3><p>The decision on build vs. buy decision for GenAI applications is not one-size-fits-all. The core of the decision-making strategy is not too different from any other build vs. buy decision, but GenAI applications have the additional complexities of a fast-changing landscape, a high pace of innovation and high-but-reducing costs of a relatively new technology.</p><p>While building offers control and customization at generally higher costs, buying provides speed and simplicity. Sometimes you may not have something available that fits your needs, but that may change in a few months. By carefully evaluating your organization’s needs, their urgency, resources, and goals, you can make a decision that drives success and long-term value.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a6828f99c922" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/the-build-vs-buy-dilemma-for-genai-applications-a6828f99c922">The Build vs. Buy Dilemma for GenAI Applications</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The not-so-hidden costs of building an AI enabled RAG based chatbot]]></title>
            <link>https://medium.com/@bhagatanurag03/the-not-so-hidden-costs-of-building-an-ai-enabled-rag-based-chatbot-edf1600a4d38?source=rss-122376ea00fe------2</link>
            <guid isPermaLink="false">https://medium.com/p/edf1600a4d38</guid>
            <category><![CDATA[busineß]]></category>
            <category><![CDATA[genai]]></category>
            <category><![CDATA[generative-ai-use-cases]]></category>
            <category><![CDATA[retrieval-augmented-gen]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Anurag Bhagat]]></dc:creator>
            <pubDate>Thu, 26 Dec 2024 06:58:46 GMT</pubDate>
            <atom:updated>2024-12-26T06:58:46.632Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Surprise surprise- Inference is not the biggest cost !</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TTK4MQcUAp9FI4YVeCCZxw.jpeg" /><figcaption>Source: Generated by AI with Gemini</figcaption></figure><h3>Introduction</h3><p>Building a Retrieval-Augmented Generation (RAG) based chatbot offers cutting-edge capabilities by combining large language models (LLMs) with powerful information retrieval systems. This is a common use case being implemented by organizations to improve their internal operations, sometimes to reduce call center loads and to improve overall experience.</p><p>However, many organizations embarking on this journey often fixate on the costs of model inference — the step where the chatbot generates responses from a language model. While inference costs are non-trivial, they are far from being the largest expense in a fully operational RAG-based system. The true cost drivers are much more nuanced, ranging from storage and database operations to development and ongoing maintenance. These typically are barriers to being able to scale such AI chatbots, and hence have to be understood early on.</p><p>In this article, we’ll explore the fully loaded costs of building, deploying, and maintaining a RAG-based chatbot, highlighting why inference is only one piece of a much larger puzzle.</p><h3>1. Infrastructure: The Foundation of the Chatbot</h3><p>The infrastructure required to run a RAG-based chatbot involves far more than simply hosting the language model. Key infrastructure components include cloud compute resources, storage, and networking, all of which significantly contribute to the overall cost.</p><h4>a) Cloud Compute (Virtual Machines and Instances)</h4><p>RAG-based chatbots typically operate on cloud platforms like AWS, Azure, or Google Cloud, where compute costs can scale rapidly. The cost of virtual machines (VMs) or compute instances depends on factors such as the complexity of the chatbot, the number of users, and the amount of data processed. Additionally, memory-intensive operations, like loading large datasets into memory and running search queries over vectorized data, require high-performance hardware, such as GPUs or high-CPU instances, increasing the cost further.</p><h4>b) Storage for Data and Embeddings</h4><p>When deploying a RAG-based chatbot, significant data storage is required — not just for the documents or knowledge base that the chatbot retrieves from, but also for the embeddings that represent this information.</p><h4>c) Networking Costs (Data Transfer)</h4><p>Networking costs, particularly for transferring data between regions or across services, can become a significant portion of the overall bill. When a user interacts with the chatbot, data is transferred across different layers: from the LLM to the document retrieval system and back to the user. If the system serves a global audience, inter-region data transfers further inflate these costs. Organizations often overlook networking costs, but they can snowball, especially as the chatbot scales to handle more users.</p><h3>2. Inference Costs: Not the Biggest Culprit</h3><p>Contrary to popular belief, inference costs — the cost of generating a response from an LLM — are not the largest part of the total expenses. While inference using APIs from providers like OpenAI, AWS Bedrock, or Azure OpenAI Service can get expensive, especially for large-scale deployments, it’s the context and number of tokens used in a response that drives the cost more than the inference itself.</p><p>For instance, while many focus on per-token pricing for API calls (e.g., $0.02 per 1000 tokens), the real driver is how much context (previous interactions or large documents) the chatbot needs to process with each request. However, with smart prompt engineering and efficient token management, inference costs can be controlled.</p><p>But even with efficient token usage, inference represents only a slice of the overall expense pie.</p><h3>3. Vector Databases: The Hidden Giant</h3><p>A key differentiator in RAG-based chatbots is the use of vector databases to store and retrieve relevant documents or passages from the knowledge base. Tools like ChromaDB, Pinecone, or Weaviate provide this functionality by enabling nearest-neighbor searches over vectorized data (embeddings). While this approach enhances the quality and relevance of the chatbot’s responses, it introduces a new cost center.</p><h4>a) Vector Search Complexity</h4><p>Performing vector searches requires powerful compute resources, especially when the database contains a large volume of embeddings. Each time a user query is received, the chatbot needs to perform a nearest-neighbor search over these vectors, which can be compute-intensive depending on the size of the dataset. As the index grows, so does the complexity of searches, leading to increased costs.</p><h4>b) Continuous Indexing</h4><p>Another underestimated aspect is the cost of continuously updating and indexing new data into the vector database. If the system needs to constantly ingest new documents, refresh embeddings, and maintain an up-to-date index, these operations can lead to significant compute and storage costs over time.</p><h3>4. Developer costs: The Gift That Keeps on Charging</h3><p>This is a relatively broad bucket which includes developer and tooling costs associate with initial development, integration, maintenance, incident management, and eventually scaling. This cost bucket is majorly comprised of precious developer time.</p><h4>a) Developer Time and Integration Costs</h4><p>The initial setup of a RAG-based chatbot requires significant developer hours for model integration, embedding generation, API usage, and vector database management. However, development costs don’t stop once the chatbot is live. Continuous optimization is needed to ensure low-latency responses, prompt improvements, and the fine-tuning of retrieval algorithms.</p><p>Many companies underestimate the <strong>integration costs</strong>, especially when connecting the chatbot to other systems (e.g., customer databases, CRMs, or ERPs). The effort required to manage these integrations, including error handling and data synchronization, adds to the total cost.</p><h4>b) Ongoing Maintenance and Support</h4><p>Once the system is in production, maintaining it becomes a continuous cost. Monitoring performance, scaling resources, managing uptime, and fixing bugs all require regular developer and DevOps input. Additionally, handling security, compliance (such as GDPR or HIPAA for data-sensitive applications), and patching vulnerabilities create an ongoing financial commitment.</p><h4>c) Incident Response</h4><p>Unexpected issues such as downtime, high latency, or system failures can occur, particularly in complex environments. Having a dedicated team on-call for incident response adds another layer of cost. This can be exacerbated if you need 24/7 support or have Service Level Agreements (SLAs) to meet.</p><h4>d) Monitoring and Logging</h4><p>To ensure the system operates smoothly, a RAG-based chatbot needs comprehensive monitoring and logging. Services like AWS CloudWatch or Azure Monitor, which track system health, response times, and resource usage, are critical for performance and reliability. However, these monitoring solutions come with recurring costs based on the volume of logs generated and the metrics monitored.</p><p>For chatbots with high traffic, log storage can grow significantly, especially if detailed logs are kept for every interaction, query, and system behavior. Retaining these logs for analysis or compliance purposes (often for extended periods) only adds to the costs.</p><h4><strong>e) Scaling</strong></h4><p>Finally, scaling a RAG-based chatbot can quickly multiply the costs, and scaling is rarely linear. As more users interact with the chatbot, not only does the load on the LLM increase, but so does the load on the vector database, storage, and compute infrastructure. Managing spikes in usage, particularly for high-traffic scenarios, often requires expensive autoscaling solutions.</p><p>Additionally, maintaining low-latency responses for global users might necessitate deploying instances in multiple regions, which further drives up the cost of compute, storage, and data transfer.</p><p>— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —</p><h3><strong>An illustrative example to make this come to life:</strong></h3><p>Let’s illustrate with an example, with a few assumptions about the organization setting up the RAG chatbot.</p><ul><li><strong>Global company</strong> with 10,000 employees</li><li>The chatbot will handle <strong>50,000 queries per month</strong> (5 queries per employee monthly, on average)</li><li>The organization stores a large amount of internal data (e.g., documentation, wikis, internal knowledge)</li><li>The system is deployed on cloud<strong> infrastructure</strong> (AWS, Azure, or GCP) with redundancy for global access</li><li><strong>RAG-based architecture</strong> includes a vector database (e.g., ChromaDB, Pinecone, or Weaviate) to handle document retrieval.</li></ul><p>Here is a rough breakdown of the monthly costs that one may expect from this RAG system</p><ol><li>Infrastructure across Compute, Storage and Networking: $3000–5000</li><li>Inference (LLM): $1000 — $3000</li><li>Vector Database (Search &amp; Index): $3,500 — $6,000</li><li>Developer costs : $10,000 — $20,000</li></ol><p><strong>Total Monthly Cost: $17,000 — $34,000</strong></p><p>As is evident, what might may been looking like a 1–3k a month project at the surface has quickly escalated into a half a million a year cost project, which will throw off most business cases. Some of these costs may be offset as one scales into multiple use cases, with shared resources, but fundamentally the vector database and developer cost buckets will not go away.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/892/1*hQ2QUzNRxFx_up7fx8-dgg.png" /><figcaption><em>Source: Author’s analysis</em></figcaption></figure><h3>Conclusion: The Full Picture</h3><p>While inference costs for using large language models in a RAG-based chatbot are significant, they pale in comparison to the hidden costs of storage, compute for vector searches, developer time, and ongoing maintenance. Building and deploying a successful RAG chatbot involves a complex web of infrastructure and operational costs that go far beyond simply generating responses.</p><p>The key to managing costs lies in understanding the entire lifecycle of the chatbot — from development and deployment to maintenance and scaling. By accurately estimating these costs, organizations can avoid unpleasant surprises and build chatbots that are not only effective but also sustainable in the long run. In short, inference is just the tip of the iceberg.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=edf1600a4d38" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[6 challenges I can’t wait for GenAI to solve]]></title>
            <link>https://generativeai.pub/6-challenges-i-cant-wait-for-genai-to-solve-717bf80989f4?source=rss-122376ea00fe------2</link>
            <guid isPermaLink="false">https://medium.com/p/717bf80989f4</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[genai]]></category>
            <category><![CDATA[generative-ai-use-cases]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Anurag Bhagat]]></dc:creator>
            <pubDate>Sun, 13 Oct 2024 07:13:30 GMT</pubDate>
            <atom:updated>2025-01-07T01:22:05.250Z</atom:updated>
            <content:encoded><![CDATA[<p>Generative AI (GenAI) is rapidly evolving, offering transformative solutions across various sectors. Despite its infancy, GenAI has already started to enhance human creativity, productivity, and problem-solving capabilities, making it a game-changer for the future. This article explores the current applications of GenAI while envisioning its potential to address 6 pressing issues in healthcare, education and climate change.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*6WUIMzVDBJnmrGEh" /><figcaption>Source: Generated with the help of AI (OpenAI’s Dall-E model)</figcaption></figure><h3><strong>1. What is Generative AI</strong></h3><p>Generative AI (GenAI) is like a human baby learning at a million times the velocity in a multidimensional way. Foundational models take data from multiple sources, learn from it and then create new content, be it text, images, video, or music, just like an accomplished artist would after years of practice and pattern recognition. It creates original work, which is often indistinguishable from one created by humans, even though there is some way to go before we reach <a href="https://www.cnbc.com/2024/01/16/openais-sam-altman-agi-coming-but-is-less-impactful-than-we-think.html">AGI</a>.</p><h3><strong>2. How is Generative AI being used today</strong></h3><p>GenAI is still in its infancy, and hence, just like a less informed human would, GenAI models do make mistakes(“hallucinate”). For instance, if I ask my 3-year-old how her day was, there is a 0.2% chance she will respond “tomato”- nothing wrong with that.</p><p>Even with its limitations, GenAI is transforming industries by enhancing human creativity and opening up bandwidth for higher intellect work.</p><ul><li><strong>Marketing</strong> is using GenAI for personalized content creation, targeted ads, and creative social media posts.</li><li><strong>Customer service</strong> has been enhanced through chatbots and RAG-enabled smart virtual assistants.</li><li><strong>Software development</strong> has been one of the major beneficiaries, revolutionizing software development and testing.</li><li><strong>Banking </strong>has been leveraging GenAI for fraud detection, risk prediction, claims processing, regulatory compliance, and financial advice assistance among others</li><li><strong>Healthcare </strong>is using it for it analysis of medical images, assisting claims processing and other high-criticality tasks like note summarization.</li></ul><h3>3. What else can we do- A wish list of 6 problems I would like GenAI to solve</h3><p>All these use cases, and other common ones, are merely scratching the surface of what GenAI can enable. As is evident from the <a href="https://www.tanayj.com/p/openai-and-anthropic-revenue-breakdown">revenue breakdown of OpenAI</a>, ~73% still comes from chatbot subscriptions(mostly from individuals), with ~27% coming from their APIs which is where enterprise usage is. There is so much more GenAI can potentially unlock — this is just the beginning!</p><p><strong>3.1. Better diagnosis and personalized treatment</strong></p><p>GenAI can help sift through vast volumes of medical journals, understand years of a patient’s medical history <a href="https://www.jnj.com/innovation/are-super-personalized-solutions-the-future-of-healthcare">or even genetics</a> as context, and help doctors diagnose rare conditions earlier or create tailored treatment plans. These could lead to better patient outcomes, especially for patients who are not benefitting from standardized treatment plans and might need a more individual approach. There is a long way to go before we would(and should) trust diagnosis without a doctor in the loop, but GenAI could be used as an assistive tool with the right guardrails.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*OM2yynbrffmGcjHC" /><figcaption>Source: Generated with the help of AI (OpenAI’s Dall-E model)</figcaption></figure><p><strong>3.2. Drug Discovery</strong></p><p>This might be one of the most talked about use cases of GenerativeAI, which can not just help cure diseases, but also reduce R&amp;D costs (which run in billions of dollars) which will in turn improve the affordability of healthcare, especially for diseases like cancer for which drug costs runs in <a href="https://prescancerpanel.cancer.gov/report/drugvalue/Part1.html">tens of thousands of dollars</a> and is increasing. Google’s Deepmind has been working on <a href="https://deepmind.google/technologies/alphafold/">AlphaFold</a> which has revealed millions of intricate 3D protein structures and is helping scientists understand how all of life’s molecules interact, reducing research time from months/years to just days.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*jYlTEYZLQ4y98JJa" /><figcaption>Source: Generated with the help of AI (OpenAI’s Dall-E model)</figcaption></figure><p><strong>3.3. Mental Health Support</strong></p><p><a href="https://www.who.int/news/item/17-06-2022-who-highlights-urgent-need-to-transform-mental-health-and-mental-health-care">According to WHO</a>, in 2019, nearly a billion people — including 14% of the world’s adolescents — were living with a mental disorder. Depression and anxiety went up by more than 25% in the first year of the COVID pandemic alone. Access to mental health resources is very limited, even if there is access, the associated stigma limits how many people leverage the resources available to them. GenerativeAI can help improve access and quality of mental health resources available to those who need them via personalized mental health support that is 24x7 accessible at low cost. This will become increasingly useful as GenAI foundational models advance multimodal functionality, and thus become more adept at recognizing emotional cues.</p><p>GenAI-powered mental health tools can help reduce associated stigma, by providing private, judgment-free therapeutic content, especially to underserved members of the community.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*JqNQl8QdwhpY8mid" /><figcaption>Source: Generated with the help of AI (OpenAI’s Dall-E model)</figcaption></figure><p><strong>3.4. Assistive Technologies for People with Disabilities</strong></p><p>We have all seen <a href="https://marvelcinematicuniverse.fandom.com/wiki/J.A.R.V.I.S.">J.A.R.V.I.S.</a> in action in the IronMan movies, it’s a natural language interface that provides Tony Stark with all the information he needs quickly. If we tone it down a little bit and focus on the essentials, multi-modal GenAI-enabled tools can have numerous users e.g.:</p><ul><li>Advanced speech recognition and text generation systems can help individuals with communication impairments express more fluently</li><li>Image/video-to-text/audio models can support those with visual impairments by describing surroundings or reading text out loud. <a href="https://about.fb.com/news/2024/09/introducing-orion-our-first-true-augmented-reality-glasses/">Meta’s Orion glasses</a> may be a big step in this direction.</li></ul><p>By continuously learning and adapting, GenAI-based assistive technologies can empower users with enhanced independence and integration into digital and physical spaces.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*E6g9MXPQhc1viw4P" /><figcaption>Source: Generated with the help of AI (OpenAI’s Dall-E model)</figcaption></figure><p><strong>3.5. Personalized Education</strong></p><p>Generative AI can revolutionize education by creating tailored learning paths for individual students, catering to diverse learning styles and needs. It can also be a tutor, guiding students step-by-step, as we saw on the<a href="https://www.youtube.com/watch?v=IvXZCocyU_M&amp;t=1s"> demo for GPT-4o</a>. By personalizing the learning journey, Generative AI empowers students to learn at their own pace, deepening understanding and enhancing educational outcomes across diverse subjects. This can break down barriers to learning which have been lowered by online learning. Some of this is already available on <a href="https://fortune.com/education/articles/coursera-ceo-jeff-maggioncalda-online-learning-in-2024/">Coursera</a> where “<em>AI is being deployed directly as a learning tool in the form of a personalized learning companion named Coach. Students can ask the chatbot to provide help on a concept, create practice problems, or summarise learning activities.”</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*TNToqoJ9oUicSCFI" /><figcaption>Source: Generated with the help of AI (OpenAI’s Dall-E model)</figcaption></figure><p><strong>3.6. Climate Change research and solutions</strong></p><p>Climate change is real, and it could worsen owing to this new GenAI era. <a href="https://blog.bosch-digital.com/generative-ai-and-its-potential-environmental-impact/">One of the most significant</a> potential environmental impacts of GenAI is the energy consumption. It requires enormous computational power to process the large amount of data, which is needed to generate new information. In January 2023, ChatGPT had roughly 590 million visits. With approximately 5 questions per user, this amounted to the same energy consumption as 175,000 people consumed in the same period. It is hence even more crucial that GenAI help solve this problem which it is currently fueling.</p><p>Gen AI can make a big dent in tackling climate change by optimizing energy consumption, accelerating scientific research, and supporting sustainable agriculture. It can enhance efficiency in smart grids, aid in discovering new materials for carbon capture, and improve renewable energy technologies. GenAI-driven models can also monitor environmental changes, such as deforestation and air quality, through real-time data analysis. Additionally, AI helps governments and organizations model climate scenarios and predict the impact of policy decisions. The avenues are endless, <a href="https://aws.amazon.com/blogs/startups/how-climate-tech-startups-use-generative-ai-to-address-the-climate-crisis/">the work has only begun.</a></p><h3>4. Conclusion</h3><p>GenAI has near-unlimited potential to tackle some of the world’s most pressing challenges. From enhancing healthcare through personalized treatments, drug discovery, mental health support and assistive technologies, to revolutionizing education GenAI is already improving lives. The potential for GenAI to address climate change is particularly crucial, especially given GenAI is now part of the problem. As we build these solutions, ensuring these are designed with user-centric guardrails is going to be crucial.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*mD_Bi840gE5rxqKZ.png" /></figure><p>This story is published on <a href="https://generativeai.pub/">Generative AI</a>. Connect with us on <a href="https://www.linkedin.com/company/generative-ai-publication">LinkedIn</a> and follow <a href="https://www.zeniteq.com/">Zeniteq</a> to stay in the loop with the latest AI stories.</p><p>Subscribe to our <a href="https://www.generativeaipub.com/">newsletter</a> and <a href="https://www.youtube.com/@generativeaipub">YouTube</a> channel to stay updated with the latest news and updates on generative AI. Let’s shape the future of AI together!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*BLBgQeqnOWm2v3TU.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=717bf80989f4" width="1" height="1" alt=""><hr><p><a href="https://generativeai.pub/6-challenges-i-cant-wait-for-genai-to-solve-717bf80989f4">6 challenges I can’t wait for GenAI to solve</a> was originally published in <a href="https://generativeai.pub/">Generative AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Scaling RAG from POC to Production]]></title>
            <link>https://medium.com/data-science/scaling-rag-from-poc-to-production-31bd45d195c8?source=rss-122376ea00fe------2</link>
            <guid isPermaLink="false">https://medium.com/p/31bd45d195c8</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[genai]]></category>
            <category><![CDATA[retrieval-augmented-gen]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[chatgpt]]></category>
            <dc:creator><![CDATA[Anurag Bhagat]]></dc:creator>
            <pubDate>Mon, 07 Oct 2024 18:40:56 GMT</pubDate>
            <atom:updated>2024-10-08T20:40:38.657Z</atom:updated>
            <content:encoded><![CDATA[<h4>Common challenges and architectural components to enable scaling</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*33pjuHPUAC-01Hik" /><figcaption><em>Source: Generated with the help of AI (OpenAI’s Dall-E model)</em></figcaption></figure><h3>1. Introduction</h3><h4>1.1. Overview of RAG</h4><p>Those of you who have been immersed in generative AI and its large-scale applications outside of personal productivity apps have likely come across the notion of Retrieval Augmented Generation or RAG. The RAG architecture consists of two key components—the retrieval component which uses vector databases to do an index based search on a large corpus of documents. This is then sent over to a large language model (LLM) to generate a grounded response based on the richer context in the prompt.</p><p>Whether you are building customer-facing chatbots to answer repetitive questions and reduce workload from customer service agents, or building a co-pilot for engineers to help them navigate complex user manuals step-by-step, RAG has become a key archetype of the application of LLMs. This has enabled LLMs to provide a contextually relevant response based on ground truth of hundreds or millions of documents, reducing hallucinations and improving the reliability of LLM-based applications.</p><h4>1.2. Why scale from Proof of Concept(POC) to production</h4><p>If you are asking this question, I might challenge you to answer why are you even building a POC if there is no intent of getting it to production. Pilot purgatory is a common risk with organisations that start to experiment, but then get stuck in experimentation mode. Remember that POCs are expensive, and true value realisation only happens once you go into production and do things at scale- either freeing up resources, making them more efficient, or creating additional revenue streams.</p><h3>2. Key challenges in scaling RAG</h3><h4>2.1. Performance</h4><p>Performance challenges in RAGs come in various flavours. The speed of retrieval is generally not the primary challenge unless your knowledge corpus has millions of documents, and even then it can be solved by setting up the right infrastructure- of course, we are limited by inference times. The second performance problem we encounter is around getting the “right” chunks to be fed to the LLMs for generation, with a high level of precision and recall. The poorer the retrieval process is, the less contextually relevant the LLM response will be.</p><h4>2.2. Data Management</h4><p>We have all heard the age-old saying “garbage in garbage out (GIGO)”. RAG is nothing but a set of tools we have at our disposal, but the real value comes from the actual data. As RAG systems work with unstructured data, it comes with its own set of challenges including but not limited to- version control of documents, and format conversion (e.g. pdf to text), among others.</p><h4>2.3. Risk</h4><p>One of the biggest reasons corporations hesitate to move from testing the waters to jumping in is the possible risks that come with using AI based systems. Hallucinations are definitely lowered with the use of RAG, but are still non-zero. There are other associated risks including risks for bias, toxicity, regulatory risks etc. which could have long term implications.</p><h4>2.4. Integration into existing workflows</h4><p>Building an offline solution is easier, but bringing in the end users’ perspective is crucial to make sure the solution does not feel like a burden. No users want to go to another screen to use the “new AI feature”- users want the AI features built into their existing workflows so the technology is assistive, and not disruptive to the day-to-day.</p><h4>2.5. Cost</h4><p>Well, this one seems sort of obvious, doesn’t it? Organisations are implementing GenAI use cases so that they can create business impact. If the benefits are lower than we planned, or there are cost overruns, the impact would be severely diminished, or also completely negated.</p><h3>3. Architectural components needed for Scaling</h3><p>It would be unfair to only talk about challenges if we don’t talk about the “so what do we do”. There are a few essential components you can add to your architecture stack to overcome/diminish some of the problems we outlined above.</p><h4>3.1. Scalable vector databases</h4><p>A lot of teams, rightfully, start with open-source vector databases like <a href="https://www.trychroma.com/">ChromaDB</a>, which are great for POCs as they are easy to use and customise. However, it may face challenges with large-scale deployments. This is where scalable vector databases come in (such as <a href="https://www.pinecone.io/">Pinecone</a>, <a href="https://weaviate.io/platform">Weaviate</a>, <a href="https://milvus.io/">Milvus</a>, etc.) which are optimised for high-dimensional vector searches, enabling fast (sub-millisecond), accurate retrieval even as the dataset size increases into the millions or billions of vectors as they use <a href="https://www.mongodb.com/resources/basics/ann-search">Approximate Nearest Neighbour</a> search techniques. These vector databases have APIs, plugins, and SDKs that allow for easier workflow integration and they are also horizontally scalable. Depending on the platform one is working on- it might make sense to explore vector databases offered by <a href="https://www.databricks.com/product/machine-learning/vector-search">Databricks</a> or <a href="https://aws.amazon.com/opensearch-service/">AWS</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*8WfH_YWkpuH4w-n-" /><figcaption><em>Source: Generated with the help of AI (OpenAI’s Dall-E model)</em></figcaption></figure><h4>3.2. Caching Mechanisms</h4><p>The concept of caching has been around almost as long as the internet, <a href="https://cacm.acm.org/opinion/ibms-single-processor-supercomputer-efforts/#:~:text=a.,computer%20system%20to%20use%20cache">dating back to</a> the 1960’s. The same concept applies to GenerativeAI as well—If there are a large number of queries, maybe in the millions (very common in the customer service function), it is likely that many queries are the same or extremely similar. Caching allows one to avoid sending a request to the LLM if we can instead return a response from a recent cached response. This serves two purposes- reduced costs, as well as better response times for common queries.</p><p>This can be implemented as a memory Cache (in-memory caches like <a href="https://redis.io/">Redis</a> or <a href="https://memcached.org/">Memcached</a>), Disk Cache for less frequent queries or distributed Cache (Redis Cluster). Some model providers like Anthropic offer <a href="https://www.anthropic.com/news/prompt-caching">prompt caching</a> as part of their APIs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*75sRhd3kFKHQYtM3" /><figcaption><em>Source: Generated with the help of AI (OpenAI’s Dall-E model)</em></figcaption></figure><h3>3.3. Advanced Search Techniques</h3><p>While not as crisply an architecture component, multiple techniques can help elevate the search to enhance both efficiency and accuracy. Some of these include:</p><ul><li><strong>Hybrid Search: I</strong>nstead of relying only on semantic search(using vector databases), or keyword search, use a combination to boost your search.</li><li><strong>Re-ranking: </strong>Use a LLM or SLM to calculate a relevancy score for the query with each search result, and re-rank them to extract and share only the highly relevant ones. This is particularly useful for complex domains, or domains where one may have many documents being returned. One example of this is <a href="https://aws.amazon.com/marketplace/pp/prodview-pf7d2umihcseq">Cohere’s Rerank</a>.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_sNyvu_DKRMbauBo" /><figcaption><em>Source: Generated with the help of AI (OpenAI’s Dall-E model)</em></figcaption></figure><h3>3.4. Responsible AI layer</h3><p>Your Responsible AI modules have to be designed to mitigate bias, ensure transparency, align with your organisation’s ethical values, continuously monitor for user feedback and track compliance to regulation among other things, relevant to your industry/function. There are many ways to go about it, but fundamentally this has to be enabled programmatically, with human oversight. A few ways it can be done that can be done:</p><ul><li><strong>Pre-processing: </strong>Filter user queries before they are ever sent over to the foundational model. This may include things like checking for bias, toxicity, un-intended use etc.</li><li><strong>Post-processing: </strong>Apply another set of checks after the results come back from the FMs, before exposing them to the end users.</li></ul><p>These checks can be enabled as small reusable modules you buy from an external provider, or build/customise for your own needs. One common way organisations have approached this is to use carefully engineered prompts and foundational models to orchestrate a workflow and prevent a result reaching the end user till it passes all checks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*rFY6iQj3GT100a6z" /><figcaption><em>Source: Generated with the help of AI (OpenAI’s Dall-E model)</em></figcaption></figure><h4>3.5. API Gateway</h4><p>An API Gateway can serve multiple purposes helping manage costs, and various aspects of Responsible AI:</p><ul><li>Provide a unified interface to interact with foundational models, experiment with them</li><li>Help develop a fine-grained view into costs and usage by team/use case/cost centre — including rate-limiting, speed throttling, quota management</li><li>Serve as a responsible AI layer, filtering out in-intended requests/data before they ever hit the models</li><li>Enable audit trails and access control</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_3O2kndIydQq9cva" /><figcaption><em>Source: Generated with the help of AI (OpenAI’s Dall-E model)</em></figcaption></figure><h3>4. Is this enough, or do we need more?</h3><p>Of course not. There are a few other things that also need to be kept in mind, including but not limited to:</p><ul><li>Does the use case occupy a strategic place in your roadmap of use cases? This enables you to have leadership backing, and right investments to support the development and maintenance.</li><li>A clear evaluation criterion to measure the performance of the application, against dimensions of accuracy, cost, latency and responsible AI</li><li>Improve business processes to keep knowledge up to date, maintain version control etc.</li><li>Architect the RAG system so that it only accesses documents based on the end user permission levels, to prevent unauthorised access.</li><li>Use design thinking to integrate the application into the workflow of the end user e.g. if you are building a bot to answer technical questions over Confluence as the knowledge base, should you build a separate UI, or integrate this with Teams/Slack/other applications users already use?</li></ul><h3>5. Conclusion</h3><p>RAGs are a prominent use case archetype, and one of the first few ones that organisations try to implement. Scaling RAG from POC to production comes with its challenges, but with careful planning and execution, many of these can be overcome. Some of these can be solved by tactical investment in the architecture and technology, some require better strategic direction and tactful planning. As LLM inference costs continue to drop, either owing to reduced inference costs or heavier adoption of open-source models, cost barriers may not be a concern for many new use cases.</p><p><em>All views in this article are the Author’s and don’t represent an endorsement of any products or services.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=31bd45d195c8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/scaling-rag-from-poc-to-production-31bd45d195c8">Scaling RAG from POC to Production</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Evaluating performance of LLM-based Applications]]></title>
            <link>https://medium.com/data-science/evaluating-performance-of-llm-based-applications-be6073c02421?source=rss-122376ea00fe------2</link>
            <guid isPermaLink="false">https://medium.com/p/be6073c02421</guid>
            <category><![CDATA[generative-ai-solution]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[generative-ai-use-cases]]></category>
            <category><![CDATA[llm-evaluation]]></category>
            <dc:creator><![CDATA[Anurag Bhagat]]></dc:creator>
            <pubDate>Mon, 30 Sep 2024 22:13:40 GMT</pubDate>
            <atom:updated>2024-10-02T18:44:49.017Z</atom:updated>
            <content:encoded><![CDATA[<h4><strong>Evaluation Framework for </strong>real-world requirements</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*92JNZxysq8pHfE_DCKTOxg.jpeg" /><figcaption><em>Source: Generated with the help of AI (OpenAI’s Dall-E model)</em></figcaption></figure><h3>Abstract</h3><p>Ever since OpenAI’s ChatGPT took the world by storm in November 2022, Large Language Models (LLMs) have revolutionized various applications across industries, from natural language understanding to text generation. However, their performance needs rigorous and multidimensional evaluation metrics to ensure they meet the practical, real-world requirements of accuracy, efficiency, scalability, and ethical considerations. This article outlines a broad set of metrics and methods to measure the performance of LLM-based applications, providing insights into evaluation frameworks that balance technical performance with user experience and business needs.</p><p>This is not meant to be a comprehensive guide on all metrics to measure the performance of LLM applications, but it provides a view into key dimensions to look at and some examples of metrics. This will help you understand how to build your evaluation criterion, the final choice will depend on your actual use case.</p><p>Even though this article focuses on LLM based applications, this could be extrapolated to other modalities as well.</p><h3>1. Introduction</h3><h4>1.1. LLM-Based Applications: Definition and Scope</h4><p>There is no dearth of Large Language Models(LLMs) today. LLMs such as GPT-4, Meta’s LLaMA, Anthropic’s Claude 3.5 Sonnet, or Amazon’s Titan Text Premier, are capable of understanding and generating human-like text, making them apt for multiple downstream applications like customer facing chatbots, creative content generation, language translation, etc.</p><h4>1.2. Importance of Performance Evaluation</h4><p>LLMs are non-trivial to evaluate, unlike traditional ML models, which have pretty standardized evaluation criteria and datasets. The black box nature of LLMs, as well as the multiplicity of downstream use cases warrants a multifaceted performance measurement across multiple considerations. Inadequate evaluation can lead to cost overruns, poor user experience, or risks for the organization deploying them.</p><h3>2. The four key dimensions of LLM Performance</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*DlvGgjRj8SdfO8Da" /><figcaption>Source: Generated with the help of AI (OpenAI’s Dall-E model)</figcaption></figure><p>There are 3 key ways to look at the performance of LLM based applications- namely accuracy, cost, and latency. It is additionally critical to make sure to have a set of criteria for Responsible AI to ensure the application is not harmful.</p><p>Just like the bias vs. variance tradeoff we have in classical Machine Learning applications, for LLMs we have to consider the tradeoff between accuracy on one side and cost + latency on the other side. In general, it will be a balancing act, to create an application that is “accurate”(we will define what this means in a bit) while being fast enough and cost effective. The choice of LLM as well as the supporting application architecture will heavily depend on the end user experience we aim to achieve.</p><h4><strong>2.1. Accuracy</strong></h4><p>I use the term “Accuracy” here rather loosely, as it has a very specific meaning, but gets the point across if used as an English word rather than a mathematical term.</p><p>Accuracy of the application depends on the actual use case- whether the application is doing a classification task, if it’s creating a blob of text, or if it is being used for specialized tasks like Named Entity Recognition (NER), Retrieval Augmented Generation (RAG).</p><p><strong>2.1.1. Classification use cases</strong></p><p>For classification tasks like sentiment analysis (positive/negative/neutral), topic modelling and Named Entity Recognition classical ML evaluation metrics are appropriate. They measure accuracy in terms of various dimensions across the confusion matrix. Typical measures include Precision, Recall, F1-Score etc.</p><p><strong>2.1.2. Text generation use cases — including summarization and creative content</strong></p><p><a href="https://en.wikipedia.org/wiki/BLEU">BLEU</a>, <a href="https://en.wikipedia.org/wiki/ROUGE_(metric)#:~:text=The%20metrics%20compare%20an%20automatically,produced%20summary%20and%20the%20reference">ROUGE</a> and <a href="https://en.wikipedia.org/wiki/METEOR">METEOR</a> scores are common metrics used to evaluate text generation tasks, particularly for translation and summarization. To simplify, people also use F1 scores by combining BLEU and ROUGE scores. There are additional metrics like Perplexity which are particularly useful for evaluating LLMs themselves, but less useful to measure the performance of full blown applications. The biggest challenge with all the above metrics is that they focus on text similarity and not semantic similarity. Depending on the use case, text similarity may not be enough, and one should also use measures of semantic proximity like <a href="https://huggingface.co/blog/g-ronimo/semscore">SemScore</a>.</p><p><strong>2.1.3. RAG use cases — including summarization and creative content</strong></p><p>In RAG based applications, evaluation requires advanced metrics to capture performance across retrieval as well as generation steps. For retrieval, one may use recall and precision to compare relevant and retrieved documents. For generation one may use additional metrics like Perplexity, Hallucination Rate, Factual Accuracy or Semantic coherence. <a href="https://medium.com/@daniel.puenteviejo/key-metrics-for-evaluating-a-rag-system-f20b8bcae318">This Article</a> describes the key metrics that one might want to include in their evaluation.</p><h4>2.2. Latency (and Throughput)</h4><p>In many situations, latency and throughput of an application determine its end usability, or use experience. In today’s generation of lightning fast internet, users do not want to be stuck waiting for a response, especially when executing critical jobs.</p><p>The lower the latency, the better the user experience in user-facing applications which require real time response. This may not be as important for workloads that execute in batches, e.g. transcription of customer service calls for later use. In general, both latency and throughput can be improved by horizontal or vertical scaling, but latency may still fundamentally depend on the way the overall application is architected, including the choice of LLM. A nice benchmark to use speed of different LLM APIs is <a href="https://artificialanalysis.ai/">Artificial Analysis</a>. This complements other leaderboards that focus on the quality of LLMs like LMSYS Chatbot Arena, Hugging Face open LLM leaderboards, and Stanford’s HELM which focus more on the quality of the outputs.</p><p>Latency is a key factor that will continue to push us towards Small Language Models for applications that require fast response time, where deployment on edge devices might be a necessity.</p><h4>2.3. Cost</h4><p>We are building LLM applications to solve business problems and create more efficiencies, with the hope of solving customer problems, as well as creating bottom line impact for our businesses. All of this comes at a cost, which could add up quickly for generative AI applications.</p><p>In my experience, when people think of the cost of LLM applications, there is a lot of discussion about the cost of inference (which is based on #tokens), the cost of find tuning, or even the cost of pre-training a LLM. There is however limited discussion on the total cost of ownership, including infrastructure and personnel costs.</p><p>The cost can vary based on the type of deployment (cloud, on-prem, hybrid), the scale of usage, and the architecture. It also varies a lot depending on the lifecycle of the application development.</p><ul><li><strong>Infrastructure costs</strong> — includes inference, tuning costs, or potentially pre-training costs as well as the infrastructure — memory, compute, networking, and storage costs associated with the application. Depending on where one is building the application, these costs may not need to be managed separately, or bundled into one if one if using managed services like AWS Bedrock.</li><li><strong>Team and Personnel cost</strong>- we may sometimes need an army of people to build, monitor, and improve these applications. This includes the engineers to build this (Data Scientists and ML Engineers, DevOps and MLOps engineers) as well as the cross functional teams of product/project managers, HR, Legal and Risk personnel who are involved in the design and development. We may also have annotation and labelling teams to provide us with high quality data.</li><li><strong>Other costs</strong>- which may include the cost of data acquisition and management, customer interviews, software and licensing costs, Operational costs (MLOps/LLMOps), Security, and Compliance.</li></ul><h4>2.4. Ethical and Responsible AI Metrics</h4><p>LLM based applications are still novel, many being mere proof of concepts. At the same time, they are becoming mainstream- I see AI integrated into so many applications I use daily, including Google, LinkedIn, Amazon shopping app, WhatsApp, InstaCart, etc. As the lines between human and AI interaction become blurrier, it becomes more essential that we adhere to responsible AI standards. The bigger problem is that these standards don’t exist today. Regulations around this are still being developed across the world (including the <a href="https://www.whitehouse.gov/briefing-room/presidential-actions/2023/10/30/executive-order-on-the-safe-secure-and-trustworthy-development-and-use-of-artificial-intelligence/">Executive Order from the White House</a>). Hence, it’s crucial that application creators use their best judgment. Below are some of the key dimensions to keep in mind:</p><ul><li><strong>Fairness and Bias</strong>: Measures whether the model’s outputs are free from biases and fairness related to race, gender, ethnicity, and other dimensions.</li><li><strong>Toxicity</strong>: Measures the degree to which the model generates or amplifies harmful, offensive, or derogatory content.</li><li><strong>Explainability</strong>: Assesses how explainable the model’s decisions are.</li><li><strong>Hallucinations/Factual Consistency</strong>: Ensures the model generates factually correct responses, especially in critical industries like healthcare and finance.</li><li><strong>Privacy</strong>: Measures the model’s ability to handle PII/PHI/other sensitive data responsibly, compliance with regulations like GDPR.</li></ul><h3>3. So are these metrics enough?</h3><p>Well… not really! While the four dimensions and metrics we discussed are essential and a good starting point, they are not always enough to capture the context, or unique user preferences. Given that humans are typically end consumers of the outputs, they are best positioned to evaluate the performance of LLM based applications, especially in complex or unknown scenarios. There are two ways to take human input:</p><ul><li><strong>Direct via human-in-the-loop: </strong>Human evaluators provide qualitative feedback on the outputs of LLMs, focusing on fluency, coherence, and alignment with human expectations. This feedback is crucial for improving the human-like behaviour of models.</li><li><strong>Indirect via secondary metrics: </strong>A|B testing from end users can compare secondary metrics like user engagement and satisfaction. E.g., we can compare the performance of hyper-personalized marketing using generative AI by comparing click through rates and conversion rates.</li></ul><h3>4. Conclusion</h3><p>As a consultant, the answer to most questions is “It depends.”. This is true for evaluation criteria for LLM applications too. Depending on the use case/industry/function, one has to find the right balance of metrics across accuracy, latency, cost, and responsible AI. This should always be complemented by a human evaluation to make sure that we test the application in a real-world scenario. For example, medical and financial use cases will value accuracy and safety as well as attribution to credible sources, entertainment applications value creativity and user engagement. Cost will remain a critical factor while building the business case for an application, though the fast dropping cost of LLM inference might reduce barriers of entry soon. Latency is usually a limiting factor, and will require right model selection as well as infrastructure optimization to maintain performance.</p><blockquote>All views in this article are the Author’s and don’t represent an endorsement of any products or services.</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=be6073c02421" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/evaluating-performance-of-llm-based-applications-be6073c02421">Evaluating performance of LLM-based Applications</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>