<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Venky Karthik on Medium]]></title>
        <description><![CDATA[Stories by Venky Karthik on Medium]]></description>
        <link>https://medium.com/@karthikvgen1?source=rss-e10c86b77188------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*VfRaYZo5VesNLPwW</url>
            <title>Stories by Venky Karthik on Medium</title>
            <link>https://medium.com/@karthikvgen1?source=rss-e10c86b77188------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 15:59:04 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@karthikvgen1/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Scaling Gen AI — Perils to avoid]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/@karthikvgen1/scaling-gen-ai-perils-to-avoid-37bbb1d8eca9?source=rss-e10c86b77188------2"><img src="https://cdn-images-1.medium.com/max/1374/1*UhW7TrI6fexQOLst-cc9NQ.jpeg" width="1374"></a></p><p class="medium-feed-snippet">Over the past year, I&#x2019;ve had the privilege of working with clients to design, build, and deploy generative AI (Gen AI) use cases. Some of&#x2026;</p><p class="medium-feed-link"><a href="https://medium.com/@karthikvgen1/scaling-gen-ai-perils-to-avoid-37bbb1d8eca9?source=rss-e10c86b77188------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://medium.com/@karthikvgen1/scaling-gen-ai-perils-to-avoid-37bbb1d8eca9?source=rss-e10c86b77188------2</link>
            <guid isPermaLink="false">https://medium.com/p/37bbb1d8eca9</guid>
            <category><![CDATA[agentic-ai]]></category>
            <category><![CDATA[gen-ai-for-business]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[enterprise-ai]]></category>
            <dc:creator><![CDATA[Venky Karthik]]></dc:creator>
            <pubDate>Sat, 04 Jan 2025 21:11:49 GMT</pubDate>
            <atom:updated>2025-01-04T21:35:00.537Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Using Open Source LLM Frameworks from UC Berkeley Skylab]]></title>
            <link>https://medium.com/@karthikvgen1/using-open-source-llm-frameworks-from-uc-berkeley-skylab-669cf51adef6?source=rss-e10c86b77188------2</link>
            <guid isPermaLink="false">https://medium.com/p/669cf51adef6</guid>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[agents]]></category>
            <category><![CDATA[openai]]></category>
            <category><![CDATA[large-language-models]]></category>
            <dc:creator><![CDATA[Venky Karthik]]></dc:creator>
            <pubDate>Sun, 23 Jun 2024 15:19:32 GMT</pubDate>
            <atom:updated>2024-06-23T15:59:22.642Z</atom:updated>
            <content:encoded><![CDATA[<h3>Agentic Applications using Open Source LLM Frameworks from UC Berkeley Skylab</h3><p>I recently had the opportunity to deep dive into a number of open source LLM frameworks from the renowned Skylab at UC Berkeley. This is the same group that incubated Spark before it became Databricks! This group works on technologies that are years ahead in terms of what developers and product companies are solving for right now, hence I believe it is very relevant for our current generation of LLM developers and architects to know what these are and how to use it in practice.</p><p>As the design of Generative AI applications evolve from simple RAG based, single LLM architectures to more advanced ones that go beyond static prompt templates and involve integrations between multiple LLMs, frameworks such as these allow application architects to reuse the design patterns, design principles and framework components to improve the robustness and maintainability of current generation of LLM based applications as the underlying model capabilities evolve and as more models become available across both proprietary as well as open source ones.</p><p>The diagram below depicts what these open source frameworks and components are and how they interact with each other. In this article, we will go over what some of the application layer components are and how they can be used to build an end to end application.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SS7dTsZJMsusNYhiBYd5qQ.png" /><figcaption>Open Source LLM Frameworks and Tools from Berkeley Skylabs</figcaption></figure><blockquote><strong>Firstly, the Prompt layer with DsPY</strong> Today, prompting an LLM application is an iterative exercise with manual hand-crafted prompts to yield the best possible response from the LLM. Imagine if the prompt generation itself is left to an optimization function that searches through a number of possible prompts to arrive at the best possible one for your task. This the basic idea behind DsPY (Declarative Self Improving Language Programs ). This has a few important advantages, namely:</blockquote><ul><li>The developer focuses on the defining the metrics of evaluation of LLM output and leaves the generation of the prompt itself to one of the optimizers used by DsPY.</li><li>Based on the evaluation metrics that have been defined and the optimal output obtained using a larger LLM, if you have enough evaluation samples, you can use the same metric and an optimization function on a different smaller sized LLM to fine tune it’s weights — thereby providing you with options to choose the LLM that you will eventually deploy. The key takeaway here is that your prompts are no longer ‘hard-wired’ to your LLM.</li><li>Most LLM applications today would likely use frameworks like Langchain or LlamaIndex that come with a set of ‘pre-defined prompts’ (aka ‘batteries included’) that work out of the box. But if you are looking to build an application that needs to adapt to it’s data in a way that static prompt templates do not, then DsPY provides a very effective way to do so.</li><li>There is a high learning curve, as is the case with most frameworks that allow you to create your own learning and inferencing architecture (e.g. using Pytorch or Tensorflow) but the ROI in terms of being able to conceive, define and build applications that can evolve with their data and LLM could be significant.</li></ul><blockquote><strong>2. Secondly, the Agentic layer with MemGPT.</strong> MemGPT provides a framework to build long running, stateful agents that have access to a long term, persistent memory. This integrates with an LLM of your choice while providing a programmatic mechanism to integrate this with other agents that are either autonomous or user controlled. So, how does one use this?</blockquote><ul><li>The simplest and fastest way is to deploy it as a personal agent on your machine. Configurations are provided to point it to an LLM of your choice and persistent memory store of your choice, for e.g. SQL lite which is built in or other DBs which support vector space embedding searches including PostgreSQL, Chroma and others.</li><li>RAG capabilities are in-built within the personal agent, so it’s fairly straightforward to point it to your document store and use it to interact with your own documents.</li><li>The framework itself is quite comprehensive allowing further customizations to build your own application including integrating with custom document retrievers and pipelines in other frameworks in Langchain or LlamaIndex.</li></ul><blockquote><strong>3. The third and last component that we will cover in this article is the Function or Task Layer.</strong> This layer determines the right API function to invoke based on inputs or task given to the LLM. The Gorilla Function or Task invocation capability consists of three distinct components:</blockquote><ol><li>The core <strong>‘Function aware’ LLM</strong> called Gorilla LLM which is a fine tuned version of the Llama3 model. This fine tuned version is one of the most accurate LLMs for API invocation and benchmark results on API invocation accuracy are available.</li><li>The <strong>Function Generation layer</strong>, called Open Functions, which formulates/generates the executable function or API call based on the natural language instructions provided and the API context. This is an open source alternative to the function calling capabilities seen in proprietary LLMs such as Open AI or Cohere.</li><li>The <strong>Function Execution layer</strong> which invokes the API returned by the Function Generation layer. The design of this layer is based on ‘post-facto LLM validation’ and provides abstractions that developers can use to quantify and assess the associated risks of executing the target function and setup undo or reversibility mechanisms in conjunction with the function execution after verification of the result of function execution e.g. to rollback a database transaction, recall a sent email or Slack message and so on. I believe this mechanism is most effective when the target function itself provides the necessary semantics for undo transactions as is the case in complex microservices settings.</li></ol><h3><strong>Use Case Scenarios</strong></h3><ol><li>Imagine a developer working on an application that needs to interact with several third-party APIs to fetch weather data, user information, and payment processing. Using Gorilla LLM and Gorilla Function Calling, the developer can:</li><li><strong>Generate API Calls</strong>: Automatically generate the required API calls with the correct syntax and parameters.</li><li><strong>Handle Responses</strong>: Ensure that the responses from these APIs are correctly parsed and utilized in the application.</li><li><strong>Error Management</strong>: Efficiently manage any errors or exceptions that occur during the API interactions.</li></ol><p>This approach not only speeds up development but also reduces the likelihood of bugs related to incorrect API usage.</p><p>In the coming days, I will further unpack the capabilities, limitations and use cases for these frameworks along with other patterns and tools that I come across in my work across different LLM projects.</p><p><strong>Reference links:</strong></p><p><a href="https://dspy-docs.vercel.app/">https://dspy-docs.vercel.app</a></p><p><a href="https://memgpt.ai/">https://memgpt.ai</a></p><p><a href="https://gorilla.cs.berkeley.edu/">https://gorilla.cs.berkeley.edu</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=669cf51adef6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building LLM applications with Regulatory Documents]]></title>
            <link>https://medium.com/@karthikvgen1/building-llm-applications-with-regulatory-documents-e11b81785684?source=rss-e10c86b77188------2</link>
            <guid isPermaLink="false">https://medium.com/p/e11b81785684</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[retrieval-augmented]]></category>
            <category><![CDATA[regulatory-compliance]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[large-language-models]]></category>
            <dc:creator><![CDATA[Venky Karthik]]></dc:creator>
            <pubDate>Sun, 23 Jun 2024 14:31:20 GMT</pubDate>
            <atom:updated>2024-06-24T21:54:38.138Z</atom:updated>
            <content:encoded><![CDATA[<p>In the course of working with LLMs, the domain of regulatory documents presents great opportunities but also some unique and complex challenges.</p><p>In this article, I will go over these in the context of a use case which we built working alongside a leading AI research organization using Retrieval Augmented Generation (RAG) techniques.</p><p>Let’s kickoff by asking what makes regulatory documents unique and special from other documents that LLMs have been used with.</p><ol><li>Accuracy and Robustness — unlike general knowledge Q&amp;A, answers to queries posed regarding regulatations and policies need a precise, accurate and consistent answer.</li><li>A-priori domain knowledge — whether it is SEC filings, municipal bylaws, or healthcare policies, each domain requires varying degrees of prior domain knowledge to provide an answer. Note though this prior domain knowledge can be taught to LLMs.</li><li>Format and representational variety — Regulations are communicated and notified using a variety of representations and formats including XBRL, XML, JSON, PDF, HTML etc. Semantic and Structure extraction of content from such regulatory documents has to make the best possible use of the design choices that such formats provide.</li></ol><p>I’ll start with the very basics. We used a technique called RAG to do some of this. RAG is a collection of methods and techniques designed to enhance the natural language capabilities of large language models (LLMs) by incorporating external data retrieved from documents, web searches, databases, and other sources.</p><p>Businesses should consider implementing RAG when they need AI systems that can leverage both pre-existing knowledge and generative capabilities to provide contextually relevant and accurate responses. There are many potential use cases in which a basic LLM can be augmented with real-time and/or domain-specific data to produce more accurate and timely results. RAG is particularly useful in scenarios requiring complex information retrieval, personalized customer support, and content creation where context plays a key role in generating meaningful outputs.</p><p>Some applications of RAG include:</p><p>· Healthcare — Assisting in diagnosis by retrieving relevant medical case studies and research.</p><p>· Financial Analysis — Incorporating real-time market data, financial reports, and news to generate insights.</p><p>· Marketing — Conducting in-depth market research by retrieving consumer insights, market trends, and competitor analysis.</p><p>· Manufacturing — Optimizing supply chain management by retrieving supplier performance data, logistics information, and demand forecasts.</p><p>· Education — Enhancing curriculum content by retrieving and integrating the latest academic research and educational resources.</p><p>By themselves, LLMs have a few limitations. They are generic and lack subject matter expertise. They can also hallucinate, generating responses that are incorrect but sound accurate. Additionally, they are limited by the amount of data used to train them as well as the point in time that their training occurred.</p><p>When combined with techniques like RAG, LLMs can overcome these limitations. They can enhance their understanding of subjects by referencing domain-specific information. They can generate more reliable responses since they are grounded in relevant content which is provided to the LLM during interactions with it (called the LLM Inference process). This avoids the need to fine-tune or train a model from scratch, both of which are time-consuming and expensive.</p><p>Our implementation spanned the three main components of RAG:</p><p>1) Ingestion — Loading data from external sources and creating vector embeddings so the data can be compared and stored efficiently.</p><p>2) Retrieval — Extracting the most relevant information from the data ingested.</p><p>3) Synthesis — Using the retrieved information to augment the LLM prompt and generate a final response.</p><p>Our application focused on extracting sections from a large volume of regulatory compliance documents, then using RAG to generate insights from them. Many organizations across industries need to submit regulatory compliance documents, which must adhere to standardized categories of information but can vary significantly in formatting. These formatting differences present a challenge from a parsing perspective.</p><p>While building our application, we noticed it became increasingly difficult to maintain high accuracy in section extraction as the volume of documents grew. This was due to the increasing variety of formatting that had to be parsed, and this challenge was amplified when working with unstructured documents, such as PDFs. We found the semi-structured nature of HTML documents helped solve this issue, enabling more consistent and reliable section extraction. It allowed hyperlinks to be used as pointers to each section, increasing resiliency to formatting differences and resulting in more accurate section extraction at scale.</p><p>Once the sections were extracted, they were converted into vector embeddings that could be ingested into a database and queried by an LLM using RAG. This end-to-end workflow provides a scalable and robust method to generate insights by analyzing specific sections and comparing them across documents. This approach can be generalized to various domains where document formatting variability poses a challenge, enabling accurate section extraction and insightful analysis across different types of documents and industries.</p><p>RAG offers transformative capabilities across a wide range of real-world scenarios. However, it is important to note that there is no one-size-fits-all implementation. The effectiveness of RAG depends on the careful alignment of its components with the unique requirements of each use-case and application. This customizability allows RAG to drive innovation across diverse industries, offering solutions that address the challenges and opportunities specific to each application.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*_ys89RwX1qPhJg-5kfAgKA.png" /><figcaption>RAG architecture for our application</figcaption></figure><p>Stay tuned for subsequent articles as we unpack different aspects of this application and go over other other LLM techniques beyond RAG…</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e11b81785684" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>