<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by UAV on Medium]]></title>
        <description><![CDATA[Stories by UAV on Medium]]></description>
        <link>https://medium.com/@databy-uav?source=rss-0de7ca78f4f8------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*apZqk9JIiY01IjhLnkhjlw@2x.jpeg</url>
            <title>Stories by UAV on Medium</title>
            <link>https://medium.com/@databy-uav?source=rss-0de7ca78f4f8------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 23:15:23 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@databy-uav/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[ Building My Own Modern Lake-house (From a Data Warehouse Engineer’s POV) — Part 2]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/@databy-uav/building-my-own-modern-lake-house-from-a-data-warehouse-engineers-pov-part-2-8ef1e602ebb2?source=rss-0de7ca78f4f8------2"><img src="https://cdn-images-1.medium.com/max/1536/1*wG6KIqe8Ie_kWal7CysNjg.png" width="1536"></a></p><p class="medium-feed-snippet">In the previous part, I set up MinIO as the blob storage foundation for my lake-house.</p><p class="medium-feed-link"><a href="https://medium.com/@databy-uav/building-my-own-modern-lake-house-from-a-data-warehouse-engineers-pov-part-2-8ef1e602ebb2?source=rss-0de7ca78f4f8------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://medium.com/@databy-uav/building-my-own-modern-lake-house-from-a-data-warehouse-engineers-pov-part-2-8ef1e602ebb2?source=rss-0de7ca78f4f8------2</link>
            <guid isPermaLink="false">https://medium.com/p/8ef1e602ebb2</guid>
            <category><![CDATA[data-lakehouse]]></category>
            <category><![CDATA[dlt]]></category>
            <category><![CDATA[docker]]></category>
            <dc:creator><![CDATA[UAV]]></dc:creator>
            <pubDate>Sun, 22 Feb 2026 07:02:12 GMT</pubDate>
            <atom:updated>2026-02-22T07:02:12.098Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[ Building My Own Modern Lake-house (From a Data Warehouse Engineer’s POV) — Part 1]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/@databy-uav/building-my-own-modern-lake-house-from-a-data-warehouse-engineers-pov-part-1-2538df082bb7?source=rss-0de7ca78f4f8------2"><img src="https://cdn-images-1.medium.com/max/1536/1*FDlCVP68NXWFSMwMuqbPTw.png" width="1536"></a></p><p class="medium-feed-snippet">I&#x2019;ve spent the last few years deep in the world of data warehousing &#x2014; building models, writing SQL, and powering BI layers that drive&#x2026;</p><p class="medium-feed-link"><a href="https://medium.com/@databy-uav/building-my-own-modern-lake-house-from-a-data-warehouse-engineers-pov-part-1-2538df082bb7?source=rss-0de7ca78f4f8------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://medium.com/@databy-uav/building-my-own-modern-lake-house-from-a-data-warehouse-engineers-pov-part-1-2538df082bb7?source=rss-0de7ca78f4f8------2</link>
            <guid isPermaLink="false">https://medium.com/p/2538df082bb7</guid>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[data-lakehouse]]></category>
            <category><![CDATA[minio]]></category>
            <dc:creator><![CDATA[UAV]]></dc:creator>
            <pubDate>Sat, 21 Feb 2026 14:15:48 GMT</pubDate>
            <atom:updated>2026-02-21T14:15:48.107Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[# Getting Started with Pyvis for Graph Visualization]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/@databy-uav/getting-started-with-pyvis-for-graph-visualization-86a0bd7d985c?source=rss-0de7ca78f4f8------2"><img src="https://cdn-images-1.medium.com/max/1536/1*vIkseT8dFeC9h35dK_bGqA.png" width="1536"></a></p><p class="medium-feed-snippet">Create stunning, interactive graph visualizations in Python using pyvis. A beginner-friendly guide to get you started fast!</p><p class="medium-feed-link"><a href="https://medium.com/@databy-uav/getting-started-with-pyvis-for-graph-visualization-86a0bd7d985c?source=rss-0de7ca78f4f8------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://medium.com/@databy-uav/getting-started-with-pyvis-for-graph-visualization-86a0bd7d985c?source=rss-0de7ca78f4f8------2</link>
            <guid isPermaLink="false">https://medium.com/p/86a0bd7d985c</guid>
            <category><![CDATA[pyvis]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[network-visualization]]></category>
            <category><![CDATA[python-visualization]]></category>
            <dc:creator><![CDATA[UAV]]></dc:creator>
            <pubDate>Sun, 13 Apr 2025 10:04:04 GMT</pubDate>
            <atom:updated>2025-04-15T08:48:05.981Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Exploring Langchain ChatModel: Building the Most Unbiased Cricket Chatbot (definitely not…]]></title>
            <link>https://medium.com/@databy-uav/exploring-langchain-chatmodel-building-the-most-unbiased-cricket-chatbot-definitely-not-50d483bc65b0?source=rss-0de7ca78f4f8------2</link>
            <guid isPermaLink="false">https://medium.com/p/50d483bc65b0</guid>
            <category><![CDATA[langchain]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[UAV]]></dc:creator>
            <pubDate>Fri, 11 Apr 2025 17:29:15 GMT</pubDate>
            <atom:updated>2025-04-11T17:29:15.201Z</atom:updated>
            <content:encoded><![CDATA[<h3>Exploring Langchain ChatModel: Building the Most Unbiased Cricket Chatbot (definitely not Kohli-biased)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7EPo8c-2nNW_OYW2jV_HJQ.png" /><figcaption>Kohlipaglu Chatbot using Langchain</figcaption></figure><p>Large language models (LLMs) are becoming easier to integrate into applications, thanks to powerful frameworks like <strong>LangChain</strong>. In this tutorial, we will build a <strong>Virat Kohli-themed chatbot</strong> using <strong>LangChain, Ollama, and Langfuse</strong>, progressively adding features to improve interactivity and tracking.</p><blockquote><strong><em>Disclaimer:</em></strong><em> This chatbot provides </em>completely neutral<em> cricket insights. It’s not biased at all… unless you count the fact that Kohli is objectively responsible for everything good that happens in cricket. 😏</em></blockquote><h3>🛠️ Pre-requisites</h3><p>Before diving in, ensure you have the following:</p><ul><li><strong>Python 3.8+</strong> installed</li><li><strong>Ollama</strong> set up for running local LLMs</li><li><strong>LangChain</strong> installed (pip install langchain langchain_core langfuse) , langfuse if going for observability</li><li><strong>Langfuse</strong> account (optional, for observability)</li><li>Basic understanding of <strong>Python</strong> and <strong>LLMs</strong></li></ul><h3>🧰 Tech Stack Overview</h3><p>Before we dive into the code, let’s take a quick look at the technologies powering our totally neutral cricket chatbot.</p><h3>🤖 Large Language Models (LLMs)</h3><p>LLMs are AI models trained on vast amounts of text data to generate human-like responses. Our chatbot uses <strong>Llama3.2</strong>, a powerful open-source LLM by Meta.</p><h3>⛓️‍💥 LangChain</h3><p>LangChain is a framework that simplifies working with LLMs by providing tools for chaining prompts, managing memory, and integrating external data sources.</p><h3>🐑 Ollama</h3><p>Ollama is a local LLM runtime that allows you to run <strong>open-source models</strong> efficiently on your machine without requiring cloud APIs.</p><h3>📊 Langfuse</h3><p>Langfuse is an observability platform for LLM applications, helping track inputs, outputs, and metadata to debug and optimize model performance.</p><h3>1️⃣ Basic Chat with invoke ()</h3><p>Let’s start by setting up a simple chatbot using <strong>LangChain’s </strong><strong>init_chat_model</strong> and <strong>Ollama&#39;s Llama3.2 model</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aMLxJhpRezHV_qsebazy0g.png" /><figcaption>Code for basic usage of Langchain’s ChatModels using .invoke() method</figcaption></figure><h3>🔹 Explanation:</h3><ul><li>We initialize an <strong>Ollama-powered Llama3.2 model</strong>.</li><li>We provide a <strong>system message</strong> (defining the chatbot’s <em>perfectly neutral</em> personality).</li><li>We use .invoke () to fetch a <strong>single response</strong>.</li><li><strong>Metadata</strong> contains useful details like <strong>token usage, response time, and model performance</strong>.</li></ul><h4>Code output</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8H0pCCeAcuGqqd602dcbsw.png" /><figcaption>output of first app</figcaption></figure><h3>2️⃣ Streaming Output for Real-time Responses</h3><p>A better approach is to <strong>stream responses</strong> rather than waiting for the entire output to load.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZVJWKMdyIsr7JpaiwCnbdw.png" /><figcaption>Code for basic usage of Langchain’s ChatModel with streaming output for ChatGPT style output to CLI</figcaption></figure><ul><li>Users get responses <strong>in real time</strong> instead of waiting.</li><li>It enhances the <strong>conversational feel</strong> of the chatbot.</li></ul><h4>Code output</h4><p>Similar as previous one but this time the output streamed in the terminal, token by token.</p><h3>3️⃣ Building a Multi-turn Chatbot</h3><p>Now, let’s add user interaction for a continuous conversation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fKrAOkuqS2ZX_mWnwNAyvA.png" /><figcaption>Code for basic usage of Langchain’s ChatModel with streaming output for ChatGPT style output to CLI and continuous conversation with previous context</figcaption></figure><h3>🔹 Features added:</h3><ul><li><strong>Multi-turn chat</strong> with /exit and /clear commands.</li><li><strong>Real-time streaming output</strong> for better interaction.</li><li><strong>Keeps context</strong> across turns unless manually cleared.</li></ul><h4>Code output</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HX5ZP3VaF_BDul7cRrzbvg.png" /></figure><h3>4️⃣ Integrating Langfuse for Logging and Analytics</h3><p>Now, let’s add <strong>Langfuse</strong>, a tool for logging and monitoring LLM-based applications. You would first need to create a project and generate API key for langfuse. <a href="https://cloud.langfuse.com/">https://cloud.langfuse.com/</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hLkFT4cu9F3z8pbp2JYDtw.png" /><figcaption>Added langfuse for observability</figcaption></figure><h3>🔹 What Langfuse adds:</h3><ul><li><strong>Tracks input and output</strong> for debugging.</li><li><strong>Stores metadata</strong> to analyze model performance.</li><li><strong>Provides observability</strong> into how the chatbot is being used — the current implementantion will create a dashboard in <a href="https://cloud.langfuse.com/">https://cloud.langfuse.com/</a></li></ul><h4>Code output</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FWzP46JmCn-g-StKPBCQ7g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*K9QxztycblS3RHyH61gbHQ.png" /></figure><h3>🛠️ Future Improvements</h3><p>Here are a few ideas to <strong>enhance our <em>neutral</em> chatbot</strong> further:</p><p>✅ <strong>Custom Memory</strong> — Implement <strong>LangChain’s memory</strong> to handle context more effectively. Right now, the chatbot:</p><ol><li><strong>Does not persist memory across session</strong>s, meaning it forgets everything when restarted.</li><li><strong>Includes all previous messages in context</strong>, even when they aren’t relevant, instead of intelligently deciding what context to retain. <br>A better approach would be to use <strong>ConversationalRetrievalChain</strong> or <strong>summary-based memory </strong>to selectively retain relevant parts of the conversation.</li></ol><p>✅ <strong>Dockerizing the Chatbot</strong> — Deploy as a <strong>Docker container</strong>.<br>✅ <strong>More Langfuse Insights</strong> — Analyze logs to see <strong>common user queries</strong> and <strong>token usage trends</strong>.<br>✅ <strong>Custom UI</strong> — Build a <strong>Streamlit</strong> or <strong>React</strong> frontend.</p><h3>🚀 What’s Next?</h3><ul><li>Using Prompt templates to <strong>dynamically adjust the system message</strong> based on user input.</li><li>Use self hosted version of langfuse</li></ul><p>This was a fun experiment in <strong>LLMs, real-time streaming, and observability</strong>. What are <strong>you</strong> building? Let me know in the comments! 🎤🔥</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=50d483bc65b0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[DBT and BashCommand: Effectively Utilizing DBT Core Functionality with Airflow]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/@databy-uav/dbt-and-bashcommand-effectively-utilizing-dbt-core-functionality-with-airflow-f369375c6ef1?source=rss-0de7ca78f4f8------2"><img src="https://cdn-images-1.medium.com/max/1984/1*66SkNc84qiO45t7IpGlEfQ.png" width="1984"></a></p><p class="medium-feed-snippet">In our previous article on DBT and Airflow, we explored how these two tools can be combined to build scalable and modular data pipelines&#x2026;</p><p class="medium-feed-link"><a href="https://medium.com/@databy-uav/dbt-and-bashcommand-effectively-utilizing-dbt-core-functionality-with-airflow-f369375c6ef1?source=rss-0de7ca78f4f8------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://medium.com/@databy-uav/dbt-and-bashcommand-effectively-utilizing-dbt-core-functionality-with-airflow-f369375c6ef1?source=rss-0de7ca78f4f8------2</link>
            <guid isPermaLink="false">https://medium.com/p/f369375c6ef1</guid>
            <category><![CDATA[bash]]></category>
            <category><![CDATA[etl-pipeline]]></category>
            <category><![CDATA[dbt]]></category>
            <category><![CDATA[airflow]]></category>
            <category><![CDATA[data-model]]></category>
            <dc:creator><![CDATA[UAV]]></dc:creator>
            <pubDate>Mon, 07 Apr 2025 07:15:47 GMT</pubDate>
            <atom:updated>2025-04-07T07:15:47.630Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Building Scalable Data Pipelines]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/@databy-uav/building-scalable-data-pipelines-47a134d8b1fc?source=rss-0de7ca78f4f8------2"><img src="https://cdn-images-1.medium.com/max/1100/0*OZgFUDn_orP3A_OB" width="1100"></a></p><p class="medium-feed-snippet">Integrating dbt and Apache Airflow for Seamless Orchestration</p><p class="medium-feed-link"><a href="https://medium.com/@databy-uav/building-scalable-data-pipelines-47a134d8b1fc?source=rss-0de7ca78f4f8------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://medium.com/@databy-uav/building-scalable-data-pipelines-47a134d8b1fc?source=rss-0de7ca78f4f8------2</link>
            <guid isPermaLink="false">https://medium.com/p/47a134d8b1fc</guid>
            <category><![CDATA[data-model]]></category>
            <category><![CDATA[airflow]]></category>
            <category><![CDATA[dbt]]></category>
            <category><![CDATA[etl]]></category>
            <category><![CDATA[orchestration]]></category>
            <dc:creator><![CDATA[UAV]]></dc:creator>
            <pubDate>Mon, 10 Feb 2025 16:46:23 GMT</pubDate>
            <atom:updated>2025-02-11T13:00:05.526Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[From Raw SQL to Lineage Insights: A RAG-Powered Approach]]></title>
            <link>https://medium.com/@databy-uav/from-raw-sql-to-lineage-insights-a-rag-powered-approach-d0afb82fdc7a?source=rss-0de7ca78f4f8------2</link>
            <guid isPermaLink="false">https://medium.com/p/d0afb82fdc7a</guid>
            <category><![CDATA[langchain]]></category>
            <dc:creator><![CDATA[UAV]]></dc:creator>
            <pubDate>Tue, 21 Jan 2025 16:01:26 GMT</pubDate>
            <atom:updated>2025-04-03T05:26:29.410Z</atom:updated>
            <content:encoded><![CDATA[<p>In our organization, we use <strong>Trino</strong> as the query engine, but it doesn’t maintain lineage. This lack of visibility makes it difficult to track which downstream tables or reports could be impacted if something goes wrong with a table. To solve this, I decided to leverage <strong>Retrieval-Augmented Generation (RAG)</strong> — a technique that combines the power of large language models (LLMs) with external data sources to provide smarter insights.</p><p>My task was clear: <strong>build table-level and column-level lineage from raw SQL code</strong>. With no pre-existing metadata to rely on, the solution needed to be both clever and efficient.</p><p>In this article, I’ll walk you through how I used the LangChain framework and OpenAI APIs to process SQL scripts, embed them into vectors, and store them in a vector database for easy retrieval. Let’s dive in and explore how RAG can help turn raw SQL into structured lineage insights!</p><p><a href="https://github.com/dataguyavi/Playing-Around-with-Langchain/tree/main/4_rags"><em>GitHub Link for this Article</em></a><em> -&gt; Complete code and dummy SQL files here with the embeddings of the docs :)</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/680/0*dl6PXozCcfbyVGd3.jpg" /></figure><h3>Setting Up: From SQL Files to a Vector Database</h3><p>To enable a language model to reason about SQL code, the first step is to process the SQL scripts into a structure it can understand — by embedding them into vectors for efficient retrieval. Here’s how I approached it:</p><h4>1. Library imports and Setting up Directories and Paths</h4><pre>import os<br>from langchain_community.document_loaders import TextLoader<br>from langchain_text_splitters import CharacterTextSplitter<br>from langchain_openai import OpenAIEmbeddings<br>from langchain_chroma import Chroma<br><br>current_path = os.path.dirname(__file__)<br>code_directory = os.path.join(current_path, &#39;sql_codes&#39;)<br>vector_db_directory = os.path.join(current_path, &#39;db&#39;)</pre><ul><li>code_directory: The folder where all the SQL files are present.</li><li>vector_db_directory: The folder where the vector database will be saved</li></ul><h4>2. Loading SQL Files</h4><p>Once the directories are setup and files are present, we begin by loading the SQL files from the sql_codes directory:</p><ul><li>First, we gather all files which are code files, ending with `.sql`.</li><li>Next, we load the content of each SQL file using the TextLoader from LangChain.</li></ul><pre>sql_files = [f for f in os.listdir(code_directory) if f.endswith(&#39;.sql&#39;)]<br><br>    sql_codes = []<br><br>    for sql_file in sql_files:<br>        sql_path = os.path.join(code_directory, sql_file)<br><br>        loader = TextLoader(sql_path)<br>        sqls = loader.load()</pre><ul><li>This loads the content of each SQL file into a list of documents. We also attach metadata, such as the source file name, for easier traceability:</li></ul><pre>  ...<br><br>  for sql in sqls:<br>      sql.metadata = {&#39;source&#39;: sql_file}<br>      sql_codes.append(sql)</pre><h4>3. Splitting SQL Code into chunks</h4><p>Since language models can struggle with large documents, we need to split the SQL code into smaller, more manageable chunks. The CharacterTextSplitter is ideal for this task:</p><pre>text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=50)<br>sql_codes_chunked = text_splitter.split_documents(sql_codes)</pre><p>Here:</p><ul><li>chunk_size=1000: Each chunk will have a maximum of 1000 characters.</li><li>chunk_overlap=50: Each chunk overlaps the next by 50 characters, ensuring context is preserved between chunks.</li></ul><p>This is crucial for the model to maintain coherence across chunks.</p><h4>4. Embedding SQL Chunks with OpenAI Embeddings</h4><p>Next, we use OpenAI’s embedding model to convert the SQL chunks into vectors. Embeddings are high-dimensional numerical representations of text, which makes them suitable for efficient similarity searches:</p><pre>embeddings = OpenAIEmbeddings(model=&#39;text-embedding-3-small&#39;)</pre><p>This line initializes the embedding model that will process each chunk of SQL code. The embeddings allow us to convert the SQL code into a form that can be easily compared and retrieved later.</p><h4>5. Creating a Vector Database</h4><p>After embedding the SQL chunks, we need to store them in a vector database for fast retrieval. For this, we use <strong>Chroma</strong>, a vector database built for use with LangChain:</p><pre>db = Chroma.from_documents(<br>    documents=sql_codes_chunked,<br>    embedding=embeddings,<br>    persist_directory=vector_db_directory<br>)</pre><p>This creates the vector database and saves it in the vector_db_directory. The embeddings of each chunk are stored, allowing for fast and efficient retrieval later when we need to query the database.</p><p>With this we are done with the setup steps, full code <a href="https://github.com/dataguyavi/Playing-Around-with-Langchain/blob/main/4_rags/rag_part1.py">here</a>.</p><h3>Next Step: Querying the Vector Database for Insights</h3><p>Now that we have set up the vector database and embedded the SQL documents, it’s time to make use of this rich, searchable database. In the next step, we’ll focus on <strong>querying the database</strong> to retrieve the most relevant documents that help answer specific questions about the SQL code — like understanding table dependencies or tracking how certain tables are used across queries.</p><p>By integrating a <strong>Retrieval-Augmented Generation (RAG)</strong> workflow, we can use <strong>Chroma</strong> for efficient document retrieval and <strong>OpenAI’s LLMs</strong> for generating precise answers based on the retrieved context. This will allow us to ask specific questions, like which tables are involved in a certain data pipeline and get meaningful insights directly from the SQL scripts.</p><p>Let’s dive into how we can query the vector database and pass the retrieved documents to the LLM to generate actionable answers.</p><h4>1. Setting Up the Environment</h4><p>The first part of the code is setting up the embeddings and the Chroma database:</p><pre>import os<br>from langchain_openai import OpenAIEmbeddings, ChatOpenAI<br>from langchain_core.messages import HumanMessage, SystemMessage<br>from langchain_chroma import Chroma<br><br>current_dir = os.path.dirname(os.path.abspath(__file__))<br>chrom_db = os.path.join(current_dir, &#39;db&#39;)<br><br>embeddings = OpenAIEmbeddings(model=&#39;text-embedding-3-small&#39;)<br><br>db = Chroma(<br>    persist_directory = chrom_db,<br>    embedding_function = embeddings<br>)</pre><ul><li>current_dir and chrom_db define the paths to the directory containing the vector database.</li><li>embeddings initialize the OpenAI embedding model, which is used to convert the SQL chunks into vector representations.</li><li>Chroma sets up the vector database, pointing it to the directory where your embedded documents are stored.</li></ul><h3>2. Querying the Database</h3><p>You then define a query and set up the retriever:</p><pre>query = &#39;&#39;&#39;<br>    What all tables are required to create the customer behavior table?<br>&#39;&#39;&#39;<br><br>retriever = db.as_retriever(<br>    search_type = &#39;similarity_score_threshold&#39;,<br>    search_kwargs={&quot;k&quot;: 1, &quot;score_threshold&quot;: 0.2},<br>)</pre><ul><li>query is the question you&#39;re asking the model.</li><li>retriever is set up to retrieve the most relevant document (or documents) based on a similarity score, with a threshold of 0.2. Only documents with a similarity score above this threshold are considered relevant. In our case only 1 doc will be fetched since K = 1.</li></ul><h3>3. Retrieving Relevant Documents</h3><pre>relevant_doc = retriever.invoke(query)</pre><p>This retrieves the most relevant document based on the query. The result is stored in relevant_doc, which contains the content of the most relevant document(s) from your Chroma database.</p><h4>4. Combining the Query and Document</h4><p>The next step is to combine the query with the retrieved document and format it as input for the LLM, this ensures that LLM receives the necessary context to generate an accurate answer.</p><pre>combined_input = f&#39;&#39;&#39;<br>    I will supply you the relevant docs to answer my query:<br>    {query}<br><br>    Doc:<br>    {relevant_doc[0].page_content}<br><br>    The first line in doc contains the file name.<br><br>    You need to answer only using the relevant docs, if not possible then pls respond with &quot;No Answer Found&quot;<br>&#39;&#39;&#39;</pre><h4>5. Invoking the LLM</h4><p>Now you prepare the LLM (using ChatOpenAI) and feed the combined input:</p><pre>llm = ChatOpenAI(model=&quot;gpt-4o-mini&quot;)<br><br>messages = [<br>    SystemMessage(&#39;You are a Data Warehouse expert&#39;),<br>    HumanMessage(combined_input)<br>]<br><br>result = llm.invoke(messages)<br><br>print(result.content)</pre><ul><li>SystemMessage sets the context for the model to act as a Data Warehouse expert.</li><li>HumanMessage contains the query and the relevant document.</li><li>llm.invoke(messages) runs the model and gets the response.</li></ul><p>Output for our example:</p><pre>To create the customer behavior table, the following tables are required:<br><br>1. **lake.sales** - This table is needed to get transaction details such as `transaction_id`, `quantity`, and `total_amount`.<br>2. **dim.customers** - This table is needed to link customer information with their respective behavior data, using the `customer_id` and `customer_key`. <br><br>Thus, the two tables required are:<br>- lake.sales<br>- dim.customers</pre><p>This is 100% accurate :) … Link to customer behaviour table <a href="https://github.com/dataguyavi/Playing-Around-with-Langchain/blob/main/4_rags/sql_codes/fact_customer_behavior.sql">code</a>.</p><h3>Some more examples:</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qqxCc-TPfdwnmDoNFLdUdQ.png" /><figcaption>Column Level Lineage.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*o3Bn9ldT0DEdW4sUt7sCjA.png" /><figcaption>Generating Query by using existing tables.</figcaption></figure><h3>Next Steps: Testing, Improving, and Scaling the Solution</h3><p>With the core system in place, it’s time to take the project to the next level. Here’s what’s on the horizon:</p><ol><li><strong>Testing with Real SQL Code (DBT)</strong>:<br>We’ll begin testing with <strong>real-world SQL scripts</strong>, particularly focusing on <strong>DBT</strong> (Data Build Tool) code. DBT already maintains table-level lineage, so testing with DBT will be straightforward and will allow us to verify that the lineage extraction process works well with real data pipelines.</li><li><strong>Improving Output Structure</strong>:<br>The next step is to refine the prompt to ensure that the output is structured more effectively. By adjusting the format and clarity of the responses from the LLM, we can ensure they are easier to interpret and ready for downstream usage, such as visualizing lineage.</li><li><strong>Exploring AI Agents for Automation</strong>:<br>To make the solution even more powerful, we’ll explore <strong>AI agents</strong> that can automate the process of pushing the results directly into a database. This would allow us to store the lineage data for future use and even build dynamic lineage graphs that show dependencies between tables.</li><li><strong>Exploring Free Embeddings and LLM Models</strong>:<br>As the project evolves, we’ll also look into <strong>free embeddings</strong> and <strong>LLM models</strong> that can help reduce costs while maintaining the performance of the system. There are plenty of open-source alternatives to explore, which can be integrated without compromising the quality of results.</li><li><strong>Designing a SaaS Product</strong>:<br>Finally, we’ll start thinking bigger. The goal is to <strong>design this solution as a SaaS product</strong> that can be scaled and used by other organizations facing similar challenges. This would involve packaging the code into an easy-to-use platform with a user-friendly interface, API access, and more!</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d0afb82fdc7a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Introduction to RAG: Bringing Context to Large Language Models]]></title>
            <link>https://medium.com/@databy-uav/introduction-to-rag-bringing-context-to-large-language-models-955d5b905b4f?source=rss-0de7ca78f4f8------2</link>
            <guid isPermaLink="false">https://medium.com/p/955d5b905b4f</guid>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[retrieval-augmented-gen]]></category>
            <dc:creator><![CDATA[UAV]]></dc:creator>
            <pubDate>Sun, 12 Jan 2025 04:57:56 GMT</pubDate>
            <atom:updated>2025-01-12T04:57:56.966Z</atom:updated>
            <content:encoded><![CDATA[<p>Large Language Models (LLMs) are transforming the way we interact with data, but their true potential lies in how effectively they integrate with private, domain-specific knowledge. This article serves as an introduction to Retrieval-Augmented Generation (RAG), a powerful approach that combines LLMs with external knowledge sources, making them more tailored and useful for specific applications. In the next article in this series, we will get hands-on by building table and column-level lineage for finance data.</p><h3>Motivation</h3><p>Over time, the <strong>context window</strong> of LLMs has grown significantly. The context window refers to the capacity of an LLM to store and process external or private knowledge alongside its training data. While LLMs are trained on vast amounts of publicly available data, we can further enhance their utility by feeding them private data. This enables LLMs to cater to niche use cases — for instance, generating SQL queries for a private data warehouse.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/689/1*UOx10TvXECyhGYO8IRhe_w.png" /><figcaption>source: <a href="https://www.youtube.com/watch?v=sVcwVQRHIc8">https://www.youtube.com/watch?v=sVcwVQRHIc8</a></figcaption></figure><h3>What is RAG?</h3><p>Retrieval-Augmented Generation (RAG) enhances LLM capabilities by integrating external knowledge retrieval into the process. The basic RAG workflow is as follows:</p><p><strong>Question → Indexing (private data) → Retrieval (relevant docs) → Generation (answer) → Answer</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SThyDRWWf3U3LyaPb7jpNg.png" /><figcaption>source: <a href="https://www.youtube.com/watch?v=sVcwVQRHIc8">https://www.youtube.com/watch?v=sVcwVQRHIc8</a></figcaption></figure><h3>Key Components of RAG</h3><ol><li><strong>Indexing</strong></li></ol><ul><li>Indexing organizes private data or knowledge into a structure that a retriever can use to fetch relevant information for queries.</li><li>Comparing relevancy between documents and queries in natural language can be complex. To simplify this, we convert both documents and queries into numerical forms, which allows for computational comparison.</li></ul><p>Two main approaches to indexing:</p><ol><li><strong>Bag of Words → Sparse Representations</strong></li></ol><ul><li>Example: <strong>BM25</strong> (a traditional search algorithm)</li></ul><ol><li><strong>Embeddings → Dense Representations</strong></li></ol><ul><li>Example: <strong>KNN</strong>, <strong>HNSW</strong></li></ul><p><strong>Preferred Approach:</strong> Use ML-based models to create embeddings, which are dense numerical representations of documents.</p><p>Since embedding models have a limited context window, we need to split large documents into smaller chunks before creating embeddings.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/790/1*Ix9c22-3DNRKlQUyXiq1Tw.png" /><figcaption>Docs are loaded, split into smaller chunks and then converted into numerical forms using embeddings</figcaption></figure><p><strong>2. Retrieval</strong></p><ul><li>Retrieval is the process of fetching the most relevant documents based on a given query.</li><li>Using techniques like <strong>KNN</strong> or <strong>HNSW</strong>, we compare the similarity between document embeddings and query embeddings. Documents with similar semantic meanings are closer to the query in the embedding space.</li><li><strong>Vector stores</strong> simplify storing and searching for embeddings, enabling efficient retrieval of similar documents.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/898/1*hQqHIuVJMnI1qVx2ywYBWA.png" /><figcaption>Toy embedding of vectors having length 3 (3-Dimensions)</figcaption></figure><p><strong>3. Generation</strong></p><ul><li>After retrieving the relevant documents, they are fed into the context window of the LLM along with the user’s query.</li><li>The LLM processes this enriched prompt and generates a response tailored to the input query and the retrieved context.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/864/1*RiAoP7mxIXCBZwSU8nDATA.png" /><figcaption>Complete flow from query to answer.</figcaption></figure><h3>Conclusion</h3><p>This article provided a foundational understanding of RAG and its basic flow. To recap:</p><ul><li><strong>Indexing</strong>: Prepare and structure private knowledge into embeddings.</li><li><strong>Retrieval</strong>: Fetch relevant documents using similarity techniques.</li><li><strong>Generation</strong>: Leverage LLMs to generate answers enriched by retrieved context.</li></ul><p>Stay tuned for the next article in this series, where we will apply these concepts by creating table and column-level lineage for a DataMart. This hands-on approach will bridge the gap between theory and practical implementation.</p><p><em>Follow for updates on the next article in this series!</em></p><p>🖊: <a href="https://in.linkedin.com/in/avi-kasliwal">Avi Kasliwal</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=955d5b905b4f" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>