<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Joseph Jing on Medium]]></title>
        <description><![CDATA[Stories by Joseph Jing on Medium]]></description>
        <link>https://medium.com/@joseph.jing?source=rss-b7eaff029be1------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*Ew5EWxCU-9_F6DDT</url>
            <title>Stories by Joseph Jing on Medium</title>
            <link>https://medium.com/@joseph.jing?source=rss-b7eaff029be1------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 22:34:25 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@joseph.jing/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Leveraging LLMs to generate AI driven dbt documentation]]></title>
            <link>https://medium.com/loblaw-digital/leveraging-llms-to-generate-ai-driven-dbt-documentation-c4735faa6ca5?source=rss-b7eaff029be1------2</link>
            <guid isPermaLink="false">https://medium.com/p/c4735faa6ca5</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[dbt]]></category>
            <category><![CDATA[analytics-engineering]]></category>
            <category><![CDATA[dbt-docs]]></category>
            <dc:creator><![CDATA[Joseph Jing]]></dc:creator>
            <pubDate>Mon, 27 Nov 2023 16:18:47 GMT</pubDate>
            <atom:updated>2023-11-27T16:29:12.482Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-y0xukadIQnQA0PbdhhDHA.png" /></figure><p><em>Co-authored by: Joseph Jing, Rohit Bathija, Michelle Qi, Indrani Gorti</em></p><h3>Introduction</h3><p>In the ever-evolving landscape of analytics engineering, finding ways to streamline workflows and increase productivity is paramount. One powerful tool that can help achieve this is Large Language Models (LLMs). Harnessing the capabilities of LLMs empowers us to automate labor-intensive tasks such as documentation, freeing up valuable time for Analytics Engineers to focus on more complex and strategic endeavors. In this article, we will delve into the practical applications of LLMs in the analytics engineering realm, exploring how they can revolutionize workflows and enhance overall efficiency.</p><h3>Background</h3><p>The open-source Extract-Load-Transform (ELT) framework we use across the data organization at Loblaw Digital is <a href="https://github.com/dbt-labs/dbt-core">dbt</a>. A model serves as an essential building block of a dbt framework. Models usually reside in a single file and contain logic that transforms data. This logic can be expressed as a SQL select statement or a Python dataframe operation. Out of the box, dbt allows us to generate a static documentation website that lists all available models across different projects or folders, detailed information like its dependent models, as well as which other models reference it, and finally a lineage graph.</p><p>Each dbt model consists of an SQL file that transforms data and a corresponding config file that contains the schema information, such as a brief description of what the model does, columns that are available, and the description of each column’s data type. This information needs to be manually written down when developing each model, and with thousands of models across different lines of businesses, the description section of the documentation is often omitted. This omission can be confusing for business users who are not familiar with the data when viewing our dbt documentation website. When dbt compiles a model, it will transform or compile the model into JSON format. Here is an example of the compiled model `snowplow_analytics_user.`</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dtFdGFLAV3_dijbUASuFTg.png" /></figure><p>In the above example, we did not add any content under <em>description</em> and <em>column</em>. However, documentation in the data engineering world is often regarded as a “necessary evil” — crucial for data integrity, compliance, and collaboration, yet burdened by manual processes that are slow and error-prone. As data models continually evolve, the task of keeping documentation up to date becomes increasingly arduous, often leading to “documentation debt.”</p><p>To address these challenges, <a href="https://github.com/TextQLLabs/dbt-documentor">dbt documentor</a> emerges as a transformative tool that combines the capabilities of traditional dbt documentation and LLMs. This innovative approach automates the otherwise cumbersome workflow and utilizes relation information generated by dbt, eliminating the need to manually cross-reference multiple files and metadata, thereby streamlining the entire documentation process.</p><p><strong>Architecture</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*G5X2SuHC7DrKlR24" /></figure><p>At Loblaw Digital, our Business Intelligence dbt repo closely resembles a typical dbt project, consisting of project level folders containing models from different teams, along with the configuration files. We wanted to make the process of documentation more efficient.</p><p>Leveraging on the ML platform, like Vertex AI, the dbt documentor ingests the dbt model schema directly from a manifest.json file. From there, the LLMs generate comprehensive documentation, detailing everything from describing the SQL queries to the intricate dependencies between models and columns. Importantly, this is achieved without reading actual data, thereby ensuring that the integrity and security of the data remain intact.</p><h3>Setup</h3><h4>Installation of the .NET SDK</h4><p>The .NET SDK version 6.0 is needed to install this project. cd into the project directory, run dotnet publish, and install the binary into your PATH.</p><p>To build a self-contained binary (not dependent on .NET), run dotnet publish — sc — runtime {RID} where the<a href="https://learn.microsoft.com/en-us/dotnet/core/rid-catalog"> RID is a valid runtime identifier</a>.</p><p>Popular RIDs are linux-x64, osx.13-x64, osx.13-arm64, win10-x64.</p><h4>Generate docs for all undocumented models</h4><p>$ DbtHelper — working-directory /home/mark/src/dbt_project</p><h4>Run dbt commands</h4><p>$ dbt run<br>$ dbt doc</p><p>All undocumented models will have [ai-gen] documentation in their YAML file.</p><h3>Putting it all together</h3><p>Let’s take the same model `snowplow_analytics_user.` We can see what our dbt docs look like for the model without any documentation; it can be very hard for an analyst to infer what <em>exactly </em>this model does.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*xjZjktUw-HLkGwO0" /></figure><p>Once we have set up the dbt documentor and run the required commands to compile and serve the docs, we can see descriptive documentation being generated by the LLMs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*QC_TyQ1aOb0f9is7" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*04XSa5b23rCECgEF" /></figure><p>With over 3,000 models in bi-dbt currently live across dev, data and business analytics teams at Loblaw, this automated way of generating dbt docs helps us increase productivity to serve our core lines of business.</p><h3>Closing words</h3><p>In conclusion, as we continue to explore the powerful capabilities of Language Model Models (LLMs) in the realm of analytics engineering, we can anticipate exciting developments on the horizon. One promising future use case for LLMs lies in enabling users to query large, complex datasets using natural language, thus democratizing data access and analysis for a broader audience. This innovation promises to further transform and simplify the data engineering landscape, ultimately paving the way for even more efficient and insightful data-driven decision-making. At Loblaw Digital, we remain committed to harnessing cutting-edge technologies like LLMs to enhance our workflows and drive progress in the world of data.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c4735faa6ca5" width="1" height="1" alt=""><hr><p><a href="https://medium.com/loblaw-digital/leveraging-llms-to-generate-ai-driven-dbt-documentation-c4735faa6ca5">Leveraging LLMs to generate AI driven dbt documentation</a> was originally published in <a href="https://medium.com/loblaw-digital">Loblaw Digital</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>