<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Nate Nowack on Medium]]></title>
        <description><![CDATA[Stories by Nate Nowack on Medium]]></description>
        <link>https://medium.com/@nate.nowack?source=rss-e8be470b144a------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*9ELXuPOGn-N6qhRyHctBpg.jpeg</url>
            <title>Stories by Nate Nowack on Medium</title>
            <link>https://medium.com/@nate.nowack?source=rss-e8be470b144a------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 04:37:23 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@nate.nowack/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Keeping  on AI tools with Marvin]]></title>
            <link>https://medium.com/the-prefect-blog/keeping-on-ai-tools-with-marvin-1dc287b8d177?source=rss-e8be470b144a------2</link>
            <guid isPermaLink="false">https://medium.com/p/1dc287b8d177</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[pydantic]]></category>
            <category><![CDATA[prefect]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Nate Nowack]]></dc:creator>
            <pubDate>Fri, 16 Jun 2023 12:48:32 GMT</pubDate>
            <atom:updated>2023-06-16T12:48:32.447Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*t6_fPRr2R21eSr-w9I6v-Q.png" /><figcaption>a tapestry of random, potentially related events</figcaption></figure><p>Since March 14, when I <a href="https://medium.com/the-prefect-blog/keeping-your-eyes-on-your-ai-tools-6428664537da">posted about instrumenting LLM tools with Prefect</a>, a whole lot has happened in the AI tooling landscape.</p><p>Most significant to my world, <a href="https://github.com/PrefectHQ/marvin">Marvin</a> has <a href="https://star-history.com/#prefecthq/marvin&amp;Date">taken on a life of its own</a>. Leveraging the diverse LLM support from the open-source community, the data engineering amenities offered by Prefect, and the type-safety of Pydantic — Marvin has grown up to be a pythonic LLM toolbox that you can drop right into traditional software contexts to solve problems. If you want to learn more about how Marvin works, <a href="https://www.askmarvin.ai/">read the docs!</a></p><h4>expanding the original use case</h4><p>Making a Slack-based assistant for our <a href="https://www.prefect.io/community/">Prefect Community</a> was the start of all things Marvin, and it has been the use-case where we prove out our concepts to make sure they work in practice.</p><p>Today we’ll look at a concrete example of observability in an AI system — namely how we can track Marvin’s global token usage with the Events and Automations systems offered by Prefect Cloud (<a href="https://docs.prefect.io/2.10.13/cloud/cloud-quickstart/#sign-in-or-register">generous free tier</a> btw).</p><h4>background</h4><p>You can chat with our Prefect assistant bot (affectionately, also Marvin) in the <strong>#ask-marvin </strong>channel in our Slack community. There’s a parallel Marvin bot we run internally to assist with tasks like support, debugging and summarizing internal process documents .</p><p>Each time you or I tag any of our Marvins, Slack will hit that Marvin app’s FastAPI endpoints with the Slack event data and prompt a Prefect-aware <a href="https://www.askmarvin.ai/guide/concepts/bots/">Bot</a> with the user question:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/956/1*CSQowEdmcDdTiKyLtbc4jw.png" /><figcaption>Marvin answering questions in the wild</figcaption></figure><p>Each time, the bot will leverage an LLM (like GPT 4) and may use an arbitrary number of tools in order to answer the user question — you can learn more about the state of Marvin’s OpenAI tool use <a href="https://www.factsmachine.ai/p/the-prodigal-json">here</a>.</p><p>Between LLM calls and plugin execution, this can be time and token intensive stuff!</p><p>Prefect Cloud offers the idea of an Event , which is just a named and timestamped payload that we can emit from anywhere and then react to.</p><h4>emitting events</h4><p>Once Marvin settles on a response and sends it to you, we’ll run a little prefect routine from inside the endpoint called emit_event:</p><pre>from prefect.events import emit_event<br><br>...<br><br>async def _slackbot_response(event: SlackEvent):<br><br>    # other things happening ...<br><br>    response = await bot.say(text)<br><br>    slack_response = await _post_message(<br>        channel=event.channel, message=response.content, thread_ts=event.ts<br>    )<br><br>    prompt_tokens = count_tokens(text)<br>    response_tokens = count_tokens(response.content)<br><br>    # this will do nothing if Prefect credentials are not configured<br>    emit_event(<br>        event=f&quot;bot.{bot.name}.responded&quot;,<br>        resource={&quot;prefect.resource.id&quot;: f&quot;bot.{bot.name}&quot;},<br>        payload={<br>            &quot;user&quot;: event.user,<br>            &quot;channel&quot;: event.channel,<br>            &quot;thread_ts&quot;: thread,<br>            &quot;text&quot;: text,<br>            &quot;response&quot;: response.content,<br>            &quot;prompt_tokens&quot;: prompt_tokens,<br>            &quot;response_tokens&quot;: response_tokens,<br>            &quot;total_tokens&quot;: prompt_tokens + response_tokens,<br>        },<br>    )<br>    <br>    # more things happening ...</pre><p>… basically saying, “hey! Marvin responded in slack” to Prefect Cloud.</p><p>As a technical note, our community Marvin bot is running as a GCP Cloud Run service and has two important environment variables set:</p><pre>PREFECT_API_KEY=pnu_XXX<br>PREFECT_API_URL=https://api.prefect.cloud/api/accounts/&lt;UUID/workspaces/&lt;UUID&gt;</pre><p>which allow it to talk to our Prefect Cloud workspace.</p><p>So after I emit my event, I can pop into the marvin-bot workspace’s Event Feed and see some details about it:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/714/1*NSC4ruKmctR8WEzeAMUEwQ.png" /></figure><p>Now this event lives in the Events DB of my Prefect Cloud workspace, so I can query events over time for token usage by user or whatever else.</p><p>Being able to track aggregate token usage is great, but most exciting to me is the idea of using this event to immediately trigger other actions.</p><h4>the automations system</h4><p>Automations in Prefect Cloud have the form: <strong>Trigger</strong> &gt; <strong>Action</strong></p><p>So, with Prefect Cloud as a listener, I can have the receipt of certain events <strong>trigger</strong> an additional <strong>action</strong>. So here’s my trigger:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/746/1*666fUAOHL7KkKBaO8Y31ew.png" /><figcaption>Where any events with resource ids like `bot.*` will trigger my automation</figcaption></figure><p>… which reads as, trigger this automation once I receive 1 event where the resource id looks like bot.* .</p><p>Now, we decide what our action will be. Let’s choose Run a deployment , since I <a href="https://github.com/PrefectHQ/marvin/blob/main/examples/flows/token_usage.py#L66-L95">already implemented a small flow to increment a token tracker</a>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/752/1*QDrkR3sNEOMozwWymycoWA.png" /><figcaption>I can use jinja to template event payload data into string fields</figcaption></figure><p>Once I finish configuring the automation, a flow will run each time a user tags Marvin in Slack, incrementing my token tracker.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AyzCZHuNYy8ygBWhsRDAKw.png" /><figcaption>Flow run timeline view for token tracker flow</figcaption></figure><p>Now, if I only ever wanted to increment a token tracker, I could have just done that directly from within the _slackbot_response method from the Marvin app itself, without emitting an event that triggers a Prefect flow.</p><p>However, the larger idea here is that I can easily <strong>do</strong> <strong>arbitrary, observable work by eventing directly off of the user’s interaction with my application.</strong></p><p>I can maintain global state with Prefect Cloud and use it in many ways:</p><ul><li><a href="https://docs.prefect.io/2.10.13/concepts/states/">To govern the state transitions</a> of tasks within my application, and observe each unit of work performed (especially when the LLM is the one deciding how and when to use additional tools)</li><li>I could moderate user-bot interactions with my application(s), in a way that I can modify without re-deploying the Marvin service (i.e., edit an ACL Block in Cloud; the app reads from the Block in Cloud)</li><li>if I were using my own self-hosted LLM for my Marvin app, I could have this interaction <a href="https://docs.prefect.io/2.10.13/api-ref/prefect/deployments/?h=run_deplo#prefect.deployments.run_deployment">trigger a flow</a> to update a domain-specific training set</li><li>I could service user requests received by the Marvin app which require work from different applications, using the Prefect Cloud <a href="https://docs.prefect.io/2.10.13/cloud/events/">Events</a> and <a href="https://docs.prefect.io/2.10.13/cloud/automations/">Automations</a> systems as the router of that work</li></ul><h4>going forward</h4><p>I’m excited to expand the set of events that are emitted by Marvin, and in turn expand the set of actionable information I can leverage as a builder.</p><p>Having spent a lot of time in batch-schedule-land historically, its lovely to enter a fully event-driven world without breaking out a whole new streaming solution / tool.</p><p>As for Marvin’s future, we’ll be working on a first-class and opt-in Event instrumentation of Marvin’s LLM utilities to make as it as easy as possible to build transparent and reactive systems to control our use of AI in code.</p><p>Let us know if you have any questions in our <a href="https://www.prefect.io/community/">Prefect Slack community</a> or our <a href="https://discord.gg/Kgw4HpcuYG">Marvin Discord</a>.</p><p>—</p><p>Happy Engineering!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1dc287b8d177" width="1" height="1" alt=""><hr><p><a href="https://medium.com/the-prefect-blog/keeping-on-ai-tools-with-marvin-1dc287b8d177">Keeping 👀 on AI tools with Marvin</a> was originally published in <a href="https://medium.com/the-prefect-blog">The Prefect Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Your Python job is already a Prefect flow, it just doesn’t know it yet]]></title>
            <link>https://medium.com/@nate.nowack/your-python-job-is-already-a-prefect-flow-it-just-doesnt-know-it-yet-745c5fb6b050?source=rss-e8be470b144a------2</link>
            <guid isPermaLink="false">https://medium.com/p/745c5fb6b050</guid>
            <dc:creator><![CDATA[Nate Nowack]]></dc:creator>
            <pubDate>Thu, 08 Jun 2023 22:33:48 GMT</pubDate>
            <atom:updated>2023-06-08T22:33:48.976Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*pFyYLUBG67kh8QUe.jpg" /></figure><p><a href="https://www.prefect.io/blog/introducing-prefect-2-0/">Prefect 2.0</a> has graduated to beta status and it’s time to start kicking the tires!</p><p>The new iteration of the workflow orchestrator sets fire to clunky concepts historically central to data pipeline creation (notably <a href="https://en.wikipedia.org/wiki/Directed_acyclic_graph#Data_processing_networks">DAG</a>s) and allows us to write regular Python <strong>code as workflows</strong> without forcing us to refactor it just to play nice with more, uhh, <em>idiosyncratic</em> frameworks.</p><p>A unique benefit of this hands-off philosophy is the control to adopt as many (or few) Prefect features as we actually want in our workflows, for example:</p><p><em>You want caching but don’t need distributed compute?</em> <strong>cool</strong>!</p><p><em>Vice versa?</em> <strong>also</strong> <strong>cool</strong></p><p><em>Both?</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/0*w5AVyKFzbEgDepAD.gif" /><figcaption>RIP Troy and Abed In the Morning</figcaption></figure><h3>example use case: <em>you like chess?</em></h3><p>To put the rubber to the road, I’ll dive into a <a href="http://www.chess.com/">chess.com</a>-themed flow, highlighting the naturally Pythonic way we can build workflows with Prefect.</p><h4>what’s my example?</h4><p>I have a python script that takes in some list of Chess.com usernames<strong> </strong>and writes those users’ games to S3 as Pandas DataFrames in <a href="https://parquet.apache.org/docs/">parquet</a> (row 1:1 with game).</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/48a3a34fb9faf2c34dc413848b405047/href">https://medium.com/media/48a3a34fb9faf2c34dc413848b405047/href</a></iframe><h4><strong>💡 Did you know?</strong></h4><blockquote><strong>Portable Game Notation</strong> (PGN) is the human-readable format in which chess games are represented as text</blockquote><h4>what do I need to follow along and run the code?</h4><ul><li>a Python environment (I used a <a href="https://docs.conda.io/en/latest/miniconda.html">conda</a> venv) like the following</li></ul><pre>pip install chess.com boto3 pandas python-chess prefect</pre><ul><li>A file path to write the games to! (I used an <a href="https://docs.aws.amazon.com/AmazonS3/latest/userguide/create-bucket-overview.html">S3 Bucket</a>)</li></ul><h3>the script, without using any Prefect features</h3><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/43985e7a24aa5444439ffc1bc34e3dde/href">https://medium.com/media/43985e7a24aa5444439ffc1bc34e3dde/href</a></iframe><p>So, okay — we have some Python code that will load a user’s chess games into S3 for later analysis… what now?</p><p>While this is just a fun use case, it can also be an illustrative example of how we can incrementally adopt Prefect features to tackle inefficiencies in our workflows. Take the following considerations for example:</p><ul><li>Every time I run the code, it’s going to pull get_games. We can avoid duplicating this work automatically, with <strong>caching!</strong></li><li>In our orchestrator function (that I named orca 😅), we call the get_games function to fetch each month’s games in a list comprehension. Using pure Python in this way, that means all our API calls happen sequentially. To make our code faster, we can <strong>parallelize</strong> just by decorating our get_games function with @task to turn it into a Prefect task and adding a DaskTaskRunner to our flow.</li></ul><p>Let’s see how we can make these changes!</p><h4>The Glow Up: python script → Prefect Flow</h4><p>First thing first — in order to reap the benefits of Prefect features, let’s first decorate our existing python functions with @task and our orca function (i.e. driver, main, etc) with @flow , which will give us:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/99a2a2c793e2b0be193a58982b480057/href">https://medium.com/media/99a2a2c793e2b0be193a58982b480057/href</a></iframe><p>… and presto change-o, we have ourselves a Prefect Flow! If you run the code now, you’ll see that we have some fancy logging action going on:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6b28e57f2e2083f2094747fe22b2818a/href">https://medium.com/media/6b28e57f2e2083f2094747fe22b2818a/href</a></iframe><p>So we’re grabbing games from the chess.com API and packing them into Pandas DataFrames right? Instead of writing and testing extra python logic to check which games we’ve already fetched and loaded, let’s use caching to save our function results (called <em>tasks</em> once wrapped with the @task decorator) so our workflow just remembers what we’ve done. That would look like:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6e133af7755ea14ddafce9db90893719/href">https://medium.com/media/6e133af7755ea14ddafce9db90893719/href</a></iframe><p>What’s going on with this cache_key_fn=task_input_hash in the decorator?</p><p>Basically, task_input_hash will create a unique string (i.e. <a href="https://www.investopedia.com/terms/h/hash.asp#:~:text=A%20hash%20is%20a%20mathematical,always%20be%20the%20same%20size.">hash</a>) based on the content of the arguments passed to the get_games task (just the url in this case). Prefect will then save the results of this get_games task and associate the hash with the results, so that if we ever see that hash again in future invocations of the task, Prefect will know that we already did that work.</p><p>To put it succinctly: if we already saw the input, we already have the output!</p><p>Having added caching functionality, let’s run it and look at the logs again:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7ec98e2bc9d57c6c3e50e0d37bd91618/href">https://medium.com/media/7ec98e2bc9d57c6c3e50e0d37bd91618/href</a></iframe><p>Notice anything different?</p><p>Remember that logger output from get_games? There’s no such output anymore since the Prefect engine realized it’s seen that url before, so it didn’t have to run that code. It just pulled the task result associated with that url from our cache, and left the get_games tasks in Cached state.</p><h4>💡 Did you know?</h4><blockquote><strong>Prefect Storage </strong>can be configured to cache your task results, try the CLI command <em>prefect storage create --help</em></blockquote><p>So far, we’ve accomplished an efficiently lazy workflow — let’s see if we can make it faster and more distributed by exploring the Prefect <a href="https://orion-docs.prefect.io/concepts/task-runners/">DaskTaskRunner</a>.</p><h4>concurrency — avoid slow sequential execution</h4><p>Prefect will execute tasks concurrently by default:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c22c532e6f82845282338f5ad38255cb/href">https://medium.com/media/c22c532e6f82845282338f5ad38255cb/href</a></iframe><p>… and you’ll notice that both the get_games and load_games tasks are now executed concurrently (even though we’re just using list comprehension and for loops)!</p><h4>… and that’s a wrap!</h4><p>I hope this conveyed how easy it is to start leveraging Prefect’s newest offering — elevating your python jobs into robust and distributed workflows with simple annotations to your code, as you’d prefer to write it!</p><p>Happy Engineering!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=745c5fb6b050" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Keeping Your Eyes on Your AI Tools]]></title>
            <link>https://medium.com/the-prefect-blog/keeping-your-eyes-on-your-ai-tools-6428664537da?source=rss-e8be470b144a------2</link>
            <guid isPermaLink="false">https://medium.com/p/6428664537da</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[openai]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[prefect]]></category>
            <category><![CDATA[langchain]]></category>
            <dc:creator><![CDATA[Nate Nowack]]></dc:creator>
            <pubDate>Tue, 14 Mar 2023 16:14:24 GMT</pubDate>
            <atom:updated>2023-03-14T16:14:24.149Z</atom:updated>
            <content:encoded><![CDATA[<h3>Keeping Your 👀 on Your AI Tools</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DWL7PzmKSSHbgGHcyFKkig.png" /><figcaption>Big Brother Marvin keeping a watchful eye over our AI minions</figcaption></figure><p>Like everyone else, I’ve been playing with AI over the last couple of months.</p><p>Projects like <a href="https://openai.com/">OpenAI</a> and <a href="https://huggingface.co/">Hugging Face</a> have made it simpler than ever to interact with LLMs like GPT3 — whether it’s popping open a new tab in my browser for <a href="https://openai.com/blog/chatgpt">ChatGPT</a> or <a href="https://openai.com/product#made-for-developers">making a simple API call</a> with Python.</p><blockquote><strong>What is an LLM? <br></strong>“Large Language Model” refers to a type of AI model that’s been trained on massive amounts of text to generate human-like language. There are <a href="https://en.wikipedia.org/wiki/Algorithmic_bias">open questions</a> about the use of LLMs.</blockquote><p>One very interesting way to leverage this new LLM accessibility is being explored by a project called <a href="https://github.com/hwchase17/langchain">LangChain</a>, the growth of which has exploded.</p><p>LangChain is open-source tooling that adds some useful abstractions for interacting with these LLMs, whether that’s:</p><ul><li>callable LLMs to simplify ad-hoc API calls</li><li><a href="https://langchain.readthedocs.io/en/latest/use_cases/agents.html">successive calls to LLMs + tools to refine answers to questions</a></li><li><a href="https://langchain.readthedocs.io/en/latest/use_cases/chatbots.html">building customized chatbots</a> by persisting information in vector <a href="https://pypi.org/project/chromadb/">embedding databases</a>, and querying them to provide LLMs with whatever context it needs to answer a question</li></ul><p>Through these LLM interactions + others, LangChain is in the business of “enabling developers to build applications that they previously could not.”</p><p>I’ll explore some functionality LangChain makes easy, and then present a first pass at a plugin powered by <a href="https://github.com/PrefectHQ/prefect">Prefect</a>, which can help us observe our use of LangChain, named <a href="https://github.com/PrefectHQ/langchain-prefect">langchain-prefect</a>.</p><p><strong>Exploring LangChain</strong></p><p>Starting barebones, if I have an OPENAI_API_KEY on my machine, I can go ahead and pip install langchain openai and do a random ol’ LLM call:</p><pre>from langchain.llms import OpenAI<br><br>llm = OpenAI(temperature=0.9)<br>answer: str = llm(<br>    &quot;What would be a good company name for &quot;<br>    &quot;a company that makes colorful socks?&quot;<br>)<br><br>print(answer) # prints something like `Socktastic!&#39;</pre><p>This is cool. But, to quote LangChain’s README:</p><blockquote>the real power comes when you can combine (LLMs) with other sources of computation or knowledge</blockquote><p>A tool like LangChain starts to gets interesting when I integrate other tools, APIs, and embedding databases — but it also starts to get complex enough that orchestration is non-trivial.</p><h3>As with any tool I’d want to build on top of…</h3><p>Before I dive headlong into building some LLM-based application using LangChain, it’s important to think pragmatically about a few things.</p><p>Let’s say I want to build a chatbot powered by LangChain and some LLM with domain-specific knowledge I can curate and regularly update.</p><p>I’d want the chatbot to handle a conversation with a user, answering their questions using the <strong>LLM + context + tools</strong> at its disposal.</p><p>Implementation details aside, I’d need to answer a couple of questions before deploying this chatbot in a production use case.</p><h4>Ballpark, how much will it cost?</h4><p>The main currency with these LLMs are tokens — for example, OpenAI charges $0.0200 / 1K tokens to use their DaVinci model. 1 token <strong><em>roughly </em></strong>corresponds to one word, depending on who you ask.</p><p>You pay for <strong>total tokens, </strong>which is <strong>prompt tokens </strong>+ <strong>completion tokens, </strong>i.e. text you send the LLM + text you get back from it.</p><p>If I’m using LangChain agents to make successive LLM calls / leverage tools, and they’re <a href="https://langchain.readthedocs.io/en/latest/modules/indexes/combine_docs.html?highlight=stuffing#stuffing">stuffing</a> extra context (tokens) in those calls, how much will that cost me?</p><p><strong>A quick experiment and some math:</strong></p><p>Let’s invoke a LangChain agent like my eventual chatbot might.</p><p>It will use the llm-math tool to do the math and fall back to asking the LLM when it doesn’t know the answer (you could add <a href="https://serpapi.com/blog/up-to-date-gpt-3-info-with/">serpapi</a> as a tool here).</p><pre>from langchain.agents import initialize_agent, load_tools<br>from langchain.llms import OpenAI<br><br>llm = OpenAI(temperature=0)<br>tools = load_tools([&quot;llm-math&quot;], llm=llm)<br>agent = initialize_agent(tools, llm)<br><br>answer: str = agent.run(<br>    &quot;How old is the current Dalai Lama? &quot;<br>    &quot;What is his age divided by 2 (rounded to the nearest integer)?&quot;<br>)<br>print(answer) # The current Dalai Lama&#39;s age divided by 2 is 43</pre><p>When I ran this, the agent made 7 calls to the LLM:</p><p>let<strong> total, p, c </strong>= <strong>total_tokens, prompt_tokens, completion_tokens</strong></p><ul><li><strong>total</strong>: 205 = <strong>p</strong>: 172 + <strong>c</strong>: 33</li><li><strong>total</strong>: 254 = <strong>p</strong>: 242 + <strong>c</strong>: 12</li><li><strong>total</strong>: 243 = <strong>p</strong>: 224 + <strong>c</strong>: 19</li><li><strong>total</strong>: 244 = <strong>p</strong>: 239 + <strong>c</strong>: 5</li><li><strong>total</strong>: 282 = <strong>p</strong>: 256 + <strong>c</strong>: 26</li><li><strong>total</strong>: 245 = <strong>p</strong>: 242 + <strong>c</strong>: 3</li><li><strong>total</strong>: 316 = <strong>p</strong>: 293 + <strong>c</strong>: 23</li></ul><p>That shakes out to a <strong>grand total </strong>of<strong> 1789 tokens = $0.03578 = 3.578 cents</strong></p><p>Not bad, but this is only one invocation of a barebones agent. If I used other <a href="https://langchain.readthedocs.io/en/latest/modules/agents/tools.html?highlight=tools">tools</a> (which could be chains/agents) or pulled more context from a <a href="https://langchain.readthedocs.io/en/latest/reference/modules/vectorstore.html">vectorstore</a> to stuff in my prompts, that number would rise pretty dramatically — depending on the math you’d have to do for your app.</p><p>If you’ve followed along and run the example above, you might wonder how I even figured out my total token usage. It didn’t come back as a return from agent.run 🧐 — but we’ll get to that in just a bit.</p><p><strong>So how much <em>does</em> it cost?</strong></p><p><strong>It depends. </strong>It depends on how well I write my application, what I want it to do, and how well LangChain is leveraging my context when talking to LLMs for me. It certainly seems like something I want to keep track of.</p><p>What I haven’t considered here yet at all is infrastructure costs, although, yeah, we will need a scalable and custom runtime for this app.</p><h4>How do I know that things are working the way I want?</h4><p>On its face, this seems obvious. It’s working if my users get a useful answer from the chatbot, right?</p><p>Well, that’s part of the story for sure. But, this question strikes me as one that’d be most difficult to nail down at production scale — where processes running LLM calls are just one part of a complex workflow.</p><p>As abstractions for interfacing with LLMs mature, it feels like it’s going get difficult to keep track of what’s going on under the hood — just like it’s non-trivial to keep track of modern ELT / ETL at production scale.</p><p><strong>Hypothetical question:</strong> A user opens a support ticket for my chatbot and says my chatbot <a href="https://youtu.be/1TO48Cnl66w?t=18">left them in the cold</a>. How do I know what went wrong?</p><ul><li>Is OpenAI having an incident? did they choke my API calls?</li><li>Is there a problem with my embeddings db?</li><li>Am I orchestrating my LangChain agents/chains incorrectly?</li></ul><p>Building tooling to frame and answer these sorts of questions sounds like a lot. But then again, <strong>Prefect <em>does</em> kinda do a lot</strong>.</p><h3>Making a Prefect plugin for LangChain</h3><p>I’ll share some progress on a plugin for LangChain to address some of the pain points I’ve mentioned above, dubbed langchain-prefect.</p><p>Let’s first go ahead and pip install langchain-prefect…</p><p>and return to that first example, with a new friend named RecordLLMCalls:</p><pre>from langchain.llms import OpenAI<br>from langchain_prefect.plugins import RecordLLMCalls<br><br>llm = OpenAI(temperature=0.9)<br><br>with RecordLLMCalls():<br>  answer: str = llm(<br>      &quot;What would be a good company name for &quot;<br>      &quot;a company that makes colorful socks?&quot;<br>  )<br>  <br>  print(answer)</pre><pre>17:45:19.782 | INFO | prefect.engine - Created flow run &#39;scarlet-chital&#39; for flow &#39;Execute LLM Call&#39;<br>17:45:21.319 | INFO | Flow run &#39;Calling langchain.llms.openai&#39; - Sending &#39;What would be a good company name for a company that makes colorful socks?&#39; to langchain.llms.openai via &lt;function BaseLLM.generate at 0x11e567a30&gt;<br>17:45:22.211 | INFO | Flow run &#39;Calling langchain.llms.openai&#39; - Recieved: NotAnArtifact(name=&#39;LLM Result&#39;, description=&#39;The result of the LLM invocation.&#39;, content=LLMResult(generations=[[Generation(text=&#39;\n\nHappy Feet Socks.&#39;, generation_info={&#39;finish_reason&#39;: &#39;stop&#39;, &#39;logprobs&#39;: None})]], llm_output={&#39;token_usage&#39;: {&#39;completion_tokens&#39;: 7, &#39;total_tokens&#39;: 22, &#39;prompt_tokens&#39;: 15}}))<br>17:45:22.336 | INFO | Flow run &#39;scarlet-chital&#39; - Finished in state Completed()<br><br>Happy Feet Socks.</pre><h4>Uhh, what just happened?</h4><p>Let’s break it down by looking at the logs.</p><p>With langchain-prefect you get prefect. With Prefect, <a href="https://docs.prefect.io/concepts/flows/#flows-overview">you can wrap arbitrary Python functions in a </a><a href="https://docs.prefect.io/concepts/flows/#flows-overview">@flow decorator</a> so when you run one of them, metadata on that execution is tracked (see footnotes).</p><p>From inside of RecordLLMCalls, the LangChain methods that actually make the API call to the LLM are wrapped in a @flow so that Prefect can track the execution when they are invoked. Prefect gives fun names like scarlet-chital to each flow run, so we see something like:</p><pre>prefect.engine - Created flow run &#39;scarlet-chital&#39; for flow &#39;Execute LLM Call&#39;</pre><p>Then we see what is actually being sent to the LLM, how the LLM is being prompted by LangChain, the LLM output, and how many tokens we used.</p><pre>Flow run &#39;Calling langchain.llms.openai&#39; - Sending &#39;What would be a good company name for a company that makes colorful socks?&#39; to langchain.llms.openai via &lt;function BaseLLM.generate at 0x11e567a30&gt;<br>Flow run &#39;Calling langchain.llms.openai&#39; - Recieved: NotAnArtifact(name=&#39;LLM Result&#39;, description=&#39;The result of the LLM invocation.&#39;, content=LLMResult(generations=[[Generation(text=&#39;\n\nHappy Feet Socks.&#39;, generation_info={&#39;finish_reason&#39;: &#39;stop&#39;, &#39;logprobs&#39;: None})]], llm_output={&#39;token_usage&#39;: {&#39;completion_tokens&#39;: 7, &#39;total_tokens&#39;: 22, &#39;prompt_tokens&#39;: 15}}))</pre><p>Finally, since our Python code (or LangChain’s) ran without errors, we see:</p><pre>Flow run &#39;scarlet-chital&#39; - Finished in state Completed()</pre><h4>Here’s where it gets fun.</h4><p>If we pop into a terminal and run prefect server start, you’ll be prompted to <strong>Check out the dashboard at </strong><a href="http://127.0.0.1:4200/"><strong>http://127.0.0.1:4200</strong></a></p><p>… where you’ll see a fancy dashboard just waiting to fill up with colors:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sIXe28O5x9Uoa9WPAVOfhw.png" /><figcaption>The default dashboard view, showing colored dots to represent states of our flow runs</figcaption></figure><p>If we click into our flow run, we see:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VVRpzTgWEuAG1XyP8w4LyA.png" /><figcaption>All the metadata and logs for a given flow run</figcaption></figure><p>The logs and a bunch of metadata on when, what, and how things happened.</p><p><strong>Prefect is uniquely interesting here.</strong></p><p>In Prefect, you can call flows from inside other flows — it’s called a <a href="https://docs.prefect.io/concepts/flows/#composing-flows">subflow</a>.<br>Look what happens if I run the agent example with RecordLLMCalls and add a flow wrapping my agent.run call:</p><pre>from langchain.agents import initialize_agent, load_tools<br>from langchain.llms import OpenAI<br>from langchain_prefect.plugins import RecordLLMCalls<br>from prefect import flow<br><br>llm = OpenAI(temperature=0)<br>tools = load_tools([&quot;llm-math&quot;], llm=llm)<br>agent = initialize_agent(tools, llm)<br><br>@flow<br>def my_flow():<br>    &quot;&quot;&quot;Flow wrapping any LLM calls made by the agent.&quot;&quot;&quot;<br>    return agent.run(<br>        &quot;How old is the current Dalai Lama? &quot;<br>        &quot;What is his age divided by 2 (rounded to the nearest integer)?&quot;<br>    )<br>with RecordLLMCalls(tags={&quot;agent&quot;}):<br>    result = my_flow()</pre><p>Remember how we ran this earlier, and it made 7 LLM calls? Here they are.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jr8R4y05SisyMTtjbgaNCg.png" /><figcaption>Prefect UI showing the subflow runs (i.e. LLM calls) of `my_flow`</figcaption></figure><p>For each one, we’re still tracking LLM input / output and token usage, and now we know where these came from. I used tags={‘agent’} to apply an agent tag to any LLM calls that agent.run created on my behalf.</p><p>The nice thing about Prefect flows is that my_flow that wraps these LLM calls could itself be called in the context of a larger workflow (even nested as a subflow of another wrapping parent flow).</p><p>Here’s what the usage page looks like in OpenAI:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/286/1*ITOT1_IIwRTBTaxkBmmAdA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MdvxydpmlY_DOoLx9FsT7g.png" /><figcaption>not the most enlightening</figcaption></figure><h3>Where do we go from here?</h3><p>Okay, hopefully, that was enough to get the blood flowing. Let’s sum up.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7DF4YubZRtsnSylGb9ttqw.png" /><figcaption>so many possibilities</figcaption></figure><p>In general, LLMs are interesting and useful — building apps that use them responsibly feels like a no-brainer. Tools like LangChain make it easier to build LLM apps. We still need to know details about how our apps work, even when we want to use tools with convenient abstractions that can obfuscate those details.</p><p>Prefect is built to help data people run, scale, and observe event-driven work wherever they want. It provides a framework for <a href="https://docs.prefect.io/concepts/deployments/">deployments</a> on a whole slew of runtime environments (from Lambda to Kubernetes) on all the main cloud providers. Productionizing LLMs is right up Prefect’s alley.</p><p>Among the more advanced concepts in Prefect are <a href="https://docs.prefect.io/concepts/results/">results</a>, which are applicable here. Caching common LLM results, aggregating token usage across many actors, and routing good and bad LLM results for use in fine-training all feel like practical choices that results could help implement.</p><p>I’m biased, but frankly I’m excited about Prefect’s potential to address problems in LLM observability. As the AI hype train pushes forward, and we try to balance easy abstractions with control, I believe it will be increasingly important to understand what exactly is happening with applications built on LLMs — the same way it is now for apps built on data warehouses. Real outcomes depend on our tools working as expected.</p><p>Stay tuned for further exploration of tools like LangChain! Questions on langchain-prefect or want to contribute? Find and fork <a href="https://github.com/PrefectHQ/langchain-prefect">the code</a>.</p><p>Happy Engineering!</p><p>—</p><h4>Footnotes</h4><p><strong>How is metadata tracked?<br></strong>By default (with no configuration outside of pip install prefect), Prefect flows spin up a lightweight ephemeral API (FastAPI) and use it to start writing details about flow runs to a SQLite db file ~/.prefect/prefect.db.</p><p>You can inspect flow runs with the dashboard after prefect server start.</p><p>It’s also super simple to <a href="https://docs.prefect.io/ui/cloud-quickstart/">let Prefect Cloud handle UI, API, and DB</a> management for you, for free — but for dev’ing on things, the ephemeral API / local SQLite DB / local UI dashboard is great.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6428664537da" width="1" height="1" alt=""><hr><p><a href="https://medium.com/the-prefect-blog/keeping-your-eyes-on-your-ai-tools-6428664537da">Keeping Your Eyes on Your AI Tools</a> was originally published in <a href="https://medium.com/the-prefect-blog">The Prefect Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Coordinate ELT in 2023 with Airbyte, dbt and Prefect]]></title>
            <link>https://medium.com/the-prefect-blog/coordinate-elt-in-2023-with-airbyte-dbt-and-prefect-ecd7547e6c1a?source=rss-e8be470b144a------2</link>
            <guid isPermaLink="false">https://medium.com/p/ecd7547e6c1a</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Nate Nowack]]></dc:creator>
            <pubDate>Tue, 31 Jan 2023 22:29:26 GMT</pubDate>
            <atom:updated>2023-02-01T00:00:59.343Z</atom:updated>
            <content:encoded><![CDATA[<h4>Orchestrate data ingestion and transformation for Snowflake data warehouse with Airbyte, dbt, and Prefect</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jaIYOW5Nv0BmKWQeWVUlRw.png" /></figure><p>When wrangling data on a small team, it’s hard not to rely on tools that help us get event data from the world to our data consumers — they’re just too useful to ignore!</p><p>Though, as a data practice scales, it can get difficult to keep track of your tools’ interactions that are not <em>automated</em> and <em>visible</em>.</p><p>This tutorial is a quick start to a simple, scalable, and maintainable ELT workflow using an open-source stack and a free-tier data warehouse. We’ll be using the following tools:</p><ul><li><a href="https://airbyte.io/">Airbyte</a> to <strong>extract</strong> data sources and <strong>load</strong> them to a data warehouse (like <a href="https://www.snowflake.com/">Snowflake</a>)</li><li><a href="https://www.getdbt.com/">dbt</a> to <strong>transform</strong> the data</li><li><a href="https://www.prefect.io/">Prefect</a> to <strong>automate</strong> and <strong>observe</strong> our ecosystem of tools</li><li>GitHub as a source of event data and a place to store our code</li></ul><p>For the purpose of illustrating an ELT pattern, we choose a fun question that serves as the motivation for the workflow:</p><blockquote>Who are the <strong>common contributors</strong> to airbyte, dbt and prefect?</blockquote><p>Alrighty, let’s get into it!</p><h4>Table of Contents</h4><p>· <a href="#3c7c">Set up an Airbyte instance</a><br>· <a href="#8743">Create some Airbyte connections</a><br> ∘ <a href="#94bb">Create a GitHub source connection</a><br> ∘ <a href="#efc4">Create a Snowflake destination connection</a><br>· <a href="#77df">Create some dbt models to transform the raw data</a><br>· <a href="#2dd8">Write a Prefect flow to orchestrate our tools</a><br> ∘ <a href="#71e7">Let’s use some Prefect collections to make our lives easier</a><br>· <a href="#2834">Define a Prefect flow</a><br>· <a href="#5aca">Running Airbyte connections</a><br>· <a href="#331d">Run dbt models</a><br>· <a href="#a523">Run some queries against the transformed data</a><br>· <a href="#58f0">The whole enchilada</a><br>· <a href="#2de2">Running our flow</a><br>· <a href="#2de6">… and that’s it!</a><br>· <a href="#5958">Extending this example</a><br>· <a href="#623e">Next steps</a></p><h3>Set up an Airbyte instance</h3><p>This tutorial is based on Open-Source Airbyte. If you are using Airbyte Cloud, you’ll need to <a href="https://app.harvestr.io/roadmap/view/pQU6gdCyc/airbyte-roadmap">wait for the Airbyte Cloud API</a> to programmatically interact with Airbyte from Prefect.</p><p>To set up a local Airbyte instance, follow the <a href="https://docs.airbyte.com/quickstart/deploy-airbyte/">Airbyte Quickstart</a> guide or <a href="https://docs.airbyte.com/deploying-airbyte/on-kubernetes-via-helm/#add-helm-repository">use a helm chart</a>.</p><h3>Create some Airbyte connections</h3><p>For this tutorial, we’ll use <a href="https://docs.airbyte.io/integrations/sources/github">Airbyte’s GitHub source connector</a> to pull data from the GitHub organizations whose repositories are used in this tutorial:</p><ul><li><a href="https://github.com/AirbyteHQ/airbyte">airbytehq/airbyte</a></li><li><a href="https://github.com/dbt-labs/dbt-core">dbt-labs/dbt-core</a></li><li><a href="https://github.com/PrefectHQ/prefect">prefecthq/prefect</a></li></ul><h4>Create a GitHub source connection</h4><p>All Airbyte needs to connect to GitHub is a valid <a href="https://docs.github.com/en/github/authenticating-to-github/creating-a-personal-access-token">personal access token</a>.</p><p>Let’s go ahead and create a new connection in the Airbyte UI by clicking on the <strong>Sources</strong> tab and click <strong>+ New Source</strong>. Select the <strong>GitHub</strong> source connector and pass along your token (or comma-separated list of tokens to distribute load across multiple actors):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Zut6_w69PwMNzqT2Rlkm2w.png" /><figcaption>Airbyte UI when creating a GitHub source connector</figcaption></figure><p>For this tutorial, we’ll create a connection for each of the three GitHub organizations we’re interested in, grabbing the repository of interest for each organization.</p><h4>Create a Snowflake destination connection</h4><p>To tell Airbyte where to drop our raw data in Snowflake, we’ll need to provide the following information:</p><ul><li><strong>account</strong>: the name of your Snowflake account</li><li><strong>username</strong>: the name of the user that Airbyte will use in Snowflake</li><li><strong>password</strong>: the password for this user</li><li><strong>database</strong>: the name of the database that Airbyte will use to write data to</li><li><strong>schema</strong>: the name of the schema that Airbyte will use to write data to</li></ul><p>Let’s go ahead and create a new connection in the Airbyte UI by clicking on the <strong>Destinations</strong> tab and then clicking on <strong>+ New Destination</strong>. Select the <strong>Snowflake</strong> destination connector and pass in the information above.</p><blockquote><em>💡 </em><strong><em>Note</em></strong><em>: While you’ve got this info handy, you can also use it to connect your dbt CLI to Snowflake. See the dbt docs for more info — psst… </em><em>dbt debug is your friend!</em></blockquote><h3>Create some dbt models to transform the raw data</h3><p>Now that we’ve given Airbyte the ability to pull data from GitHub and write it to Snowflake, we can use dbt to transform the raw data into a format that’s more useful for our downstream consumers (like BI tools, fun dashboards and downstream dbt models).</p><p>While not the focus of this tutorial, the source code for the dbt models we’ll be using can be found in the linked <a href="https://github.com/zzstoatzz/blogs/tree/master/prefect-airbyte-dbt-snowflake/github_common_contributors">repository</a>.</p><p>In this tutorial, we’ll create two models: <strong>common_commiters</strong> and <strong>common_issue_submitters</strong>. These will create views in Snowflake containing the committers and issues submitters that are common across all three repositories. We’ll then be able to query these views and check out the overlap between the three communities.</p><h3>Write a Prefect flow to orchestrate our tools</h3><p>It’s time to connect our tools with Prefect!</p><h4>Let’s use some Prefect collections to make our lives easier</h4><pre>pip install prefect-airbyte prefect-dbt prefect-snowflake</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*h5tyGPqX197jlJLCIj43Aw.jpeg" /><figcaption>it’s 2023, use a venv</figcaption></figure><p>… and now let’s import some of their useful functionality for our flow:</p><pre>from prefect_airbyte import AirbyteConnection, AirbyteServer<br>from prefect_dbt.cli.commands import trigger_dbt_cli_command<br>from prefect_snowflake.database import SnowflakeConnector, snowflake_multiquery</pre><h3>Define a Prefect flow</h3><p>It’s often useful to take a step back and think about each step we want to take in our flow. In this case, we want to:</p><ul><li><strong>trigger</strong> existing <strong>Airbyte</strong> connections to drop raw data into our Snowflake database</li><li><strong>run</strong> <strong>dbt</strong> models to transform the raw data into something more useful</li><li><strong>query</strong> the transformed data in <strong>Snowflake</strong> to find the answer to our original question</li></ul><p>Let’s dive into the code for each of these steps!</p><h3>Running Airbyte connections</h3><p>Lucky for us, the AirbyteConnection block defined in prefect-airbyte takes care of the hard work for us. If we tell it which connection to trigger and which AirbyteServer to use, it&#39;ll do the rest!</p><p>Let&#39;s create our AirbyteConnection blocks, all on the same AirbyteServer:</p><pre>airbyte_server = AirbyteServer(server_host=&quot;localhost&quot;, server_port=8000)<br><br>airbyte_connections = [<br>    AirbyteConnection(<br>        connection_id=connection_id,<br>        airbyte_server=airbyte_server<br>    )<br>    for connection_id in [<br>        &quot;fa8d5164-ca22-47da-83e0-829cc86a70b8&quot;, # Airbyte Github Stats<br>        &quot;8e3a3ef1-6c2d-4255-99d4-21e9e542d853&quot;, # DBT Github Stats<br>        &quot;980c14d9-2992-46a7-a1dc-2b92e60e1475&quot; # Prefect Github Stats<br>    ]<br>]</pre><p>…and a task to manage their execution:</p><pre>@task<br>def run_airbyte_sync(connection: AirbyteConnection) -&gt; AirbyteSyncResult:<br>    job_run = connection.trigger()<br>    job_run.wait_for_completion()<br>    return job_run.fetch_result()</pre><blockquote><em>💡 </em><strong><em>Note</em></strong><em>: All Prefect blocks that implement the </em><em>JobBlock interface (like </em><em>AirbyteConnection) have a </em><em>trigger method that returns a </em><em>JobRun , which has </em><em>wait_for_completion and </em><em>fetch_result methods.</em></blockquote><h3>Run dbt models</h3><p>To trigger dbt run, you can leverage the predefined Prefect task from the dbt collection called trigger_dbt_cli_command. You can pass the command to run and the path to your dbt project as follows:</p><pre>dbt_result = trigger_dbt_cli_command( # run dbt models<br>    command=&quot;dbt run&quot;,<br>    project_dir=&quot;github_common_contributors&quot;,<br>    wait_for=airbyte_results<br>)</pre><p>Notice the use of wait_for here. This tells Prefect that we want to make sure the airbyte tasks have successfully dropped raw data in Snowflake before we try to transform it with dbt.</p><blockquote><em>💡 </em><strong><em>Note</em></strong><em>: If you’re using </em><a href="https://www.getdbt.com/product/what-is-dbt/"><em>dbt Cloud</em></a><em>, you can use the functionality defined in </em><em>prefect_dbt.cloud instead.</em></blockquote><h3>Run some queries against the transformed data</h3><p>Assuming we now have transformed data, we need to tell Prefect how to connect to Snowflake and which queries to run. The first part can be accomplished by creating a SnowflakeConnector block:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FrSHjINFgF3OICZGOydY4g.png" /><figcaption>Prefect UI while creating a `SnowflakeConnector` block</figcaption></figure><blockquote><em>💡 Note: A </em><em>SnowflakeConnector needs a set of </em><em>SnowflakeCredentials to authenticate with Snowflake. As with any </em><a href="https://docs.prefect.io/concepts/blocks/"><em>block</em></a><em>, you can create this in the UI or programmatically.</em></blockquote><p>Now we can use our SnowflakeConnector to connect to the right place in Snowflake and snowflake_multiquery (another pre-built task) to ask questions about the transformed data that lives there:</p><pre>common_authors, common_issue_submitters = snowflake_multiquery(<br>    queries=[<br>        &quot;select login from commit_authors&quot;,<br>        &quot;select login from issue_submitters&quot;<br>    ],<br>    snowflake_connector=SnowflakeConnector.load(&quot;github-contributors&quot;),<br>    wait_for=[dbt_result]<br>)</pre><p>In the same way as before, we use wait_for to make sure the dbt models have successfully run before we try to query the transformed data.</p><h3>The whole enchilada</h3><p>Pretty nifty, huh? Let’s put it all together into a @flow:</p><pre>@flow(log_prints=True)<br>def my_elt_flow():<br><br>    # run airbyte syncs<br>    airbyte_results = run_airbyte_sync.map(airbyte_connections)<br><br>    # run dbt models<br>    dbt_result = trigger_dbt_cli_command(<br>        command=&quot;dbt run&quot;,<br>        project_dir=&quot;github_common_contributors&quot;,<br>        wait_for=airbyte_results<br>    )<br><br>    # query our fancy new tables<br>    common_authors, common_issue_submitters = snowflake_multiquery(<br>        queries=[<br>            &quot;select login from commit_authors&quot;,<br>            &quot;select login from issue_submitters&quot;<br>        ],<br>        snowflake_connector=SnowflakeConnector.load(&quot;github-contributors&quot;),<br>        wait_for=dbt_result<br>    )<br>    <br>    # log the common contributors among our repos<br>    print(<br>        f&quot;Common authors: {common_authors} &quot;<br>        f&quot;Common issue submitters: {common_issue_submitters}&quot;<br>    )<br><br>if __name__ == &quot;__main__&quot;:<br>    my_elt_flow()</pre><h3>Running our flow</h3><p>To run our flow locally, we can simply run the script:</p><pre>python elt_orchestrator.py</pre><p>…and watch it go 🚀</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VBomeueviU1__UBziNfwsw.png" /><figcaption>shell output after running our flow locally</figcaption></figure><p>Turns out dependabot is the only common contributor since last August — laaaaaaaame.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*RAh09bSkOEV10luN" /><figcaption>Photo by <a href="https://unsplash.com/@thomascpark?utm_source=medium&amp;utm_medium=referral">Thomas Park</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>… but oh well — we have a nice and sturdy ELT pattern now.</p><p>Let’s create a <a href="https://docs.prefect.io/concepts/deployments/">deployment</a> for our flow so we can run it when and where we want.</p><p>To do this, we need to make two decisions:</p><ul><li><strong>where</strong> do we want <strong>to</strong> <strong>run</strong> our flow?</li><li><strong>where</strong> do we want <strong>to</strong> <strong>store</strong> our flow?</li></ul><p>To keep things simple, I’ll run our code locally and keep it in GitHub (where it is, anyway!).</p><p>In terms of prefect deployments, that means I’m using the default <a href="https://docs.prefect.io/concepts/infrastructure/#process">Process infrastructure</a> and a <a href="https://docs.prefect.io/api-ref/prefect/filesystems/#prefect.filesystems.GitHub">GitHub filesystem block</a>.</p><p>The command to create the deployment is:</p><pre>prefect deployment build flows/elt_orchestrator.py:my_elt_flow \<br>	-n &quot;elt-pipeline&quot; \<br>	-sb github/demo-repo \<br>	-a</pre><blockquote><em>💡 </em><strong><em>Note</em></strong><em>: If we wanted to use a different runtime infrastructure (like Kubernetes), we could specify that in the </em><em>prefect deployment build command by passing </em><em>-ib kubernetes/my-k8s-job-template. </em><em>-a tells prefect to go ahead and </em><em>apply the deployment.</em></blockquote><p>With our new deployment created, we can hop over to the UI and run it manually or attach an interval/cron schedule.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5UhgxbYIdeAKiXt6Xk8_rw.png" /><figcaption>Note the “<strong>Schedule</strong>: ✏️ Add” button on the right side</figcaption></figure><h3>… and that’s it!</h3><p>We now have an intelligent Prefect workflow that will not waste time/resources running downstream tasks if upstream tasks fail.</p><p>When the flow fails (as it probably will at some point), it’s easy to identify and triage issues by observing the failing flow run in the Prefect UI!</p><p>If we ever want to change how or when our flow executes, we can do so without ever changing our code!</p><ul><li>change how often it runs → update the deployment schedule in the UI</li><li>change where we store our code → add new storage block, rebuild deployment with prefect deployment build ... -sb block-type/name (same goes for infrastructure block determining where it runs)</li></ul><h3>Extending this example</h3><p>While this pattern is sophisticated enough to handle many teams’ ELT workflows, there are many ways we could extend it:</p><ul><li>parameterize my_elt_flow to accept an arbitrary number of Airbyte Connections or dbt commands to run</li><li>create flow runs from this deployment via API calls when you know Airbyte has new data to sync instead of on a schedule</li><li>use the Airbyte sync row count in AirbyteSyncResult to decide how many threads dbt should run!</li><li>do basically anything you want in Python at any point.</li></ul><h3>Next steps</h3><p>If you haven’t already, you can <a href="https://app.prefect.cloud/">create a Prefect Cloud account</a> for free to serve as your launch pad into a world of more effective data engineering!</p><p>Questions? Feel free to ask them in <a href="https://www.prefect.io/slack/">our Slack Community</a>!</p><p>Happy Engineering!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ecd7547e6c1a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/the-prefect-blog/coordinate-elt-in-2023-with-airbyte-dbt-and-prefect-ecd7547e6c1a">Coordinate ELT in 2023 with Airbyte, dbt and Prefect</a> was originally published in <a href="https://medium.com/the-prefect-blog">The Prefect Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Orchestrating Airbyte with Prefect 2.0]]></title>
            <link>https://medium.com/the-prefect-blog/orchestrating-airbyte-with-prefect-2-0-35501997a974?source=rss-e8be470b144a------2</link>
            <guid isPermaLink="false">https://medium.com/p/35501997a974</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[orchestration]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Nate Nowack]]></dc:creator>
            <pubDate>Mon, 22 Aug 2022 13:56:26 GMT</pubDate>
            <atom:updated>2022-12-31T03:53:51.511Z</atom:updated>
            <content:encoded><![CDATA[<h4>Introducing the prefect-airbyte collection</h4><figure><img alt="A blue duck on a roller coaster" src="https://cdn-images-1.medium.com/max/1024/1*SR27bhZuP298PECIcOJibA.png" /></figure><p><a href="https://airbyte.com/">Airbyte</a> is an open-source data integration tool that’s pretty great when you want to move batch data from A to B. You can define connections between <a href="https://docs.airbyte.com/">source</a>s<a href="https://docs.airbyte.com/"> and </a><a href="https://docs.airbyte.com/">destination</a>s and then trigger a sync of those connections — moving data from each source to each destination in streams.</p><p>In Prefect 1.0, we had <a href="https://docs.prefect.io/api/latest/tasks/airbyte.html">Airbyte Tasks</a> to perform a trigger of a sync and export of all the defined connections as a gzip file.</p><p>Here we’ll introduce prefect-airbyte, the s<a href="https://pypi.org/project/prefect-airbyte/">tandalone PyPI package</a> and the Prefect 2.0 analogue — let’s look at what we can do with this collection!</p><p><strong>Want to try out Airbyte locally?</strong></p><p>If you’ve got Docker, this should spin up Airbyte for you on localhost:8000</p><pre>git clone https://github.com/airbytehq/airbyte.git<br>cd airbyte<br>docker-compose up</pre><h4>Triggering an Airbyte connection sync</h4><p>Go ahead and grab the connection ID from your Airbyte UI:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*n4BnyLNzmWtWc8R-_fZeww.png" /><figcaption>The connector ID can be found in the URL when navigated to your Connection in the UI</figcaption></figure><p>Here’s an example using blocks in prefect-airbyte &gt;= 0.2.0</p><ul><li>AirbyteServer block representing a running Airbyte instance at <a href="http://localhost:8000/">http://localhost:8000</a></li><li>AirbyteConnection representing an active connection defined on your Airbyte instance</li></ul><pre>from prefect import flow<br>from prefect_airbyte.server import AirbyteServer<br>from prefect_airbyte.connections import AirbyteConnection<br>from prefect_airbyte.flows import run_connection_sync<br><br>server = AirbyteServer(server_host=&quot;localhost&quot;, server_port=8000)<br><br>connection = AirbyteConnection(<br>    airbyte_server=server,<br>    connection_id=&quot;e1b2078f-882a-4f50-9942-cfe34b2d825b&quot;,<br>    status_updates=True,<br>)<br><br>@flow<br>def airbyte_syncs():<br>    # do some setup<br><br>    sync_result = run_connection_sync(<br>        airbyte_connection=connection,<br>    )<br><br>    # do some other things, like trigger DBT based on number of records synced<br>    print(f&#39;Number of Records Synced: {sync_result.records_synced}&#39;)</pre><p>Here’s another example using trigger_sync(still available in 0.2.0):</p><pre>from prefect import flow, task<br>from prefect_airbyte.connections import trigger_sync<br><br>@flow(name=&quot;Airbyte Sync&quot;)<br>def example_trigger_sync_flow(): <br>  connection_id = &quot;e1b2078f-882a-4f50-9942-cfe34b2d825b&quot;<br>    trigger_sync(<br>          connection_id=connection_id,<br>          status_updates=True<br>    )</pre><p>each of which will run your airbyte sync to completion as a prefect flow:</p><pre>❯ python airbyte_syncs.py<br>03:46:03 | prefect.engine - Created flow run &#39;thick-seahorse&#39; for flow &#39;Airbyte Sync&#39;<br>03:46:03 | Flow run &#39;thick-seahorse&#39; - Using task runner &#39;ConcurrentTaskRunner&#39;<br>03:46:03 | Flow run &#39;thick-seahorse&#39; - Created task run &#39;trigger_sync-35f0e9c2-0&#39; for task &#39;trigger_sync&#39;<br>03:46:03 | prefect - trigger airbyte connection: e1b2078f-882a-4f50-9942-cfe34b2d825b, poll interval 3 seconds<br>03:46:03 | prefect - pending<br>03:46:06 | prefect - running<br>03:46:09 | prefect - running<br>03:46:12 | prefect - running<br>03:46:16 | prefect - running<br>03:46:19 | prefect - running<br>03:46:22 | prefect - Job 26 succeeded.<br>03:46:22 | Task run &#39;trigger_sync-35f0e9c2-0&#39; - Finished in state Completed(None)<br>03:46:22 | Flow run &#39;thick-seahorse&#39; - Finished in state Completed(&#39;All states completed.&#39;)</pre><p><strong>Exporting your Airbyte configuration archive</strong></p><blockquote>(<strong>EDIT</strong>) <strong>Note</strong>: As of Airbyte <strong>v0.40.7-alpha</strong>, the API shipped with open-source Airbyte no longer supports this feature — their docs seem to recommend checking out the <a href="https://github.com/airbytehq/airbyte/tree/master/octavia-cli">Ocatavia CLI</a> for connector configuration management.</blockquote><p>Many self-hosted Airbyte users will be running on EC2 or Kubernetes, where instances can sometimes die unexpectedly due to <a href="https://en.wikipedia.org/wiki/Out_of_memory">OOM errors</a> or provider outages. With that situation in mind, it can be useful to backup your Airbyte connectors’ configuration somewhere so that you can pick up where you left off on a new instance.</p><p>We can write our instance’s gzip archive somewhere with the export_configuration task from prefect_airbyte.configuration like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/91fabe614f4b1c4e2984927cb4d63e2f/href">https://medium.com/media/91fabe614f4b1c4e2984927cb4d63e2f/href</a></iframe><h4>💡 Did you know?</h4><blockquote>You can pass any valid URI prefix into somewhere , like s3://**/my_destination.gz , <em>gs://**/my_destination.gz , </em>etc, as long as you’re authenticated with wherever you’re trying to write to!</blockquote><h4>… and that’s a wrap!</h4><p>If you encounter any bugs while using prefect-airbyte, feel free to open an issue in the <a href="https://github.com/PrefectHQ/prefect-airbyte">prefect-airbyte</a> repository.</p><p>If you have any questions or issues, you can find help in either the <a href="https://discourse.prefect.io/">Prefect Discourse forum</a> or the <a href="https://prefect.io/slack">Prefect Slack community</a>.</p><p>Happy Engineering!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=35501997a974" width="1" height="1" alt=""><hr><p><a href="https://medium.com/the-prefect-blog/orchestrating-airbyte-with-prefect-2-0-35501997a974">Orchestrating Airbyte with Prefect 2.0</a> was originally published in <a href="https://medium.com/the-prefect-blog">The Prefect Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A modern ETL pattern using Prefect’s S3 Storage and Kubernetes runtime]]></title>
            <link>https://medium.com/the-prefect-blog/a-modern-etl-pattern-using-prefects-s3-storage-and-kubernetes-runtime-816a4ed2772c?source=rss-e8be470b144a------2</link>
            <guid isPermaLink="false">https://medium.com/p/816a4ed2772c</guid>
            <category><![CDATA[etl]]></category>
            <category><![CDATA[prefect]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Nate Nowack]]></dc:creator>
            <pubDate>Wed, 02 Feb 2022 21:59:35 GMT</pubDate>
            <atom:updated>2022-02-02T21:59:35.146Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/873/1*RmPNIFSTprbl-fnybzQlVQ.png" /><figcaption>Look at these beautiful cakes, would be a shame if someone were to DROP them</figcaption></figure><p>Since I’m fond of cats, the <a href="https://www.netflix.com/watch/80063224?source=35">Great British Baking Show</a>, and dragging out analogies… I’ll say “<em>there are many ways to bake a cake</em>” when it comes to creating modern ETL<em>-</em>like jobs moving data from point A to point B.</p><p>However, for data engineers, the challenge is rarely baking a single cake once— it’s in baking many types of cake on inter-dependent schedules, in a customizable manner that’s easy to <strong>repeat</strong>, <strong>manage</strong>, <strong>scale</strong>, and <strong>understand</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*mVLPC8xn7tJVzuFU.png" /><figcaption>prefect.io</figcaption></figure><p><a href="http://prefect.io/"><strong>Prefect</strong></a> is an orchestration engine that takes your family-favorite cake recipes and makes them easy to <strong>repeat</strong>, <strong>manage</strong>, <strong>scale</strong>, and <strong>understand</strong> — erm… well, I mean, just as long as the cake recipes are actually Python jobs.</p><p>Below I’ll lay out a Prefect ETL pattern with 3 main considerations in mind:</p><ul><li><strong>rapid deployment </strong>— <a href="https://docs.prefect.io/orchestration/execution/storage_options.html#non-docker-storage-for-containerized-environments">addressing Docker storage’s drawbacks at scale</a>, RE: building / pushing a Docker image to a registry whenever code changes</li><li><strong>scalability</strong> — offered by containerized runtime environments like EKS</li><li><strong>modularization</strong> — clear categorization of various Python jobs (<a href="https://docs.prefect.io/core/concepts/flows.html">flows</a>) into Prefect <a href="https://docs.prefect.io/orchestration/concepts/projects.html">Projects</a>, based on intended outcomes and required dependencies</li></ul><h3>What do I need in order to use a pattern like this?</h3><p>This specific example will require a few pieces of supporting infrastructure:</p><ul><li><strong>AWS S3</strong> <strong>Bucket</strong> for storing our python jobs <em>as scripts</em> (<a href="https://docs.aws.amazon.com/AmazonS3/latest/userguide/create-bucket-overview.html">how to provision</a>)</li><li><strong>AWS EKS</strong> <strong>Cluster</strong> for job execution (<a href="https://learn.hashicorp.com/tutorials/terraform/eks">how to provision using Terraform</a>)</li><li><strong>AWS ECR</strong> <strong>Repository</strong> for custom base-image — <em>optional</em> (<a href="https://docs.aws.amazon.com/AmazonECR/latest/userguide/get-set-up-for-amazon-ecr.html">how to set up</a>)</li></ul><p>I’m using AWS for cloud, but this same pattern works for GCP / Azure.</p><h3>Why are we using Prefect again?</h3><p>We’ll use a few of Prefect’s convenient features (* → Prefect Cloud only):</p><ul><li><a href="https://docs.prefect.io/api/latest/storage.html#s3">the S3 Storage object</a> (used to automatically upload flows to an S3 Bucket)</li><li><a href="https://docs.prefect.io/api/latest/run_configs.html#kubernetesrun">the <em>KubernetesRun</em> run configuration</a> (used to specify an image &amp; more)</li><li><a href="https://docs.prefect.io/orchestration/concepts/secrets.html"><em>PrefectSecrets</em></a> (for accessing sensitive values like credentials, API keys, etc)</li><li><a href="https://docs.prefect.io/orchestration/concepts/kv_store.html">the key-value store</a>* (e.g. for stateful flow input <a href="https://docs.prefect.io/core/concepts/parameters.html">Parameters</a>)</li></ul><h3>How does all of this piece together? i.e code plz</h3><p>Most directly, we can build our flows using <a href="https://docs.prefect.io/core/concepts/flows.html#functional-api">Prefect’s Functional API</a>. In less true but generally Pythonic terms, we throw some specially @task-decorated Python functions in a Flow context manager and let Prefect go to work:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5a28ec684603cbc9830a3554bff72936/href">https://medium.com/media/5a28ec684603cbc9830a3554bff72936/href</a></iframe><h3>Wait, when does it run?</h3><p>Here’s what this Flow context looks like as a <a href="https://en.wikipedia.org/wiki/Directed_acyclic_graph">DAG</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/434/1*AhCEHaph_sMEXXbU25dWBw.png" /><figcaption>You can generate this using <a href="https://docs.prefect.io/core/advanced_tutorials/visualization.html">flow.visualize()</a></figcaption></figure><p>By running the Python file as is, Prefect will use our Flow context definition (lines 47–50 from above) to build a DAG (Directed Acyclic Graph, i.e. execution plan). The DAG is inferred by Prefect based on how our task objects are called and how return values are passed between them.</p><p>Let’s break down these 4 lines, since there’s a fair amount happening here:</p><ul><li>We defined a data<strong> sources</strong> <a href="https://docs.prefect.io/core/concepts/parameters.html">Parameter</a> in a JSON-like List[Dict]in terms of what requests.get() would need to extract each source.</li><li>We tell Prefect to dynamically map this extract() task across each source in data<strong> sources, </strong>producing raw_data: List[raw_piece_of_data]</li><li>We map our transform() task over the raw_data from extract() (which is inferred to be an upstream task, i.e. transform iff extract succeeds)</li><li>The terminal task will load() our squeaky clean_data to our destination of choice — storing any relevant flow results in a bucket.</li></ul><p>Now that our flow is built, we can use flow.run() outside the flow context to trigger flow runs locally:</p><pre>if __name__ == &quot;__main__&quot;:<br>     flow.run()</pre><p>… or we can also <a href="https://docs.prefect.io/api/latest/cli/run.html">use the CLI</a> or the Prefect Cloud UI to trigger flow runs.</p><h3>Where?</h3><p>At this point, we can create a Prefect Cloud project and assign this flow to it:</p><pre>prefect register -p flow.py --project DEMO</pre><p>This registration will upload our flow as a script to the bucket using the {PROJECT}/flow.py key defined in the S3_storage object — no need to build and push a whole new Docker image to deploy iterative changes or fixes.</p><p>The flow will now be loaded from S3 and executed on the EKS cluster as prompted by our <a href="https://docs.prefect.io/orchestration/agents/kubernetes.html">Kubernetes Agent</a>, which can be based on a <a href="https://docs.prefect.io/core/concepts/schedules.html">Schedule</a>, manual triggers from the UI, or event-driven Client / GraphQL <a href="https://docs.prefect.io/core/idioms/flow-to-flow.html">API calls</a>.</p><p>To handle dependencies, I built a custom image on top of the <a href="http://docker pull prefecthq/prefect:0.15.9-python3.9">Prefect Python 3.9 base image</a> including hypothetically common Python libraries to this (demo) Prefect project. I pushed that project’s image to an AWS ECR repo, stored the image URL in Prefect KV store as {PROJECT}_BASE_IMAGE , and provided it to KubernetesRun() as the image to run on a pod at runtime.</p><h3>A couple thoughts on scalability</h3><p>These data <strong>sources</strong> could easily be defined elsewhere as a JSON-like object, loaded at runtime, and passed to flow runs as <a href="https://docs.prefect.io/core/concepts/parameters.html">Parameter</a>s. A good example of ‘elsewhere’ would be the Prefect KV store, where you can store and update stateful values for use in Flow runs, like <a href="https://docs.prefect.io/orchestration/concepts/kv_store.html#using-key-value-pairs-in-flows">this example</a>.</p><p>In order to further parallelize our mapped tasks in production, we could use Dask Kubernetes’ KubeCluster, as outlined in <a href="https://medium.com/slateco-blog/prefect-x-kubernetes-x-ephemeral-dask-power-without-responsibility-6e10b4f2fe40">a great article</a> by <a href="https://medium.com/u/50e1841103ae">George Coyne</a>.</p><p>This pattern was designed to be extensible across projects, teams and organizations. I chose to store my flows in sub-divisions of my S3 bucket according to Prefect Project , but flows can be stored in buckets in whatever logical grouping of S3 blobs is most convenient for the project at hand.</p><p>—</p><p>Any questions on patterns for python ETLs leveraging Prefect workflow orchestration? Don’t hesitate in reaching out to us in <a href="https://prefect.io/slack">our Slack Community</a>.</p><p>Happy engineering!</p><p>Nathan Nowack — nate@prefect.io</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=816a4ed2772c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/the-prefect-blog/a-modern-etl-pattern-using-prefects-s3-storage-and-kubernetes-runtime-816a4ed2772c">A modern ETL pattern using Prefect’s S3 Storage and Kubernetes runtime</a> was originally published in <a href="https://medium.com/the-prefect-blog">The Prefect Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Commoditizing data integration alongside Airbyte]]></title>
            <link>https://medium.com/slateco-blog/commoditizing-data-integration-alongside-airbyte-340317b4b4d1?source=rss-e8be470b144a------2</link>
            <guid isPermaLink="false">https://medium.com/p/340317b4b4d1</guid>
            <category><![CDATA[data-integration]]></category>
            <category><![CDATA[etl]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[open-source]]></category>
            <dc:creator><![CDATA[Nate Nowack]]></dc:creator>
            <pubDate>Tue, 18 May 2021 16:55:32 GMT</pubDate>
            <atom:updated>2021-05-18T16:55:32.644Z</atom:updated>
            <content:encoded><![CDATA[<p>Nathan Nowack — SLATE Data + Design</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MNp6_enNf3mkIldJdCvMiA.png" /></figure><p>After a few months of working with Airbyte and contributing to their connector ecosystem I wanted to give a little background on the value that Airbyte brings to organizations like us and to demonstrate how to extend its functionality to suit your use-case and to benefit the Airbyte community.</p><h3><strong>What is Airbyte?</strong></h3><p><a href="https://airbyte.io/">Airbyte</a> soft-launched in late September 2020 as an open-source data integration platform. It&#39;s backed by a friendly group of engineers, enthusiastic community contributions, and some <a href="https://airbyte.io/articles/our-story/we-raised-a-5m-seed-round-with-accel-to-commoditize-data-integration/">impressive seed funding</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HFTdaEej4mUdg3Ftsx4NAw.png" /></figure><p>Their core product, a self-hosted EL(T) engine, effectively addresses the established need for an enterprise-scale solution to moving enterprise-scale amounts of data. It does this by abstracting the EL(T) process, using connector instances between sources, destinations and the all-orchestrating Airbyte instance.</p><h3>Airbyte&#39;s Unique Usefulness</h3><p>If Airbyte wasn’t around to do my job, I&#39;d probably be writing and repurposing all of our clients’ ETLs in <a href="https://www.prefect.io/">Prefect</a>-orchestrated Python or advise clients’ that to pay a subscription to a low-code platform that likely exceeds their use-case.</p><p>Thankfully, Airbyte does exist. At SLATE, we leverage their core EL(T) tool in dozens of use-cases to automatically move data at a near-arbitrary scale, while our clients pay for nothing but the virtual machines that run Airbyte.</p><p>So, to summarize how we find Airbyte useful:</p><ul><li>Airbyte allows us to avoid the potentially unsavory hours involved in re-purposing Python code to build one-off ET(L)s that are fast, schedule-able, and reliable</li><li>Unlike many ET(L) services with subscription or usage-based pricing, Airbyte&#39;s open-source ET(L) engine is freely customizable and easily integrated into your own application or simple use-case.</li></ul><p>Chances are that if you need for large-scale data integration, you&#39;re probably using a platform to store data that Airbyte handles out-of-the-box. <a href="https://docs.airbyte.io/integrations">Here</a> you can find the full list of currently supported source and destination connectors.</p><h4>I&#39;ll admit, that&#39;s cool... but I don&#39;t see the connector I want!</h4><p>..but, if you&#39;re like I was, then you need something beyond the standard integrations.</p><p>Now I&#39;ll outline how to add a connector to Airbyte, based on my experience implementing <a href="https://github.com/airbytehq/airbyte/pull/2880">Airbyte&#39;s Smartsheets source connector</a>.</p><p>If at any point you need help, feel free to reach out to the good folks in the <a href="https://airbyte.io/community/">Airbyte slack community</a> or <a href="/service/mailto:nate.nowack@slateco.io">me</a>.</p><h3>Develop a Connector (Python source)</h3><p>In Airbyte-speak, source and destination connectors are different things - they require a different set of steps to implement. However, I found that learning to interact with Airbyte&#39;s deployment and OOP framework took the most time, which is common to both development workflows.</p><p>So, hopefully you can still benefit from this article if you&#39;re here looking to make a new destination connector.</p><h3>Setup</h3><h4>Gathering your Source&#39;s required Credentials</h4><p>You first need to figure out what&#39;s required to programmatically connect and retrieve information from your desired data source. In my case, I needed:</p><ul><li>an API access Key that I obtained from the source platform</li><li>the ID of the spreadsheet that I was going to replicate</li></ul><p>What&#39;s required here is entirely dependent on how you choose to connect to your source, but keep in mind that ideally it shouldn&#39;t be too difficult for an end-user to configure an instance of your connector.</p><h4>Development Environment</h4><p>See <a href="https://docs.airbyte.io/contributing-to-airbyte/developing-locally#troubleshooting">here</a> for the setup I outline below, but only from the horse&#39;s mouth.</p><p><strong>Install</strong> (if you don&#39;t have them already):</p><ul><li><a href="https://www.java.com/en/">Java (version 14 or above)</a></li><li><a href="https://nodejs.org/en/download/">npm (version 14 or above)</a></li><li>Python 3.7 or above</li><li>Docker</li></ul><p>Next: <strong>fork</strong> Airbyte&#39;s master branch, <strong>clone</strong> the fork onto your machine, and from the root directory, <strong>run the following</strong> to get started:</p><pre>$ cd airbyte-integrations/connector-templates/generator<br><br>$ npm install<br><br>$ npm run generate</pre><p>You&#39;ll be prompted to select a bootstrapped source-template (Python, in my case) and then name your connector. Finally, to build your template and prepare a virtual environment with the required dependencies, run the folliwing:</p><pre>$ cd airbyte-integrations/connectors/source-&lt;name&gt;<br><br>$ python -m venv .venv <br><br>$ source .venv/bin/activate<br><br>$ pip install -r requirements.txt</pre><p>The code scaffolding that you will need to edit now exists in a directory called source-&lt;yourconnectorname&gt; within airbyte/airbyte-integrations/connectors/.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/1*dfsm5Qm_HEeSHcUiOzYHrw.gif" /><figcaption><em>IT’S TIME TO IMPLEMENT!</em></figcaption></figure><h3><strong>Implementation</strong></h3><p>While certain aspects of your connector&#39;s implementation are going to be platform specific, all Airbyte source connectors are organized as a class with the 3 methods outlined below:</p><pre><em># main class definition for your source</em><br>class SourceSmartsheets(Source):<br><br>    <em># &quot;check&quot; that your credentials give a good connection</em><br>    def check(self, logger, config) -&gt; AirbyteConnectionStatus:<br>        # TODO<br><br>    <em># create the JSONschema catalog from your source&#39;s columns</em><br>    def discover(self, logger, config) -&gt; AirbyteCatalog:<br>        # TODO<br><br>    <em># create an AirbyteMessage for each record from your catalog</em><br>    def read(self, logger, config, catalog, state) -&gt; Generator:<br>        <em># TODO</em></pre><p>It&#39;s likely going to be helpful to check out <a href="https://docs.airbyte.io/tutorials/building-a-python-source">these docs</a> during these implementations, but I&#39;ll describe where I got stuck during the process and how I got around it.</p><h3>spec.json</h3><p>The first file to edit in our newly-generated directory is the JSON Schema configuration. This where you specify what credentials an end-user would need to use your connector in Airbyte. In the case of my source connector, I&#39;m using the Smartsheets API under the hood so users of my connector need to provide both <strong>an API token</strong> and <strong>their Smartsheet&#39;s ID</strong>.</p><pre>{<br>  &quot;documentationUrl&quot;: &quot;https://docs.airbyte.io&quot;,<br>  &quot;connectionSpecification&quot;: {<br>    &quot;$schema&quot;: &quot;http://json-schema.org/draft-07/schema#&quot;,<br>    &quot;title&quot;: &quot;Smartsheets Source Spec&quot;,<br>    &quot;type&quot;: &quot;object&quot;,<br>    &quot;required&quot;: [&quot;access_token&quot;, &quot;spreadsheet_id&quot;],<br>    &quot;additionalProperties&quot;: false,<br>    &quot;properties&quot;: {<br>      &quot;access_token&quot;: {<br>        &quot;title&quot;: &quot;API Access token&quot;,<br>        &quot;description&quot;: &quot;Found in Profile &gt; Apps &amp; Integrations &gt; API Access within Smartsheet app&quot;,<br>        &quot;type&quot;: &quot;string&quot;,<br>        &quot;airbyte_secret&quot;: true<br>      },<br>      &quot;spreadsheet_id&quot;: {<br>        &quot;title&quot;: &quot;Smartsheet ID&quot;,<br>        &quot;description&quot;: &quot;Found in File &gt; Properties&quot;,<br>        &quot;type&quot;: &quot;string&quot;<br>      }<br>    }<br>  }<br>}</pre><p><strong>NOTE</strong> that the elements of the required list are the pieces of information an end-user of your connector would need during configuration, so if you&#39;re not using a token / ID combo like me then make sure to name these appropriately (e.g. username, password, hostname, port, etc).</p><p>The description field for each required will pop up as alongside its corresponding field in Airbyte during setup, so give hints for acquiring this info or direct users to what will be your documentation.</p><p>There! You&#39;ve implemented the first part of your new Airbyte connector.</p><h3>check</h3><p>Like the rest below, this function is one of those scaffolded methods of your connector&#39;s class, just waiting for you to implement it. The check method serves to take user credentials outlined in spec.json and somehow check with the data source that they are valid.</p><p>If you&#39;re using an API to connect to your source, like Smartsheets or Google Sheets, the request response code can typically be found in the object returned by the client&#39;s connect method. Try:</p><pre>print(yourClient.yourConnectionObject.__dict__)</pre><p>... and see if you have a response code in there somewhere! Unless you successfully connect to the wrong source, chances are that a 200 status code in your response is what you need.</p><p>If you&#39;re not using an API, you&#39;ll have to decide what constitutes a successful connection to your source.</p><p>As outlined below, that&#39;s conceptually it for this step. To follow <a href="https://docs.airbyte.io/contributing-to-airbyte/building-new-connector/best-practices#principles-of-developing-connectors">Airbyte best practices</a> and wrap up this step: add exception handling, actionable error messages for the AirbyteLogger, and return an AirbyteConnectionStatus object with status set to either Status.SUCCEEDED or Status.FAILED, depending on your definition of a &quot;successful connection&quot;. Here&#39;s what my check method looks like:</p><pre>def check(self, logger: AirbyteLogger, config: json) -&gt; AirbyteConnectionStatus:<br>    try:<br>        access_token = config[&quot;access_token&quot;]<br>        spreadsheet_id = config[&quot;spreadsheet_id&quot;] + 1<br><br>        smartsheet_client = smartsheet.Smartsheet(access_token)<br>        smartsheet_client.errors_as_exceptions(True)<br>        smartsheet_client.Sheets.get_sheet(spreadsheet_id)<br><br>        return AirbyteConnectionStatus(status=Status.SUCCEEDED)<br>    except Exception as e:<br>        if isinstance(e, smartsheet.exceptions.ApiError):<br>            err = e.error.result<br>            code = 404 if err.code == 1006 else err.code<br>            reason = f&quot;{err.name}: {code} - {err.message} | Check your spreadsheet ID.&quot;<br>        else:<br>            reason = str(e)<br>        logger.error(reason)<br>    return AirbyteConnectionStatus(status=Status.FAILED)</pre><p><strong>NOTE</strong> that one of the inputs of check is config, which is basically a JSON object that you can assume contains the user&#39;s credentials as outlined in spec.json.</p><h3>discover</h3><p>Next, we need to specify to Airbyte how to find column names and types from our source. To do this, we create an AirbyteCatalog that lists the distinct streams of data to be replicated by the connector.</p><p>For context:</p><ul><li>The MSSQL connector has a stream for each selected table in the relevant database schema</li><li>The Google Sheets connector has a stream for each individual sheet within the spreadsheet</li><li>In my case, Smartsheets can only have one sheet, so I only had one stream</li></ul><p>To make a stream, you need a name and a json_schema. The name can be anything but <strong>make it descriptive. T</strong>he json_schema reflects the types of the data source&#39;s columns.</p><p>Here&#39;s a sample of what a properly configured json_schema looks like:</p><pre>{<br>    &quot;$schema&quot;: &quot;http://json-schema.org/draft-07/schema#&quot;,<br>    &quot;type&quot;: &quot;object&quot;,<br>    &quot;properties&quot;: {<br>        &quot;column1&quot;: {<br>            &quot;type&quot;: &quot;string&quot;<br>        },<br>        &quot;column2&quot;: {<br>            &quot;type&quot;: &quot;number&quot;<br>        }<br>    }<br>}</pre><p><strong>NOTE</strong> that the &quot;$schema&quot; key references a <a href="https://json-schema.org/">JSON Schema</a> template adopted by Airbyte, so just copy that and the &quot;type&quot;: &quot;object&quot; key / value pair. All you need to do is figure out how to populate the properties object with type information for each column in a given stream.</p><p>I used a couple helper functions to do this (shown below):</p><ul><li>get_prop for conforming Smartsheet typing convention to <a href="https://docs.airbyte.io/architecture/basic-normalization#typing">Airbyte types</a></li><li>get_json_schema for generating the json_schema in the required format given the type-mapping.</li></ul><pre><em># Mapping Smartsheet types to acceptable Airbyte types</em><br>def get_prop(col_type: str) -&gt; Dict[str, any]:<br>    props = {<br>        &quot;TEXT_NUMBER&quot;: {&quot;type&quot;: &quot;string&quot;},<br>        &quot;DATE&quot;: {&quot;type&quot;: &quot;string&quot;, &quot;format&quot;: &quot;date&quot;},<br>        &quot;DATETIME&quot;: {&quot;type&quot;: &quot;string&quot;, &quot;format&quot;: &quot;date-time&quot;},<br>    }<br>    if col_type in props.keys():<br>        return props[col_type]<br>    else:  <em># assume string</em><br>        return props[&quot;TEXT_NUMBER&quot;]<br><br><em># Create dictionary representing the JSON schema for this sheet</em><br>def get_json_schema(sheet: Smartsheet) -&gt; Dict:<br>    column_info = {i[&quot;title&quot;]: get_prop(i[&quot;type&quot;]) for i in sheet[&quot;columns&quot;]}<br>    json_schema = {<br>        &quot;$schema&quot;: &quot;http://json-schema.org/draft-07/schema#&quot;,<br>        &quot;type&quot;: &quot;object&quot;,<br>        &quot;properties&quot;: column_info,<br>    }<br>    return json_schema</pre><p>With the JSON Schema created, you can then instantiate an AirbyteStream for each distinct source of data within your configured source - if, unlike me, you have more than one per data source!</p><p>Again, fail early and actionably to help out the end-user trying to use your connector. Now just return your AirbyteCatalog as shown below, and let Airbyte do the discovery from now on.</p><p><strong>NOTE</strong> You&#39;re going to want to test this step by printing the json_schema to somewhere you can check that it looks right. You can also use a pydantic <a href="https://pydantic-docs.helpmanual.io/usage/dataclasses/">dataclass</a> if you want to specify upfront what it should look like!</p><p>Here&#39;s the full discover method I ended up with:</p><pre>def discover(self, logger: AirbyteLogger, config: json) -&gt; AirbyteCatalog:<br>    access_token = config[&quot;access_token&quot;]<br>    spreadsheet_id = config[&quot;spreadsheet_id&quot;]<br>    streams = []<br><br>    smartsheet_client = smartsheet.Smartsheet(access_token)<br>    try:<br>        sheet = smartsheet_client.Sheets.get_sheet(spreadsheet_id)<br>        sheet = json.loads(str(sheet))  <em># make it subscriptable</em><br>        sheet_json_schema = get_json_schema(sheet)<br><br>        logger.info(f&quot;Discovering:{sheet[&#39;name&#39;]}:{spreadsheet_id}&quot;)<br><br>        try:<br>            stream = AirbyteStream(<br>                name=sheet[&quot;name&quot;], <br>                json_schema=sheet_json_schema)<br>            streams.append(stream)<br>        except Exception as e:<br>            rec = &quot;Source column names must not contain spaces or _&quot;<br>            logger.error(f&quot;Stream creation failed: {str(e)}\n{rec}&quot;)<br>    except Exception as e:<br>        raise Exception(f&quot;Could not run discovery: {str(e)}&quot;)<br><br>    return AirbyteCatalog(streams=streams)</pre><p>Don&#39;t worry just yet about testing your methods in series, each method can be developed and tested in isolation using the provided main_dev.py or running the Docker-based test of that method. Both methods of iterative testing are outlined <a href="https://docs.airbyte.io/tutorials/building-a-python-source#iterating-on-your-implementation">here</a>.</p><h3>read</h3><p>Here we are - this is the last method of our class SourceYourFancySourceName that we have to write!</p><p>Essentially, the logic here is:</p><p>for each stream:</p><p>— for each row in a stream:</p><p>— —define dict elements like: {column name : column&#39;s value in row}</p><p>— — pass this dict with it’s elements to Airbyte as an AirbyteRecord</p><p>For example, say that your source table has 5 records with columns names date, stock_ticker, and price. Then, the below is the data that you would need to pass to Airbyte.</p><pre>{&quot;date&quot;: &quot;2020-12-15&quot;, &quot;stock_ticker&quot;: &quot;TSLA&quot;, &quot;price&quot;: 633.25}<br>{&quot;date&quot;: &quot;2020-12-16&quot;, &quot;stock_ticker&quot;: &quot;TSLA&quot;, &quot;price&quot;: 622.77}<br>{&quot;date&quot;: &quot;2020-12-17&quot;, &quot;stock_ticker&quot;: &quot;TSLA&quot;, &quot;price&quot;: 655.9}<br>{&quot;date&quot;: &quot;2020-12-18&quot;, &quot;stock_ticker&quot;: &quot;TSLA&quot;, &quot;price&quot;: 695}<br>{&quot;date&quot;: &quot;2020-12-21&quot;, &quot;stock_ticker&quot;: &quot;TSLA&quot;, &quot;price&quot;: 649.86}</pre><p>How this actually happens in the Airbyte OOP framework is slightly weird, but not too difficult at all.</p><ul><li>For each row of data, create an AirbyteRecordMessage with your stream name, data, and current time.</li><li>Instantiate an AirbyteRecord by passing in your AirbyteRecordMessage along with it&#39;s type as Type.RECORD</li></ul><p>Below is my implementation.</p><pre>def read(<br>    self,<br>    logger: AirbyteLogger,<br>    config: json,<br>    catalog: ConfiguredAirbyteCatalog,<br>    state: Dict[str, any]<br>) -&gt; Generator[AirbyteMessage, None, None]:<br><br>    access_token = config[&quot;access_token&quot;]<br>    spreadsheet_id = config[&quot;spreadsheet_id&quot;]<br>    smartsheet_client = smartsheet.Smartsheet(access_token)<br><br>    for configured_stream in catalog.streams:<br>        stream = configured_stream.stream<br>        properties = stream.json_schema[&quot;properties&quot;]<br>        if isinstance(properties, list):<br>            columns = tuple(key for dct in properties for key in dct.keys())<br>        elif isinstance(properties, dict):<br>            columns = tuple(i for i in properties.keys())<br>        else:<br>            logger.error(&quot;Could not read properties from the JSONschema in this stream&quot;)<br>        name = stream.name<br><br>        try:<br>            sheet = smartsheet_client.Sheets.get_sheet(spreadsheet_id)<br>            sheet = json.loads(str(sheet))  <em># make it subscriptable</em><br>            logger.info(f&quot;Starting syncing spreadsheet {sheet[&#39;name&#39;]}&quot;)<br>            logger.info(f&quot;Row count: {sheet[&#39;totalRowCount&#39;]}&quot;)<br><br>            for row in sheet[&quot;rows&quot;]:<br>                values = tuple(i[&quot;value&quot;] for i in row[&quot;cells&quot;])<br>                try:<br>                    data = dict(zip(columns, values))<br><br>                    yield AirbyteMessage(<br>                        type=Type.RECORD,<br>                        record=AirbyteRecordMessage(stream=name, data=data, emitted_at=int(datetime.now().timestamp()) * 1000),<br>                    )<br>                except TypeError as e:<br>                    logger.error(f&quot;{e}: BOGUS!&quot;)<br><br>        except Exception as e:<br>            logger.error(f&quot;Could not read smartsheet: {name}&quot;)<br>            raise e<br>    logger.info(f&quot;Finished syncing spreadsheet with ID: {spreadsheet_id}&quot;)</pre><p><strong>NOTE</strong> You might notice that I have a weird few if else blocks in there - that is just to catch TypeError inconsistencies between my dev environment and Airbyte&#39;s Docker-based environment. Suffice to say, try to use the Docker-based environment for iterative test and ... <a href="https://www.google.com/url?sa=i&amp;url=https%3A%2F%2Fwww.designboom.com%2Fart%2Fthumbs-ammo-replaces-movie-guns-with-a-thumbs-up%2F&amp;psig=AOvVaw3fwWO0qgFkWRoH51ymRGaQ&amp;ust=1619741469993000&amp;source=images&amp;cd=vfe&amp;ved=0CAIQjRxqFwoTCLCvtq2VovACFQAAAAAdAAAAABAD">don&#39;t use a mutable type as a dictionary key</a>.</p><p>You got through all the implementation steps, it is time to test your connector!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*6fzIOM-UXo8MwlcILZJPTw.jpeg" /></figure><h3>Testing</h3><p>To check that your connector passes Airbyte&#39;s baseline Integration Tests, navigate back out to the root directory and run the following:</p><pre>$ ./gradlew clean:airbyte-integrations:connectors:source-&lt;name&gt;:build<br><br>$ ./gradlew clean :airbyte-integrations:connectors:source-&lt;name&gt;:integrationTest</pre><p>This is going to package your connector into a Docker image and check your code&#39;s basic functionality and compliance with the Airbyte framework. It&#39;ll likely take a couple minutes to build your code and run all the tests.</p><p>If it errors out, use the prompts to run it again with a stack-trace to debug (and/or use the suggested flag to generate a helpful gradle build summary in your browser).</p><h3>Unit Tests</h3><p>To make your connector extra robust, you can add unit tests to check your methods&#39; functionalities. See the <a href="https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-google-sheets/unit_tests/test_helpers.py">Google Sheets connector&#39;s repo</a> for a good reference.</p><h3>Test your connector in the UI</h3><p>Before you open that PR and pop that bubbly, you might want to actually try your new connector in the Airbyte UI to make sure everything works as expected.</p><p><a href="https://docs.airbyte.io/tutorials/building-a-python-source#step-11-add-the-connector-to-the-api-ui">Here</a> is a description of how to do this that I could only make worse.</p><h3>Write Documentation</h3><p>Use another connector&#39;s documentation as a template to write your own connector&#39;s docs, focusing on the unique behaviors.</p><p>Below is an where documentation files live:</p><pre>airbyte/docs/integrations/sources/&lt;source-name&gt;.md</pre><h3>Open a merge request</h3><p>Now, you just have to open a merge request: your fork &gt; Airbyte&#39;s master branch. Coordinate on this with Airbyte&#39;s friendly engineers - they will help you through the rest of the process.</p><p>You have now helped Airbyte improve open-source data integration - for yourself, the world, or both!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*0sjNeSYNk080qw4h7xRaDQ.jpeg" /><figcaption><em>VICTORY SCREECH!!!</em></figcaption></figure><p>For more info on how to use out-of-the-box Airbyte, explore <a href="https://docs.airbyte.io/">here</a>.</p><p>originally published at <a href="https://github.com/zzstoatzz/blogs">https://github.com/zzstoatzz/blogs</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=340317b4b4d1" width="1" height="1" alt=""><hr><p><a href="https://medium.com/slateco-blog/commoditizing-data-integration-alongside-airbyte-340317b4b4d1">Commoditizing data integration alongside Airbyte</a> was originally published in <a href="https://medium.com/slateco-blog">SlateCo</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>