<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Gill on Medium]]></title>
        <description><![CDATA[Stories by Gill on Medium]]></description>
        <link>https://medium.com/@gillsantos?source=rss-437749168948------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*x3Y-QDiELJgnQDQCBa-MNw.png</url>
            <title>Stories by Gill on Medium</title>
            <link>https://medium.com/@gillsantos?source=rss-437749168948------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 10:54:11 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@gillsantos/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[DuckDB+DBT Observability]]></title>
            <link>https://medium.com/@gillsantos/duckdb-dbt-observability-ef6557318d09?source=rss-437749168948------2</link>
            <guid isPermaLink="false">https://medium.com/p/ef6557318d09</guid>
            <category><![CDATA[duckdb]]></category>
            <category><![CDATA[observability]]></category>
            <category><![CDATA[dbt]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[elementary]]></category>
            <dc:creator><![CDATA[Gill]]></dc:creator>
            <pubDate>Fri, 28 Feb 2025 18:58:44 GMT</pubDate>
            <atom:updated>2025-02-28T18:58:44.527Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fAK1Rc2Rjo25n_TV6ejexg.png" /></figure><p>Hello everyone! I’m back to share my love for the DBT + DuckDB duo and the amazing observability package I recently discovered, Elementary. I had a local project already using DBT + DuckDB, but it lacked observability. That’s where Elementary comes in; I successfully used it in a project at work (Thanks, Blue Orange! I’m always learning from you).</p><p>There was just one problem: the Elementary package doesn’t officially support DuckDB yet. So, I got a bit inspired and decided to make these two work together. What I’m presenting to you isn’t official, but it works! You might ask if that was thoroughly tested. No, it wasn’t, this is a quick and dirty implementation. But I’m excited enough to ask for opinions and share my findings. I’ve submitted my snippet in a GitHub issue (STALE for almost a year now), and we’ll see if I can grab their attention. Who knows, maybe I’ll make more time and contribute to the official solution.</p><p>So, that’s all folks! I’d appreciate a little support from those who love DBT, DuckDB, and observability to give a nudge on the GitHub issue <a href="https://github.com/elementary-data/elementary/issues/1124#issuecomment-2671761877">here</a>. Let’s get their attention! In the meantime, check out my quick-and-dirty duckdb_elemetary macro. Just place it under your macros subfolder ;)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/686/1*ollVRcWav6su69o8XUoAbg.gif" /></figure><h3>There you have it!</h3><p><em>./macros/duckdb_elementary.sql</em></p><pre>{# /elementary/macros/utils/cross_db_utils/timeadd.sql #}<br>{# FIX: A Snowflake macro was set as default timestamp behavior #}<br>{% macro snowflake__edr_timeadd(date_part, number, timestamp_expression) %}<br>    dateadd({{ date_part }}, {{ elementary.edr_cast_as_int(number) }}, {{ elementary.edr_cast_as_timestamp(timestamp_expression) }})<br>{% endmacro %}<br><br>{# FIX: Postgres as default timestamp behavior #}<br>{% macro default__edr_timeadd(date_part, number, timestamp_expression) %}<br>    {{ elementary.edr_cast_as_timestamp(timestamp_expression) }} + {{ elementary.edr_cast_as_int(number) }} * INTERVAL &#39;1 {{ date_part }}&#39;<br>{% endmacro %}<br><br>{% macro duckdb__edr_timeadd(date_part, number, timestamp_expression) %}<br>    {{ elementary.edr_cast_as_timestamp(timestamp_expression) }} + {{ elementary.edr_cast_as_int(number) }} * INTERVAL &#39;1 {{ date_part }}&#39;<br>{% endmacro %}<br><br>{# elementary/macros/utils/table_operations/delete_and_insert.sql #}<br>{# FIX: Removed begin/end transaction behavior #}<br>{% macro duckdb__get_delete_and_insert_queries(relation, insert_relation, delete_relation, delete_column_key) %}<br>    {% set query %}<br>        {% if delete_relation %}<br>            delete from {{ relation }}<br>            where<br>            {{ delete_column_key }} is null<br>            or {{ delete_column_key }} in (select {{ delete_column_key }} from {{ delete_relation }});<br>        {% endif %}<br>        {% if insert_relation %}<br>            insert into {{ relation }} select * from {{ insert_relation }};<br>        {% endif %}<br>    {% endset %}<br>    {% do return([query]) %}<br>{% endmacro %}<br><br><br>{# elementary/macros/utils/cross_db_utils/generate_elementary_profile_args.sql #} <br>{# FIX: Duckdb arguments #}<br>{% macro duckdb__generate_elementary_profile_args(method, elementary_database, elementary_schema) %}<br>  {% do return([<br>    _parameter(&quot;type&quot;, target.type),<br>    _parameter(&quot;path&quot;, target.path),<br>    _parameter(&quot;schema&quot;, elementary_schema),<br>    _parameter(&quot;threads&quot;, target.threads),<br>  ]) %}<br>{% endmacro %}<br><br><br>{# elementary/macros/utils/table_operations/insert_rows.sql #} <br>{# FIX: Duckdb escape single quote #}<br>{%- macro duckdb__escape_special_chars(string_value) -%}<br>    {{- return(string_value | replace(&quot;\\&quot;, &quot;\\\\&quot;) | replace(&quot;&#39;&quot;, &quot;&#39;&#39;&quot;) | replace(&quot;\n&quot;, &quot;\\n&quot;) | replace(&quot;\r&quot;, &quot;\\r&quot;)) -}}<br>{%- endmacro -%}</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ef6557318d09" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[StreamFlake: Real-Time CDC Pipeline with Kafka and Snowflake]]></title>
            <link>https://medium.com/blue-orange-digital/streamflake-real-time-cdc-pipeline-with-kafka-and-snowflake-256f2408be62?source=rss-437749168948------2</link>
            <guid isPermaLink="false">https://medium.com/p/256f2408be62</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[cdc]]></category>
            <category><![CDATA[postgres]]></category>
            <category><![CDATA[snowflake]]></category>
            <dc:creator><![CDATA[Gill]]></dc:creator>
            <pubDate>Tue, 21 May 2024 18:48:40 GMT</pubDate>
            <atom:updated>2025-10-28T18:15:14.324Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Building a Real-Time CDC Pipeline with Kafka and Snowflake (Part 1)</strong></p><p>🚀 In a modern enterprise there are several compelling reasons why you would/should want to replicate changes from one place to another, and even more for doing it constantly, reliably, efficiently and fast. In this article, we will show how we replicated not one, but several databases into snowflake in real-time with a lot of cool features like: automatic table replication, schema evolution and deduplication.</p><p>🚲 I know that mentioning Kafka and Kubernetes in the same sentence could feel a little bit overwhelming at first, and while I agree, the right way to start is to <strong><em>Keep It Simple Stupid</em></strong> 💋. So let’s start our journey small, as if we were learning to ride a bike. The beauty of kubernetes is that we can always turn this Bike into a swarm of Bikes.</p><figure><img alt="High Level Architecture Diagram" src="https://cdn-images-1.medium.com/max/720/1*Al8hLwke-cW_CuoXBYquTA.png" /></figure><p>📚 In this 3 part article we’ll present a real time CDC, Change Data Capture, pipeline with Kafka and Debezium to stream changes from a Postgres database to a Snowflake lake house in Real Time. We’ll start small, so let’s begin with a minimal confluent kafka setup in your local machine as you would ideally start as-in a PoC environment.</p><h3>Part 1 — Install and Deploy Confluent Kafka on Kubernetes</h3><p>💻 Requirements:</p><p>Kubectl + Minikube with 2 cpus and 4G memory</p><p>At the time of this tutorial I was running on a very modest 4 cpus 8Gb memory laptop with Linux Ubuntu 22.04, kubectl v1.29.3 and minikube v1.32.0. You should be fine with earlier versions of linux, kubectl and minikube 🙂</p><h3>Section 1 — Installing kubectl</h3><p>To run kubectl download and install kubectl for linux, or your OS<a href="https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/"> https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/</a></p><p>1. Download kubectl.</p><pre>curl -LO &quot;https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl&quot;</pre><p>2. Install kubectl.</p><pre>sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl</pre><p>3. Check kubectl’s version.</p><pre>kubectl version --client --output=yaml</pre><h3>Section 2 — Installing Minikube</h3><p>To deploy kubernetes locally Install and start minikube, we’ll follow instructions in here: <a href="https://minikube.sigs.k8s.io/docs/start/">https://minikube.sigs.k8s.io/docs/start/</a>.</p><p>1. Download minikube.</p><pre>curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64</pre><p>2. Install minikube.</p><pre>sudo install minikube-linux-amd64 /usr/local/bin/minikube &amp;&amp; rm minikube-linux-amd64</pre><p>3. Check minikube’s version.</p><pre>minikube version</pre><p>4. Start minikube.</p><pre>minikube start<br># or dev/local minimum cpu/memory configuration<br>minikube start --cpus 2 --memory 4096</pre><h3>Section 3 — Deploying Confluent Kafka</h3><p>In order to deploy confluent Kafka on Kubernetes check out the examples here,<a href="https://github.com/confluentinc/confluent-kubernetes-examples"> https://github.com/confluentinc/confluent-kubernetes-examples</a>, but we’ll provide our pre-configured helm chart for this project:</p><p>❄️<a href="https://github.com/gillsantos/streamflake"> https://github.com/gillsantos/streamflake</a></p><pre>git clone https://github.com/gillsantos/streamflake.git</pre><p>📔 <strong>IMPORTANT NOTE:</strong> Some configurations were made to confluent helm charts in order to enable automatic topic creation and schema evolution.</p><p>1. Create a confluent namespace.</p><pre>kubectl create namespace confluent</pre><p>2. Set confluent as your current context.</p><pre>kubectl config get-contexts<br>kubectl config set-context --current --namespace confluent</pre><p>3. Set up the Helm Chart.</p><pre>helm repo add confluentinc https://packages.confluent.io/helm</pre><p>4. Install Confluent Operator For Kubernetes using Helm.</p><pre>helm upgrade --install confluent-operator confluentinc/confluent-for-kubernetes --namespace confluent</pre><p>5. Deploy Confluent Platform with the provided configuration.</p><pre>kubectl apply -f ./helm/confluent-platform.yaml<br># or dev/local minimal configuration<br>kubectl apply -f ./helm/confluent-platform-dev.yaml</pre><p>6. Check that the Confluent For Kubernetes pod comes up and is running.</p><pre>kubectl get pods</pre><p>7. Check that all Confluent Platform resources are deployed.</p><pre>kubectl get confluent</pre><p>8. Finally access control center on your web browser.</p><pre># get kubernetes ip<br>minikube ip<br># get exposed port<br>kubectl describe service -n confluent | grep -i nodeport</pre><p>And access control center on <strong>ip:port</strong>.<br>Or forward desired pod port to your local machine, here are the main ones.</p><pre>#kubectl port-forward &lt;pod-name&gt; &lt;localhost-port&gt;:&lt;pod-port&gt;<br>kubectl port-forward schemaregistry-0 8081:8081<br>kubectl port-forward controlcenter-0 9021:9021<br>kubectl port-forward connect-0 8083:8083</pre><p>Voilá! You should see a screen like this one.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YNGQxRYPawa-nL75UFezQg.png" /></figure><h3>🐞 TROUBLESHOOTING makes perfect:</h3><p>When troubleshooting I personally like the macro-to-micro approach, and believe me, while working on this post I used these A LOT.</p><pre># 1. macro (all pods)<br>kubectl get po -n confluent<br># 2. pod specific<br>kubectl describe pod connect-0<br># 3. log error messages<br>kubectl logs connect-0 | grep -i error</pre><h3>♻️ WHEN DONE:</h3><p>In case you need to remove everything we’ve done, or want to do it all over again here is how to clean up everything.</p><p>In case you want to stick around, I recommend you just Stop Minikube and wait until our next episode (Just step 4).</p><p>1. Delete kubernetes resources.</p><pre>kubectl delete -f ./helm/confluent-platform.yaml<br># or<br>kubectl delete -f ./helm/confluent-platform-dev.yaml</pre><p>2. Check if ports are being forwarded.</p><pre>ps -aux | grep kubectl<br># Kill process<br>pkill kubectl</pre><p>3. Uninstall Confluent Operator.</p><pre>helm uninstall confluent-operator</pre><p>4. Stop minikube (if necessary).</p><pre>minikube stop</pre><p>5. Delete minikube (⚠️ This will destroy everything deployed).</p><pre>minikube delete</pre><h3>🎬 To be continued…</h3><p>This is the first of a three-part article. Stay tuned for what’s next!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=256f2408be62" width="1" height="1" alt=""><hr><p><a href="https://medium.com/blue-orange-digital/streamflake-real-time-cdc-pipeline-with-kafka-and-snowflake-256f2408be62">StreamFlake: Real-Time CDC Pipeline with Kafka and Snowflake</a> was originally published in <a href="https://medium.com/blue-orange-digital">Blue Orange Digital</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>