<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Coste Virgile on Medium]]></title>
        <description><![CDATA[Stories by Coste Virgile on Medium]]></description>
        <link>https://medium.com/@virgile.coste?source=rss-d0d50f1457b3------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*ZVowCGoTOYaWQyhkJH8WuA.jpeg</url>
            <title>Stories by Coste Virgile on Medium</title>
            <link>https://medium.com/@virgile.coste?source=rss-d0d50f1457b3------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 05:24:13 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@virgile.coste/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Unit testing SQL queries with DuckDB]]></title>
            <link>https://medium.com/clarityai-engineering/unit-testing-sql-queries-with-duckdb-23743fd22435?source=rss-d0d50f1457b3------2</link>
            <guid isPermaLink="false">https://medium.com/p/23743fd22435</guid>
            <category><![CDATA[sql-queries]]></category>
            <category><![CDATA[property-based-testing]]></category>
            <category><![CDATA[pandas-dataframe]]></category>
            <category><![CDATA[unit-testing]]></category>
            <category><![CDATA[duckdb]]></category>
            <dc:creator><![CDATA[Coste Virgile]]></dc:creator>
            <pubDate>Wed, 18 Jan 2023 19:02:26 GMT</pubDate>
            <atom:updated>2023-02-06T10:05:21.532Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*G_P3ZbdtsgIFuMpQp6SHNg.png" /></figure><p>If you work in data wonderland, chances are that SQL is one of your main programming languages: combined with a powerful engine (BigQuery, Snowflake, Redshift…), SQL queries let you quickly transform datasets the way you want.</p><p>However, SQL queries do not seem to be unit-tested that much… And pushing code (SQL queries) to production without proper unit testing seems dangerous to me: when you or other teammates have to modify existing queries it can quickly become a tedious and error-prone process. This situation can quickly lead team members to create new and often duplicated tables, instead of modifying existing ones, to avoid the risk of corrupting or even breaking pipelines that run fine in production…</p><p>There must be a better way! And this is what I would like to discuss in this article. Before jumping on how to use DuckDB for unit testing SQL queries, let me briefly introduce the options I came across. We’ll then see how we could use DuckDB with pandas as a good complementary solution.</p><h3>Zero data cloning</h3><p>The best solution that I’ve seen involves zero data cloning. We used it in some of my previous experiences and a friend and former teammate wrote a <a href="https://medium.com/doctolib/datawarehouse-tests-automation-strategy-in-redshift-324df1fecca7">great article</a> about it.</p><p>The idea is to clone your production data warehouse into much lighter testing warehouses (with a limited lifespan) and run your queries against those testing warehouses.</p><p>It works well and it will definitely help you create a safer working environment. It really protects you when you modify a table structure: adding/renaming/deleting columns and/or changing a column data type. With the testing warehouse, you can check if all the downstream queries of a modified query (table) are still correctly running.</p><p>However, because you don’t use data here, you are not really testing the output of your query given a set of inputs. Of course, you could add data to your testing warehouses. But this can quickly become a hassle. How/where do you store your testing inputs? How/where do you store your expected outputs given your set of inputs? How to automate this in your CI/CD pipeline to have the possibility to quickly and easily change your input/output data?</p><h3>Production data validation</h3><p>To do data validation you can use open-source tools (such as <a href="https://docs.greatexpectations.io/docs/tutorials/getting_started/tutorial_overview/">Great Expectations</a> or <a href="https://docs.soda.io/soda-core/overview-main.html">Soda Core</a>). They are awesome and will clearly do a great job at checking your output datasets.</p><p>However, in a typical setup, you would use them in your production data pipeline. So you will be checking that your production outputs have some expected properties and break if that’s not the case. It’s great but it doesn’t really serve the same purposes as unit tests.</p><p>Now that I present some of the available options I know about and what seems to be missing, let’s see how DuckDB fits in the picture and how it can help you with testing queries against a set of inputs.</p><h3>DuckDB for data unit testing</h3><p>Recently, I found <a href="https://dagster.io/blog/duckdb-data-lake">this blog post</a> about the usage of DuckDB to create a “poor man data warehouse”. What stroke me was how the author designed his <a href="https://github.com/petehunt/dagster-poor-mans-data-lake/blob/main/jaffle/duckpond.py">code</a> to be used either against his data warehouse or against some pandas DataFrame for unit testing.</p><p>To make more sense of what the author is doing, let’s take a simple example:</p><pre>import duckdb<br>import pandas as pd<br><br><br>def order_table_fixture() -&gt; pd.DataFrame:<br>    return pd.DataFrame(<br>        [<br>            {&quot;order_id&quot;: 1, &quot;order_value&quot;: 9.9, &quot;user_id&quot;: &quot;a&quot;},<br>            {&quot;order_id&quot;: 2, &quot;order_value&quot;: 8.9, &quot;user_id&quot;: &quot;b&quot;},<br>            {&quot;order_id&quot;: 3, &quot;order_value&quot;: 15.5, &quot;user_id&quot;: &quot;c&quot;},<br>        ]<br>    )<br><br><br>def user_table_fixture() -&gt; pd.DataFrame:<br>    return pd.DataFrame(<br>        [<br>            {&quot;id&quot;: &quot;a&quot;, &quot;email&quot;: &quot;user_a@domain.fr&quot;},<br>            {&quot;id&quot;: &quot;b&quot;, &quot;email&quot;: &quot;user_b@domain.de&quot;},<br>        ]<br>    )<br><br># Retrieve email addresses of users having place an order of more than 9 euros<br>QUERY = &quot;&quot;&quot;<br>SELECT<br>    order_table.order_id<br>    , order_value<br>    , email<br>FROM dfA AS order_table                        -- &lt;= queriying dfA !<br>LEFT JOIN dfB AS user_table                    -- &lt;= queriying dfB !<br>ON order_table.user_id = user_table.id<br>WHERE order_table.order_value &gt; 9<br>AND user_table.email IS NOT NULL<br>&quot;&quot;&quot;<br><br><br>if __name__ == &quot;__main__&quot;:<br>    dfA = order_table_fixture()<br>    dfB = user_table_fixture()<br>    result = duckdb.query(QUERY)</pre><p>Take a look at the query and notice how we referenced tables (<strong>dfA</strong> and <strong>dfB</strong>). DuckDB will automatically resolve these table names with the python variables in our codes: <strong>dfA</strong> and <strong>dfB</strong> are pandas DataFrame containing our data examples. That’s great! But how do we take it to the next level?</p><h3>A viable testing scenario</h3><p>Building on the previous example, we could design the following scenario where we write our production queries that will run against our data warehouse. But we could take those exact queries and run them against panda DataFrames using DuckDB and some string replacement.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/778/1*OBB8WK6Nj_scvBQ9U6MBIA.png" /><figcaption>Production vs Unit tests pipeline</figcaption></figure><p>Here is how we could implement this in a quick and dirty way:</p><pre>import duckdb<br>import pandas as pd<br>import pytest<br>from pandas.testing import assert_frame_equal<br>from sql_metadata import Parser<br><br><br>class TestOrderUser:<br>    def setup(self):<br>        self.replacements = {<br>            &quot;my_database.my_schema.my_order_table&quot;: &quot;order_table&quot;,<br>            &quot;my_database.my_schema.my_user_table&quot;: &quot;user_table&quot;,<br>        }<br>        with open(&quot;s3duckdb/order_user.sql&quot;) as f:<br>            self.query = f.read()<br><br>    def test_query(self, order_table: pd.DataFrame, user_table: pd.DataFrame):<br>        query = self.query<br>        tables = Parser(self.query).tables<br>        for table in tables:<br>            if table in self.replacements:<br>                query = query.replace(table, self.replacements[table])<br><br>        result = duckdb.query(query).to_df()<br><br>        assert_frame_equal(<br>            result,<br>            pd.DataFrame(<br>                [{&quot;order_id&quot;: 1, &quot;order_value&quot;: 9.9, &quot;email&quot;: &quot;user_a@domain.fr&quot;}]<br>            ),<br>        )<br><br>    @pytest.fixture<br>    def order_table(self) -&gt; pd.DataFrame:<br>        return pd.DataFrame(<br>            [<br>                {&quot;order_id&quot;: 1, &quot;order_value&quot;: 9.9, &quot;user_id&quot;: &quot;a&quot;},<br>                {&quot;order_id&quot;: 2, &quot;order_value&quot;: 8.9, &quot;user_id&quot;: &quot;b&quot;},<br>                {&quot;order_id&quot;: 3, &quot;order_value&quot;: 15.5, &quot;user_id&quot;: &quot;c&quot;},<br>            ]<br>        )<br><br>    @pytest.fixture<br>    def user_table(self) -&gt; pd.DataFrame:<br>        return pd.DataFrame(<br>            [<br>                {&quot;id&quot;: &quot;a&quot;, &quot;email&quot;: &quot;user_a@domain.fr&quot;},<br>                {&quot;id&quot;: &quot;b&quot;, &quot;email&quot;: &quot;user_b@domain.de&quot;},<br>            ]<br>        )</pre><pre>-- production_queries/order_user.sql<br><br>SELECT<br>    order_table.order_id<br>    , order_value<br>    , email<br>FROM my_database.my_schema.my_order_table AS order_table<br>LEFT JOIN my_database.my_schema.my_user_table AS user_table<br>ON order_table.user_id = user_table.id<br>WHERE order_table.order_value &gt; 9<br>AND user_table.email IS NOT NULL</pre><p>Interesting parts:</p><ul><li>we use <a href="https://github.com/macbre/sql-metadata">sql_metadata</a> to automatically extract tables used in a given query: query parsing is a solved problem and it’s now very easy to extract all kinds of information from a SQL query (tables used, columns used, …).</li><li>we create a <em>replacements</em> dictionary to map the extracted table names with what we want to replace them, and we do the replacement.</li><li>we use some fixtures to return pandas DataFrames with interesting values we want to test our query against.</li></ul><h3>Pros and cons</h3><p>✅ You could now unit test your SQL queries along with testing how your queries behave with each other (Zero data cloning option) and monitoring the quality of your production outputs (Data validation option)!</p><p>✅ You can design fixtures DataFrames with your own convoluted data (for some edge cases you want to handle). But you could also leverage data-generating tools (<a href="https://faker.readthedocs.io/en/master/">Faker</a> for example) or even use property-based testing (with <a href="https://hypothesis.readthedocs.io/en/latest/">hypothesis</a> for example; check out <a href="https://medium.com/clarityai-engineering/property-based-testing-a-practical-approach-in-python-with-hypothesis-and-pandas-6082d737c3ee">this article</a> to know more about it)</p><p>❌ You use a different query engine for production and for unit tests. I’m not sure how big of an issue this is. If anyone has some examples that would work on Snowflake/BigQuery and not on DuckDB I would really appreciate you letting me know in the comment section.</p><p>ℹ️ You do table name replacements. I feel I need to talk a bit about it but, because I don’t really think it’s a con, I put it as information. This is why I don’t think it’s a con:</p><ul><li>you do the replacement only at the testing unit stage. The SQL queries that you write are production queries with production table names. You can copy/paste those queries and run them as is on your production data warehouse.</li><li>if you use dbt (fantastic tool) you kind of do the same thing. You use Jinja templating to specify the table name placeholders in your query. By the way, you are no longer able to copy-paste your query directly to your production data warehouse (see <a href="https://maxhalford.github.io/blog/dbt-ref-rant/">this article</a> about some limitations of dbt which is still an awesome tool)</li><li>as said, extracting used table names from a query (for lineage or dynamic replacement) is a solved problem with very good libraries doing the job for you</li></ul><h3>Next steps</h3><p>Let me know if you also think that this feature of DuckDB is useful! I think seeing how to use it within a dbt project could be a nice follow-up article.</p><p>Thanks for reading and happy to take your feedback!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=23743fd22435" width="1" height="1" alt=""><hr><p><a href="https://medium.com/clarityai-engineering/unit-testing-sql-queries-with-duckdb-23743fd22435">Unit testing SQL queries with DuckDB</a> was originally published in <a href="https://medium.com/clarityai-engineering">Clarity AI Tech</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to move data between your Snowflake environments in under 5 minutes]]></title>
            <link>https://medium.com/qonto-way/how-to-move-data-between-your-snowflake-environments-in-under-5-minutes-9c37ed59d091?source=rss-d0d50f1457b3------2</link>
            <guid isPermaLink="false">https://medium.com/p/9c37ed59d091</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[snowflake]]></category>
            <dc:creator><![CDATA[Coste Virgile]]></dc:creator>
            <pubDate>Tue, 20 Jul 2021 08:58:58 GMT</pubDate>
            <atom:updated>2021-07-20T08:58:58.308Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*W4gbjhGRIA1TMKJyGn5EqQ.png" /></figure><p>Production can be a rough place, especially when dealing with heavy migrations of data pipelines. Testing pipelines, in different environments and with actual data can save you lots of trouble down the road. In this article, we’ll show you a quick solution that we implemented at Qonto to copy data between Snowflake environments for testing purposes.</p><p>At Qonto, we are building our brand new data stack by migrating all existing data pipelines to Airflow. Our goal is to have a highly dynamic and easily configurable data platform hence allowing other tech teams to add new data sources in our Snowflake warehouse and to get an always up-to-date data lineage from ingestion to activation (more on that in an upcoming article — stay tuned).</p><p>To achieve this migration with confidence, we choose to create multiple environments with dedicated Snowflake accounts. So in addition to the existing production environment, we now have continuous integration (CI) ones for automated testing and staging ones for manual testing.</p><p>Working on data pipelines in different and hermetic environments is more secure because it allows experimentation and greatly reduces the risk of production problems later down the road. However, because these environments are hermetic, it also means that data from one environment is not accessible in another environment. And while experimenting with your data pipelines, you might need to access and copy data from different test environments.</p><p>This situation is where Snowflake user stages come in handy. We’ll see in this article how we used them to drastically reduce the time needed to move data from one environment to another.</p><h3>Did you know about internal stages?</h3><p>Stages are Snowflake objects used to load data into or from Snowflake. There are 2 types of stages for different purposes. We often hear of Snowflake stages as references to cloud storage buckets (Google Cloud Storage, Amazon S3, …). Those are <a href="https://docs.snowflake.com/en/user-guide/data-load-s3-create-stage.html">external stages</a> and they work like a symlink between a Snowflake folder and a bucket.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*y6RRch0xIbXJD_3puUCouQ.png" /></figure><p>But Snowflake also offers ready-to-use stages known as internal stages. If you have a Snowflake account, you already have an internal stage. Let’s see the difference between those two objects and how you can use them for different purposes.</p><h3>External stages: elegant but time-consuming</h3><p>To work with those external stages, you’ll need to configure a cloud storage bucket and create a Snowflake external stage referencing this bucket. This step is straightforward but might take you a bit of time (you can find detailed examples <a href="https://docs.snowflake.com/en/user-guide/data-load-bulk.html">here</a>).</p><p>First, you’ll need to find the correct Snowflake configuration depending on your use case: like most Snowflake objects, a stage is associated with a ROLE and created in a SCHEMA inside a DATABASE. You’ll then probably need to liaise with your SRE team to create the appropriate cloud storage bucket (which might require a Terraform merge request).</p><p>Those stages are ideal for automating a specific use case. But if you’re looking for a quick and dirty solution to copy data from one test environment to another, <a href="https://docs.snowflake.com/en/user-guide/data-load-local-file-system-create-stage.html">internal stages</a> can be a better choice.</p><h3>Internal stages: quick and dirty</h3><p>Unlike external stages, internal stages are only available within Snowflake and do not reference any external cloud storage bucket.</p><p>And as <a href="https://docs.snowflake.com/en/user-guide/data-load-local-file-system-create-stage.html">Snowflake documentation</a> states, if you have a Snowflake account, you already have a personal stage that you can use to quickly move data around. You can run the following command in Snowflake UI to list the files already present in your personal stage:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/af125a5d537f37e3601373f49baf7809/href">https://medium.com/media/af125a5d537f37e3601373f49baf7809/href</a></iframe><p>Stages are named and you would list their content by running the following command:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c3de3d75665d6449cdac8ca4ba95f710/href">https://medium.com/media/c3de3d75665d6449cdac8ca4ba95f710/href</a></iframe><p>By using your already configured personal stage, you greatly reduce the overheads of creating and configuring cloud buckets and external stages.</p><h3>The 5 steps copy workflow</h3><p>Here is the plan we’ll follow to copy a table from one environment to the other, using a personal stage:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mahfuzUhveYS6r2596F2-A.png" /></figure><p>As you can see, we have :</p><ul><li>1 step on your LOCAL computer</li><li>2 steps on the SOURCE environment</li><li>2 steps on the DESTINATION environment</li></ul><p>Almost all of those steps can be done using the Snowflake UI. But to perform the GET files and PUT files operations, you’ll need <a href="https://docs.snowflake.com/en/user-guide/snowsql-install-config.html">snowsql</a>. Snowsql is a command-line tool that will allow us to execute Snowflake instructions with the terminal.</p><p>Let’s detail a bit more the 5 steps:</p><h3>Locally</h3><p>Create a local folder in which we’ll download the data files:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6e7c35197411cb7f9b43847e8166dd66/href">https://medium.com/media/6e7c35197411cb7f9b43847e8166dd66/href</a></iframe><h3>On the source account</h3><p>Connect with snowsql on the source account.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b78727181354ab70bcdaf811a1f18dfc/href">https://medium.com/media/b78727181354ab70bcdaf811a1f18dfc/href</a></iframe><p>Export the content of MY_DATABASE.MY_SCHEMA.MY_TABLE in our personal stage (@~) in a “my_folder” folder (automatically created). In this folder, single or multiple files prefixed with “my_table” (depending on the size of the table) will be exported.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e5c25df7174cff1536c78207658e83f5/href">https://medium.com/media/e5c25df7174cff1536c78207658e83f5/href</a></iframe><p>Get the exported files in our local folder.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/86e27ba049e6f71f73c86571812041c4/href">https://medium.com/media/86e27ba049e6f71f73c86571812041c4/href</a></iframe><p>Copy the result of the following query (GET_DDL) on the source table (we’ll pass it in the following step):</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/66be8b96f06656527d8cdd8cdcdea2c2/href">https://medium.com/media/66be8b96f06656527d8cdd8cdcdea2c2/href</a></iframe><p>We’ve downloaded the files locally. We are now ready to upload those files to our destination.</p><h3>On the destination account</h3><p>Connect with snowsql on the destination account.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a08ef66da0bee8ee4917be23aed82367/href">https://medium.com/media/a08ef66da0bee8ee4917be23aed82367/href</a></iframe><p>Create the table on the destination account by pasting the DDL (copied in the last step).</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/be091a44bc224e2fbf753b1659678db9/href">https://medium.com/media/be091a44bc224e2fbf753b1659678db9/href</a></iframe><p>Upload the local data files in the destination user stage.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e7778fa18edf1be3968a97ccef81795c/href">https://medium.com/media/e7778fa18edf1be3968a97ccef81795c/href</a></iframe><p>Load the staged files in the destination table.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/15142cc8ca3125fd57b171fc3c487f3b/href">https://medium.com/media/15142cc8ca3125fd57b171fc3c487f3b/href</a></iframe><p>The destination table is now created and populated. You can start using it in your data pipeline.</p><h3>Automating those 5 steps</h3><p>We detailed the 5 steps process to get a better understanding of what need to be done in order to copy data from one Snowflake environment to another. Even if those steps are fairly easy, it can be cumbersome to run all those steps each time you need to copy a table from a source to a destination. Below, we show a way to automate the process using a shell script with 2 SQL queries.</p><h3>Prerequisite</h3><p>First, you need to have snowsql installed. See <a href="https://docs.snowflake.com/en/user-guide/snowsql-install-config.html">here</a> for more information. You’ll also need to configure some automated connections to snowsql. Let’s see how we can do this.</p><p>When installing snowsql, you’ll have a config file at ~/.snowsql/config where you can configure connections. You can add your credentials by adding and adapting the following lines:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d15445924670729e82d0d47733951bdb/href">https://medium.com/media/d15445924670729e82d0d47733951bdb/href</a></iframe><p>Doing so, you can connect to snowsql using the following command:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/49ea70af0495a886f3fa04a4f57249b6/href">https://medium.com/media/49ea70af0495a886f3fa04a4f57249b6/href</a></iframe><h3>The code</h3><p>The code is composed of 3 files:</p><ul><li>an export_data_query.sql file to export the source table data and its DDL</li><li>an import_data_query.sql file for creating the destination table and loads the data in it</li><li>a copy_table.sh file for the shell code</li></ul><p>Note that this script shows a way of automating. You might need to adapt it depending on your need (for instance the FILE_FORMAT that we choose in the example might not be the correct one for you).</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b12b75c2ab14269c1599d44b85946856/href">https://medium.com/media/b12b75c2ab14269c1599d44b85946856/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/18f12cfabbcf0378930b1e2ace400593/href">https://medium.com/media/18f12cfabbcf0378930b1e2ace400593/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e4b798283604edd2292dbba71467725d/href">https://medium.com/media/e4b798283604edd2292dbba71467725d/href</a></iframe><p>Before running the shell script, make the file executable by running the following command:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/eb084017f280d5fc0a25ae9148e1f63b/href">https://medium.com/media/eb084017f280d5fc0a25ae9148e1f63b/href</a></iframe><p>You can then run the script by launching one of the following commands:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2212299123b7ddf2dbe62263b406b71e/href">https://medium.com/media/2212299123b7ddf2dbe62263b406b71e/href</a></iframe><h3>Conclusion</h3><p>Testing data pipelines with actual data and having hermetic testing environments can make a huge difference when delivering code in production. When we started to work on our different testing environments, it seemed difficult to have access to data and we were not sure how to achieve this quickly.</p><p>With internal Snowflake stages, we can now experiment very quickly with actual data by copying tables between testing environments or even load hand-made CSV files in testing tables. The best part is that we didn’t need to configure anything: those stages are already there!</p><p><strong>About Qonto</strong> <a href="https://qonto.com/en">Qonto</a> is a finance solution designed for SMEs and freelancers founded in 2016 by Steve Anavi and Alexandre Prot. Since our launch in July 2017, Qonto has made business financing easy for more than 150,000 companies.</p><p>Business owners save time thanks to Qonto’s streamlined account set-up, an intuitive day-to-day user experience with unlimited transaction history, accounting exports, and a practical expense management feature.</p><p>They have had more control, whilst being able to give their teams autonomy via real-time notifications and a user-rights management system.</p><p>They have improved visibility on cash-flows through tools such as smart dashboards, transaction auto-tagging, and cash-flow monitoring.</p><p>They also enjoy stellar customer support at a fair and transparent price.</p><p>Interested in joining a challenging and game-changing company? Consult our <a href="https://qonto.com/en/careers">job offers</a>!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9c37ed59d091" width="1" height="1" alt=""><hr><p><a href="https://medium.com/qonto-way/how-to-move-data-between-your-snowflake-environments-in-under-5-minutes-9c37ed59d091">How to move data between your Snowflake environments in under 5 minutes</a> was originally published in <a href="https://medium.com/qonto-way">The Qonto Way</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A local Kubernetes cluster orchestrated with Airflow]]></title>
            <link>https://medium.com/@virgile.coste/a-local-kubernetes-cluster-orchestrated-with-airflow-5b216de684b8?source=rss-d0d50f1457b3------2</link>
            <guid isPermaLink="false">https://medium.com/p/5b216de684b8</guid>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[airflow]]></category>
            <category><![CDATA[redis]]></category>
            <category><![CDATA[python3]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Coste Virgile]]></dc:creator>
            <pubDate>Tue, 28 Apr 2020 15:11:36 GMT</pubDate>
            <atom:updated>2020-04-28T15:11:36.504Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/641/1*DYlk0CkN50k5gIMlSzSTFw.png" /></figure><p>Docker is an amazing tool ! When I first heard of it in 2016, it seemed to be barely used. Sure everyone was excited about it and some colleagues played with it locally. But no one was using it for production purposes: deploying containers in production was not a trivial task let alone the challenge of interacting containers…</p><p>Then came Kubernetes… For production purposes, it’s a real game changer and it’s definitely the go to container-orchestration system ! However the tool is still complex because of the vast variety of tasks it can handle.</p><p>So to get a deeper understanding of Kubernetes, I explored some of its features through a fairly simple use case that I would like to share with you in the form of a tutorial (available <a href="https://github.com/coste-v/kubernetes_infra_tutorial">here</a>).</p><p>Here is the general workflow I worked with :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/561/1*fBNCvW11ADhDsYxurZnOTw.png" /></figure><p>This workflow consists of :</p><ul><li>a <strong>redis feeder</strong> task, taking 2 arguments (<em>first_name</em> and <em>last_name</em>) and one environment variable (<em>ENVIRONMENT</em>) and pushing those values to a redis server</li><li>a <strong>redis server</strong>, storing the first name, last name and environment pushed by the feeder</li><li>a <strong>dummy api</strong>, reading the first name, last name and environment on the redis server and displaying it with Flask</li></ul><p>Following the tutorial, we’ll slowly build this simple workflow on a local Kubernetes cluster. Once done, we’ll use an external instance of Airflow running on a docker container outside our Kubernetes cluster to trigger our redis feeder.</p><p>If everything goes smoothly and works as expected, we should get the following environment :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/641/1*DYlk0CkN50k5gIMlSzSTFw.png" /></figure><p>I also wrote an introduction on canary deployments and on horizontal pod autoscaling.</p><p>For this tutorial, you’ll need :</p><ul><li>a local Kubernetes cluster, such as docker desktop or minikube, up and running (see <a href="https://medium.com/containers-101/local-kubernetes-for-mac-minikube-vs-docker-desktop-f2789b3cad3a">here</a>),</li><li>a local version of my code (available on my <a href="https://github.com/coste-v/kubernetes_infra_tutorial">github repo</a>),</li><li>k9s as a nice to have (some of my screenshots will display this tool). See <a href="https://github.com/derailed/k9s">here</a>.</li></ul><p>You’ll find the tutorial <a href="https://github.com/coste-v/kubernetes_infra_tutorial">here</a>.</p><p>Hope it helps in your understanding of Kubernetes and it’s orchestration with Airflow !</p><p>Thanks for reading !</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5b216de684b8" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>