<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by raphaelauv on Medium]]></title>
        <description><![CDATA[Stories by raphaelauv on Medium]]></description>
        <link>https://medium.com/@raphaelauv?source=rss-62310886853b------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*1NENG87AEuakN_pJ-p6v2Q.png</url>
            <title>Stories by raphaelauv on Medium</title>
            <link>https://medium.com/@raphaelauv?source=rss-62310886853b------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 15:57:40 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@raphaelauv/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Apache Airflow : 10 rules to make it work ( scale )]]></title>
            <link>https://medium.com/data-science/apache-airflow-in-2022-10-rules-to-make-it-work-b5ed130a51ad?source=rss-62310886853b------2</link>
            <guid isPermaLink="false">https://medium.com/p/b5ed130a51ad</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[airflow]]></category>
            <category><![CDATA[dagster]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[raphaelauv]]></dc:creator>
            <pubDate>Sun, 13 Feb 2022 14:37:31 GMT</pubDate>
            <atom:updated>2023-09-22T07:20:25.129Z</atom:updated>
            <content:encoded><![CDATA[<h3>Apache Airflow : 10 Rules to Make It Work ( scale )</h3><h4>Airflow is by default very permissive and without strict rules you are likely to create a chaotic code base that is impossible to scale and administrate.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*2E2jHrw6Snd461Vs5HwjAQ.jpeg" /><figcaption>if you are not careful your shortcuts will cost you a lot afterwards</figcaption></figure><p>Airflow permissive approach will let you schedule any custom code (jobs) but you will create a spaghetti stack if you do not follow very strict <strong>SEPARATION OF CONCERN design</strong> between the airflow dags and your jobs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/717/1*Cfa_L5O114F8U3MjR5uBZw.jpeg" /><figcaption>Airflow allow you to run your jobs without isolation with the framework itself</figcaption></figure><p>At the origin Airflow was sort of a “super cron”, so the way to run jobs was heavily coupled to the framework itself. Today the biggest challenge that you must overcome is remove coupling between scheduling and running jobs.</p><p><strong><em>Nowadays (2023) GCP, AWS , AZURE provide managed Airflow v2. It’s proof enough that if you correctly setup and use the framework, it works!</em></strong></p><h3><strong>The 10 rules</strong> :</h3><p><em>so you can scale your Airflow deployment only depending on the number of dags and tasks to run (and not depending of what you run)</em></p><p><strong>1) Airflow is an orchestration framework, not an execution framework:</strong></p><p>For your jobs you should use only operators <strong>that trigger computation out of Airflow </strong>in a separate processing solution like :</p><ul><li>a container: K8S, GKE, EKS, ECS, Cloud RUN …</li><li>a serverless function: AWS Lambda, Cloud Function</li><li>a spark job: EMR, DATAPROC …</li><li>a query: BigQuery, Redshift , ClickHouse , Trino , Databend …</li><li>…</li></ul><p>Because running directly your jobs (performing substantial CPU, Memory or IO operations) inside Airflow would put pressure on your airflow_workers. Airflow should just run operators launching and asserting tasks running in a separate processing solution.</p><p><strong>2) Do not use the PythonOperator (and other magic) for your jobs:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*tl-xlrcOuVAMxgTMJwtbsQ.jpeg" /></figure><p>If you use a PythonOperator then only run <strong>very very simple code</strong>, that must only do simple IO operations (like transform a small XCOM), otherwise run your job with an operator of the rule 1.</p><p><strong>Do not use the VirtualVenvOperator or ExternalPythonOperator or BashOperator or the DockerOperator </strong>(except if you trigger the run in a host not part of your airflow deployment), <strong>you should use an operator like a KubernetesPodOperator , EmrCreateJobFlowOperator …</strong></p><p><strong>Do not use taskflow decorators</strong> ( <strong><em>@task.kubernetes</em></strong> ...) that are nice magic for POC but it will lack of <strong>separation of concerns</strong> ( having your job logic (code) inside the scheduling logic (code) is high coupling that will reduce your ability of administrating your airflow DAGS</p><p><strong>3) Choose wisely the operators:</strong></p><p><strong>3.1) Check the existing operators before creating one</strong></p><p>In 99% of cases you should not create a new Airflow operator, look very carefully for the existing operators. Check whether what you are trying to do is possible with a combination of existing operators (example, first task is a SubmitXXOperator, second task is a SensorXXOperator)</p><p>If you really feel the need for a new operator, ask the Airflow community on GitHub or slack, in most cases they will show you a way with existing operators.</p><p>If you absolutely need to customize an existing operator, extend an existing class, do not recreate everything from zero.</p><p><em>In my current company we created only 1 operator a BigQueryInsertJobToXCOMOperator (to store the small first line result of a query inside the XCOM return_value)</em></p><p><strong>3.2) Do not use every existing operators ( many are inefficient )</strong></p><p>Some open source airflow operators are just helpers (running directly python code and not triggering the operation to an external solution)</p><p>example the <strong>SimpleHttpOperator</strong> or <strong>GCSToGoogleDriveOperator</strong> run the operation directly with python code on your airflow_worker(s) <em>if you are not using the KubernetesExecutor or CeleryKubernetesExecutor</em>.</p><p>In that case it’s better to use a KubernetesPodOperator (running a container doing the desired operation) and not used theses kind of operators.</p><p><strong>3.3) Use Deferrable mode (AsyncOperators)</strong></p><p>All the long running operators <strong>( KubernetesPodOperator … ) </strong>are using a airflow_worker slot during all the run of the triggered task. So if you want run 50 <strong>KubernetesPodOperator</strong> at the same time and your airflow_worker can run 6 operators at maximum then you are going to need 9 airflow_worker nodes.</p><p>But if you use 50 <strong>KubernetesPodOperator in deferrable mode </strong>then you only need 1 airflow_triggerer node and 1 airflow_worker node</p><p><strong>4) Do not install any custom dependency in your Airflow deployment:</strong></p><p>The only allowed dependencies are the Airflow community supported providers <strong>apache-airflow-providers-XXX</strong></p><p>like :</p><p>- apache-airflow-providers-cncf-kubernetes<br>- apache-airflow-providers-google<br>- apache-airflow-providers-slack</p><p>Because these are the only packages that the community of Airflow ensures the good compatibility with Airflow itself.<br>installing any other dependency will put the deployment in a dangerous state and can result in dependency hell when upgrading.</p><p>I do not recommend installing custom Airflow provider that are not part of the <a href="https://airflow.apache.org/docs/#providers-packages-docs-apache-airflow-providers-index-html">official list</a> ( also because most of them are inefficient helper operators running code directly in your airflow_worker(s) ) :</p><p>- https://pypi.org/project/airflow-dbt/<br>- https://github.com/great-expectations/airflow-provider-great-expectations</p><p><strong>5) Airflow is NOT a data lineage solution:</strong></p><p>Airflow is a scheduler running tasks defined in operators, currently Airflow does have very limited (in beta) lineage capabilities. These allow Airflow to integrate with third party solutions using the Open Lineage standard (such as Marquez).<br>You should absolutely avoid re-implementing a custom/home made solution for this, no matter how tempting.</p><p><strong>6) Airflow is NOT a data storage solution:</strong></p><p>Airflow operators can return data that Airflow will store in its internal database airflow_db (backed by a traditional RDBS such as Postgresql). We call the data stored in the airflow_db a <strong>XCOM</strong>.</p><p>But the airflow_DB is not supposed to store custom data but only <strong>very small metadata </strong>(like our <em>BigQueryToXCOMOperator </em>usually returning a single value, like a timestamp).</p><p>The best practice is to NOT return data from operators. If you have a <strong>special cases</strong> where the operator must return data (like a complex json of multiples lines), you need to use a custom <strong>xcom_backend</strong> to write the data not in airflow_db but in another place (like S3 or GCS )</p><p>Your <strong>custom_xcom_backend</strong> must only be used for the task that you explicitly choose (be careful by default a xcom_backend will be use by Airflow for all XCOM)</p><p>In all cases if you run a job with an external processing solution like a <strong>KubernetesPodOperator then the job is responsible to write itself the result to a storage </strong>(like S3, GCS) at a path depending on the context provided by the scheduler.</p><p><strong>7) Do not put secrets in Airflow variables or connections:</strong></p><p>Airflow can store variables that DAG’s can access with templating, so the DAG will always retrieve the latest version of the variable at running time, same for connection.</p><p>But if a variable or a connection stores a secret value (like a private key), do NOT register this variable or connection directly in Airflow, you MUST register that variable or connection in a secret store (Vault , GCP secret-manager …) and use a <strong>secret_backend</strong> in Airflow.</p><p>Your <strong>custom_secret_backend</strong> must only retrieve variable or connection that you defined (be careful by default Airflow will try first to find the secret for all variables and connections in the custom_secret_backend giving warnings log for normal Airflow variables or connections)</p><p><strong>8) Your jobs must be at maximum agnostic of scheduling:</strong></p><p>If you rerun a job it must be <strong>idempotent</strong> and you must also be capable of running your job outside Airflow.</p><p>Example: if you have a small job crawling an API and writing the result on S3, you must be able to run that job in a container on your computer because the job takes all the context from env_vars or args. This ensures complete decoupling from airflow and the actual computation (clear boundaries between the scheduler and the task implementation).</p><p>A good practice is to code your jobs in a separate repository from your dags (because the dags only need to know the name and version of the container implementing the jobs).</p><p>Then in the jobs repository : associate to every job a small docker-compose to be able to run the job in a fixed context.</p><p><strong>9) Do not deploy together the different elements of Airflow:</strong></p><p>If you deploy Airflow in K8S with the official HELM file, it’s the default behavior.</p><p>But if you go for manual deployment then you must isolate the airflow_scheduler, the airflow_webserver and airflow_workers so you can scale them depending your use-cases.</p><p><strong>10) LocalExecutor and KubernetesExecutor are not flexible:</strong></p><p>The LocalExecutor is only vertically scalable because you will run tasks at the same location than the scheduler ( having HA with multiple schedulers doesn’t mean you should have N schedulers to horizontally scale your Airflow stack). Thus, it is discouraged for production environments.</p><p>The KubernetesExecutor will put the pressure directly on the K8S cluster (a new pod for every task ! very overkill ), it’s harder to detect and solve problems of scaling with that executor.</p><p>The <strong>CeleryExecutor</strong> is a good trade-off , all operators will run directly in airflow_workers and only create pods in your K8S if you use KubernetesPodOperator.</p><p>More details on the -&gt; <a href="https://airflow.apache.org/docs/apache-airflow/stable/executor/index.html#executor-types">Airflow e</a><a href="https://airflow.apache.org/docs/apache-airflow/stable/executor/index.html#executor-types">xecutors</a><a href="https://airflow.apache.org/docs/apache-airflow/stable/executor/index.html#executor-types"> documentation</a></p><p>Thank you and a big thanks to all Apache Airflow committers<br>My Github -&gt; <a href="https://github.com/raphaelauv">https://github.com/raphaelauv</a></p><p>* All images by Author *</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b5ed130a51ad" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/apache-airflow-in-2022-10-rules-to-make-it-work-b5ed130a51ad">Apache Airflow : 10 rules to make it work ( scale )</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[It’s time to do infra as REAL code]]></title>
            <link>https://medium.com/@raphaelauv/its-time-to-do-infra-as-real-code-987d4b3f5664?source=rss-62310886853b------2</link>
            <guid isPermaLink="false">https://medium.com/p/987d4b3f5664</guid>
            <category><![CDATA[pulumi]]></category>
            <category><![CDATA[code]]></category>
            <category><![CDATA[iac]]></category>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[cloudformation]]></category>
            <dc:creator><![CDATA[raphaelauv]]></dc:creator>
            <pubDate>Sat, 12 Feb 2022 11:48:45 GMT</pubDate>
            <atom:updated>2026-05-26T23:19:55.753Z</atom:updated>
            <content:encoded><![CDATA[<h3>Infra as REAL code</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/880/1*9Q1Y4WtfE8R8xy7cYCP_mA.png" /></figure><blockquote>Static configuration languages like <a href="https://en.wikipedia.org/wiki/JSON">JSON</a>, YAML .. are limited in expressivity and prone to errors as configurations become more complex. Often, these limitations lead to the introduction of auxiliary tools or custom logic, making the configuration resemble a programming language but with added complexity.</blockquote><blockquote>Alternatively, using general-purpose languages like Kotlin, Ruby, or JavaScript for configuration can be powerful but tricky, as they are not originally designed for defining and validating data.</blockquote><blockquote><strong>Aditya Kulkarni</strong></blockquote><p>With Pulumi or AWS-CDK you can use a real programming languages (python , go , java , typescript …) and no more declarative configuration languages ( HCL , YAML , JSON … ) to write your IAC.</p><p>The Wikipedia definition of IAC is :</p><blockquote><strong>Infrastructure as code</strong> is the process of managing and provisioning computer <a href="https://en.wikipedia.org/wiki/Data_center">data centers</a> through machine-readable definition files, rather than physical hardware configuration or interactive configuration tools</blockquote><p><strong>Some history :</strong></p><p>2014 - <strong>Terraform</strong> first release, it was the first big standard for doing IAC but it’s base on a home made <strong>DSL</strong> ( domain specific language ) from the creators : <strong>HCL</strong> (HashiCorp Configuration Language)</p><p>End of 2018 - <strong>Pulumi</strong> have open sourced a wrapper on Terraform so you can use a real programming language like Python.</p><p>Start of 2019 - <strong>AWS</strong> released AWS-CDK to do the same than Pulumi ( but it’s only for AWS )</p><p>End of 2019 - <strong>Hashicorp</strong> is working (with the team of AWS-CDK) on a CDK for Terraform: tfcdk</p><p>End of 2021 - <strong>Pulumi</strong> is starting to provide native API to stop using the Terraform providers (that are always late compared to the real cloud providers API).</p><p>September 2023 - <strong>Opentofu</strong> fork without official support for tfcdk</p><p>June 2024 - <strong>Terraform</strong> is also providing a “native API” provider for aws<br><a href="https://github.com/hashicorp/terraform-provider-awscc">https://github.com/hashicorp/terraform-provider-awscc</a></p><p>End of 2025 - <strong>Terraform </strong>deprecate tfcdk , <strong>Opentofu</strong> is not going to maintain the project</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=987d4b3f5664" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Scale vertically Kafka consumers]]></title>
            <link>https://medium.com/@raphaelauv/scale-vertically-kafka-consumers-e676ff458c3d?source=rss-62310886853b------2</link>
            <guid isPermaLink="false">https://medium.com/p/e676ff458c3d</guid>
            <category><![CDATA[scaling]]></category>
            <category><![CDATA[thread-safe]]></category>
            <category><![CDATA[consumer]]></category>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[multithreading]]></category>
            <dc:creator><![CDATA[raphaelauv]]></dc:creator>
            <pubDate>Tue, 22 Sep 2020 08:34:45 GMT</pubDate>
            <atom:updated>2025-06-27T00:31:35.191Z</atom:updated>
            <content:encoded><![CDATA[<h3>Scale vertically java Kafka consumers</h3><p><em>Low footprint multi-threaded java Kafka Consumers using CompletionService for graceful shutdown.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BlBbWcXnRIbYNzr3S6NjCA.jpeg" /><figcaption>Multiple <strong>carousel </strong>running in the same terminal</figcaption></figure><p>This article is not about :</p><ul><li><em>consuming with an unique thread and then splitting the compute of the messages in multiples threads( this </em><a href="https://www.confluent.io/blog/kafka-consumer-multi-threaded-messaging/?utm_source=blog&amp;utm_medium=social&amp;utm_campaign=ch.multi-threaded-message-consumption_type.community_content.clients"><em>article</em></a><em> talk about)</em></li><li>using an async kafka client -&gt; <a href="https://github.com/confluentinc/parallel-consumer">https://github.com/confluentinc/parallel-consumer</a></li></ul><blockquote>We will talk about scaling <strong>Kafka consumer</strong> but this logic can be apply to any sync (not async) code.</blockquote><h3>1) Intro :</h3><p>With Kubernetes we usually look for horizontal scaling. One way is running an unique process in your container and deploy N containers.</p><blockquote><strong>pros</strong> : easy , and compatible with horizontalpodautoscaler<br><strong>cons</strong> : memory overhead were you could have multi threading running on an unique container</blockquote><p>For a given Kafka topic with N partitions, if you have identified that to maintain your lag at minimum you need to run at least 5 consumers (in the same consumer group) you will deploy 5 containers. <strong>But in the case of java</strong>, that would be 5 isolated JVM (without any optimization or GraalVM, it’s around 300 Mo of RAM). So you’re already using 1.5 Go of ram.<br><em>There is also some CPU overhead for your server/instance to support all theses containers/JVM</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*sdDP-3u9bFPY2mswIANKDA.jpeg" /><figcaption>each container run 1 Kafka consumer</figcaption></figure><p><strong>But what if we launch multiple consumers in an unique process thank to multi-threading ?</strong></p><p>Example : Deploy 2 containers and each run a process launching N/2 threads (so each thread consume 1 partition of the topic)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3EY0sWsbntd4O7M9qTizwA.png" /><figcaption>each container run N/2 consumers</figcaption></figure><blockquote><strong>pros</strong> : better memory use (only few Mo for each thread)<br><strong>cons</strong> : code is much more complicate and no native auto-scaler for this use case in Kubernetes (that will be the subject of a future article)</blockquote><p><em>So what I’m about to explain comes with advantages and disadvantages !</em></p><h3>2) Implementation :</h3><p>The main process of the program will start threads and if any one of them stop then we will stop all the others and terminate the program. We delegate the restart management to our container orchestration solution (Kubernetes).<br><br>So we need a mechanism ensuring that if any thread finish all the others will close, for that :</p><ul><li>We will run the threads inside a <strong>CompletionService</strong> (pool of threads that give the result of a terminated thread).</li><li>And each thread will register a <strong>ShutDownHook</strong> so it can catch that the program is closing and gracefully close his Kafka consumer.</li></ul><h3>3) Code :</h3><h4>First the declaration of a Kafka consumer Class implementing the <strong>Callable</strong> interface</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/db96407063951c6a0cf3f64caed91407/href">https://medium.com/media/db96407063951c6a0cf3f64caed91407/href</a></iframe><h4>Then the condition of the loop running the Kafka consumer and how run multiple threads and wait for any close</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/95da1564eeff0c2b00b1da817bd2b804/href">https://medium.com/media/95da1564eeff0c2b00b1da817bd2b804/href</a></iframe><h4>And finally a MAIN to run the code</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2685c776f082b68bf694fc6a48857800/href">https://medium.com/media/2685c776f082b68bf694fc6a48857800/href</a></iframe><h4>Bonus</h4><p>The code of the <strong>ShutDownHook </strong>wrapper ( since the java kafka client is not thread-safe )</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/85a2aa2b2e4d6ebbe17c1aee5a484c17/href">https://medium.com/media/85a2aa2b2e4d6ebbe17c1aee5a484c17/href</a></iframe><p>Thank you , I hope you have remarks and comments !</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e676ff458c3d" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>