<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Chen Meyouhas on Medium]]></title>
        <description><![CDATA[Stories by Chen Meyouhas on Medium]]></description>
        <link>https://medium.com/@chen_13104?source=rss-206662d8af05------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*moJ4hXrlKXf-alq5m1xHuA.png</url>
            <title>Stories by Chen Meyouhas on Medium</title>
            <link>https://medium.com/@chen_13104?source=rss-206662d8af05------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 06:08:04 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@chen_13104/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Airflow On Azure Kubernetes — Part 1]]></title>
            <link>https://medium.com/@chen_13104/airflow-on-azure-kubernetes-part-1-6ded98d6b8d6?source=rss-206662d8af05------2</link>
            <guid isPermaLink="false">https://medium.com/p/6ded98d6b8d6</guid>
            <category><![CDATA[civalue]]></category>
            <category><![CDATA[azure]]></category>
            <category><![CDATA[airflow]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[helm]]></category>
            <dc:creator><![CDATA[Chen Meyouhas]]></dc:creator>
            <pubDate>Wed, 13 Nov 2019 11:57:27 GMT</pubDate>
            <atom:updated>2020-01-22T08:46:56.851Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>Airflow On Azure Kubernetes — Part 1</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gzOciZYkBVA_yWjYAYAQ2A.png" /></figure><h4>Background</h4><p>At <a href="http://www.civalue.com/">ciValue</a>, our various data pipelines and maintenance workflows needs drove us to explore some of the widely adopted workflow solutions out there.</p><p>After several POCs, here’s why we chose <a href="https://airflow.apache.org/">Apache Airflow</a>:</p><ul><li>It’s an open source project with wide community support</li><li>It’s cloud agnostic</li><li>Workflows<strong> </strong>as code</li><li>Written in Python and has many operators for different services, such as Databricks, PostgreSQL, SSH, Bash, Slack and more.</li></ul><h4><strong>Airflow Architecture</strong></h4><p>In general, Airflow can be deployed in two different kinds of architecture:</p><ol><li>Single node – all airflow components are installed on one machine</li><li>Multi node – each airflow component is installed on a different machine</li></ol><p>The single node architecture might be good for development while multi node architecture for production environments.</p><p>In this post I’m going to show how to easily deploy and manage multi node Airflow architecture on AKS (Azure Kubernetes service), using the official Airflow Helm chart.</p><p>First, let’s have a quick review on how Airflow components interact in a multi node architecture:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/506/1*jIBWbyetpZbvDhPI8lZ8yg.png" /></figure><p>There are five different kinds of Airflow components:</p><ol><li><strong>Webserver<br></strong>Exposing the Airflow WebUI to let a user manage his workflows, configure global variables and connections interactively.<br>It also accepts REST API requests to interact with a DAG (trigger a DAG, get information of a DagRun, and more…)<br><a href="https://airflow.apache.org/api.html">https://airflow.apache.org/api.html</a></li><li><strong>Database<br></strong>The database is responsible for storing all the Airflow metadata such as Dags, DagRuns, task instances, XCOMs (for sharing data between tasks), Variables, users and connections.</li><li><strong>Scheduler<br></strong>The scheduler periodically checks if there are new DAGs that should be registered and if a DAG and its task instances should be triggered and sent to the queue.</li><li><strong>Queue<br></strong>In order to work at scale with multiple workers, Airflow uses a queue to store tasks which should be then polled and executed by the Airflow workers.</li><li><strong>Worker<br></strong>A worker polls a task from the queue and executes the task logic.<br>Number of concurrent tasks a worker can handle at a time can be configured.</li></ol><p>Here are some reasons why deploying such an architecture on Kuberenetes with Helm is a good idea:</p><ul><li><strong>Fast initial deployment:</strong><br>Kubernetes pulls the requested container images from a container registry and deploys them on the Airflow pods. It builds the internal network between the Airflow components and exposes the Airflow WebUI as a network service. It also provisions cloud storage, mounts it to the database pod and creating web server and worker replicas for a highly available Airflow application. In our test environment, all the above took 5 minutes.</li><li><strong>Easy to upgrade and rollback versions:<br></strong>Thanks to the Helm package manager, deploying new versions or rolling back to previous versions becomes an easy and fast task.</li><li><strong>Kubernetes autoscaler:<br></strong>Kubernetes automatically changes the number of cluster nodes to meet the pods resources demand.<br>For example, changing the number of Airflow workers or changing its required resources, may trigger the autoscaler.</li><li><strong>Cloud independent</strong>:<br>Deploying the same Airflow architecture with the same Helm chart in every cloud provider</li></ul><h4><strong>How to deploy Airflow on AKS?</strong></h4><p><strong>Prerequisites</strong></p><ul><li>Python (v2.7)</li><li>Azure CLI (v2.0.76)</li><li>kubectl CLI (v1.15.3)</li><li>Helm CLI (v2.12.1)</li></ul><p><strong>Configure a new AKS<br></strong>First, let’s create a service principal <em>“service-principal-demo”</em> for the cluster. A service principal is needed for the cluster to interact with Azure resources. For example: creating a persistent volume for the database pod.</p><pre>$ az ad sp create-for-rbac --skip-assignment --name service-principal-demo</pre><p>Save the response JSON, we will need it when creating the AKS.</p><p>Create a new resource group <em>“airflow-aks-demo-rg”</em></p><pre>$ az group create --name airflow-aks-demo-rg --location westeurope</pre><p>Now, let’s create a new AKS <em>“airflow-aks-demo”</em> in the new resource group <em>“airflow-aks-demo-rg”</em></p><p>We will provide the following arguments:</p><ul><li>Service principal application id and password as the <em>“service-principal”</em> and <em>“client- secret”</em> accordingly</li><li>AKS version as the <em>“kubernetes-version”</em></li><li>Initial number of cluster nodes as <em>“node-count”</em></li><li>Cluster nodes type and size as <em>“node-vm-size”</em></li><li><em>“enable-cluster-autoscaler”</em> if we want the cluster to autoscale upon new resources demand</li><li>“VirtualMachineScaleSets” as <em>“vm-set-type”</em> – (if we enable cluster autoscaler)</li><li>Minimum and maximum number of cluster nodes as <em>“min-count“</em> and “<em>max-count”</em> (if we enable cluster autoscaler)</li><li>Location where we want the cluster to be deployed as <em>“location”</em></li></ul><p><strong>Note:<br></strong>The following command will automatically deploy a new virtual network with default address space <em>10.0.0.0/8</em>. If we want to use an existing virtual network, we should provide <em>“vnet-subnet-id”</em> as well.<br>Also, the docker bridge address defaults to <em>172.17.0.1/16</em>, so we need to make sure it doesn’t overlap with any other subnet in our subscription. If it does overlap, we might want to provide an existing address space as<em> “docker-bridge-address”</em></p><pre>$ az aks create \<br>--resource-group airflow-aks-demo-rg \<br>--name airflow-aks-demo \<br>--service-principal ABCDABCD-ABCD-ABCD-ABCD-ABCDABCDABCD \<br>--client-secret ABCDABCD-ABCD-ABCD-ABCD-ABCDABCDABCD \<br>--kubernetes-version 1.13.12 \<br>--node-count 1 \<br>--vm-set-type VirtualMachineScaleSets \<br>--node-vm-size Standard_D2s_v3 \<br>--load-balancer-sku basic \<br>--enable-cluster-autoscaler \<br>--min-count 1 \<br>--max-count 2 \<br>--location westeurope</pre><p>The output is a large JSON object describing the AKS deployment.</p><p>Configure the <em>kubectl </em>to connect to <em>“airflow-aks-demo” </em>AKS by downloading credentials and adding them to <em>~/.kube/config</em></p><pre>$ az aks get-credentials --resource-group airflow-aks-demo-rg --name airflow-aks-demo</pre><p>Bind kubernetes-dashboard service to cluster-admin role to get access to the dashboard:</p><pre>$ kubectl create clusterrolebinding kubernetes-dashboard --clusterrole=cluster-admin --serviceaccount=kube-system:kubernetes-dashboard</pre><p><strong>Installing Helm server<br></strong>Create service account <em>“tiller”</em></p><pre>$ kubectl -n kube-system create serviceaccount tiller</pre><p>In order to let Helm manage the cluster resource, the tiller service needs a cluster-admin role:</p><pre>$ kubectl create clusterrolebinding tiller --clusterrole cluster-admin --serviceaccount=kube-system:tiller</pre><p>Deploy the Helm server (Tiller)</p><pre>$ helm init --service-account tiller --wait</pre><p>Let’s verify the Tiller has been successfully deployed</p><pre>$ kubectl get deployments -n kube-system</pre><p>We should see a new deployment called <em>“tiller-deploy”</em>.</p><p>Alternatively, we can verify this using the Kubernetes dashboard:</p><pre>$ az aks browse --resource-group airflow-aks-demo-rg --name airflow-aks-demo</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/939/1*_Pk4fVBru-IOuNpUSpV7QA.png" /></figure><p><strong>Installing Airflow using Helm package manager<br></strong>Let’s create a new Kubernetes namespace <em>“airflow”</em> for the Airflow application</p><pre>$ kubectl create ns airflow</pre><p>Generate secrets for postgres and redis components and add them under <em>“airflow”</em> namespace:</p><pre>$ kubectl create secret generic airflow-postgres -n airflow --from-literal=postgres-password=$(openssl rand -base64 13)</pre><pre>$ kubectl create secret generic airflow-redis -n airflow --from-literal=redis-password=$(openssl rand -base64 13)</pre><p>Check if they were created</p><pre>$ kubectl get secrets -n airflow</pre><p>Clone the following helm chart:<br><a href="https://github.com/helm/charts/tree/master/stable/airflow">https://github.com/helm/charts/tree/master/stable/airflow</a></p><p>Configure the following secret names for postgres and redis components in the <em>values.yaml</em> file of the Airflow chart.</p><pre><strong>postgresql:</strong><em><br>## The name of an existing secret that contains the postgres password.<br> </em><strong>existingSecret: </strong>airflow-postgres</pre><pre><strong>redis:</strong><em><br>## The name of an existing secret that contains the redis password.<br> </em><strong>existingSecret: </strong>airflow-redis</pre><p>Generate fernet key to enable password encryption when creating a new connection.<br>First, install the crypto package:</p><pre>$ pip install &#39;apache-airflow[crypto]&#39;</pre><p>Then, get the fernet key:</p><pre>$ python -c &quot;from cryptography.fernet import Fernet; FERNET_KEY = Fernet.generate_key().decode(); print(FERNET_KEY)&quot;</pre><p>Update the generated key in the <em>values.yaml</em> file:</p><pre><strong>airflow:<br>  fernetKey: </strong>ABCDABCDABCDABCDABCDABCDABCDABCDABCDABCD</pre><p>If we explore the <em>requierments.yaml</em> file of the Airflow chart, we will notice that this chart has two dependencies, postgresql and redis.<br>Let’s install these dependencies:</p><p>Execute under the Airflow chart directory:</p><pre>$ helm dep update</pre><p>Make sure the dependencies are in status <em>“ok”</em>:</p><pre>$ helm dep list</pre><p>Now we are ready to install the Airflow application.<br>First, let’s install it in a <em>“dry-run”</em> mode to make sure the generated charts are valid:</p><pre>$ helm install --name airflow  --namespace airflow --debug . --dry-run</pre><p>The output is a large YAML describing the airflow deployment.<br>Let’s run it again without the “dry-run” flag and check out the pods statuses</p><pre>$ helm install --name airflow  --namespace airflow --debug .</pre><pre>$ kubectl get pods -n airflow --watch</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/589/1*WQOiM_LAFtoL_pd2e4BMgA.png" /></figure><p><strong>Open access to the Airflow WebUI</strong></p><p>Usually, this kind of deployment (internal workflows) should not be accessed through the public network, therefore in this post the way to access the Airflow WebUI is by using VPN gateway that is peered to the AKS virtual network.<br>Assuming you have a connection from your local machine to your Azure VPN gateway, and the gateway is peered to the AKS virtual network, let’s configure a load balancer that will expose the WebUI to a private IP in the AKS subnet:</p><p>In this example the AKS subnet is <em>10.97.0.0/16</em> so I’m going to use <em>10.97.0.200</em> as the load balancer IP. (make sure the chosen IP is not already taken by another resource)</p><p>Update the following values in the <em>values.yaml</em> file:</p><pre><strong>airflow:<br>  service:<br>    annotations: </strong>{<strong>service.beta.kubernetes.io/azure-load-balancer-internal: </strong>&quot;true&quot;}<br>    <strong>type: </strong>LoadBalancer<br>    <strong>loadBalancerIP: </strong>10.97.0.200<br>    <strong>externalPort: </strong>8080<br>    <strong>nodePort:<br>      http:</strong></pre><p>Now we can upgrade the Airflow application:</p><pre>$ helm upgrade airflow . --debug</pre><p>Verify that the <em>“airflow-web”</em> service got the correct external-IP</p><pre>$ kubectl get services -n airflow –watch</pre><p>Then, navigate to <em>10.97.0.200:8080...</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/939/1*V_29GP7BsHxEvTMRbcU4lQ.png" /></figure><p>Congratulations! We’ve just deployed Apache Airflow on Azure Kubernetes Service!</p><p><strong>Triggering the AKS autoscaler</strong></p><p>If we look again how we configured the AKS, we will notice that:</p><ul><li>Cluster nodes type is Standard_D2s_v3 (2 cores and 8 GB memory)</li><li>Default node count is 1</li><li>Cluster autoscaler is enabled.</li></ul><p>List the current pods and its nodes:</p><pre>$ kubectl get pods -n airflow -o wide --watch</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/939/1*HLKqU9cbJrHhQOThqz7puw.png" /></figure><p>We can see that all the Airflow pods are deployed on the cluster node<br><em>“aks-nodepool1-12545537-vmss000000”</em>.<br>Let’s deploy a new version to trigger the autoscaler to add another cluster node.</p><p>Add another Airflow worker and configure each worker to have 1 CPU.</p><pre><em>##<br> ## Workers configuration<br> </em><strong>workers:<br>   enabled: </strong>true<br>   <em>##<br>   ## Number of workers pod to launch<br>   </em><strong>replicas: </strong>2<br>   <em>##<br>   ## Custom resource configuration<br>   </em><strong>resources:<br>      limits:<br>        cpu: </strong>&quot;1&quot;<br>        <strong>memory: </strong>&quot;2G&quot;<br>      <strong>requests:<br>        cpu: </strong>&quot;1&quot;<br>        <strong>memory: </strong>&quot;2G&quot;</pre><p>This change should trigger the autoscaler as the AKS has just one cluster node with 2 CPUs and we are now going to request more than 2 CPUs (2 for the workers and some more for the other Airflow components)</p><p>Upgrade the airflow application and watch the new pod creation:</p><pre>$ helm upgrade airflow . –debug</pre><pre>$ kubectl get pods -n airflow -o wide –watch</pre><p>First, we will see a new worker pod in a <em>“pending”</em> status as it’s actually waiting for new resources</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/939/1*N_-7GuZvFWvFwaQfkgEDyg.png" /></figure><p>Then, after several minutes, a new cluster node is automatically added and the new worker pod is running on the new cluster node<br><em>“aks-nodepool1-12545537-vmss000003”</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/939/1*qycRTdSqbyyu8YOjiS1Jlg.png" /></figure><p><strong>Rolling back to a previous version</strong></p><p>Let’s rollback to a version when we had only one Airflow worker.<br>First, check the revisions statuses</p><pre>$ helm history airflow</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/725/1*NDtMse_IjHudaLzcNwyeeA.png" /></figure><p>We can see that revision 3 of the “airflow” release is currently deployed.<br>Revision 2 is the version when we had one worker and the load balancer is configured, so rollback to revision 2:</p><pre>$ helm rollback airflow 2</pre><p>Check out the pods statuses</p><pre>$ kubectl get pods -n airflow --watch</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/583/1*9IafVOoDxGMKQUYM0axlBg.png" /></figure><p>Pod <em>“airflow-worker-1”</em> has changed its status to <em>“Terminating” </em>and about to disappear</p><p>Check the airflow revisions again</p><pre>$ helm history airflow</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/745/1*RXLO5CnGbjlpSLZBpqnPoA.png" /></figure><p>We rolled back to revision 2!</p><p>That’s it for now! I hope you found this post useful and informative!<br>In part II of the post, I’ will overview advanced Airflow configuration topics, including:</p><ul><li>Best practice for deploying DAGs in production</li><li>Azure Container Register integration for deploying private docker images</li><li>Configuring Azure file as a shared storage between Airflow workers</li><li>Configuring static Azure disk as the Airflow database storage</li><li>Azure key vault integration for saving secrets</li></ul><h4>References:</h4><p><a href="https://airflow.apache.org/">https://airflow.apache.org/</a><br><a href="https://github.com/helm/charts/blob/master/stable/airflow/README.md">https://github.com/helm/charts/blob/master/stable/airflow/README.md</a><br><a href="https://docs.microsoft.com/bs-latn-ba/azure/aks/configure-azure-cni">https://docs.microsoft.com/bs-latn-ba/azure/aks/configure-azure-cni</a><br><a href="https://docs.microsoft.com/en-us/azure/aks/cluster-autoscaler">https://docs.microsoft.com/en-us/azure/aks/cluster-autoscaler</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6ded98d6b8d6" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>