<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Carlos Escura on Medium]]></title>
        <description><![CDATA[Stories by Carlos Escura on Medium]]></description>
        <link>https://medium.com/@carlosescura?source=rss-965f4a3bd4a8------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*LSf3GpjBUukf6e8qZvxZew.jpeg</url>
            <title>Stories by Carlos Escura on Medium</title>
            <link>https://medium.com/@carlosescura?source=rss-965f4a3bd4a8------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 15:08:00 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@carlosescura/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Run Spark History Server on Kubernetes using Helm]]></title>
            <link>https://medium.com/@carlosescura/run-spark-history-server-on-kubernetes-using-helm-7b03bfed20f6?source=rss-965f4a3bd4a8------2</link>
            <guid isPermaLink="false">https://medium.com/p/7b03bfed20f6</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[k8s]]></category>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[web-ui]]></category>
            <dc:creator><![CDATA[Carlos Escura]]></dc:creator>
            <pubDate>Mon, 05 Oct 2020 15:23:00 GMT</pubDate>
            <atom:updated>2020-10-06T09:00:46.759Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*m7OO5icGzlWvlR2uR9JNyw.png" /></figure><p>Big data means managing tons of data, and one of the most used tools nowadays is Apache Spark. And guess what? At <a href="https://typeform.com/">Typeform</a> we deal with lots of data.</p><p>But have you ever wondered what information can you extract from your running Spark jobs? Do you really know how your data is being processed and moved back and forth among all the pieces of the puzzle?</p><p>Having a tool that helps developers to understand what’s happening under the hood of Spark applications is something really interesting, isn’t it?</p><p>So here is where <a href="https://spark.apache.org/docs/latest/web-ui.html">Spark Web UI</a> and <a href="https://spark.apache.org/docs/latest/monitoring.html">History Server</a> come into the scene. You will learn how these two tools that Spark created work, being the latter shipped by default with YARN clusters (oh wait, let me remember the EMR era and think about how easy is to fall asleep now after <a href="https://medium.com/albert-franzi/spark-structured-streaming-in-k8s-with-argo-cd-de4942846161">moving our Spark jobs</a> and <a href="https://medium.com/typeforms-engineering-blog/deploy-airflow-1-10-10-in-kubernetes-using-terraform-and-helm-2476f03f07d0">migrating Airflow</a> to K8s )</p><p>Starting from the web UI, and according to the <a href="https://spark.apache.org/docs/3.0.0-preview/web-ui.html">official documentation</a>, this is what it is:</p><blockquote>Apache Spark provides a suite of web user interfaces (UIs) that you can use to monitor the status and resource consumption of your Spark cluster.</blockquote><p>Great! So the tool is already in place, <strong>but all that glitters is not gold, and we still face two different issues while working with K8s</strong>:</p><ul><li>If running Spark in cluster mode (or via Spark Operator), driver’s and executor pods’ life is too short, and they will gracefully die and disappear once the job is completed. So not a big deal if we want to monitor or debug a completed job. Right?</li><li>Accessing a web interface from an exposed K8s service requires a little bit of <a href="https://kubernetes.io/docs/tasks/access-application-cluster/port-forward-access-application-cluster/">CLI magic</a> and a port mapping/forward using kubectl. Let’s be honest, we are lazy, and we don’t want to do it every time we need it.</li></ul><h4>In summary:</h4><ul><li>We have a tool in place, that offers the functionality we need. 👍</li><li>But we can’t easily use it due to the ephemeral nature of pods in K8s 👎</li></ul><p>The only remaining doubt is: where to store the logs so we don’t loose them once pods are removed so the Spark History Server can read them?</p><p>Guess it, <strong>we will be storing Spark logs on AWS S3</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/580/0*wK0kM69L6TIGbOcT" /></figure><h3>Kubernetes meets Helm, and invites Spark History Server to the party</h3><p>I will not discover anything new to you all if I tell you how cool is to use <a href="https://helm.sh/">Helm</a> for packaging service deployments on K8s clusters, so let’s get our hands dirty and start preparing our Helm chart to deploy the history server.</p><blockquote>But wait, you said AWS is involved right?</blockquote><p>Yes, and that means that <strong>we will not use the base Spark 3.0.1 Docker image</strong>. We need to create a custom one in order to add two crucial dependencies:</p><ul><li>Hadoop, which will allow access to file systems in the form of s3a://</li><li>Refresh a little bit the AWS SDK for Java so we can authenticate our Docker image using an IAM role assumed from a web identity (*<strong><em>WebIdentityTokenCredentialsProvider</em></strong>)</li></ul><p>*If you want to know more about <strong><em>WebIdentityTokenCredentialsProvider </em></strong><em>and why we use it at </em><a href="https://typeform.com/"><em>Typeform</em></a><em>, check </em><a href="https://aws.amazon.com/blogs/opensource/introducing-fine-grained-iam-roles-service-accounts/"><em>this AWS post</em></a><em> about </em><strong><em>IRSA</em></strong></p><h4>Building new Spark Docker image</h4><p>In order to build a new Spark image with all the requirements in place, you can use this two docker files that we use: <a href="https://gist.github.com/afranzi/4685518e24fd81e07639b97c4a5a2757">Hadoop Base image</a> → <a href="https://gist.github.com/afranzi/85ff3bf47632fc650cec17b0cc16bbca">Spark</a></p><h4>Deploying Spark History Server to Kubernetes</h4><p>Once our images are in place, we can start the Helm party:</p><p>Clone the helm repository from <a href="https://github.com/carlosescura/spark-history-server-helm">here</a></p><pre>git clone <a href="/service/mailto:git@github.com">git@github.com</a>:carlosescura/spark-history-server-helm.git</pre><p>Now we only need to take into account a couple of parameters:</p><p>image.repository substitute the base Spark image by the one that contains Hadoop and the new AWS SDK</p><p>S3logPath which indicates the S3 bucket where log files will be stored and read by the History Server</p><p>serviceAccount.name remember when we where talking about accessing AWS resources with <strong><em>WebIdentityTokenCredentialsProvider</em> </strong>using<strong> IRSA? </strong>So now is when it makes sense. In case that you want to use old style keys, you can always render the corresponding <a href="https://github.com/carlosescura/spark-history-server-helm/blob/master/templates/spark-configmap.yaml">configMap.yaml</a> with the helm of some K8s secret and an init container.</p><p>If you want to access the service from your network (outside K8s cluster) you should also modify the ingress spec from <a href="https://github.com/carlosescura/spark-history-server-helm/blob/master/values.yaml">values.yaml</a> according to your needs.</p><h4>Sending logs from Spark jobs to S3</h4><p>Last but not least, the remaining step is to configure Spark to send logs to S3</p><p>The appropriate configuration options are:</p><pre>spark.eventLog.enabled=true<br>spark.eventLog.dir=s3a://youtbucket/spark-logs<br>spark.eventLog.rolling.enabled=true<br>spark.eventLog.rolling.maxFileSize=128m</pre><p>Launch your Spark applications and enjoy checking how they work behind the scenes with your brand new Spark History Server</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eT8mhwci_lSPFwQ5saVJLA.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7b03bfed20f6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Deploy Airflow 1.10.10 in Kubernetes using Terraform and Helm]]></title>
            <link>https://medium.com/typeforms-engineering-blog/deploy-airflow-1-10-10-in-kubernetes-using-terraform-and-helm-2476f03f07d0?source=rss-965f4a3bd4a8------2</link>
            <guid isPermaLink="false">https://medium.com/p/2476f03f07d0</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[airflow]]></category>
            <category><![CDATA[k8s]]></category>
            <category><![CDATA[helm]]></category>
            <dc:creator><![CDATA[Carlos Escura]]></dc:creator>
            <pubDate>Fri, 08 May 2020 10:10:57 GMT</pubDate>
            <atom:updated>2020-05-20T22:31:38.681Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tAnz9bE2xdC53P_22iZ2Bw.png" /></figure><p>The Data Platform team at Typeform is a combination of multidisciplinary engineers, that goes from Data to Tracking and DevOps specialists. We use Airflow, love Kubernetes, and deploy our infrastructure using Terraform, so we asked ourselves:</p><blockquote>Can we move from our current Airflow running on a single EC2 instance to a new system that combines all the good parts of Kubernetes (and new Airflow’s Kubernetes Operator) and Terraform?</blockquote><p>Of course, the answer was “at least, let’s try it”. And few weeks later, we are delighted to share with you our migration journey.</p><p>This article is the first part of our series of posts related with the migration, and next releases will also cover some cool features like running Spark jobs from Airflow using the benefits of K8s.</p><h3>Migration scheduling and phase definition</h3><p>First step on such an important migration, is defining the scope of the project. What do we want to achieve? How do we want to do it? When do we want to have a “stable” solution in place?</p><h4>Our specific scenario</h4><p>At Typeform, we use Airflow to run almost all our data ETL jobs, as well as some interesting jobs related with ML model training among others, so our technology stack uses a combination of Python, Bash, SQL, Docker and Spark operators.</p><p>Given the fact that we wanted to divide the migration in several phases, our idea was to start with:</p><ul><li>Phase 1: Migrate Airflow, and just make it work</li><li>Phase 2: Migrate Python, SQL and Bash operator jobs</li><li>Phase 3: Migrate Docker and Spark operators</li></ul><p>The complex part on this journey we knew that it was going to be Docker (do you remember headaches trying to run docker in docker, with the complexity of accessing AWS credentials available in instance profile metadata?) and Spark (here we wanted to move away from classic EMR clusters and run Spark natively in K8s)</p><h3>Phase 1. About the magic of combining Airflow, Kubernetes and Terraform</h3><p>Before this migration, we also completed one of our biggest projects, which consisted in migrating almost all our services to new Kubernetes data clusters. We wanted to do it in a simple but elegant way and we decided to use the Terraform modular approach. We defined our EKS clusters using Terraform, and also a couple of modules for what we called “Cluster bootstraping” like: installing all the required base namespaces, monitoring tools, ingress controllers, and so on.</p><p>That approach worked so well for us, so we wanted to keep using it for Airflow.</p><h4>Where Terraform meets Helm</h4><p>Terraform is such a powerful tool, and most of you probably know it because you use it to define your AWS or GCP infrastructure, but in fact, if we scratch a little bit the surface of this monster, we will find something called providers and specially, one really interesting for our use case: <a href="https://www.terraform.io/docs/providers/helm/index.html">Helm provider</a></p><p>Helm helps us a lot with the heavy lifting of complex Kubernetes deployments, and you know, whenever you can follow the <a href="https://en.wikipedia.org/wiki/KISS_principle">KISS</a> principle, just follow it. We decided to use the <a href="https://github.com/helm/charts/tree/master/stable/airflow">community Helm chart for Airflow</a>, to which we also had the need to <a href="https://github.com/helm/charts/pull/21748">contribute fixing some bugs</a> (good things of Open Source projects)</p><h4>Ok, but show me the code</h4><p>Let’s start with an overview of our Airflow Terraform module</p><pre># This is a brief overview of the complete project structure</pre><pre>├── data_platform<br>│   └── k8s<br>│       ├── airflow.tf --&gt; The interesting one for this article<br>│       ├── eks.tf<br>│       ├── eks_bootstraping.tf<br>│       ├── irsa_airflow.tf<br>│       ├── main.tf<br>│       ├── outputs.tf<br>│       ├── secrets.tf<br>│       └── variables.tf<br>└── modules<br>    └── k8s<br>        ├── airflow --&gt; The interesting one for this article<br>        │   ├── airflow.tf<br>        │   ├── helm_values<br>        │   │   └── values.yaml<br>        │   ├── main.tf<br>        │   └── variables.tf<br>        ├── eks_cluster<br>        │   ├── ...<br>        │   └── ...<br>        └── eks_cluster_bootstraping<br>            ├── ...<br>            └── ...</pre><p>As the code is based on a module, and the instance of that module, we will focus first of all on the module itself, understanding all the parts involved</p><p>modules/k8s/airflow/variables.tf</p><p>The content of this file includes all the required variables that allow the possibility to launch as many times as needed (or in different clusters) this module</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d0d38cfe6dcedfb7610f66780c30e8c2/href">https://medium.com/media/d0d38cfe6dcedfb7610f66780c30e8c2/href</a></iframe><p>Most important variables we should focus on here:</p><ul><li>cluster_id used for obtaining an access token and interacting with kubelet via Terraform (more information later on)</li><li>airflow_dns_name we use <a href="https://github.com/kubernetes-sigs/external-dns">this pice of cake</a>, so really useful in combination of nginx ingress controllers</li><li>ingress_class allows us to expose each Airflow instance to different access levels or specific configurations defined in a <a href="https://kubernetes.github.io/ingress-nginx/user-guide/multiple-ingress/">specific ingress class</a></li><li>irsa_assumable_role_arn this is where things start to get a little bit more tricky. At Typeform data platform we use one of the new coolest features of EKS called <a href="https://aws.amazon.com/blogs/opensource/introducing-fine-grained-iam-roles-service-accounts/">IRSA</a>, which means that a K8s service account can be integrated with IAM and assume a certain set of roles. Benefits of this approach? A LOT, starting from the assumption that you no longer need to grant EKS worker nodes (instance profiles) a wide set of permissions that all running pods can use, or even worse, have to deal with 3rd party tools like <a href="https://github.com/jtblin/kube2iam">Kube2IAM</a> or <a href="https://github.com/uswitch/kiam">KIAM</a>. Don’t worry for now about IRSA, it’s explained later on in more depth in this article.</li></ul><p>Now it’s time to explain the magic under /modules/k8s/airflow/main.tf</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a94fd2bc094e2934e10eae12c1595b1d/href">https://medium.com/media/a94fd2bc094e2934e10eae12c1595b1d/href</a></iframe><p>Nothing really special here, but the fun fact of using kubernetes provider by only passing a simple cluster_id variable and with the help of the data resource aws_eks_cluster_auth. Thank you Terraform for making our lifes so easy!! No more kubeconfig files for dealing with clusters. As long as your initiating Terraform command is assuming a role (or using an IAM user) that is allowed to interact with EKS IAM Authenticator, you’re safe :)</p><p>It’s time to define the proper Airflow Helm chart in Terraform, so let’s move on into the next file:</p><p>/modules/k8s/airflow/airflow.tf</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e0e49990cb14a11c11370e718c07fa9b/href">https://medium.com/media/e0e49990cb14a11c11370e718c07fa9b/href</a></iframe><p>In this file, we define all the helm variables that need to be dynamic, leaving all the rest for the imported yaml file in:</p><p>/modules/k8s/airflow/helm_values/values.yaml</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e2a4ccb1f863e30f9cd1f1d1e0a8cc82/href">https://medium.com/media/e2a4ccb1f863e30f9cd1f1d1e0a8cc82/href</a></iframe><p>Ok, you maybe got information overflow, right?</p><p>So let’s stripe it in smaller pieces:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6e0c338a4c8b0a57560aaa1bf923857c/href">https://medium.com/media/6e0c338a4c8b0a57560aaa1bf923857c/href</a></iframe><p>Service accounts come into scene! Maybe one of the most important things to get IRSA working. All pods will be executed using this service account, so it’s crucial to define it.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2acfeb9493ef82c0eddd169d496c42e2/href">https://medium.com/media/2acfeb9493ef82c0eddd169d496c42e2/href</a></iframe><p>Dags section can vary a lot depending on your needs, but we went for one of the simplest ways to configure and sync DAG folders into Airflow volumes: Git sync (a simple container that runs a git pull from your desired repository every refreshTime seconds)</p><p>As we are using a custom Airflow docker image that has all our requirements already installed (and therefore allowing us faster startup times) we disable the installRequirements feature as it’s not needed. Resources are defined because:</p><p>a) It’s a good practice</p><p>b) It’s a requirement in order to activate later on HPA on Airflow worker nodes</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/689c82f8ea99b2bc4d79e7e7c8588d85/href">https://medium.com/media/689c82f8ea99b2bc4d79e7e7c8588d85/href</a></iframe><p>Airflow section is other piece that can be really different depending on your special needs, but as a brief explanation, we use a custom Docker image and have Google oAuth activated as default auth backend. Important note on ENABLE_PROXY_FIX feature, as it’s needed if you’re running your Airflow webserver behind a LoadBalancer with ProxyV2 enabled, as in our case with an AWS Classic Load Balancer.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/34cdd34318fa7f4fe8a5b658a8aa5ffb/href">https://medium.com/media/34cdd34318fa7f4fe8a5b658a8aa5ffb/href</a></iframe><p>Workers, web and scheduler sections share pretty much all the parameters, so let’s focus on workers, which are the special case.</p><p>As part of the first iteration, we wanted to keep using Celery executor (instead of Kubernetes Executor) for a simple and smooth migration in Phase 1 so it made lot of sense to enable HPA for worker nodes and replicate an “old” elastic load balancer for EC2 instances, inside K8s. More info about how Airflow can schedule jobs can be found <a href="https://www.astronomer.io/guides/airflow-executors-explained/">here</a>.</p><p>The autoscaling section enables that HPA based on our standard or custom defined metrics. This is also the main reason to have resources limits and requests defined for all the pods.</p><p>Really special mention to the securityContext. We had to fight against issues with the file reading permissions related to the AWS session token that EKS injects into the container and stores the STS credentials used by IRSA and the corresponding language SDK. Without this “ugly” fix, the default mount mode ( 600 ) for the token made it impossible to be read if you don’t start your container running as root. More info <a href="https://github.com/kubernetes-sigs/external-dns/pull/1185">here</a></p><p>Finally it’s time to instantiate our new module in /data_platform/k8s/airflow.tf</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9fd0d0601d02000700bb4a1451430571/href">https://medium.com/media/9fd0d0601d02000700bb4a1451430571/href</a></iframe><p>And finally, let’s get some fun with the coolness of integrating K8s service accounts and IAM roles in /data_platform/k8s/irsa_airflow.tf</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6fde175de6886845d4356387bf5ed4e2/href">https://medium.com/media/6fde175de6886845d4356387bf5ed4e2/href</a></iframe><p>Using a custom external module that allows creating a role, which can be assumed by an OIDC realm (our K8s service account) and accepts as many IAM policies attached as we want to use :)</p><p>With all that ingredients in place, it’s time to terraform apply and have fun with a very basic, but functional Airflow deployment running on Kubernetes and using HPA for workers autoscaling.</p><p>In the next chapters of this article series, we will talk about how to migrate current DAGs to use kubernetesOperator , how to use kubernetesExecutor , Spark on K8s clusters and many more interesting things.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/870/1*IudEp7CaKH2mjCH2pNMyAA.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2476f03f07d0" width="1" height="1" alt=""><hr><p><a href="https://medium.com/typeforms-engineering-blog/deploy-airflow-1-10-10-in-kubernetes-using-terraform-and-helm-2476f03f07d0">Deploy Airflow 1.10.10 in Kubernetes using Terraform and Helm</a> was originally published in <a href="https://medium.com/typeforms-engineering-blog">Typeform&#39;s RnD Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>