<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Vitor Carra on Medium]]></title>
        <description><![CDATA[Stories by Vitor Carra on Medium]]></description>
        <link>https://medium.com/@vitorcarra?source=rss-6462b8fc37------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*qlBJsSKTqlAy_GwFhVn9kQ.jpeg</url>
            <title>Stories by Vitor Carra on Medium</title>
            <link>https://medium.com/@vitorcarra?source=rss-6462b8fc37------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 10:23:57 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@vitorcarra/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Deploying Airflow 2 on EKS using Terraform, Helm and ArgoCD — Part 2/2]]></title>
            <link>https://vitorcarra.medium.com/deploying-airflow-2-on-eks-using-terraform-helm-and-argocd-part-2-2-4d734f98f8df?source=rss-6462b8fc37------2</link>
            <guid isPermaLink="false">https://medium.com/p/4d734f98f8df</guid>
            <category><![CDATA[argo-cd]]></category>
            <category><![CDATA[aws-eks]]></category>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[airflow-2]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Vitor Carra]]></dc:creator>
            <pubDate>Wed, 22 Dec 2021 03:40:35 GMT</pubDate>
            <atom:updated>2021-12-22T03:51:22.197Z</atom:updated>
            <content:encoded><![CDATA[<h3>Deploying Airflow 2 on EKS using Terraform, Helm and ArgoCD — Part 2/2</h3><h4>Learn how to deploy Apache Airflow 2.x on Kubernets using ArgoCD, git-sync, Terraform and helm charts.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ByND-cPHqqRtGA4z2RNphg.png" /><figcaption>airflow 2 deployment</figcaption></figure><p>This is part 2/2 of the article. If you missed the first part, please check it <a href="https://vitorcarra.medium.com/deploying-airflow-2-0-on-eks-using-terraform-helm-and-argocd-part-1-2-3f7fb3d31da6">here</a>.</p><blockquote>In this article we are going to deploy Airflow on the EKS Cluster we have created before. Instead of manually deploy it, we will use the ArgoCD application we also installed on Kubernetes cluster.</blockquote><h4><strong>Project Structure</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/630/1*Q-CWDT3pcGMj96hJ_lqFNQ.png" /><figcaption>kubernetes folder</figcaption></figure><p>Let’s take a look on a folder we haven’t talked about in part 1 of this article.</p><ul><li><strong>terraform: </strong>it’s the terraform configuration project we have talked about on part 1 of this artigle</li><li><strong>kubenetes: </strong>here we have the helm projects of the applications we want to deploy. In this case, <strong>apps/airflow</strong></li></ul><h3>Airflow Deployment</h3><p>As mentioned before, we are going to use Airflow Helm chart. We can use the project directly from artifacthub.io or download the source and customize as we want. We going to deploy using the later option.</p><ol><li>Go to <strong>kubertes/apps/airflow </strong>folder</li><li>We will use this Helm: <a href="https://artifacthub.io/packages/helm/airflow-helm/airflow">https://artifacthub.io/packages/helm/airflow-helm/airflow</a></li><li>In order to download, please follow the commands below:</li></ol><pre>helm repo add airflow-helm <a href="https://airflow-helm.github.io/charts">https://airflow-helm.github.io/charts</a><br>helm pull airflow-helm/airflow --version 8.5.2</pre><p>4. Once you have downloaded the Helm project your folder will look like below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/712/1*l-lVOTkBF13aHbpUG5hLiA.png" /><figcaption>airflow helm folder</figcaption></figure><p>This path in our repository will be used in next steps to deploy Airflow in Kuebernetes.</p><h4>Helm Chart Configuration</h4><p>There are several customization we can do since we have the source helm chart project into our repository.</p><p>For most of the scenarios, using the default helm chart is good enough. The only file we must change as we need is the <strong>values.yaml.</strong></p><p>It allows us to configure the Apache Airflow as we want. Even though Airflow Configuration is a topic for another entire article, let’s take a look in some of the configurations I’ve changed for this tutorial:</p><ul><li><strong>executor: </strong>we will use KubernetesExecutor;</li><li><strong>fernetKey: </strong>it’s a critical security value. In this example, the raw value is exposed. Don’t do that in production environment. Please consider using AWS Secret Manager, Kubernetes secrets or any other way you prefer to hide this value;</li><li><strong>postgresql</strong>: enabled: false ( we are setting false because we want to use the RDS we have created before)</li><li><strong>redis: </strong>enable: false</li><li><strong>externalDatabase: </strong>we need to setup the RDS we have created before. Remember we have set up a kubernetes secret for this</li></ul><p>There are a lot more you can configure like e-mail, ssl, gitSync and more. For more details about the configuration, please <a href="https://artifacthub.io/packages/helm/airflow-helm/airflow">check this link</a>.</p><h3>ArgoCD Configuration</h3><p>It’s finally time to configure the ArgoCD and set up the Airflow Helm chart to be deployed on Kubernetes Cluster.</p><p>For the next steps, I will assume you have the cluster up and running with all the configuration made into part 1 of this article.</p><ol><li>Login into ArgoCD Web UI (<a href="http://localhost:8001/api/v1/namespaces/argocd/services/https:argocd-server:443/proxy/">http://localhost:8001/api/v1/namespaces/argocd/services/https:argocd-server:443/proxy/</a>)</li><li>On the left bar, click on “Manage your repositories, projects, settings”</li><li>You will see a page like below. Click on “Repositories”</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hZBZuGoY18wZ0GJNKWNbOw.png" /><figcaption>Argo CD Configuration</figcaption></figure><p>4. We are going to configure our repository using SSH</p><p>5. We need to generate ssh keys and configure our repository as well. Let’s follow the <a href="https://docs.github.com/en/authentication/connecting-to-github-with-ssh/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent">github documentation</a> (commands below are related to Linux):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/842/1*nPlKcilq-sysTFy1vtv2Bg.png" /><figcaption>Generating ssh-keygen</figcaption></figure><p>6. Get the public key with following command:</p><pre>cat /home/your-user/.ssh/id_rsa_argo.pub</pre><p>7. Go to your <a href="https://github.com/settings/keys">github settings</a> on “SSH and GPG Keys”</p><p>8. Click on “New SSH Key” and add the value you got on step 6</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/758/1*t2uGpjud3tW5xUbfoIwUpQ.png" /><figcaption>adding new key</figcaption></figure><p>9. Go back to ArgoCD and click on “CONNECT REPO USING SSH”</p><p>10. Add a <strong>name</strong>: Apache Airflow</p><p>11. Select a <strong>project</strong>: default</p><p>12. <strong>Repository URL</strong>: paste your ssh git path (<em>mine is -&gt; </em><a href="/service/mailto:git@github.com"><em>git@github.com</em></a><em>:vitorcarra/airflow-kubernetes-iac.git</em>)</p><p>13. Paste the private SSH key you have generated before</p><pre>cat /home/your-user/.ssh/id_rsa_argo</pre><p>14. It should look like below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/903/1*pGbJHf2eOI21PIkPvPK1iA.png" /><figcaption>adding new git</figcaption></figure><p>15. Click on “Connect”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qLrXHfmG5JvMWxccZR2NZQ.png" /><figcaption>Repo already configured</figcaption></figure><p>16. On left menu click on “Manage your applications”</p><p>17. Click on <strong>new app</strong></p><p>18. Fill the configuration as below:</p><ul><li><strong>Application Name: </strong>airflow2</li><li><strong>Project</strong>: default</li><li><strong>Sync Policy</strong>: Automatic</li><li><strong>Check Prune Resources</strong></li><li><strong>Prune Propagation Policy</strong>: foreground</li><li><strong>Source</strong>: select the repository we have configured before</li><li><strong>Revision</strong>: HEAD</li><li><strong>Path</strong>: infrastructure/kubernetes/apps/airflow/</li><li><strong>Destination</strong>: select the kubernetes cluster (you will see only one)</li><li><strong>Namespace</strong>: airflow</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5HPUrSlsKI7d-RS7mOON2Q.png" /><figcaption>adding new app</figcaption></figure><p>19. Click on “<strong>Create</strong>” button</p><p>The next screen you will see should look like below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2Rlvdrtuz1sfHhGkT4rBcw.png" /><figcaption>Applications home screen</figcaption></figure><p>20. Click on “airflow2” project and will see a screen like below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ByND-cPHqqRtGA4z2RNphg.png" /><figcaption>airflow 2 deployment in progress</figcaption></figure><p>21. When you see app health as “<strong>Healthy” </strong>you can follow the next steps and access Airflow Web UI</p><h3>Access Airflow Web UI</h3><p>Now we have Airflow up and running in our AWS EKS cluster. It’s time to see Airflow web UI.</p><ol><li>Open a terminal and run below command. It will map the Airflow running on Kubernetes to your localhost:8080</li></ol><pre>kubectl port-forward svc/airflow2-web 8080:8080 -n airflow</pre><p>2. Open a web browser and access below url:</p><pre>http://localhost:8080</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_3BpXX3PIlkqdS9VHuZz5g.png" /><figcaption>airflow login page</figcaption></figure><p>3. Username: admin</p><p>4. Password: admin</p><p>5. Now you have a Airflow 2.x running on EKS and ready to be used</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1008/1*gd264OktNCnHHI_zuLk5MA.png" /><figcaption>airflow home screen</figcaption></figure><h3>Summary</h3><p>At the end of this article you have learned how to deploy a Kubernetes Cluster on AWS EKS using Terraform, how to deploy ArgoCD on EKS using Helm and Terraform and how to deploy Apache Airflow 2.x on EKS using ArgoCD.</p><p>There are much more configurations to be done on Airflow and ArgoCD but I hope you now have a direction about how to integrate all these amazing tools.</p><p>Below you can find great links that can help you with next steps:</p><ul><li><a href="https://airflow.apache.org/docs/apache-airflow/stable/kubernetes.html"><strong>Airflow Kubernetes Official Documentation</strong></a></li><li><a href="https://artifacthub.io/packages/helm/airflow-helm/airflow#how-to-store-dags"><strong>Configure Airflow Git-sync using Helm Chart</strong></a></li></ul><h3>Attention</h3><p>If you let this infrastructure up and running you will be billed by the time it was kept up. So please make sure you destroy everything after your study.</p><pre>terraform destroy -var-file terraform.tfvars</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4d734f98f8df" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Deploying Airflow 2.0 on EKS using Terraform, Helm and ArgoCD — Part 1/2]]></title>
            <link>https://vitorcarra.medium.com/deploying-airflow-2-0-on-eks-using-terraform-helm-and-argocd-part-1-2-3f7fb3d31da6?source=rss-6462b8fc37------2</link>
            <guid isPermaLink="false">https://medium.com/p/3f7fb3d31da6</guid>
            <category><![CDATA[airflow]]></category>
            <category><![CDATA[aws-eks-using-terraform]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[helm-chart]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Vitor Carra]]></dc:creator>
            <pubDate>Tue, 21 Dec 2021 02:54:28 GMT</pubDate>
            <atom:updated>2021-12-22T03:42:14.746Z</atom:updated>
            <content:encoded><![CDATA[<h3>Deploying Airflow 2 on EKS using Terraform, Helm and ArgoCD — Part 1/2</h3><h4>Learn how to deploy Apache Airflow 2.x on Kubernets using ArgoCD, git-sync, Terraform and helm charts.</h4><blockquote>This is part 1/2 of the article. If you are looking for the second part, please check it <a href="https://vitorcarra.medium.com/deploying-airflow-2-on-eks-using-terraform-helm-and-argocd-part-2-2-4d734f98f8df">here</a>.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FJsMPN5kPMI7JuqhsaP7rA.png" /></figure><p>There are plenty of available tools to be used in Data Engineering tasks. Thanks to internet we can find several tutorials about how to install those tools and create some deployments by ourselves.</p><p>Nevertheless, integrating such tools in a complete deployment is not straightforward sometimes. The idea of this article is to show you how to deploy Apache Airflow 2.x on an AWS EKS Cluster using Helm charts and ArgoCD.</p><blockquote>In this article I hope to help you understanding how to integrate these amazing tools and how to deploy helm charts on EKS using Terraform. Also, how to deploy other helm charts using ArgoCD with GitSync.</blockquote><h4><strong>Prerequisites</strong></h4><ul><li>Create an <strong>AWS Account</strong>: <a href="https://aws.amazon.com/premiumsupport/knowledge-center/create-and-activate-aws-account/?nc1=h_ls">https://aws.amazon.com/premiumsupport/knowledge-center/create-and-activate-aws-account/?nc1=h_ls</a></li><li>Install <strong>Terraform</strong>: <a href="https://learn.hashicorp.com/tutorials/terraform/install-cli">https://learn.hashicorp.com/tutorials/terraform/install-cli</a></li><li>Having a <strong>Github</strong> Account: <a href="https://github.com/">https://github.com/</a></li><li>Install <strong>kubectl: </strong><a href="https://kubernetes.io/docs/reference/kubectl/">https://kubernetes.io/docs/reference/kubectl/</a></li></ul><h4><strong>Important</strong></h4><blockquote>In case you follow this tutorial, you can be billed by AWS resources you use.</blockquote><h4>The Stack</h4><p>Since there are many tutorials on internet about each of the tools we are going to use here, I will briefly describe each of them:</p><ul><li><a href="https://www.terraform.io/"><strong>Terraform</strong></a>: it is an open-source infrastructure as code software tool that provides a consistent CLI workflow to manage hundreds of cloud services. Terraform codifies cloud APIs into declarative configuration files.</li><li><a href="https://docs.aws.amazon.com/eks/latest/userguide/what-is-eks.html"><strong>AWS EKS</strong></a><strong>: </strong>Amazon Elastic Kubernetes Service (Amazon EKS) is a managed service that you can use to run Kubernetes on AWS without needing to install, operate, and maintain your own Kubernetes control plane or nodes. Kubernetes is an open-source system for automating the deployment, scaling, and management of containerized applications.</li><li><a href="https://helm.sh/docs/topics/charts/"><strong>Helm Chart</strong></a>: A chart is a collection of files that describe a related set of Kubernetes resources. A single chart might be used to deploy something simple, like a memcached pod, or something complex, like a full web app stack with HTTP servers, databases, caches, and so on.</li><li><a href="https://airflow.apache.org/"><strong>Apache Airflow</strong></a>: Airflow is a platform created by the community to programmatically author, schedule and monitor workflows.</li><li><a href="https://argo-cd.readthedocs.io/en/stable/"><strong>ArgoCD</strong></a><strong>: </strong>Argo CD automates the deployment of the desired application states in the specified target environments. Application deployments can track updates to branches, tags, or pinned to a specific version of manifests at a Git commit.</li></ul><h4>Project Structure</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/191/1*WPfyFEpIn-FfmpabSQ6zKQ.png" /><figcaption>Root folder structure</figcaption></figure><ul><li><strong>airflow</strong>: it contains the dags we want to deploy in our Apache Airflow environment;</li><li><strong>infrastructure</strong>: it contains all infrastructure configuration files for Terraform and Kubernetes</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/434/1*1q4DI3eWk1ok4dmstUQ1tA.png" /><figcaption>infrastructure folder</figcaption></figure><ul><li><strong>kubernetes: </strong>it contains all the helm configuration we want to deploy using ArgoCD</li><li><strong>terraform: </strong>it contains terraform configuration files to bring the cloud infra up</li></ul><h4><strong>Terraform Setup</strong></h4><p>Inside terraform folder we will create two more folders: <strong>infra </strong>and <strong>applications</strong></p><ul><li><strong>infra: </strong>contains general infrastructure configuration like VPC, EKS and RDS</li><li><strong>applications: </strong>contains terraform configuration files for <strong>Helm Provider</strong></li></ul><p>Since we have a lot of files to configure, I will mention only the most important parts of the code here. The <strong>full project</strong> you can find on link below:</p><p><a href="https://github.com/vitorcarra/airflow-kubernetes-iac">GitHub - vitorcarra/airflow-kubernetes-iac</a></p><ul><li><strong>infrastructure/terraform/versions.tf</strong></li></ul><p>We need to set the terraform version we are going to use and which providers we want to be installed.</p><pre>terraform {<br>  required_version = &quot;&gt;= 0.13.1&quot;</pre><pre>required_providers {<br>    aws        = &quot;&gt;= 3.22.0&quot;<br>    local      = &quot;&gt;= 1.4&quot;<br>    random     = &quot;&gt;= 2.1&quot;<br>    kubernetes = &quot;&gt;= 1.13&quot;<br>  }<br>}</pre><ul><li><strong>infrastructure/terraform/main.tf</strong></li></ul><pre>provider &quot;aws&quot; {<br>    region = var.aws_region<br>    profile = var.aws_profile<br>}</pre><pre>module &quot;vpc&quot; {<br>    source = &quot;terraform-aws-modules/vpc/aws&quot;<br>    version = &quot;3.7.0&quot;<br>    name = &quot;data_plataform&quot;<br>    cidr = &quot;192.168.0.0/16&quot;</pre><pre>    azs             = var.azs<br>    private_subnets = var.private_subnets<br>    public_subnets  = var.public_subnets</pre><pre>    enable_nat_gateway = true<br>    single_nat_gateway   = true<br>    enable_dns_hostnames = true</pre><pre>    tags = {<br>      Terraform = &quot;true&quot;<br>      Environment = &quot;dev&quot;<br>    }<br>}</pre><pre>module &quot;eks&quot; {<br>  source = &quot;./infra/eks&quot;</pre><pre>  vpc = module.vpc<br>  airflowdb_host = module.rds.rds_host<br>  airflowdb_username = module.rds.rds_username<br>  airflowdb_password = module.rds.rds_password<br>  airflowdb_dbname = module.rds.rds_dbname<br>}</pre><pre>module &quot;ec2&quot; {<br>  source = &quot;./infra/ec2&quot;<br>  vpc = module.vpc<br>}</pre><pre>module &quot;rds&quot; {<br>  source = &quot;./infra/rds&quot;</pre><pre>  vpc = module.vpc<br>  vpc_security_group_ids = [module.ec2.rds_security_group.id]<br>}</pre><pre>module &quot;kubernetes-dashboard&quot; {<br>  source = &quot;./applications/kubernetes/kubedashboard&quot;<br>  eks_module = module.eks.eks_all<br>}</pre><p><strong>Modules:</strong></p><ul><li><strong>vpc: </strong>it creates a vpc and we use it to deploy all the resources.</li><li><strong>ec2: </strong>there is a security group that we need to create in order to enable communication between EKS and RDS Postgres</li><li><strong>rds: </strong>airflow needs a database to handle metadata and other configurations. Using RDS is a good choice for this purpose.</li><li><strong>eks: </strong>it contains the cluster configuration and application deployments</li></ul><p><strong>EKS Module</strong></p><p>Let’s talk about the EKS Cluster configuration.</p><pre>module &quot;eks&quot; {<br>  source = &quot;terraform-aws-modules/eks/aws&quot;</pre><pre>  cluster_name    = var.eks_cluster_name<br>  cluster_version = &quot;1.21&quot;</pre><pre>  vpc_id          = local.vpc.vpc_id<br>  subnets         = [local.vpc.private_subnets[0],       local.vpc.public_subnets[1]]<br>  fargate_subnets = [local.vpc.private_subnets[2]]</pre><pre>cluster_log_retention_in_days = 3</pre><pre>cluster_endpoint_public_access  = true</pre><pre>worker_additional_security_group_ids = [aws_security_group.all_worker_mgmt.id]</pre><pre>worker_groups = [<br>    {<br>      name = &quot;worker-group-1&quot;<br>      instance_type = &quot;t3.small&quot;<br>      asg_desired_capacity = 2<br>      asg_max_size  = 5<br>      platform = &quot;linux&quot;<br>    }<br>  ]</pre><pre>tags = {<br>    Environment = &quot;dev&quot;<br>    Name = &quot;eks-dep-cluster&quot;<br>  }<br>}</pre><pre>resource &quot;kubernetes_service_account&quot; &quot;eksadmin&quot; {<br>  metadata {<br>    name = &quot;eks-admin&quot;<br>    namespace = &quot;kube-system&quot;<br>  }<br>}</pre><pre>resource &quot;kubernetes_cluster_role_binding&quot; &quot;eksadmin&quot; {<br>  metadata {<br>    name = &quot;eks-admin&quot;<br>  }<br>  role_ref {<br>    api_group = &quot;rbac.authorization.k8s.io&quot;<br>    kind      = &quot;ClusterRole&quot;<br>    name      = &quot;cluster-admin&quot;<br>  }<br>  subject {<br>    kind      = &quot;ServiceAccount&quot;<br>    name      = &quot;eks-admin&quot;<br>    namespace = &quot;kube-system&quot;<br>  }<br>}</pre><pre>resource &quot;kubernetes_namespace&quot; &quot;airflow&quot; {<br>  metadata {<br>    name = &quot;airflow&quot;<br>  }<br>}</pre><pre>resource &quot;kubernetes_secret&quot; &quot;airflow_db_credentials&quot; {<br>  metadata {<br>    name = &quot;airflow-db-auth&quot;<br>    namespace = kubernetes_namespace.airflow.metadata[0].name<br>  }</pre><pre>data = {<br>      &quot;postgresql-password&quot; = &quot;${var.airflowdb_password}&quot;<br>    }<br>}</pre><pre>#############<br># Kubernetes<br>#############</pre><pre>data &quot;aws_eks_cluster&quot; &quot;cluster&quot; {<br>  name = module.eks.cluster_id<br>}</pre><pre>data &quot;aws_eks_cluster_auth&quot; &quot;cluster&quot; {<br>  name = module.eks.cluster_id<br>}</pre><pre>provider &quot;kubernetes&quot; {<br>  host                   = data.aws_eks_cluster.cluster.endpoint<br>  cluster_ca_certificate = base64decode(data.aws_eks_cluster.cluster.certificate_authority[0].data)<br>  token                  = data.aws_eks_cluster_auth.cluster.token<br>}</pre><pre>locals {<br>  vpc = var.vpc<br>}</pre><pre>resource &quot;aws_security_group&quot; &quot;all_worker_mgmt&quot; {<br>  name_prefix = &quot;all_worker_management&quot;<br>  vpc_id      = local.vpc.vpc_id</pre><pre>ingress {<br>    from_port = 22<br>    to_port   = 22<br>    protocol  = &quot;tcp&quot;</pre><pre>cidr_blocks = [<br>      &quot;10.0.0.0/8&quot;,<br>      &quot;172.16.0.0/12&quot;,<br>      &quot;192.168.0.0/16&quot;,<br>    ]<br>  }<br>}</pre><p>We first need to setup the cluster. Terraform provides some open-source and well implemented modules in order to make complex deployments easier. If you are going to use them or not depends on your environment policies and customization level you need.</p><p>For this tutorial, we can go ahead since we need only the most common configurations.</p><p><strong>Cluster Configuration</strong></p><ul><li><strong>cluster_name</strong>: the name of your cluster</li><li><strong>cluster_version</strong>: I am using 1.21</li><li><strong>vpc_id</strong>: the VPC we created using VPC module</li><li><strong>subnets</strong>: the subnets we created using VPC module</li><li><strong>cluster_log_retention_in_days: </strong>how many days you want to keep the logs</li><li><strong>cluster_endpoint_public_access: </strong>we are using True in this tutorial but <em>this is not recommended in production.</em></li><li><strong>worker_additional_security_group_ids</strong>: security group to be used among worker nodes</li><li><strong>worker_groups: </strong>we set the server worker nodes configuration</li></ul><p>This file is also setting up a security group to enable communication between working nodes and a <strong>Kubernetes Namespace for Airflow.</strong></p><p>Below code snippet shows this configuration:</p><pre>resource &quot;kubernetes_namespace&quot; &quot;airflow&quot; {<br>  metadata {<br>    name = &quot;airflow&quot;<br>  }<br>}</pre><pre>resource &quot;kubernetes_secret&quot; &quot;airflow_db_credentials&quot; {<br>  metadata {<br>    name = &quot;airflow-db-auth&quot;<br>    namespace = kubernetes_namespace.airflow.metadata[0].name<br>  }</pre><pre>data = {<br>    &quot;postgresql-password&quot; = &quot;${var.airflowdb_password}&quot;<br>  }<br>}</pre><p>Also, a <strong>Kubernetes Secret</strong> is created to handle sensitive database connection settings.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/630/1*rKP3brbHm1vb37eGge0HUw.png" /><figcaption>terraform folder structure</figcaption></figure><h3>Now it’s time to deploy an application into our Kubernetes Cluster.</h3><p>As you can see in the picture above, we have the <strong>applications folder. </strong>Let’s take a look into it.</p><p>You will find a folder called <strong>kubernetes </strong>with the following content:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/685/1*g6esV4Ycf6UZORyyCXpjjg.png" /><figcaption>Applications to be deployed on Kubernetes</figcaption></figure><p>Both folders have terraform files and we are going to use helm provider in order to deploy the applications we want.</p><p>Let’s take argocd as an exaple:</p><pre>locals {<br>  eks = var.eks_module<br>}</pre><pre>data &quot;aws_eks_cluster&quot; &quot;cluster&quot; {<br>  name = local.eks.cluster_id<br>}</pre><pre>data &quot;aws_eks_cluster_auth&quot; &quot;cluster&quot; {<br>  name = local.eks.cluster_id<br>}</pre><pre>provider &quot;kubernetes&quot; {<br>  host                   = data.aws_eks_cluster.cluster.endpoint<br>  cluster_ca_certificate = base64decode(data.aws_eks_cluster.cluster.certificate_authority.0.data)<br>  token                  = data.aws_eks_cluster_auth.cluster.token<br>}</pre><pre>resource &quot;kubernetes_namespace&quot; &quot;argocd&quot; {<br>  metadata {<br>    name = &quot;argocd&quot;<br>  }<br>}</pre><pre>provider &quot;helm&quot; {<br>  kubernetes {<br>    host                   = data.aws_eks_cluster.cluster.endpoint<br>    cluster_ca_certificate = base64decode(data.aws_eks_cluster.cluster.certificate_authority.0.data)<br>    exec {<br>      api_version = &quot;client.authentication.k8s.io/v1alpha1&quot;<br>      args        = [&quot;eks&quot;, &quot;get-token&quot;, &quot;--cluster-name&quot;, data.aws_eks_cluster.cluster.name]<br>      command     = &quot;aws&quot;<br>    }<br>  }<br>}</pre><pre>resource &quot;helm_release&quot; &quot;argocd&quot; {<br>  name       = &quot;argocd&quot;<br>  repository = &quot;<a href="https://argoproj.github.io/argo-helm">https://argoproj.github.io/argo-helm</a>&quot;<br>  chart      = &quot;argo-cd&quot;</pre><pre>  namespace = kubernetes_namespace.argocd.metadata.0.name<br>}</pre><ul><li>We are creating a <strong>kubernetes_namespace </strong>called argocd and we will use it to deploy argocd using helm;</li><li>We need to configure the helm provider and set which cluster we are going to use;</li><li>Last but not least, we need to create a “helm_release” resource and set the helm repository from which we want to deploy the ArgoCD application;</li><li>There are more detailed configuration that can be changed at this point, but we are going with the default installation for ArgoCD</li></ul><pre><a href="https://argoproj.github.io/argo-helm">https://argoproj.github.io/argo-hel</a>m</pre><ul><li><a href="https://github.com/argoproj/argo-helm/tree/master/charts/argo-cd"><strong>argocd</strong></a>: it is the application we will use to deploy other projects into our Kubernetes Cluster</li><li><a href="https://artifacthub.io/packages/helm/k8s-dashboard/kubernetes-dashboard"><strong>kubedashboard</strong></a>: it’s an application to help us to visualize what is happening in our cluster</li></ul><h4><strong>Deploy Terraform</strong></h4><p>In order to deploy the terraform project follow below commands inside your terraform folder (airflow-kubernetes-iac/infrastructure/terraform):</p><ol><li>terraform init</li><li>terraform validate</li><li>terraform apply -var-file terraform.tfvars</li></ol><p>After sometime, you should see a message like below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/450/1*ZCZkSZZxZVniPrjKoVHQYw.png" /></figure><p>This is how your EKS cluster should look like after the deployment:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*waleQv1zLqTJLySWsgor5A.png" /></figure><h4>How to access the Kubernetes Dashboard</h4><ol><li>First you need to configure your aws cli to access the EKS cluster</li></ol><pre>aws eks update-kubeconfig \ <br>  --region us-east-2 \ <br>  --name &lt;cluster_name&gt;</pre><p>2. Get temporary token to access the Kubernetes Dashboard</p><pre>kubectl -n kube-system describe secret $(kubectl -n kube-system get secret | grep eks-admin | awk &#39;{print $1}&#39;)</pre><p>3. Open another terminal and run the following script:</p><pre>kubectl proxy</pre><p>Leave this terminal running. Don’t close it otherwise you will interrupt the connection between your local machine and EKS cluster</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/392/1*4GmY1G25gwGnDcJvCeQ2CA.png" /><figcaption>kubectl proxy configuration</figcaption></figure><p>4. Open the Kubernetes Dashboard on your web browser:</p><p><a href="http://localhost:8001/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:443/proxy/#/login">http://localhost:8001/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:443/proxy/#/login</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*e2ubcX71lxR8Z5xgWvfQ3Q.png" /></figure><p>5. Type the token you got on Step 2 into the “Enter token *” field</p><p>6. Click on <strong>Sign In</strong></p><p>7. Great! Now you can see what is going on in your Kubernetes cluster running on EKS!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ef0AXe_ueccLCqzCnk0JZA.png" /></figure><p>8. In the combobox you see the “default” namespace selected. Change it to “argocd”. Now you can see the ArgoCD which we deployed using terraform as same as the Dashboard.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*B0lX0hArcA_AGo3TjeqsjQ.png" /></figure><h4>How to access ArgoCD UI</h4><ol><li>Open the following URL in your web browser:</li></ol><p><a href="http://localhost:8001/api/v1/namespaces/argocd/services/https:argocd-server:443/proxy/">http://localhost:8001/api/v1/namespaces/argocd/services/https:argocd-server:443/proxy/</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CRdWCeesXnsIuCgYI3RHEg.png" /><figcaption>ArgoCD Login Page</figcaption></figure><p>2. <strong>Username</strong>: admin</p><p>3. <strong>Password:</strong></p><p>Run in your terminal to get the password:</p><pre>kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath=&quot;{.data.password}&quot; | base64 -d</pre><p>4. Click on Sign In</p><p>5. Great! You have just access the ArgoCD UI which we have deployed using IaC with Terraform!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Mdt7WLTOU1wUMLiFZEIvuw.png" /></figure><h3>Summary</h3><p>Great! Now you know how to start deploying EKS and Helm charts using Terraform. This knowledge can be reproduced with any other Helm Chart you want to deploy.</p><p>There a lot of helm charts and you can find some of them on this website:</p><p><a href="https://artifacthub.io/">Artifact Hub</a></p><p>In part 2/2 of this article we are going to configure the ArgoCD to access our repository and deploy Apache Airflow 2.0 using helm chart. It will use GitOps to get any change we commit into our repo and trigger the deployment automatically.</p><h3><strong>Attention</strong></h3><p>If you let this infrastructure up and running you will be billed by the time it was kept up. So please make sure you destroy everything after your study.</p><pre>terraform destroy -var-file terraform.tfvars</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3f7fb3d31da6" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>