<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Xing Du on Medium]]></title>
        <description><![CDATA[Stories by Xing Du on Medium]]></description>
        <link>https://medium.com/@fixit-xdu?source=rss-3fa1b8497132------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*PkZIkyBrSa71QoKf_vmCIw.jpeg</url>
            <title>Stories by Xing Du on Medium</title>
            <link>https://medium.com/@fixit-xdu?source=rss-3fa1b8497132------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 14:19:27 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@fixit-xdu/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[My 2nd assessment of istio]]></title>
            <link>https://fixit-xdu.medium.com/my-2nd-assessment-of-istio-355c343cdecb?source=rss-3fa1b8497132------2</link>
            <guid isPermaLink="false">https://medium.com/p/355c343cdecb</guid>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[service-mesh]]></category>
            <category><![CDATA[istio]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Xing Du]]></dc:creator>
            <pubDate>Tue, 20 Aug 2024 07:04:34 GMT</pubDate>
            <atom:updated>2024-08-24T22:53:46.108Z</atom:updated>
            <content:encoded><![CDATA[<p>5yrs after my first assessment of istio (2018-2019), I recently had the opportunity to assess the latest version of istio(1.21 and 1.22) again.</p><p>In this post, I&#39;ll share what led me to istio again, what I tested, and what&#39;s my assessment for istio as of Q2/Q3 2024.</p><p><strong>TL;DR; conclusion</strong>:</p><ul><li>istio sidecar mode <strong>works but does not scale </strong>well on big clusters (# of services).</li><li>latest stable (still in beta)istio ambient mode <strong>may not work</strong> for you.</li><li>linkerd can be a great alternative: less is more.</li></ul><h3>Context</h3><p>I’m working on building a new compute platform and service mesh is needed to solve several problems:</p><ol><li>GPRC load balancing</li><li>Mutual TLS for east-west traffic</li><li>Network policy</li><li>Weighted routing between different instances/versions of the same service</li></ol><p>Service mesh has many other L4/L7 features that any k8s cluster could benefit from and I’m sure an initiative like service mesh can find its angle to fit into your roadmap.</p><p>istio is my first preference due to its popularity and my familiarity. Despite my experience, I must thoroughly test all the requirements in an environment that is equivalent to my expectations.</p><p>Real test data speak better than reputation.</p><h4>Test environment &amp; requirements</h4><ul><li>AWS EKS 1.29/1.30</li><li>standard core add-ons: aws-vpc-cni + coredns + kube-proxy + karpenter + nginx-ingress-controller + external-dns</li><li>hosting web server applications (Http + GRPC), deployed with helm chart</li></ul><p>Scale: 1000 instances of the same “HelloWorld” web server: 1000 ClusterIP-typed Sevices + 1000 Ingresses + 1000 Deployments</p><p>The istio setup depends on the mode I tested. I&#39;ve tested both sidecar mode and ambient mode, and both were tested using the latest stable version (istio@1.22 as of testing), installed with helm:</p><ul><li>sidecar mode: istiod + istio-base (no istio-ingress). sidecar injection is enabled for all the service namespaces.</li><li>ambient mode: istio-cni + istiod + istio-base + ztunnel. ambient mode is enabled for all the service namespaces.</li></ul><p>istio custom resources (applies to testing for both modes): <strong>1000</strong> VirtualService + <strong>1000</strong> ServiceEntry + <strong>1000</strong> DestinationRule custom resource objects (1 set per instance of service):</p><ul><li>VirtualService + DestinationRule are needed for routing features</li><li>ServiceEntry is needed to access services outside the mesh. This deserves its own discussion, but the short version is that setting meshConfig.outboundTrafficPolicy.mode to ALLOW_ANY didn&#39;t work as expected and ServiceEntry must be explicitly defined (in order to access external URLs)</li></ul><h3>sidecar mode</h3><p><strong>TL;DR: works but does not scale</strong></p><p>When I tested istio with 2 instances of the “Hello World” application, everything works and all requirements are checked: GRPC load balancing, mTls, weighted routing, etc.</p><p>Several problems surfaced as I increased the number of service instances and eventually led me to test ambient mode as a solution.</p><h4>1st problem with sidecar mode</h4><p>The 1st issue that surfaced was <strong>a long pod initialization</strong> time (with 1000 instances of services)</p><p>pods with sidecar injected took <strong>2~10mins</strong> (measured from istio-proxy log &quot;Readiness succeeded in&quot;) to get ready. As you can imagine from running a &quot;hello world&quot; container, this delay is from the istio-proxy sidecar container <strong>exclusively</strong>.</p><p>All detailed context on this issue can be found <a href="https://github.com/istio/istio/issues/51160">here</a> if you’re interested.</p><p>I’ve tested with several earlier versions of istio (as early as 1.2.x) under the same setup: the issue received noticeable improvement as version progressed but it&#39;s not fully addressed.</p><p>After trying several hypotheses, <a href="https://github.com/ramaraochavali">@ramaraochavali</a> pointed out that envoy configuration wait_for_warm_on_init(<a href="https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/cluster/v3/cluster.proto">reference</a>) may have played a role due to the ServiceEntry objects I created in my test.</p><blockquote><em>If true, cluster readiness blocks on warm-up. If false, the cluster will complete initialization whether or not warm-up has completed. Defaults to true.</em></blockquote><p>I managed to disable wait_for_warm_on_init for all envoy clusters created from ServiceEntrys:</p><pre>apiVersion: networking.istio.io/v1alpha3<br>kind: EnvoyFilter<br>metadata:<br>  name: my-filter<br>spec:<br>  configPatches:<br>  - applyTo: CLUSTER<br>    match:<br>      context: SIDECAR_OUTBOUND<br>    patch:<br>      operation: MERGE<br>      value:<br>        wait_for_warm_on_init: false</pre><p>The EnvoyFilter patch above <strong>significantly improved</strong> the long initialization delay. It&#39;s reduced to 1min~5mins (nearly <strong>50% reduction</strong>) but it&#39;s <strong>still unacceptable</strong>. In addition, a second problem surfaced and made me give up on sidecar mode.</p><h4>2nd problem with sidecar mode</h4><p>I noticed severe memory issues for both istiod and istio-proxy sidecars: many OOMKill pods and pods stuck in a crash loop.</p><p>This memory consumption is ridiculous because there’s <strong>0 traffic</strong> despite having 1000 instances of services.</p><p>Resource-related values.yaml for istio chart:</p><pre>pilot:<br>  autoscaleEnabled: true<br>  autoscaleMin: 6<br>  autoscaleMax: 12<br>  resources:<br>    requests:<br>      cpu: &quot;1&quot;<br>      memory: 2Gi<br>    limits:<br>      cpu: &quot;4&quot;<br>      memory: 8Gi<br>global:<br>  proxy:<br>    logLevel: info<br>    resources:<br>      requests:<br>        cpu: 500m<br>        memory: 1024Mi<br>      limits:<br>        cpu: 2000m<br>        memory: 4096Mi</pre><p>I continued testing by scaling down every instance of service to 0 pods and scaling up 1 instance progressively (50 pods at a time), and the per-container memory consumption graph I got was:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*RiZifEa_NwPXj4kF" /><figcaption>istiod</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*xnSeAZbZaEo0LtPI" /><figcaption>istio-proxy</figcaption></figure><p>We’re looking at nearly 1GB of memory allocated per sidecar instance and nearly 2GB of memory per pilot instance. This is a cluster of not even 300 pods with 0 traffic, and 95+% of consumed memory was allocated for running istio data plane.</p><p>What’s worse is with my setup, I can’t go above hosting more than ~250 instances of istio-proxy sidecars. By committing more CPU+memory for isito-proxy , I may be able to push this number to be higher, but the price I have to pay is committing 90+% of the physical resources just to host the istio data plane.</p><p>My analysis of what’s observed:</p><ol><li>istiod memory spike as concurrent pod launch is <strong>reasonable</strong>. In every wave, 50 sidecars are injected and they all talk to the same istiod service to do discovery. As the image shows, the spike converged to a stable value in about 5 minutes: not ideal but acceptable.</li><li>istiod&#39;s converged memory usage grows <strong>proportional</strong>(maybe <strong>linearly</strong>) to the number of istio-proxy sidecars. This indicated the number of clusters in envoy configuration grows as the number of istio-proxy sidecars grow.</li><li>similarly, istio-proxy&#39;s memory usage grows as the number of istio-proxy sidecars. Every cluster in envoy configuration would require some resources and every sidecar uses the same envoy configuretion.</li><li>The root of the problem is the sidecar architecture of istio. All sidecars are homogenous (same configuration) and every instance of sidecar knows about every discovery it possibly needs (despite only a few being needed). This determined that the resource (especially memory) for istio data plane will <strong>grow </strong><strong>n^2 to the number of sidecars</strong></li></ol><p>After going through istio and envoy configuration to see if heterogenous envoy configuration can be supported, I realized that this is not a solvable problem.</p><p>The only “alleviation” is to try istio ambient mode</p><h3>ambient mode</h3><p><strong>TL;DR: still in beta and does not work in my setup</strong></p><p>The concept of ambient mode could alleviate the issue above in theory:</p><ul><li>the instance of istio-sidecar proxy is limited to 1 per namespace</li><li>ztunnel is a daemonset: 1 per node</li><li>despite using homogenous envoy configuration, the number of instances of envoy configuration can be greatly reduced</li></ul><p>With optimism and confidence, I started my test with ambient mode and soon I found a problem that killed istio for me for now</p><h4>Problems with ambient mode</h4><p><strong>Unlike sidecar mode</strong>, the sample application (example from istio official guide) does <strong>NOT</strong> work within my setup.</p><p>More details on the issue can be found <a href="https://github.com/istio/istio/issues/51342">here</a></p><p>I started with my interpretation of the installation and ran into connectivity issues, and I turned to the sample application only to find out the same problem. As of writing this issue down (nearly 3 months after istio team acknowledged the issue), <strong>0 progress has been made</strong>.</p><h3>What I end up doing</h3><p>I can’t afford to hold this project indefinitely waiting for istio maintainers to get back to me with a solution, so I turned to <a href="https://linkerd.io/">linkerd</a> (version 2.15), set up linkerd and performed the same test:</p><ul><li>0 connectivity issue as I experienced with istio</li><li>meets all the feature requirements.</li><li>data plane is extremely performant: during <strong>traffic load testing</strong> (under the same 1000 instance of service set up), linkerd sidecar used (<strong>consistently</strong>) <strong>no more than 10MB memory</strong> (and very little CPU) under 500 qps per service traffic pressure</li></ul><h3>Conclusion</h3><ul><li>istio has a lot of great features</li><li>istio sidecar mode works well if your cluster is small (don’t have many services) and don&#39;t need to scale</li><li>istio ambient mode is <strong>still not mature/GA after years of work</strong> (I first heard about it in 2019) and you probably don&#39;t want to run your production workloads on a product in beta version</li><li>I would love to know the context behind istio&#39;s decision to use a homogenous envoy configuration in its data plane.</li><li>Using <strong>open-source</strong> software as <strong>enterprise </strong>solutions <strong>can be a liability</strong> and long-term tech debt if the maintainers don’t get paid. Concrete incentivization goes hand in hand with the quality of the product.</li><li>linkerd isn&#39;t as L7-feature-rich as istio but if it meets all the requirements you need, it&#39;s a simpler and much more performant alternative.</li><li>don’t blindly trust reputation, trust your test data</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=355c343cdecb" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Running kube-proxy in IPVS mode: why and how]]></title>
            <link>https://fixit-xdu.medium.com/running-kube-proxy-in-ipvs-mode-why-and-how-2d5d4c39cc75?source=rss-3fa1b8497132------2</link>
            <guid isPermaLink="false">https://medium.com/p/2d5d4c39cc75</guid>
            <category><![CDATA[performance]]></category>
            <category><![CDATA[aws-eks]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[kube-proxy]]></category>
            <dc:creator><![CDATA[Xing Du]]></dc:creator>
            <pubDate>Sat, 10 Aug 2024 20:11:42 GMT</pubDate>
            <atom:updated>2024-08-10T20:11:42.145Z</atom:updated>
            <content:encoded><![CDATA[<p>I finally got some time to write down and share some of the things I learned in the past 3months about kube-proxy: something I was aware of but didn&#39;t quite understand why (or why not)</p><p>To me, “why” is always more important than “how”. I’ll dedicate my focus to explaining why you should configure kube-proxy to IPVS mode in this post.</p><h3>Conclusion</h3><p>TL;DR</p><ul><li>if your k8s cluster hosts more than 100 Service objects, you should switch kube-proxy to IPVS mode</li><li>you should almost always prefer using IPVS mode over iptables mode</li></ul><p>By “almost always” I mean: if you have reasons to favor iptables over IPVS mode, please comment below and I&#39;d love to learn more from you.</p><h3>Why</h3><p>Every decision should be contextual and I’ll start with sharing some of the context.</p><h4>Environment set up</h4><p>I was working on provisioning an EKS cluster in preparation for production workloads (decent traffic load) using a pretty standard AWS stack (probably the most popular):</p><ul><li>AWS EKS latest stable (1.29 / 1.30 as I applied the IPVS mode change)</li><li>EKS addons for core add-ons: kube-proxy, aws-vpc-cni, coredns with default values. Latest stable compatible version.</li><li>Other add-ons are driven by helm: aws-load-balancer-controller, cert-manager, external-dns, karpenter, etc</li></ul><p>An issue surfaced during load testing services that led me to this discovery. The load test is set up with a similar scale (more than) to expected peak traffic to provide valuable insights:</p><ul><li>1000 instances of the same helloworld web service was deployed to the same instance. every instance consists of 1 ingress + 1 service + 1 deploymenet + 1 HPA with a very-high ceiling.</li><li>karpenter is configured with a ceiling equivalent to 100 production-sized instances</li><li>deployment from my helloworld service is configured with nodeAffinity to match the labels of the kapenter pool</li><li>API load is configured to more than production peak, which is on the order of thousands of requests per second</li></ul><h4>The issue</h4><p>Pod autoscaling and node autoscaling are also triggered as the load ramps up. Many pods were concurrently scheduled and so were the nodes.</p><p>There’s a separate unresolved <a href="https://github.com/kubernetes-sigs/karpenter/issues/666">karpenter issue</a> on “how many nodes should be scheduled for concurrently scheduled pods” (and you&#39;ll see my comments on what I think should be done) but I won&#39;t expand here since it&#39;s not the focus of this post.</p><p>During concurrent node launch (identical ec2NodeClass / &quot;launch template&quot;), I noticed most of the nodes were stuck in NotReady state indefinitely.</p><h4>Investigation</h4><p>My first guess was hitting an EC2 API rate limit. Some nodes passed the API rate limit, but others got gated, leading to what I observed. After checking EC2 API rate limit metrics on CloudWatch, I found out there weren’t any API rate limit events.</p><p>Then I looked into aws-vpc-cni <a href="https://github.com/aws/amazon-vpc-cni-k8s">repo</a> for helpful resources: nodes stuck in NotReady is nearly exclusively caused by vpc-cni failure based on my experience. While trying to submit an issue, I found this helpful log collector tool: &quot;CNI Log Collection tool&quot; sudo bash /opt/cni/bin/aws-cni-support.sh, which collects logs from all pods placed on the node.</p><p>I didn’t find anything suspicious other than long intervals between blocks of logs from vpc-cni logs and I started cross-checking logs from other pods. By ordering log entries based on timestamps across different log streams/files, I eventually figured out why nodes are stuck in NotReady state:</p><p>Both vpc-cni and kube-proxy need to modify iptables and they starve each other due to lock contention. A single modification took<strong> tens of seconds</strong> (if not a few minutes) while the application (kube-proxy specifically) is configured to wait for the xtable lock up to 5 seconds with a 30s checking interval.</p><p>The issue does not look like a deadlock and I’m still unsure why the lock contention <strong>lasted hours</strong>. If I were to guess, it could be the retry interval less than the operation duration/timeout, blocking the initialization sequence from completion.</p><p>The root of the issue is that iptables modification is extremely slow on a large cluster. Since my cluster contains 1000+ k8s Services, the iptabls update command used by kube-proxy and vpc-cni both took dozens of seconds (same, if not more, order as the retry interval) to complete, saturating the xtable lock&#39;s availability.</p><p>If you’re interested in the context of this particular issue (e.g. detailed logs, how to reproduce, etc), 2 issues can be found here: <a href="https://github.com/aws/amazon-vpc-cni-k8s/issues/2945">issue#2945</a>, <a href="https://github.com/aws/amazon-vpc-cni-k8s/issues/2948">issue#2948</a></p><h4>Solution</h4><p>After doing more research and consulting AWS, it’s clear that I should set kube-proxy to IPVS mode to avoid contention.</p><p>I believe in theory one can alleviate (to a certain degree) this situation by increasing the retry interval but IPVS mode is a far better solution.</p><p>I have not collected data to help put things in perspective, but the same contention issue (which leads to nodes stuck in NotReady state) has not happened since the switch.</p><p><a href="https://www.tigera.io/blog/comparing-kube-proxy-modes-iptables-or-ipvs/">This post</a> from Tigera has an in-depth performance analysis which matches my observation, and I don&#39;t think I could have done a better job explaining than reading this post.</p><h3>How</h3><p>This is the easy part. The only thing worth calling out is that this change will result in downtime. Note that downtime does not linearly translate to a negative impact.</p><p>How long the downtime is depends on:</p><ul><li>how large your cluster is (# of nodes)</li><li>updateStrategy for kube-proxy</li><li># of services your k8s cluster hosts</li></ul><p>If you have the luxury of not worrying about downtime, e.g. preparing a new cluster for production, do this as early as possible.</p><p>Detailed change has <strong>2 components</strong>:</p><ol><li>addon configuration: tell the addon to use IPVS mode</li><li>update the kernel (AMI) if needed to enable the scheduler of your preference.</li></ol><p>It’s listed in the reverse order to help understand why but you should start with 2 and then work on 1.</p><h3>addon value override</h3><p>I’ll walk through the change based on my setup (using EKS addon distribution of kube-proxy). What your cluster needs depends on the context but it should be roughly the same.</p><ul><li>Run aws eks describe-addon-configuration --region us-east-1 --addon-name kube-proxy --addon-version &lt;version&gt; | jq -r &#39;.configurationSchema&#39; | jq .definitions to view the schema of the addon values/configuration</li><li>Set mode: &quot;ipvs&quot; and ipvs.scheduler: &quot;&lt;scheduler_of_your_choice&gt;&quot; via config</li></ul><p>Not <a href="https://keepalived-pqa.readthedocs.io/en/latest/scheduling_algorithms.html">all schedulers</a> are available by default. Matching kernel modules need to be turned on.</p><h3>kernel module</h3><p>I’m using the official EKS AMI which comes with ipvsadm installed. The following instructions are based on my setup, YMMV.</p><p>I&#39;m also using packer to build AMI but that may not be the case for everyone. I&#39;ll share what needs to be done and you can implement the same change based on your setup:</p><ul><li>install ipvsadm if not available</li><li>run lsmod to see what kernel modules are loaded, this will help identify what needs to be added in the next step</li><li>also worth checking ls -l /lib/modules/&lt;kernel_version&gt;/kernel/net/netfilter/ipvs/ to see what kernel modules can be loaded</li><li>make sure ip_vs and nf_conntrack are loaded and one or more scheduler modules (at least the one matching your addon configuration). You can test loading with sudo modprobe &lt;module_name&gt;</li><li>to ensure AMI loads the kernel modules correctly, you should add the list of modules to /etc/modules-load.d/ipvs.conf.</li></ul><p>Alternatively, you can read <a href="https://repost.aws/knowledge-center/eks-configure-ipvs-kube-proxy">this post</a> from AWS which covers most of the steps above.</p><p>Lastly, here’s my /etc/modules-load.d/ipvs.conf:</p><pre>ip_vs<br>nf_conntrack<br>ip_vs_rr<br>ip_vs_mh<br>ip_vs_wlc</pre><p>and my addon config ipvs.scheduler is set to mh</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2d5d4c39cc75" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Send AWS VPC CNI metrics to Datadog]]></title>
            <link>https://fixit-xdu.medium.com/send-aws-vpc-cni-metrics-to-datadog-a567a97c8bb1?source=rss-3fa1b8497132------2</link>
            <guid isPermaLink="false">https://medium.com/p/a567a97c8bb1</guid>
            <category><![CDATA[datadog]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[aws-eks]]></category>
            <category><![CDATA[observability]]></category>
            <dc:creator><![CDATA[Xing Du]]></dc:creator>
            <pubDate>Mon, 03 Jun 2024 07:47:15 GMT</pubDate>
            <atom:updated>2024-06-03T07:47:15.726Z</atom:updated>
            <content:encoded><![CDATA[<p>I’ve been polishing my EKS observability recently and noticed missing coverage on VPC CNI.</p><p>After doing some quick research, I failed to find an official integration for AWS VPC CNI on Datadog. Despite the disappointment, it’s not difficult to implement an equivalent solution (at a small price to pay).</p><p>I’m sharing this quick solution hoping that someone can correct me or Datadog folks can adopt this as an official integration.</p><h3>Why</h3><p>AWS VPC CNI is the add-on responsible for pod network on AWS EKS (or self-managed k8s on AWS).</p><p>It’s one of the most important add-on for your k8s cluster: when an instance fails, the node can no longer launch or terminate pods: they are stuck indefinitely. (see my other post on what happens when it fails) Therefore, having observability covering the insights of VPC CNI is equally important.</p><h3>How</h3><h4>cni-metrics-helper</h4><p>You can deploy cni-metrics-helper <a href="https://github.com/aws/amazon-vpc-cni-k8s/tree/master/charts/cni-metrics-helper">helm chart</a> and try to port the metrics from CW to Datadog.</p><p>This solution seems more official but is also a bit complicated to me, and I didn’t try to validate if it works.</p><h4>prometheus crawler</h4><p>VPC CNI has its own metrics endpoint and the port is exposed (and bound to the host): metrics from each aws-node pod can be accessed from the node.</p><p>Datadog agent is also a daemonset and supports prometheus crawling based on configuration.</p><p>All we need is to configure datadog-agent with one additional prometheus instance for AWS VPC CNI at localhost:61678/metrics and properly configure the metric mapping.</p><p>You don’t need to set anything in VPC CNI’s helm chart values / EKS addon configurations, the only thing you need is to add the following to the Datadog helm chart:</p><pre>agent:<br>  confd:<br>    prometheus.yaml: |-<br>      init_config:<br>      instances:<br>        - prometheus_url: http://localhost:61678/metrics<br>          namespace: kube-system<br>          metrics:<br>            # array of metric name, optionally mapping to a new name.<br>            - awscni_add_ip_req_count: awscni.add_ip_req_count<br>            - awscni_assigned_ip_addresses: awscni.assigned_ip_addresses<br>            ...<br>            # skipping the rest for brevity</pre><p>To view all the metrics available, you can:</p><ul><li>ssh to the node and run curl http://localhost:61678/metrics, or</li><li>kubectl exec daemonset/datadog -n &lt;datadog_namespace&gt; -- curl <a href="http://localhost:61678/metrics">http://localhost:61678/metrics</a></li></ul><h3>Results</h3><p>After deploying the updated Datadog helm chart, you&#39;ll have VPC CNI metrics available in Datadog under kube_system.awscni.*. E.g.:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*GAUD6jhDvOAoldOp" /></figure><p>Now I can proceed to create a monitor to help me catch issues when awscni_eni_allocated is approaching awscni_eni_max or when awscni_no_available_ip_addresses is not 0, preventing major incidents on my EKS clusters.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a567a97c8bb1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Using kubernetes_groups in EKS access entry: when and how]]></title>
            <link>https://fixit-xdu.medium.com/using-kubernetes-groups-in-eks-access-entry-when-and-how-5180fd178e91?source=rss-3fa1b8497132------2</link>
            <guid isPermaLink="false">https://medium.com/p/5180fd178e91</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[rbac]]></category>
            <category><![CDATA[aws-eks]]></category>
            <dc:creator><![CDATA[Xing Du]]></dc:creator>
            <pubDate>Mon, 06 May 2024 05:31:18 GMT</pubDate>
            <atom:updated>2024-05-06T05:31:18.663Z</atom:updated>
            <content:encoded><![CDATA[<p>AWS EKS access entry has an attribute called <a href="https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/eks_access_entry#kubernetes_groups">kubernetes_groups</a>. If you are familiar with the managed access policies (see my <a href="https://fixit-xdu.medium.com/aws-eks-access-entry-4a7e25ed6c3a">other post</a>), you may wonder why this is needed.</p><p>I’ll explain with a valid use cases on when to use <a href="https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/eks_access_entry#kubernetes_groups">kubernetes_groups</a> and how to.</p><h3>What problem does kubernetes_groups solve</h3><p>As I mentioned <a href="https://fixit-xdu.medium.com/aws-eks-access-entry-4a7e25ed6c3a">before</a>, the 4 managed EKS access policies are very coarse and do <strong>not</strong> allow customization. Courtesy of @mikestef9 (<a href="https://github.com/aws/containers-roadmap/issues/185#issuecomment-1863025784">explanation</a>): the 4 access entry policies map to the upstream Kubernetes user-facing <a href="https://kubernetes.io/docs/reference/access-authn-authz/rbac/#user-facing-roles">roles</a>:</p><ul><li>cluster-admin</li><li>admin</li><li>edit</li><li>view</li></ul><p>If your use case requires customization or finer granularity than these 4 roles, you’ll need to join EKS access entry with native k8s RBAC to achieve what you need to do.</p><h3>Example</h3><p>Your CD solution needs to run releases for your services to your EKS cluster. A dedicated IAM role (e.g. eks-service-deploy-role) was created and added to EKS access entries with AmazonEKSEditPolicy. This role is widely used by different teams working on different services.</p><p>It’s scope to get, create, update, and delete k8s resources for these services, no more and no less. AmazonEKSEditPolicy nearly worked, however:</p><ul><li>it does not have permission to create namespace</li><li>it has no rules for custom resources and some of your helm charts contain custom resources</li></ul><p>As a result, your helm upgrade --install --create-namespace failed due to permission errors. Thankfully, kubernetes_groups in EKS access entry solves this problem.</p><h3>How to use kubernetes_groups</h3><ul><li>add kubernetes_groups = [&quot;&lt;group_name&gt;&quot;] to your EKS access entry</li><li>create additive custom role(s)</li><li>create role binding with a group subject matching the &quot;&lt;group_name&gt;&quot; assigned to EKS access entry</li></ul><p>Any combination of ClusterRole/Role and ClusterRoleBinding/RoleBinding work for this solution. It comes down to how you design your k8s RBAC. I will <strong>not</strong> expand here, this topic deserves its own post.</p><p>In terraform source code, your change should look similar to:</p><pre>access_entries = {<br>    service_deploy_role = {<br>      principal_arn = &lt;my_role_arn&gt;<br>+     kubernetes_groups = [&quot;my-group&quot;]<br>      policy_associations = {<br>        ...ch<br>      }<br>    }<br>  }</pre><pre>+resource &quot;kubernetes_cluster_role&quot; &quot;custom_role&quot; {<br>+  metadata {<br>+    name   = &quot;my-custom-cluster-role&quot;<br>+  }<br>+<br>+  rule {<br>+    ...<br>+  }<br>+}</pre><pre>+resource &quot;kubernetes_cluster_role_binding&quot; &quot;custom_role_bind&quot; {<br>+  metadata {<br>+    ...<br>+  }<br>+  role_ref {<br>+    ...<br>+    name      = &quot;my-custom-cluster-role&quot;<br>+  }<br>+<br>+  subject {<br>+    api_group = &quot;rbac.authorization.k8s.io&quot;<br>+    kind = &quot;Group&quot;<br>+    name = &quot;my-group&quot; # use `dynamic` + eks module output if you want to go fancy<br>+  }<br>+}</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5180fd178e91" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AWS EKS Access entry]]></title>
            <link>https://fixit-xdu.medium.com/aws-eks-access-entry-4a7e25ed6c3a?source=rss-3fa1b8497132------2</link>
            <guid isPermaLink="false">https://medium.com/p/4a7e25ed6c3a</guid>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[ek]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Xing Du]]></dc:creator>
            <pubDate>Tue, 12 Mar 2024 06:25:25 GMT</pubDate>
            <atom:updated>2024-06-22T21:06:15.332Z</atom:updated>
            <content:encoded><![CDATA[<p>My EKS clusters were created with terraform <a href="https://registry.terraform.io/modules/terraform-aws-modules/eks/aws/19.21.0">EKS module@v19</a> and I recently upgraded to version 20, which introduced a <a href="https://github.com/terraform-aws-modules/terraform-aws-eks/blob/master/CHANGELOG.md#2000-2024-02-02">breaking change</a> on how to set up cluster auth.</p><p>I didn’t find any good documentation on how this upgrade process should be and I decided to write down the things I learned in the process. Hopefully, it will benefit others who need to perform the same upgrade.</p><h3>aws-auth</h3><p>aws-auth ConfigMap has been the only option for EKS auth. The cluster creator will be granted cluster admin permission without having to be added to this ConfigMap explicitly.</p><p>From the cluster provisioning perspective, I’m not a fan of this approach. Almost everything else are AWS resources and are created with aws provider, this is the only thing (correct me if I missed anything) that’s <strong>inconsistent</strong> when standing up a vanilla EKS cluster with terraform.</p><p>The cluster creator’s access is not reflected anywhere explicitly, which is another thing I don’t like.</p><h3>EKS Access entry</h3><p>EKS access entry is the replacement of aws-auth for the exact same pain point I mentioned above.</p><p>EKS access entry offloads RBAC from k8s domain to aws domain. For each entry, you can either use <a href="https://docs.aws.amazon.com/eks/latest/userguide/access-policies.html">managed EKS access policy</a> or continue with binding with k8s groups.</p><p>AWS documentation has mentioned that the cluster creator has its implicit entry in EKS access entry, which is consistent with how the other authentication would be set up.</p><h4>EKS access policy</h4><p>4 options to choose from, very coarse as of this writing. However, I believe this will get more granular in the future and the best part (could be a double-edged sword) is that it’s managed.</p><h4>k8s RBAC</h4><p>Used for customizing RBAC for the access entry policy (e.g. custom resources). See my <a href="https://fixit-xdu.medium.com/using-kubernetes-groups-in-eks-access-entry-when-and-how-5180fd178e91">other post</a> on when &amp; how to k8s groups in EKS access entry</p><h3>Upgrade</h3><p>First, bump up aws provider version to the latest stable. I suggest applying this as a stand-alone change and validating it before moving forward.</p><p>Second, address the terraform complaints. Mainly coming from 3 things:</p><ol><li>aws-auth related variables should be removed: manage_aws_auth_configmap , aws_auth_roles , etc</li><li>authentication_mode</li><li>access_entries</li></ol><p>For EKS clusters created using the EKS module before version 20, authentication_mode is defaulted to CONFIG_MAP . I suggest taking this opportunity to upgrade to API if you don’t want to have a hard time upgrading to version 21.</p><p>To upgrade from CONFIG_MAP to API , you need to run 2 terraform applys since this transition is <strong>not allowed by AWS</strong>. You need to make a change from CONFIG_MAP to API_AND_CONFIG_MAP , terraform apply, and make another change from API_AND_CONFIG_MAP to API and terraform apply.</p><p>After this change, AWS converts your existing aws-auth data into access entries. They will NOT be in your tfstate and you will run into multiple “resource already exist” errors if you try to apply directly.</p><p>Note that the cluster creator role would have a matching EKS access entry If enable_cluster_creator_admin_permissions is set to true . If you created another entry under access_entries block, you’ll get an error during terraform apply .</p><p>It’s worth taking the time to run a terraform state list after creating all the access entries (creator + additional ones under access_entries ). In the case of upgrading an existing cluster, I noticed duplicated tfstate resources: 2 resources (aws_eks_access_entry, different addresses) in tfstate share the same content. If the same thing happen to you, use terraform state rm to deduplicate.</p><p>Draft your entries based on how you configured aws-auth previously, run terraform plan to get the accurate address of the aws_eks_access_entry <a href="https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/eks_access_entry">resources</a> to be created, and use import <a href="https://developer.hashicorp.com/terraform/language/import">block</a> (alternatively, terraform import) to import these converted access entries to your tfstate and adjust the values (e.g. if you want to use managed policy instead of k8s group)</p><p>Now you should be able to run your terraform without other issues after the upgrade.</p><h3>Post upgrade</h3><p>2 important things to callout:</p><ol><li>the upgrade process will empty the aws-auth ConfigMap but it will NOT delete the object. Run kubectl delete configmap/aws-auth -n kube-system to avoid issues. I believe this is a bug from AWS as of this writing: if aws-auth exists in an API access-mode EKS cluster, the aws-auth still takes precedence in some occasions (I ran into an “unauthorized” error from terraform (authenticating using get-token API)</li><li>the submodules inside EKS module, e.g. karpenter (<a href="https://github.com/terraform-aws-modules/terraform-aws-eks/tree/master/modules/karpenter">ref</a>) or eks-managed-node-group (<a href="https://github.com/terraform-aws-modules/terraform-aws-eks/tree/master/modules/eks-managed-node-group">ref</a>) should be upgraded to the matching version if you’re utilizing these modules.</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4a7e25ed6c3a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Per-pod PIDs limit on EKS]]></title>
            <link>https://awstip.com/per-pod-pids-limit-on-eks-fe320638c7e9?source=rss-3fa1b8497132------2</link>
            <guid isPermaLink="false">https://medium.com/p/fe320638c7e9</guid>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[aws-eks]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[kubelet]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Xing Du]]></dc:creator>
            <pubDate>Mon, 01 Jan 2024 11:13:15 GMT</pubDate>
            <atom:updated>2024-01-02T17:59:34.008Z</atom:updated>
            <content:encoded><![CDATA[<p>I recently set up pod PIDs limit for my EKS clusters and I’m surprised that I couldn’t find good examples/instructions online on how to do this (it’s been enabled since k8s@1.20 )</p><p>In this post, I’ll walk through how to set this up for an AWS EKS cluster using the <a href="https://github.com/terraform-aws-modules/terraform-aws-eks">official EKS terraform module</a>. Other kubelet configurations or non-EKS k8s clusters would essentially require very similar steps so I’m sure it will benefit more than just podPidsLimit on EKS.</p><h3>Why Pod PIDs Limit is necessary</h3><p>Similar to CPU/memory, PIDs (Process IDs) are fundamental resources on kubernetes nodes.</p><p>By default, kubernetes does not limit how many PIDs a pod can consume. When mistakes (e.g. application thread leak) happen, pods will exhaust PIDs from the node (in a few minutes or a few weeks). Similar to how &quot;noisy neighbor&quot; consumes CPU/Memory, it will take down all the critical daemonset pods (e.g. vpc-cni for pod networking) as well as your application pods.</p><p>If the root of the problem comes from a deployment that has many replicas, it will cause a lot of damage to your entire kubernetes cluster.</p><p>On one hand, application owners should prevent this issue from happening by paying close attention to application architecture and thread model changes of their applications. On the other hand, a kubernetes cluster should not let workloads exhaust PIDs without a limit, similar to what CPU/Memory employs LimitRange for.</p><h3>How to check your current pod PIDs limit</h3><pre>kubectl get --raw &quot;/api/v1/nodes/&lt;nodename&gt;/proxy/configz&quot; | jq &#39;.kubeletconfig.podPidsLimit&#39;</pre><h3>k8s version</h3><p>Per-pod PIDs limit is available since k8s@1.20: I&#39;ve worked on alternatives for k8s below that version but this won&#39;t be the focus of this post.</p><p>I can cover that in a different post, leave a comment if you need this.</p><h3>AMI</h3><p>using official AL2-based EKS AMI.</p><h3>OS/Node-level PIDs limit</h3><p>On AL2 this was set to a relatively low value, however, this was updated in the EKS ami project in late September 2023.</p><p>If your issue comes from node level PIDs limit being too low, consider upgrading your EKS ami or adding a step in node provisioning. Details on node PIDs limit won’t be the focus of this post.</p><h3>Options</h3><blockquote>Kubernetes allows you to limit the number of processes running in a Pod. You specify this limit at the node level, rather than configuring it as a resource limit for a particular Pod. Each Node can have a different PID limit.</blockquote><blockquote><em>(from: </em><a href="https://kubernetes.io/docs/concepts/policy/pid-limiting/"><em>reference</em></a><em>)</em></blockquote><p>2 options to implement this:</p><ul><li>--pod-max-pids argument for kubelet</li><li>PodPidsLimit in kubelet configuration file</li></ul><p>According to k8s, option 1 is being deprecated in favor of option 2. If your k8s cluster doesn&#39;t need to be upgraded for the next few years, it&#39;s fine to use the 1st option. I&#39;ll briefly cover option 1 and focus on option 2.</p><h3>Option 1: --pod-max-pids</h3><p>Using AWS EKS terraform module:</p><pre>bootstrap_extra_args = &quot;--kubelet-extra-args &#39;--pod-max-pids &lt;your_limit&gt;&#39;&quot;</pre><h3>Option 2: PodPidsLimit</h3><p>The options for implementation depends on the version of k8s.</p><h4>Before k8s@1.28</h4><p>You need to inject PodPidsLimit into the kubelet configuratio. The default configuration is located at: /etc/kubernetes/kubelet/kubelet-config.json</p><p>You can inject PodPidsLimit (with jq) into this configuration file in your pre_bootstrap_user_data:</p><pre>pre_bootstrap_user_data = &lt;&lt;-EOT<br>  echo &quot;$(jq &#39;.PodPidsLimit = &lt;your_limit&gt;&#39; /etc/kubernetes/kubelet/kubelet-config.json&quot; &gt; /etc/kubernetes/kubelet/kubelet-config.json<br>EOT</pre><p>Alternatively, you can create your own kubelet configuration file (make sure it&#39;s compatible with EKS), which includes PodPidsLimit and point kubelet to use that configuration:</p><pre>pre_bootstrap_user_data = &lt;&lt;-EOT<br>  # create your own kubelet config<br>EOT<br>bootstrap_extra_args = &quot;--kubelet-extra-args &#39;--config &lt;path_to_your_kubelet_config&gt;&#39;&quot;</pre><h4>k8s@1.28 or newer</h4><p>kubelet started accepting --config-dir argument (referred to as drop in configuration) since k8s@1.28 (with some caveats): this allows you to specify kubelet configuration overrides in a similar way to other linux system services.</p><blockquote><em>You can only set — config-dir if you set the environment variable KUBELET_CONFIG_DROPIN_DIR_ALPHA for the kubelet process (the value of that variable does not matter)</em></blockquote><p>You need to drop a .conf file for kubelet to load the KUBELET_CONFIG_DROPIN_DIR_ALPHA environment variable. kubelet-envvar.conf:</p><pre>[Service]<br>Environment=&quot;KUBELET_CONFIG_DROPIN_DIR_ALPHA=true&quot;</pre><p>You also need a kubelet configuration file (as override to the default one). kubelet-config-override.yaml:</p><pre>apiVersion: kubelet.config.k8s.io/v1beta1<br>kind: KubeletConfiguration<br>podPidsLimit: &lt;your_limit&gt;</pre><pre>cgroupDriver: &quot;systemd&quot;<br>hairpinMode: &quot;hairpin-veth&quot;<br>serializeImagePulls: false</pre><p>Note that a few other things are specified in addition to podPidsLimit: this is due to kubelet configuration v1beta1 comes with default value (e.g. <a href="https://github.com/kubernetes/kubernetes/blob/63eb5028ba613003f044f0391912be356e0f1b97/pkg/kubelet/apis/config/v1beta1/defaults.go#L152">cgroupDriver default</a>), and the default value does NOT work on EKS.</p><p>You can check kubelet config of a vanilla EKS AMI node to see what values are specified (necessary) to EKS. Alternatively, you can read the <a href="https://github.com/awslabs/amazon-eks-ami/blob/569c6b146cbf28af71485a9f30171241643174c3/files/bootstrap.sh">bootstrap.sh</a> to see which ones are necessary.</p><p>Once these 2 files are created, you need to use pre_bootstrap_user_data to put them into the right place and use bootstrap_extra_args to tell kubelet to use --config-dir:</p><pre>pre_bootstrap_user_data = &lt;&lt;-EOT<br>  cat &lt;&lt; EOF &gt; /etc/systemd/system/kubelet.service.d/kubelet-envvar.conf<br>  ${file(&quot;${path.module}/kubelet-envvar.conf&quot;)}<br>  EOF<br>  <br>  mkdir -p /etc/kubernetes/kubelet/drop-in-config<br>  <br>  cat &lt;&lt; EOF &gt; /etc/kubernetes/kubelet/drop-in-config/10-override.conf<br>  ${file(&quot;${path.module}/kubelet-config-override.yaml&quot;)}<br>  EOF<br>EOT</pre><pre>bootstrap_extra_args = &quot;--kubelet-extra-args &#39;--config-dir=/etc/kubernetes/kubelet/drop-in-config/&#39;&quot;</pre><h3>Conclusion</h3><p>Echoing what I mentioned in the beginning, the steps here can be easily adjusted for kubelet configuration other than podPidsLimit or non-EKS k8s clusters.</p><p>With multiple options, pick the right option based on your use case. The option of using drop-in configuration requires the most work but may not be the best solution for your project.</p><p>If you find this to be helpful, give it a clap and it would mean the world to me. Please share this with whoever needs this, and I’d appreciate it if you want to <a href="http://buymeacoffee.com/fixitxdu">buy me a coffee</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fe320638c7e9" width="1" height="1" alt=""><hr><p><a href="https://awstip.com/per-pod-pids-limit-on-eks-fe320638c7e9">Per-pod PIDs limit on EKS</a> was originally published in <a href="https://awstip.com/">AWS Tip</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Debug EKS Unauthenticated Error-Part II]]></title>
            <link>https://fixit-xdu.medium.com/debug-eks-unauthenticated-error-part-ii-22f3205995ca?source=rss-3fa1b8497132------2</link>
            <guid isPermaLink="false">https://medium.com/p/22f3205995ca</guid>
            <category><![CDATA[terraform-cloud]]></category>
            <category><![CDATA[aws-eks]]></category>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[troubleshooting]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Xing Du]]></dc:creator>
            <pubDate>Thu, 16 Nov 2023 23:14:11 GMT</pubDate>
            <atom:updated>2024-01-01T11:14:14.204Z</atom:updated>
            <content:encoded><![CDATA[<p>More context was detailed in my other post: <a href="https://fixit-xdu.medium.com/debug-eks-unauthenticated-error-2d21dcc8f54">part I</a></p><p>After moving my EKS provisioning project to Terraform Cloud (<a href="https://fixit-xdu.medium.com/migrate-to-terraform-cloud-part-1-46c04d32976">part I</a>, <a href="https://fixit-xdu.medium.com/migrate-to-terraform-cloud-part-2-1bb42ad14fbb">part II</a>), I was faced with another “unauthenticated” error.</p><p>The difference from last time is that <strong>it happens only sometimes</strong>.</p><h3>Troubleshooting</h3><p>The TFC agent knows about what IAM role to assume, and the desired IAM role has been whitelisted in aws-auth . With limited insights available from Terraform Cloud, I need to look deeper into my tool bag.</p><h4>A different tool</h4><p><a href="https://docs.aws.amazon.com/eks/latest/userguide/control-plane-logs.html">EKS control plane logging</a> is available on CloudWatch, and the <a href="https://registry.terraform.io/modules/terraform-aws-modules/eks/aws/latest">EKS Terraform module</a> from AWS provisions it for you by default. Specifically, you need these configurations:</p><pre># The log group name format is /aws/eks/&lt;cluster-name&gt;/cluster<br>create_cloudwatch_log_group = true # creates CW log group<br>cluster_enabled_log_types = [“audit”, “api”, “authenticator”, “controllerManager”, “scheduler”] # defaults to [&quot;audit&quot;, &quot;api&quot;, &quot;authenticator&quot;]<br>cloudwatch_log_group_retention_in_days = 180 # in days<br># use `cloudwatch_log_group_kms_key_id` if logs need to be encrypted</pre><h4>Discovery</h4><p>Go to AWS CloudWatch, select “log group”, and filter to view authenticator log streams. Narrowing down the time window to match the TFC agent apply step, I found what I was looking for:</p><blockquote>2023–11–111T11:11:11.111–08:00 time=”2023–11–11T11:11:11Z” level=warning msg=”access denied” arn=”arn:aws:iam::&lt;aws_account_id&gt;:role/my-tfc-agent-instance-profile-role” client=”127.0.0.1:37232&quot; error=”ARN is not mapped” method=POST path=/authenticate</blockquote><p>where a successful run (same workspace, same change) shows:</p><blockquote>2023–11–11T11:11:11.111–08:00 time=”2023–11–11T11:11:11Z” level=info msg=”access granted” arn=”arn:aws:iam::&lt;aws_account_id&gt;:role/my-tfc-agent-role” client=”127.0.0.1:55540&quot; groups=”[system:masters]” method=POST path=/authenticate uid=”aws-iam-authenticator:&lt;aws_account_id&gt;:&lt;my_tfc_agent_user_id&gt;” username=my-tfc-agent-username</blockquote><p>The expected part is that the instance profile isn’t being whitelisted in aws-auth , therefore getting aUnauthorized error.</p><p>The unexpected part is: <strong>why my terraform agent does not assume the specified role?</strong></p><h3>Explanation</h3><p>The following is <strong>my hypothesis</strong>.</p><p>Going through the Terraform run, I noticed that there’s no change in all aws resources (depends on aws provider, where assume_role is specified), only kubernetes resources changes.</p><p>Terraform Cloud tries to streamline runs by decoupling different steps of a run with the agents. Different steps (terraform plan , terraform apply) <strong>can be picked up by different agents</strong> from the same pool. To improve performance, the output diff of terraform plan is saved on the disk and streamed to the agent (if needed) to be fed into terraform apply .</p><p>My guess (<strong>I could be wrong, hopefully someone from Hashicorp can throw some light on this</strong>) is:</p><p>When a diff does not contain aws changes, the terraform runtime may choose to optimize by not initializing the provider. Therefore, it uses what’s in the environment variables (e.g. AWS_ACCESS_KEY_ID) and what’s on the disk (e.g. if $HOME/.aws/credentials is mounted to terraoform container at runtime) authenticate with AWS at runtime, and these may have been affected by a previous run on the same agent.</p><p>This is why it falls back to using instance profile <strong>sometimes</strong></p><h3>Solution</h3><p>I believe the problem is specific to EKS: kubernetes resources have an <strong>implicit</strong> <strong>dependency</strong> on aws provider but terraform can not figure it out due to a lack of explicit specifications.</p><h4>What I wish was possible</h4><p>Maybe there’s an existing way to achieve what these options provide, I’d love to learn if anyone can share how to do it currently. It would be really nice to have one or more of the following options (Terraform / TFC folks, please give it a read!)</p><ol><li>adding a force_init meta argument for all providers to always initialize a provider regardless of the content of the diff</li><li>adding a flag to terraform apply to always initialize a provider regardless of the content of the diff, and make it configurable on TFC.</li><li>support reuse_agent configuration on TFC to allow specifying <strong>stickiness</strong> between terraform plan step and terraform apply step</li></ol><h4>What I did</h4><p>Given the unpredictability of this issue, I end up whitelisting the instance profile on my EKS:</p><pre># in EKS module<br>aws_auth_roles = [<br> {<br> rolearn = &quot;arn:aws:iam::&lt;aws_account_id&gt;:role/my-tfc-agent-role&quot;<br> username = “my-tfc-agent”<br> groups = [“system:masters”]<br> - }<br> + },<br> + {<br> + rolearn = &quot;arn:aws:iam::&lt;aws_account_id&gt;:role/my-tfc-agent-instance-profile-role&quot;<br> + username = “my-tfc-agent-instance-profile”<br> +groups = [“system:masters”]<br> + }<br>]</pre><h3>Conclusion</h3><ol><li>Take advantage of k8s audit logs, it’s very powerful.</li><li>terraform and TFC has its limitations, but this one is very niche.</li><li>be practical and focus on what you have control over.</li></ol><p>If you find this to be helpful, give it a clap and it would mean the world to me. Please share this with whoever needs this, and I’d appreciate it if you want to <a href="http://buymeacoffee.com/fixitxdu">buy me a coffee</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=22f3205995ca" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Migrate to Terraform Cloud: part 2]]></title>
            <link>https://fixit-xdu.medium.com/migrate-to-terraform-cloud-part-2-1bb42ad14fbb?source=rss-3fa1b8497132------2</link>
            <guid isPermaLink="false">https://medium.com/p/1bb42ad14fbb</guid>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[walkthrough]]></category>
            <category><![CDATA[migration]]></category>
            <category><![CDATA[terraform-cloud]]></category>
            <dc:creator><![CDATA[Xing Du]]></dc:creator>
            <pubDate>Thu, 24 Aug 2023 01:49:26 GMT</pubDate>
            <atom:updated>2024-01-01T11:15:13.055Z</atom:updated>
            <content:encoded><![CDATA[<p>For benefits of TFC and my assessment, see <a href="https://fixit-xdu.medium.com/migrate-to-terraform-cloud-part-1-46c04d32976">part 1</a></p><h3>Migration walkthrough</h3><h3>Prerequisites</h3><ul><li>no state drifts: running terraform plan prior to migrating to TFC should give you a &quot;no changes&quot; output.</li><li>have a valid account and organization on TFC: this walkthrough won’t cover the TFC account setup.</li><li>configure TFC to access your GitHub repos / connected to your GitHub.</li><li>agents/agent pools are already set: you can use terraform to provision the agent pool, but I won’t cover that part in this post.</li><li>Generate a token (one-time setup): this can be done by running terraform login and the token is expected to be stored at ~/.terraform.d/credentials.tfrc.json.</li></ul><h3>Components</h3><h3>TFC provisioning: workspaces / projects</h3><p>Prior to migrating to TFC, you can use the same workspace names for different terraform projects(tfstates). For example, you may have a staging workspace in tf_project0 and a staging workspace in tf_project1.</p><p>On TFC: as of this is written (08/2023) TFC workspace name is unique per organization.</p><p>This means when you migrate multiple terraform projects to TFC, you need to define a TFC workspace naming convention to prevent collision. You can optionally create projects to group / organize TFC workspaces.</p><p>My recommendation is to add your terraform project’s name as a prefix to the original terraform workspace name to construct the TFC workspace name. E.g. the staging workspace for your tf_project0 should be named as tf_project0-staging on TFC.</p><p>Although this step can be done by terraform CLI (will be covered later in switching backend), I recommend doing this using <a href="https://registry.terraform.io/providers/hashicorp/tfe/latest/docs">tfe provider</a> and provision the TFC projects and workspaces using terraform.</p><p>Workspace configuration</p><p>Important configurations for TFC workspaces:</p><ul><li>tags. tags are what&#39;s used for terraform to filter/group workspaces. E.g.</li><li>on a s3 backend, if you specify workspace_key_prefix to be &quot;my_team/my_project/&quot;, you&#39;ll find all your workspaces (staging, dev, etc) listed under this directory in s3</li><li>on TFC, all workspaces that are tagged with [&quot;my_team&quot;, &quot;my_project&quot;] are considered as the workspace for your project. You can query to list all of them via terraform workspace list</li><li>execution mode:</li><li>remote: runs on instances managed by TFC. I didn&#39;t go with this approach and I guess you&#39;ll have to make some networking level whitelisting changes to grant external access.</li><li>local: still runs on your machine.</li><li>agent: runs on a pool of agents provisioned by you. This is what I find to be the best option but will save the agent provisioing piece for another post.</li><li>apply method: use “auto apply” for non-production environment and “manual apply” for production environment (enforce human review)</li><li>remote state sharing: will cover a bit later.</li></ul><h3>unpublished private modules</h3><p>If your terraform uses unpublished private modules, you need to publish these modules to TFC.</p><p>TFC can pull GitHub repo based terraform modules as long as the repo name follows the pattern: terraform-&lt;provider&gt;-&lt;module&gt;. terraform module repos are also expected to have release(s) using semver versions.</p><p>For example, tf_project0/modules/my_module0 need to be moved to a standalone GitHub repo terraform-my_provider-my_module0 and a release v1.0.0 is published.</p><p>After modules are published, change your module reference:</p><p>from:</p><pre>module &quot;my_module0&quot; {<br>  source  = &quot;../modules/my_module0&quot;<br>  ...<br>}</pre><p>to:</p><pre>module &quot;my_module0&quot; {<br>  source  = &quot;app.terraform.io/my_org/my_module0/my_provider&quot;<br>  version = &quot;1.0.0&quot;<br>  ...<br>}</pre><h3>terraform_remote_state</h3><p>TFC has an equivalent data source for terraform_remote_state: <a href="https://registry.terraform.io/providers/hashicorp/tfe/latest/docs/data-sources/outputs">tfe_outputs</a></p><p>This is assuming the remote tfstate has been migrated to TFC. If the remote tfstate is still on a remote backend (e.g. s3), you need to provide access between TFC agent (or equivalent, depending on your execution mode) and your remote backend.</p><p>In addition (if the other state is already on TFC), per-workspace configuration needs to be updated: “Remote state sharing&quot; need to include the TFC workspace for the other tfstate.</p><p>Using an example:</p><p>my_project0 depends on terraform_remote_state from my_project1:</p><pre>data &quot;terraform_remote_state&quot; &quot;my_project1&quot; {<br>  backend = &quot;s3&quot;<br>  config = {<br>    bucket = &quot;my-terraform-s3&quot;<br>    key    = &quot;my_org/my_project1/terraform.tfstate&quot;<br>  }<br>}</pre><p>after my_project1 is migrated to TFC, change the above (for my_project0) to:</p><pre>data &quot;tfe_outputs&quot; &quot;my_project1&quot; {<br>  organization = &quot;my_org&quot;<br>  # &quot;staging&quot;: my_project1-staging -&gt; my_project0-staging<br>  workspace    = [&quot;my_team&quot;,&quot;my_project1&quot;,&quot;staging&quot;]<br>}</pre><p>In addition, remote state sharing needs to be updated. This is another reason why I recommend using tfe to provision TFC workspaces.</p><p>tfe_output requires authentication, which is taken care of in the tfe provider specification. will cover a bit later.</p><h3>workspace rename</h3><p>Due to TFC limitation on workspace names, you may have to change your workspace name.</p><p>This will affect any references of ${terraform.workspace} and need to be updated to avoid unnecessary destroy + recreate operations.</p><h3>TFC authentication</h3><p>Both tfe provider and TFC cloud backend requires token authentication. Options are environment variables, variables and token from disk:</p><ul><li>environment variable: TFE_TOKEN for initializing tfe provider and TF_TOKEN_app_terraform_io (or TF_TOKEN_&lt;tfc_host&gt; initializing cloud backend.</li><li>variable: token is accepted for both tfe provider and cloud backend.</li><li>token from disk: ~/.terraform.d/credentials.tfrc.json from your local, or agent token for agents.</li></ul><p>My recommendation is to use the 3rd option. This approach can avoid creating duplicated copies of the token per TFC workspace, making token rotation much easier to operate.</p><h3>point backend to cloud</h3><p>run terraform plan and make sure &quot;no changes&quot; is in the output: your local .terraform contains a copy of the latest tfstate</p><p>Change backend from:</p><pre>backend &quot;s3&quot; {<br>  bucket = &quot;my-terraform-s3&quot;<br>  workspace_key_prefix = &quot;my_org/my_project0/&quot;<br>  key = &quot;terraform.tfstate&quot;<br>}</pre><p>to:</p><pre>cloud {<br>  organization = &quot;my_org&quot;<br>  workspaces {<br>    tags = [&quot;my_team&quot;, &quot;my_project0&quot;, &quot;dev&quot;]<br>  }<br>}</pre><p>run terraform init: you&#39;ll be asked if you want to migrate, choose yes, and another question for a prefix for your workspace. Use the same TFC naming convention and use &lt;project&gt;-* as the prefix here so your TFC workspace will look like &lt;project&gt;-&lt;original_workspace_name&gt;</p><h3>Validation</h3><ul><li>check the “states” of your TFC workspace: you should see the same tfstate files as your previous remote backend.</li><li>run terraform plan from CLI: it will trigger a new run on your TFC. If all things are taken care of, you should see a passed plan.</li></ul><h3>Conclusion</h3><p>If you find this to be helpful, give it a clap and it would mean the world to me. Please share this with whoever needs this, and I’d appreciate it if you want to <a href="http://buymeacoffee.com/fixitxdu">buy me a coffee</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1bb42ad14fbb" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Migrate to Terraform Cloud: part 1]]></title>
            <link>https://fixit-xdu.medium.com/migrate-to-terraform-cloud-part-1-46c04d32976?source=rss-3fa1b8497132------2</link>
            <guid isPermaLink="false">https://medium.com/p/46c04d32976</guid>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[migration]]></category>
            <category><![CDATA[terraform-cloud]]></category>
            <dc:creator><![CDATA[Xing Du]]></dc:creator>
            <pubDate>Wed, 23 Aug 2023 15:47:22 GMT</pubDate>
            <atom:updated>2024-01-01T11:14:51.916Z</atom:updated>
            <content:encoded><![CDATA[<p>I managed to migrate a complicated terraform project (remote backend, s3) to Terraform Cloud (refer to as TFC) last week with lots of learnings.</p><p>In this post I’ll share:</p><ul><li>why you want to use TFC / benefit of TFC</li><li>things to consider before committing to TFC</li><li>a walkthrough on how to migrate your existing terraform projects to TFC : see <a href="https://fixit-xdu.medium.com/migrate-to-terraform-cloud-part-2-1bb42ad14fbb">part 2</a></li></ul><h3>Benefits of TFC</h3><p>1. backend authoritative tfstate version</p><p>A terraform project that has a team of developers working on it but don’t have an approach to enforce the local terraform CLI version may run into a remote state being upgraded because 1 developer used a newer version. This happens very often during new hire onboarding.</p><p>The same problem can be solved by other approaches but it’s great to have this solution and don’t need to maintain any solutions on our own.</p><p>2. agent execution mode / run consistency</p><p>TFC supports configuring pools of agents to run your terraform, ensuring all the dependencies are checked in. Agent provisioning is a forcing function to explicitly call out all the previously implicit dependencies, ensuring all developers a consistent experience running terraform for projects on TFC</p><p>3. GitOps CI/CD</p><p>You can easily configure GitHub webhook/app to use TFC as CI/CD solution for your terraform projects. This solves the problem of “what happens after my PR is merged” and increases productivity.</p><p>Apply would require human user approval, similar to how terraform CLI will ask for human user input.</p><p>4. Up to per-workspace granularity permission control</p><p>You can easily define who has what permission to each workspace and project if you need fine granularity in permission control.</p><p>5. Shared variable set among different workspaces/projects</p><p>You no longer need to make copies of the same variable values across many projects/workspaces (e.g. a service account token). The concept of “variable set” provides reusability for variable inputs among different projects/workspaces. This reduces the number of copies of variables/secrets and makes rotation much easier.</p><p>6. Policy enforcement</p><p>TFC natively supports <strong>Sentinel</strong> and <strong>OPA</strong>(Open Policy Agent) and one can set up policies to enforce rules for terraform projects in your organization. The rules can be global or applied to a set of workspaces/projects for scope control. Runs that fail to comply with affected policies cannot be applied.</p><p>7. Run history: visibility and traceability</p><p>I’m not sure how long the retention is but runs from different triggers (CLI, PR merge, etc) are preserved on TFC for people with the right permission to view and inspect.</p><p>This added trackability can be extremely valuable in identifying what caused a mysterious issue on production.</p><p>8. Enable collaboration</p><p>The improved visibility of each run enables a team of developers to review your work / troubleshoot your issues</p><p>9. Module Registry</p><p>TFC hosts private modules and you can publish your private modules on TFC, and reference them in your project with version.</p><p>10. and more.</p><p>The above is a non-exhaustive list based on my own usage / assessment and I definitely missed some other reasons to prefer using TFC. Please a comment: let me know what you think and what I missed.</p><h3>Things to consider</h3><p>A couple of things I want to point out for you to consider before jumping into migrating to TFC.</p><p>1. Productivity</p><p>You can still trigger terraform plan/apply from local CLI but it will happen on TFC. One TFC agent will pick up your run and provide an output for you. Depending on the number of workspaces, the number of agents and the concurrent human users, the delay between hitting &quot;enter&quot; to a terraform run is started varies between tens of seconds to maybe a few minutes.</p><p>In addition, when VCS connection is configured (for version control), terraform apply from CLI <strong>will be rejected</strong>. You can optionally disconnect VCS connection to use TFC as a runner only to bypass this issue.</p><p>If your project is still in the stage of frequent iteration, this added overhead may be a productivity killer. When you’re just setting up something new, you may go through some refactoring and may even change which tfstate to place things, using TFC will definitely slow you down in those cases.</p><p>My recommendations:</p><ul><li>stick with a remote backend (S3) + local terraform runs until your project hierarchy and progression is ironed out.</li><li>use different configuration for different workspaces(environments) from the same project:</li><li>dev: disconnect VSC and turn on auto apply if it matters, sine this is the most frequently iterated workspace</li><li>staging: connect VSC (GitOps) and turn on auto apply</li><li>production: connect VSC (GitOps) and turn on manual apply</li></ul><p>Migrating to TFC isn’t a lot of work (if you follow my guide) and I recommend optimizing your setup for productivity.</p><p>2. Cost</p><blockquote><em>there is no such thing as a free lunch</em></blockquote><p>TFC has a free tier plan and if it fits your organization&#39;s size, great! However, for bigger organizations, you may have to pay for TFC: more on <a href="https://www.hashicorp.com/products/terraform/pricing">pricing</a></p><p>Your mileage may vary and I won’t advise on what the cost should be. Rightsize your plan commitment/usage to what you need to get a good estimate of how much you’ll pay for the benefits.</p><p>3. +1 Point of failure</p><p>TFC becomes a single point of failure. TFC SLA on availability can be found <a href="https://portal.cloud.hashicorp.com/sla">here</a> but it won’t be 100%.</p><p>Local CLI delegates to TFC so if TFC goes down, there’s no easy way to run your terraform if you have anything urgent. Keep on reading and I&#39;ll provide a non-trivial option to achieve this as a backdoor for admins.</p><h3>Conclusion</h3><p>If you find this to be helpful, give it a clap and it would mean the world to me. Please share this with whoever needs this, and I’d appreciate it if you want to <a href="http://buymeacoffee.com/fixitxdu">buy me a coffee</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=46c04d32976" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Debug EKS Unauthenticated Error]]></title>
            <link>https://fixit-xdu.medium.com/debug-eks-unauthenticated-error-2d21dcc8f54?source=rss-3fa1b8497132------2</link>
            <guid isPermaLink="false">https://medium.com/p/2d21dcc8f54</guid>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[terraform-cloud]]></category>
            <dc:creator><![CDATA[Xing Du]]></dc:creator>
            <pubDate>Tue, 22 Aug 2023 10:15:38 GMT</pubDate>
            <atom:updated>2024-01-01T11:15:30.255Z</atom:updated>
            <content:encoded><![CDATA[<p>I ran into an “unauthorized” error while migrating my EKS terraform provisioning project to terraform cloud last week.</p><p>The debugging process is somewhat interesting and I hope writing it down would help whoever runs into the same problem in the future</p><h3>Context</h3><h4>Project Setup</h4><p>A simplified version for my terraform project, which should be a fairly common setup:</p><pre>provider &quot;aws&quot; {<br>  region              = var.region<br>  allowed_account_ids = [&quot;${var.aws_account_id}&quot;]<br>  <br>  assume_role {<br>    role_arn = var.role_arn<br>  }<br>}<br><br>provider &quot;tfe&quot; {<br>  hostname = &quot;app.terraform.io&quot;<br>}<br><br>provider &quot;kubernetes&quot; {<br>  host                   = module.eks.cluster_endpoint<br>  cluster_ca_certificate = base64decode(module.eks.cluster_certificate_authority_data)<br>  token                  = data.aws_eks_cluster_auth.this.token<br>}<br><br>data &quot;aws_eks_cluster_auth&quot; &quot;this&quot; {<br>  name = module.eks.cluster_name<br>}</pre><h4>Migration</h4><ul><li><strong>prior</strong> to migrating to terraform cloud, terraform runs from local with my local AWS credential</li><li>the project contains AWS resources and k8s resources: a running EKS cluster has been provisioned by terraform already</li><li>this migration contains no further change than necessary changes that modify where terraform is executed</li><li>existing tfstate (on a s3 backend) is automatically migrated to Terraform cloud with backend change and terraform init</li><li>on Terraform cloud, the workspace is configured to use agent Execution mode: terraform runs from an agent, which is hosted on a dedicated EC2 instance</li></ul><h4>Changes</h4><p>More details on how to migrate to Terraform cloud will be covered in a different post. For context, some high-level changes:</p><ul><li>point backend to cloud</li><li>IAM role change</li><li>workspace name change</li><li>connectivity: VPC peering connection changes</li><li>Security group changes</li></ul><h3>Problem</h3><h4>Symptom</h4><p>After making the necessary changes (see above) to stop terraform from complaining, I arrived at the last error:</p><blockquote><em>Error: Unauthorized<br>with kubernetes_some_resource.my_resource<br>on my_tf.tf line xx, in resource “kubernetes_some_resource” “my_resource”:<br>resource “kubernetes_some_resource” “my_resource” {</em></blockquote><h4>Debugging</h4><p>A couple of thoughts to narrow down the investigation:</p><ul><li>This didn’t happen when using s3 backend and local terraform, so it&#39;s caused by the environmental difference.</li><li>the project contains AWS and k8s resources and this only affects k8s resources</li><li>connectivity issue would have surfaced in a tcp dial timeout instead</li></ul><p>The root cause is in how terraform runtime authorizes with EKS, i.e. related to this block:</p><pre>provider &quot;kubernetes&quot; {<br>  host                   = module.eks.cluster_endpoint<br>  cluster_ca_certificate = base64decode(module.eks.cluster_certificate_authority_data)<br>  token                  = data.aws_eks_cluster_auth.this.token<br>}<br><br>data &quot;aws_eks_cluster_auth&quot; &quot;this&quot; {<br>  name = module.eks.cluster_name<br>}</pre><p>Reasons could be (but not limited to):</p><ul><li>auth token is <strong>expired</strong> when terraform gets to k8s resources: a token is valid for <strong>15mins</strong> and it’s possible (but unlikely) if the TFC run is using a stale token from tfstate instead of issuing a new one</li><li>auth token is valid but does not have permission to access EKS</li></ul><h4>Validate token expiration</h4><p>To verify/rule out 1st guess, I:</p><ul><li>commented out the aws_eks_cluster_auth data source</li><li>used a sensitive variable to pass in the token to k8s provider</li><li>issued the token via AWS CLI (aws eks get-token --cluster-name &lt;cluster_name&gt; --role-arn &lt;tfc_agent_role&gt;) and pass in using variable</li><li>ensure this process is done within 15mins</li></ul><p>i.e:</p><pre>provider &quot;kubernetes&quot; {<br>  host                   = module.eks.cluster_endpoint<br>  cluster_ca_certificate = base64decode(module.eks.cluster_certificate_authority_data)<br>  token                  = var.eks_auth_token<br>}<br><br>var &quot;eks_auth_token&quot; {<br>  type = string<br>  sensitive = true<br>}</pre><p>This run ends up with <strong>the same error message</strong>, meaning it’s probably not related to a valid but expired token.</p><h4>Validate token permission</h4><p>To verify: we can try hitting cluster k8s API using auth token issued from AWS CLI without going through terraform:</p><ul><li>fetch cluster_endpoint from either terraform output or AWS web console</li><li>fetch the cluster_certificate_authority_data from either terraform output or AWS web console, decode it and save it locally: echo -n &#39;&lt;cluster_certificate_authority_data&gt;&#39; | base64 --decode &gt; /tmp/mycert</li><li>aws eks get-token --cluster-name &lt;cluster_name&gt; --role-arn &lt;tfc_agent_role&gt;</li><li>curl -s --cacert &lt;(cat /tmp/mycert) --header &quot;Authorization: Bearer &lt;token&gt;&quot; --request GET &#39;https://&lt;cluster_endpoint&gt;/openapi/v2&#39;</li></ul><p>This setup returned <strong>the same unauthorized error message</strong>, however, if I replace tfc_agent_role with the role I used from local runs and repeat the steps above, I get a successful response.</p><p>Confirmed that <strong>this is caused by the difference in the AWS IAM role being used</strong> and this part is not explicitly covered in my terraform source.</p><h3>Root cause</h3><p>After some research, I found <a href="https://repost.aws/knowledge-center/eks-api-server-unauthorized-error">this page</a> explaining the issue. I provisioned my EKS cluster with an OIDC provider to drive RBAC with SSO and intentionally didn’t specify aws-auth related configurations.</p><p>The IAM role used for creating the cluster will have system:master permission to k8s and won&#39;t need to be explicitly added to aws-auth config map, while other IAM roles need to be added to aws-auth explicitly for authorization.</p><p>In this case, the role I used for local runs was used during cluster creation. Therefore, despite not specifying that to aws-auth I got permission to do other k8s provisioning with this role. When migrating to TFC, the TFC agent role needs to be added to aws-auth to prevent this error. i.e.:</p><pre># in EKS module<br>aws_auth_roles = [<br>  {<br>    rolearn  = local.tfc_role_arn<br>    username = &quot;tfc-agent&quot;<br>    groups   = [&quot;system:masters&quot;]<br>  },<br>  {<br>    rolearn  = local.local_role_arn<br>    username = &quot;local-user&quot;<br>    groups   = [&quot;system:masters&quot;]<br>  }<br>]</pre><p>After adding the above block, the unauthorized error message went away and I successfully migrated this project to terraform cloud.</p><h3>Conclusion</h3><p>If you find this to be helpful, give it a clap and it would mean the world to me. Please share this with whoever needs this, and I’d appreciate it if you want to <a href="http://buymeacoffee.com/fixitxdu">buy me a coffee</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2d21dcc8f54" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>