<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Oz Lamdan on Medium]]></title>
        <description><![CDATA[Stories by Oz Lamdan on Medium]]></description>
        <link>https://medium.com/@ozlamdan?source=rss-57d35446953a------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*gxj5W7guP8lXV5GY_l04eA.jpeg</url>
            <title>Stories by Oz Lamdan on Medium</title>
            <link>https://medium.com/@ozlamdan?source=rss-57d35446953a------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 01:36:02 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@ozlamdan/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[ArgoCD as a Service: A New Approach to Client-Based ArgoCD]]></title>
            <link>https://medium.com/@ozlamdan/argocd-as-a-service-a-new-approach-to-client-based-argocd-2cff98a6bc2f?source=rss-57d35446953a------2</link>
            <guid isPermaLink="false">https://medium.com/p/2cff98a6bc2f</guid>
            <category><![CDATA[gitops]]></category>
            <category><![CDATA[paas]]></category>
            <category><![CDATA[argo-cd]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[architecture]]></category>
            <dc:creator><![CDATA[Oz Lamdan]]></dc:creator>
            <pubDate>Sat, 28 Jun 2025 18:29:23 GMT</pubDate>
            <atom:updated>2025-06-28T18:29:23.213Z</atom:updated>
            <content:encoded><![CDATA[<p>As our use of ArgoCD scaled to support multiple clients and thousands of applications, we encountered serious performance and reliability issues. This led us to rethink our entire deployment model — and ultimately, to build <strong>ArgoCD as a Service (ArgoCDaaS)</strong>: an architecture with isolated instances per client. Here’s how we got there.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DvPDRNQNVU9BsIRiLGv6mQ.jpeg" /></figure><h3>Background</h3><p>ArgoCD is a powerful platform used to deploy and manage Kubernetes applications declaratively. It can support high-scale application deployments across a large number of clusters.<br><br> In our team, we originally had a single ArgoCD instance managing over 1,600 applications across numerous clusters. As our usage grew, we began to experience performance issues:<br> - The platform became slow.<br> - The controller queues remained stuck for long periods.<br> - Client actions interfered with one another: for example, if one client deployed an application that caused the platform to hang, all other clients were affected.<br><br> We also wanted to allow our customers to deploy ApplicationSet resources. However, according to ArgoCD documentation, ApplicationSet is considered an admin-only resource - since it can deploy applications across any AppProject.<br><br> The breaking point came when a client deployed an ApplicationSet that triggered an ArgoCD bug. This completely filled the controller queue, severely affecting all users.</p><p>These challenges ultimately led us to design a new architecture for deploying ArgoCD in our environment — an approach we now call ArgoCDaaS.</p><h4>What Is ArgoCDaaS?</h4><p><strong>ArgoCDaaS (ArgoCD as a Service)</strong> is a new architecture where each client gets their own dedicated ArgoCD instance. All instances are managed centrally using GitOps, via a management ArgoCD instance that also manages itself.</p><p>By approaching the platform with a service-oriented mindset — emphasizing scalability, isolation, and maintainability — we were able to resolve many of the challenges in our original shared setup.</p><h3>Our Implementation</h3><h4>Platform Optimization</h4><p>First of all, we conducted in-depth research in order to optimize the platform. Our main findings can be found <a href="https://medium.com/@ozlamdan/argocd-optimization-guide-key-strategies-for-enhancing-your-argocd-instance-f872aa795bcb">here</a>. We implemented these optimizations across all ArgoCD instances in our new architecture.</p><h4>Disaster Recovery Mechanism</h4><p>We created a full Disaster Recovery mechanism that performs automated backup and restore of all application data. This is done using a Kubernetes CronJob that regularly runs the `argocd admin export` and `argocd admin import` commands. In case of failure, this allows us to quickly restore the entire state of ArgoCD, including applications, projects, repositories, and settings.</p><h4>Git Structure</h4><p>We designed a Git repository with two main directories:<br> - argocd/: Contains manifests needed to deploy a new optimized ArgoCD instance (ArgoCD CR, Roles, RoleBindings, ServiceMonitor, etc.)<br> - applicationset/: Contains a Helm-based deployment configuration (values.yaml) for each client and an ApplicationSet to deploy the client-specific ArgoCD instance.</p><h4>Helm-Based Automation</h4><p>We created a custom Helm chart that simplifies the deployment of new ArgoCD instances per client. The chart includes:</p><ol><li>A default values file (used for base configuration)</li><li>Individual values files per client</li><li>Templates to generate:</li></ol><ul><li>Namespace</li><li>AppProject</li><li>ArgoCD-specific RBAC</li><li>Resource whitelists \ blacklists</li><li>Cluster and namespace permissions</li><li>Monitoring components (e.g., ServiceMonitor, PrometheusRule)</li><li>DR mechanism</li></ul><p>Onboarding a new client requires only a minimal Git change to trigger instance provisioning.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8nxSA8kC1rT5O2eJvD8oKQ.png" /></figure><h4>Deployment</h4><p>Our team uses a dedicated management ArgoCD instance to deploy the entire platform stack, including all ArgoCDaaS instances and the management ArgoCD instance itself.</p><p>This way, we’ve built a fully automated, scalable, and resilient ArgoCD-based platform. Each client is isolated in their own dedicated instance, minimizing blast radius and ensuring high reliability. Built-in monitoring and alerting provide full visibility into the health of each deployment, while the Disaster Recovery mechanism ensures we can quickly recover from failure scenarios with minimal effort. The platform is easy to maintain, simplifies onboarding, and provides end-to-end GitOps-driven control — both for our team and for the clients we support.</p><h4>Advantages of ArgoCDaaS</h4><p>1. Client-Specific Restricted Permissions<br> Permissions are configured per ArgoCD instance, not just per AppProject, allowing for more fine-grained access control.<br><br> 2. Full Isolation Between Clients<br> In shared platforms, one bad deployment can block the entire system. With ArgoCDaaS, clients cannot affect one another. Troubleshooting is easier and faster.<br><br> 3. Advanced Monitoring<br> We created Grafana dashboards with templated variables to monitor each ArgoCD instance independently. Alerts are scoped per-client, reducing noise.<br><br> 4. Client-Centric Methodology<br> This architecture fosters a product-oriented mindset, delivering a robust, reliable, and tailored deployment experience for each client.</p><h4>Disadvantages</h4><p>1. Higher Resource Utilization<br> Running multiple ArgoCD instances naturally consumes more compute and memory. Usage scales with the number of clusters and apps per client.<br><br> 2. Increased Cost<br> More resource consumption means higher infrastructure costs.<br><br> 3. More Instances to Manage<br> Operational overhead increases with the number of ArgoCD instances. You&#39;ll need processes to manage updates, versioning, and consistency across them.</p><h3>Summary</h3><p>ArgoCDaaS provides a scalable, client-focused solution to the limitations of a shared ArgoCD model. By isolating clients at the ArgoCD instance level, we gain improved security, reliability, and flexibility—at the cost of increased operational complexity and resources.</p><p>We encourage teams facing scaling challenges with ArgoCD to consider this approach. Our GitOps repository and Helm charts are available for customization, and we’re happy to share insights or support adoption.</p><p>Many thanks to Romy Kessler for her tremendous help with this project!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2cff98a6bc2f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ArgoCD Optimization Guide: Key Strategies for Enhancing Your ArgoCD Instance]]></title>
            <link>https://medium.com/@ozlamdan/argocd-optimization-guide-key-strategies-for-enhancing-your-argocd-instance-f872aa795bcb?source=rss-57d35446953a------2</link>
            <guid isPermaLink="false">https://medium.com/p/f872aa795bcb</guid>
            <category><![CDATA[security]]></category>
            <category><![CDATA[gui̇de]]></category>
            <category><![CDATA[optimization]]></category>
            <category><![CDATA[argo-cd]]></category>
            <dc:creator><![CDATA[Oz Lamdan]]></dc:creator>
            <pubDate>Tue, 21 Jan 2025 11:38:05 GMT</pubDate>
            <atom:updated>2025-01-21T11:38:05.750Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vikwYnYgkJpSOYHfQFUbMA.png" /></figure><p>ArgoCD is one of the most popular platforms for implementing GitOps. It allows you to easily create applications and deploy multiple Kubernetes manifests at once, monitor their status, automatically synchronize applications to a Git repository and more.</p><p>At my work, we deliver managed ArgoCD instances to our customers. We operate in a high-scale environment, serving a diverse range of clients, managing thousands of applications. Through extensive research and experience, we’ve identified key metrics important for optimizing the ArgoCD platform.</p><p>This article is designed for platform engineers responsible for deploying and maintaining ArgoCD. While we’ll briefly introduce its components, the primary focus is on strategies to optimize and enhance the platform’s overall performance. Note that these techniques are based on our long-term experience delivering ArgoCD as a service, rather than standardized best practices. For official best practices, refer to<a href="https://argo-cd.readthedocs.io/en/stable/user-guide/best_practices/"> ArgoCD documentation</a>.</p><h3><strong>ArgoCD Components</strong></h3><p>ArgoCD consists of the following components:</p><ul><li>Application Controller</li><li>ApplicationSet Controller</li><li>Repo Server</li><li>Redis</li><li>Server</li><li>Dex Server</li></ul><p>Each component functions as a microservice deployed in a separate pod, with distinct responsibilities. Below are our optimization recommendations for each component:</p><h4><strong>ArgoCD Application Controller:</strong></h4><p>ArgoCD application controller (or just ArgoCD controller) is an important component of ArgoCD. It is responsible for interacting with edge clusters, retrieving their status, and synchronizing applications.</p><p><strong>Key optimizations:</strong></p><ul><li>Client and Burst QPS: Increase Client QPS and Burst QPS to enhance performance and ensure smoother operations.¹ ²</li><li>Change status and operation processors: Adjust the number of status and operation processors. Increasing these values speeds up operations but requires additional resources. The<a href="https://argo-cd.readthedocs.io/"> official documentation</a> suggests increasing this number for instances with over 1,000 applications.³</li><li>Sharding algorithm: Sharding determines how clusters are distributed among controller replicas. Choosing an optimal sharding strategy tailored to your use case is required for platform efficiency.⁴</li><li>Allocate Enough Memory: The Application Controller is susceptible to OOM kills if not allocated sufficient memory. When deploying a new ArgoCD instance, it’s essential to allocate resources thoughtfully to ensure stability and performance. We recommend provisioning a minimum of <strong>300MB of memory per cluster</strong> connected to a single application. However, if your setup involves managing multiple applications, you’ll need to allocate additional resources to accommodate the increased load. Proper memory planning can help prevent disruptions and ensure the smooth operation of your ArgoCD environment.</li><li>Configure timeouts: By default, the controller has no timeout set, which can lead to operations, such as application synchronization, getting stuck indefinitely. Setting timeouts (even high timeouts) prevents these issues. from the ArgoCD docs⁵:</li></ul><blockquote>The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don’t timeout requests. (default “0”)</blockquote><p><strong>Essential Notes:</strong></p><ul><li>Queue not empty: If the queue stays uncleared (not reaching to 0 once in a while), roll the controller pods where operations are stuck. This clears the queue and restores normal functionality.</li><li>Sharding is cluster based: Sharding in ArgoCD operates at the cluster level. If applications are deployed unevenly across clusters, some controller shards may experience higher resource consumption than others.</li><li>Controller-Cluster Communication: The Application Controller begins communication with a cluster only after the first application is deployed on that cluster.</li></ul><h4><strong>ArgoCD Repo Server:</strong></h4><p>The Repo Server handles Git repository interactions, such as cloning, syncing, and manifest generation (e.g., via Helm or Kustomize). It also caches manifests, by default for 24 hours, in a dedicated Redis server.</p><p><strong>Key optimizations:</strong></p><ul><li>Configuring Parallelism Limit: By default, there’s no limit on parallel operations, which can overwhelm the server. Setting a parallelism limit can smooth operations and prevent OOM kills . Use the <em>–parallelismlimit</em> flag to configure it.⁶ ⁷</li></ul><h4><strong>ArgoCD Server:</strong></h4><p>The server is the microservice that runs the ArgoCD API. It is also responsible for the UI.</p><p><strong>Key optimizations:</strong></p><ul><li>ArgoCD’s Server High Availability: Increase server replicas to three or more for zero downtime during updates.⁸</li></ul><p><strong>Essential Notes:</strong></p><ul><li>Change ArgoCD’s URL: After changing the server URL, roll the server pods to avoid errors.</li></ul><h3>Security &amp; Permissions:</h3><p>Permissions in ArgoCD can be configured at three levels: namespace, instance (application), and edge cluster. The most secure configuration involves setting permissions at all three levels.</p><h4><strong>Edge Clusters Level:</strong></h4><p>At the edge cluster level, assign only the <strong>minimum required permissions</strong> to the service account connecting the edge cluster to ArgoCD. Use an RBAC configuration that restricts access to only the necessary operations and resources to reduce risk.</p><h4><strong>ArgoCD Namespace Level</strong></h4><p>In the ArgoCD namespace, enforce restrictions using Limits &amp; Requests and ResourceQuotas to prevent excessive resource consumption. Additionally, avoid granting clients view or edit permissions within the ArgoCD namespace. Doing so can expose your configuration to unauthorized changes and grant higher permissions than needed.</p><h4><strong>Instance Level</strong></h4><p>For instance-level permissions, leverage AppProjects, RBAC, and resource whitelists/blacklists for fine-grained control. Each client should have its own AppProject, ensuring isolation. Configure RBAC rules for each AppProject to grant specific permissions. Use the clusterResourceWhitelist/Blacklist fields in AppProjects to restrict which Kubernetes resources can be deployed, enhancing security by limiting the scope of resource creation.</p><h3><strong>ApplicationSets:</strong></h3><p>ApplicationSet is a useful tool for generating multiple applications, across one or more clusters. here are some important considerations about ApplicationSets:</p><h4><strong>Administration</strong></h4><p>ApplicationSet is an administrative tool. by the ArgoCD documentation⁹:</p><blockquote><em>“</em><strong><em>only admins</em></strong><em> may be given permission (via Kubernetes RBAC or any other mechanism) to create, update, or delete ApplicationSets.”</em></blockquote><p>This is because ApplicationSets can generate applications across all projects, create and delete multiple applications simultaneously, and expose sensitive information, such as Secrets in the ArgoCD namespace. For clients, we recommend in most cases blocking the ability to deploy application sets (unless in specific circumstances) and using the app-of-apps pattern.¹⁰</p><h4><strong>Preserve Resources On Deletion:</strong></h4><p>Add preserveResourcesOnDeletion to your ApplicationSets in order to deny deletion of child-applications, while deleting the ArgoCD applicationSet.</p><h3><strong>Monitoring:</strong></h3><p>Use the free and deep dive grafana dashboard to monitor your ArgoCD instance.</p><p><a href="https://github.com/argoproj/argo-cd/blob/master/examples/dashboard.json">argo-cd/examples/dashboard.json at master · argoproj/argo-cd</a></p><p>By implementing these optimization strategies, you can enhance the performance, reliability, and security of your ArgoCD instance, ensuring a smoother GitOps workflow for your team and your ArgoCD clients.</p><h4>References:</h4><p>¹<em> </em><a href="https://aws.amazon.com/blogs/opensource/argo-cd-application-controller-scalability-testing-on-amazon-eks/">https://aws.amazon.com/blogs/opensource/argo-cd-application-controller-scalability-testing-on-amazon-eks/</a></p><p>² <a href="https://cnoe.io/blog/argo-cd-application-scalability">https://cnoe.io/blog/argo-cd-application-scalability</a></p><p>³ <a href="https://argo-cd.readthedocs.io/en/stable/operator-manual/high_availability/#argocd-application-controller">https://argo-cd.readthedocs.io/en/stable/operator-manual/high_availability/#argocd-application-controller</a></p><p>⁴ <a href="https://cnoe.io/blog/argo-cd-application-scalability#experiment-6-sharding-deep-dive">https://cnoe.io/blog/argo-cd-application-scalability#experiment-6-sharding-deep-dive</a></p><p>⁵ <a href="https://argo-cd.readthedocs.io/en/stable/operator-manual/server-commands/argocd-application-controller/">https://argo-cd.readthedocs.io/en/stable/operator-manual/server-commands/argocd-application-controller/</a></p><p>⁶ <a href="https://argo-cd.readthedocs.io/en/stable/operator-manual/high_availability/#argocd-repo-server">https://argo-cd.readthedocs.io/en/stable/operator-manual/high_availability/#argocd-repo-server</a></p><p>⁷ <a href="https://medium.com/@keska.damian/optimizing-argocd-repo-server-to-work-with-kustomize-in-monorepo-974c3a97334d">https://medium.com/@keska.damian/optimizing-argocd-repo-server-to-work-with-kustomize-in-monorepo-974c3a97334d</a></p><p>⁸ <a href="https://argo-cd.readthedocs.io/en/stable/operator-manual/high_availability/#argocd-server">https://argo-cd.readthedocs.io/en/stable/operator-manual/high_availability/#argocd-server</a></p><p>⁹ <a href="https://argo-cd.readthedocs.io/en/stable/operator-manual/applicationset/Security/">https://argo-cd.readthedocs.io/en/stable/operator-manual/applicationset/Security/</a></p><p>¹⁰ <a href="https://argo-cd.readthedocs.io/en/stable/operator-manual/declarative-setup/#app-of-apps">https://argo-cd.readthedocs.io/en/stable/operator-manual/declarative-setup/#app-of-apps</a></p><h4><strong>Special Thanks</strong></h4><p>A special thanks to Neev Geffen and Romy Kessler for their valuable contributions and support in writing this article.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f872aa795bcb" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Using Thanos Receive Controller to Solve Common Thanos Receive Problems]]></title>
            <link>https://medium.com/@ozlamdan/using-thanos-receive-controller-to-solve-common-thanos-receive-problems-3bccdd73e06f?source=rss-57d35446953a------2</link>
            <guid isPermaLink="false">https://medium.com/p/3bccdd73e06f</guid>
            <category><![CDATA[receiver]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[monitoring]]></category>
            <category><![CDATA[thanos]]></category>
            <category><![CDATA[features]]></category>
            <dc:creator><![CDATA[Oz Lamdan]]></dc:creator>
            <pubDate>Tue, 05 Nov 2024 14:59:31 GMT</pubDate>
            <atom:updated>2024-11-05T15:00:13.802Z</atom:updated>
            <content:encoded><![CDATA[<p>As a Thanos user, I encountered challenges while setting up the Thanos Receive Controller, primarily due to the limited documentation available. In this article, I’ll share the steps and insights I gained to help streamline the process for others looking to implement this component.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*LaKRQWlbE4OoghIGh2Zlag.jpeg" /></figure><h3><strong>Problems With Thanos Receive</strong></h3><p>The Thanos Receive component uses a hashrings.json ConfigMap to determine which receivers are available and part of the same hashring. Communication between receivers occurs only when they belong to the same hashring. However, this setup presents several issues:</p><p>1. <strong>Downtime Vulnerability</strong>: If a receiver is down for any reason (for example node not ready), all of the receivers in the same hashring will stop writing and saving metrics. Continuously, it can cause data loss.</p><p>2. <strong>Static Configuration</strong>: The hashrings.json config map is static, meaning that it does not change based on the active receivers — it only changes by manually editing the ConfigMap.</p><p>3. <strong>No Automatic Reload:</strong> Even if the ConfigMap is updated, the receivers do not automatically reload it, meaning that they still have the old configuration file and communicate with the old hashring. The hashrings file is reloaded when the pod restarts.</p><p>The Thanos Receive Controller should solve all of the problems mentioned above.</p><h3><strong>What is Thanos Receive Controller?</strong></h3><p>Thanos Receive controller is a component intended for updating the hashrings if the receivers change. The component watches the receivers and when a change in their availability occurs, it automatically updates the hashrings ConfigMap. It also can automatically reload it to the receivers. Thus, Thanos Receive Controller is the solution to all the receivers problem mentioned before.</p><h3><strong>How to Implement Thanos Receive Controller</strong></h3><p>The following guide explains how to deploy Thanos Receive Controller on kubernetes.</p><p><strong>Step 1: Create an Initial ConfigMap</strong></p><p>Begin by defining an initial mapping of tenants to hashrings in a ConfigMap. The controller will use this as a template:</p><pre>apiVersion: v1<br>kind: ConfigMap<br>metadata:<br>  name: thanos-receive<br>  namespace: thanos<br>  labels:<br>    app.kubernetes.io/name: thanos-receive<br>data:<br>  hashrings.json: |<br>    [<br>        {<br>            &quot;hashring&quot;: &quot;default&quot;,<br>            &quot;tenants&quot;: [<br>    ]<br>        }<br>    ]</pre><p>This, used by the controller as a template — and the controller adds the active receivers to the hashrings.</p><p><strong>Step 2: Create Role, Role Binding And ServiceAccount</strong></p><p>To configure the necessary permissions for the Thanos Receive Controller, deploy a Role, RoleBinding, and ServiceAccount.</p><pre>apiVersion: rbac.authorization.k8s.io/v1<br>kind: Role<br>metadata:<br>  labels:<br>    app.kubernetes.io/name: thanos-receive-controller<br>  name: thanos-receive-controller<br>  namespace: thanos<br>rules:<br>- apiGroups:<br>  - &quot;&quot;<br>  resources:<br>  - configmaps<br>  verbs:<br>  - list<br>  - watch<br>  - get<br>  - create<br>  - update<br>  - delete<br>- apiGroups:<br>  - &quot;&quot;<br>  resources:<br>  - pods<br>  verbs:<br>  - list<br>  - get<br>  - update<br>- apiGroups:<br>  - apps<br>  resources:<br>  - statefulsets<br>  verbs:<br>  - list<br>  - watch<br>  - get</pre><pre>apiVersion: rbac.authorization.k8s.io/v1<br>kind: RoleBinding<br>metadata:<br>  labels:<br>    app.kubernetes.io/name: thanos-receive-controller<br>  name: thanos-receive-controller<br>  namespace: thanos<br>roleRef:<br>  apiGroup: rbac.authorization.k8s.io<br>  kind: Role<br>  name: thanos-receive-controller<br>subjects:<br>- kind: ServiceAccount<br>  name: thanos-receive-controller<br>  namespace: thanos</pre><pre>apiVersion: v1<br>kind: Service<br>metadata:<br>  labels:<br>    app.kubernetes.io/name: thanos-receive-controller<br>  name: thanos-receive-controller<br>  namespace: thanos<br>spec:<br>  ports:<br>  - name: http<br>    port: 8080<br>    targetPort: 8080<br>  selector:<br>    app.kubernetes.io/name: thanos-receive-controller</pre><p><strong>Step 3: Deploy the Thanos Receive Controller</strong></p><p>Next, deploy the Receive Controller with the following configuration:</p><pre>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  name: thanos-receive-controller<br>  namespace: thanos<br>  labels:<br>    app.kubernetes.io/name: thanos-receive-controller<br>spec:<br>  replicas: 1<br>  selector:<br>    matchLabels:<br>      app.kubernetes.io/name: thanos-receive-controller<br>  template:<br>    metadata:<br>      labels:<br>        app.kubernetes.io/name: thanos-receive-controller<br>    spec:<br>      containers:<br>      - args:<br>        - --configmap-name=thanos-receive<br>        - --configmap-generated-name=thanos-receive-generated<br>        - --file-name=hashrings.json<br>        - --allow-only-ready-replicas<br>        - --allow-dynamic-scaling<br>        - --annotate-pods-on-change<br>        image: quay.io/observatorium/thanos-receive-controller<br>        name: thanos-receive-controller<br>        env:<br>        - name: NAMESPACE<br>          valueFrom:<br>            fieldRef:<br>              fieldPath: metadata.namespace<br>        ports:<br>        - containerPort: 8080</pre><p><strong>Flag Explanations:</strong></p><p>● — configmap-name: Specifies the name of the ConfigMap containing the hashrings file template.</p><p>● — configmap-generated-name: Defines the name of the ConfigMap that the controller generates.</p><p>● — file-name: Specifies the name of the file in the ConfigMap managed by the controller.</p><p>● — allow-only-ready-replicas: Adds only Ready receiver replicas to the hashrings file.</p><p>● — allow-dynamic-scaling: Updates the hashrings file during scale-down events.</p><p>● — annotate-pods-on-change: Adds an annotation to receivers when the controller updates the hashrings file, triggering them to reload the hashrings ConfigMap.</p><p>The last 3 flags are not mandatory, but they improve the Receive Controller abilities.</p><p>In addition, deploy the following Service to support the controller:</p><pre>apiVersion: v1<br>kind: Service<br>metadata:<br>  labels:<br>    app.kubernetes.io/name: thanos-receive-controller<br>  name: thanos-receive-controller<br>  namespace: thanos<br>spec:<br>  ports:<br>  - name: http<br>    port: 8080<br>    targetPort: 8080<br>  selector:<br>    app.kubernetes.io/name: thanos-receive-controller</pre><p><strong>Step 4: Configure Receiver Labels</strong></p><p>Finally, add the following label controller.receive.thanos.io=thanos-receive-controller to the StatefulSet of the receivers under .spec.template.metadata.labels . This label tells the controller which receivers it manages. You also need to map the controller-generated hashrings ConfigMap into the receivers as the hashrings ConfigMap ( in this example — thanos-receive-generated).</p><h4>That’s it! You successfully implemented the Thanos Receive Controller!</h4><p><strong>Appendix: List of Thanos Receive Controller Flags</strong></p><p>It was hard to find the flags of the receive controller. Finally I managed to do it by adding a wrong flag and then all the existing flags appeared in the receiver controller’s logs.</p><p>The flags are:</p><pre>-allow-dynamic-scaling<br>Update the hashring configuration on scale down events.<br>-allow-only-ready-replicas<br>Populate only Ready receiver replicas in the hashring configuration<br>-annotate-pods-on-change<br>Annotates pods with current timestamp on a hashring change<br>-cluster-domain string<br>The DNS domain of the cluster (default &quot;cluster.local&quot;)<br>-configmap-generated-name string<br>The name of the generated and populated ConfigMap<br>-configmap-name string<br>The name of the original ConfigMap containing the hashring tenant configuration<br>-file-name string<br>The name of the configuration file in the ConfigMap<br>-internal-addr string<br>The address on which internal server runs (default &quot;:8080&quot;)<br>-kubeconfig string<br>Path to kubeconfig<br>-label string<br>The label workloads must have to be watched by the controller. (default &quot;controller.receive.thanos.io=thanos-receive-controller&quot;)<br>-namespace string<br>The namespace to watch (default &quot;default&quot;)<br>-pod-az-annotation-key string<br>pod annotation key for AZ Info, If not specified or key not found, will use sts name as AZ key<br>-port int<br>The port on which receive components are listening for write requests (default 10901)<br>-scale-timeout duration<br>A timeout to wait for receivers to really start after they report healthy (default 5s)<br>-scheme string<br>The URL scheme on which receive components accept write requests (default &quot;http&quot;)<br>-statefulset-label string<br>[DEPRECATED] The label StatefulSets must have to be watched by the controller<br>-use-az-aware-hashring<br>A boolean to use az aware hashring to comply with Thanos v0.32+</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3bccdd73e06f" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>