<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Moran Weissman on Medium]]></title>
        <description><![CDATA[Stories by Moran Weissman on Medium]]></description>
        <link>https://medium.com/@moran.weissman?source=rss-fe44884b7c71------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*u-qnCkYoD6yqFtrx6V6uiw.jpeg</url>
            <title>Stories by Moran Weissman on Medium</title>
            <link>https://medium.com/@moran.weissman?source=rss-fe44884b7c71------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 22:43:51 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@moran.weissman/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Multi-Cluster Addon Management with ArgoCD: A GitOps Approach — Part 3]]></title>
            <link>https://medium.com/@moran.weissman/multi-cluster-addon-management-with-argocd-a-gitops-approach-part-3-b325df01b947?source=rss-fe44884b7c71------2</link>
            <guid isPermaLink="false">https://medium.com/p/b325df01b947</guid>
            <dc:creator><![CDATA[Moran Weissman]]></dc:creator>
            <pubDate>Sun, 26 Jan 2025 00:35:06 GMT</pubDate>
            <atom:updated>2025-01-26T00:35:06.193Z</atom:updated>
            <content:encoded><![CDATA[<h3>Multi-Cluster Addon Management with ArgoCD: A GitOps Approach — Part 3</h3><p>“In Parts 1 and 2, we explored the challenges of multi-cluster management and dove deep into the architecture of our solution. Now, let’s roll up our sleeves and walk through the practical implementation steps, common challenges you might face, and battle-tested solutions from real-world usage.”</p><h3>Setting Up Your Foundation</h3><p>Before we begin implementing the solution, let’s understand what we need to have in place. Think of this like preparing your kitchen before cooking a complex meal — having everything ready makes the process smoother.</p><h3>Step 1: ArgoCD Configuration</h3><p>Let’s start with configuring ArgoCD to support our addon management system. One of the most crucial aspects is setting up the Vault Plugin correctly. Think of this as installing the right tools in your workshop — they need to be configured properly to work effectively.</p><pre>server:<br>  config:<br>    configManagementPlugins: |<br>      - name: argocd-vault-plugin-helm<br>        init:<br>          command: [&quot;/bin/sh&quot;, &quot;-c&quot;]<br>          args: [&quot;helm dependency build&quot;]<br>        generate:<br>          command: [&quot;sh&quot;, &quot;-c&quot;]<br>          args: [&quot;helm template $ARGOCD_APP_NAME ${HELM_VALUES} . | argocd-vault-plugin generate -&quot;]</pre><p>This configuration tells ArgoCD how to process our Helm charts and inject secrets during deployment. Notice how we combine Helm templating with the Vault Plugin — this allows us to manage both configuration and secrets in a single flow.</p><h3>Step 2: Setting Up AWS IAM Roles</h3><p>Security is like the foundation of a house — it needs to be solid before you build anything on top. Let’s set up the necessary AWS IAM roles:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cCc0veB0ProC_Mac3BR6Ew.png" /></figure><p>The IAM policy should look like this:</p><pre>{<br>    &quot;Version&quot;: &quot;2012-10-17&quot;,<br>    &quot;Statement&quot;: [<br>        {<br>            &quot;Effect&quot;: &quot;Allow&quot;,<br>            &quot;Action&quot;: [<br>                &quot;secretsmanager:GetSecretValue&quot;,<br>                &quot;secretsmanager:DescribeSecret&quot;<br>            ],<br>            &quot;Resource&quot;: [<br>                &quot;arn:aws:secretsmanager:*:*:secret:datadog-api-keys-integration*&quot;,<br>                &quot;arn:aws:secretsmanager:*:*:secret:cluster-*&quot;<br>            ]<br>        }<br>    ]<br>}</pre><p>Notice how we’re following the principle of least privilege — the role has access only to the specific secrets it needs.</p><h3>Step 3: Implementing the Directory Structure</h3><p>Organization is key when managing multiple clusters and addons. Let’s look at how to structure your repository for maximum clarity and maintainability. Think of this like organizing a library — everything needs a logical place:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aE168XD_j-popJFuBdZ8VA.png" /></figure><h3>Step 4: Configuring Your First Addon</h3><p>Let’s walk through adding a new addon to your system. We’ll use Datadog as an example since it’s a common use case with interesting complexity. This process involves several steps:</p><p>Define the addon in your addons-list.yaml:</p><pre>applicationsets:<br>  - appName: datadog<br>    repoURL: https://helm.datadoghq.com<br>    chart: datadog<br>    environments:<br>      - env: dev<br>        version: 3.25.3</pre><p>Create cluster-specific configurations:</p><pre># values/addons-values/clusters/cluster-1/datadog.yaml<br>datadog:<br>  containerIncludeLogs: &#39;kube_namespace:cluster-1&#39;</pre><p>Set up the necessary secrets:</p><pre>apiVersion: v1<br>kind: Secret<br>metadata:<br>  name: datadog-tags<br>  annotations:<br>    avp.kubernetes.io/path: {{ .Values.clusterName }}</pre><p>Notice how each piece fits together like a puzzle, with clear separation between configuration, secrets, and deployment logic.</p><h3>Advanced Configuration Patterns: Building Flexibility into Your System</h3><p>When managing multiple clusters, flexibility in configuration becomes crucial. Let’s explore some sophisticated patterns that make our system adaptable and maintainable.</p><h4>The Power of Configuration Layering</h4><p>Think of configuration layering like creating a custom sandwich — you start with a base layer and add specific ingredients for different tastes. In our solution, we’ve implemented a sophisticated layering system that provides both consistency and flexibility.</p><p>Consider how we implement this for the Datadog addon. Each layer serves a specific purpose:</p><pre># 1. Global defaults (values/addons-values/defaults.yaml)<br>datadog:<br>  logLevel: INFO<br>  logs:<br>    enabled: true<br>    containerCollectAll: true<br><br># 2. Environment-specific settings (values/addons-list.yaml)<br>environments:<br>  - env: dev<br>    version: 3.25.3<br>    valuesObject:<br>      datadog:<br>        env:<br>          - name: DD_ENV<br>            value: development<br><br># 3. Cluster-specific overrides (values/addons-values/clusters/cluster-1/datadog.yaml)<br>datadog:<br>  containerIncludeLogs: &#39;kube_namespace:cluster-1&#39;</pre><p>This layering system provides several benefits:</p><ul><li>Common configurations stay consistent across all clusters</li><li>Environment-specific needs are easily accommodated</li><li>Individual clusters can be customized without affecting others</li><li>Changes can be made at the appropriate scope</li></ul><h3>Troubleshooting Common Issues: A Systematic Approach</h3><p>When managing multiple clusters, issues can arise in various places. Let’s explore a systematic approach to troubleshooting, organized by the common categories of problems you might encounter.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KXX_wXlSIjjn9bMwdEIZ0Q.png" /></figure><p>Let’s examine each category:</p><h4>Sync Failures</h4><p>When ArgoCD fails to sync an application, follow these steps:</p><p>Check ArgoCD Application status:</p><pre>argocd app get addon-name-cluster-1 -o yaml</pre><p>Verify the Vault Plugin is working:</p><pre># Check AVP logs in the argocd-repo-server pod<br>kubectl logs -n argocd -l app.kubernetes.io/name=argocd-repo-server -c vault-plugin</pre><p>Examine the Git repository state:</p><pre># Verify the manifests locally<br>helm template . -f values.yaml | argocd-vault-plugin generate -</pre><h4>Secret Management Issues</h4><p>When dealing with secret-related problems:</p><p>Verify External Secrets Operator status:</p><pre># Check ESO logs<br>kubectl logs -n external-secrets -l app.kubernetes.io/name=external-secrets<br><br># Check ExternalSecret status<br>kubectl get externalsecret -n target-namespace my-secret -o yaml</pre><p>Validate AWS IAM permissions:</p><pre># Test AWS credentials<br>aws secretsmanager get-secret-value --secret-id test-secret</pre><h3>Day-2 Operations: Maintaining a Healthy System</h3><h4>Version Management Strategy</h4><p>Implementing a robust version management strategy is crucial. Consider this approach for updating add-ons:</p><ol><li>Test in development clusters first</li><li>Use cluster labels for gradual rollout</li><li>Maintain version compatibility matrix</li></ol><p>For example:</p><pre>clusters:<br>  - name: cluster-dev<br>    labels:<br>      env: dev<br>      datadog-version: &quot;3.70.7&quot;  # Testing new version<br>  - name: cluster-prod<br>    labels:<br>      env: prod<br>      datadog-version: &quot;3.25.3&quot;  # Stable version</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b325df01b947" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Multi-Cluster Add-on Management with ArgoCD: A GitOps Approach — Part 2]]></title>
            <link>https://medium.com/@moran.weissman/multi-cluster-add-on-management-with-argocd-a-gitops-approach-part-2-aeb6c3908c6d?source=rss-fe44884b7c71------2</link>
            <guid isPermaLink="false">https://medium.com/p/aeb6c3908c6d</guid>
            <dc:creator><![CDATA[Moran Weissman]]></dc:creator>
            <pubDate>Sat, 25 Jan 2025 23:53:07 GMT</pubDate>
            <atom:updated>2025-01-25T23:53:07.356Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>Multi-Cluster Add-on Management with ArgoCD: A GitOps Approach — Part 2</strong></h3><p>In Part 1, we explored the challenges of managing add-ons across multiple Kubernetes clusters. Now, let’s dive deep into the architecture that makes this solution powerful and scalable. We’ll explore how the combination of ArgoCD’s App of Apps pattern, ApplicationSets, and secure secret management creates a robust system for managing cluster add-ons.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*15rpHBvSZ5-eLiIzANgLuA.png" /></figure><p>Now, let’s break down each major component of the architecture:</p><h3>The App of Apps Pattern: A Foundation for Scale</h3><p>The App of Apps pattern forms the backbone of our solution. At its core, we have a root application that manages all other applications and configurations. Let’s look at the key components of the implementation:</p><p>This pattern provides several benefits:</p><ol><li>Centralized management of all cluster configurations</li><li>Consistent deployment across environments</li><li>Single point of truth for the entire system</li></ol><h3>Dynamic Management with ApplicationSets</h3><p>ApplicationSets are where the magic happens. They enable dynamic application generation based on cluster metadata.</p><pre>apiVersion: argoproj.io/v1alpha1<br>kind: ApplicationSet<br>metadata:<br>  name: {{ $appset.appName }}-{{ $environment.env }}<br>spec:<br>  generators:<br>    - clusters:<br>        selector:<br>          matchLabels:<br>            env: &#39;{{ $environment.env }}&#39;<br>            {{ $appset.appName }}: enabled</pre><p>This configuration demonstrates several advanced patterns:</p><ol><li>Environment-based deployment control</li><li>Feature flag system through labels</li><li>Version management per cluster</li></ol><h3>Secure Secret Management</h3><p>This solution implements a sophisticated secret management system combining:</p><ol><li>External Secrets Operator (ESO)</li><li>AWS Secrets Manager</li><li>ArgoCD Vault Plugin (AVP)</li></ol><h3>Flexible Configuration Management</h3><p>One of the most impressive aspects of your solution is the elegant handling of cluster-specific configurations. The system implements a layered approach:</p><ol><li>Default configurations (values/addons-values/defaults.yaml)</li><li>Environment-specific values (from addons-list.yaml)</li><li>Cluster-specific overrides (values/addons-values/clusters/&lt;cluster&gt;/)</li></ol><p>This approach provides:</p><ul><li>Sensible defaults for all clusters</li><li>Environment-specific customizations</li><li>Granular control when needed</li><li>Clear configuration hierarchy</li></ul><h3>Cluster-Specific Configuration: A Deep Dive</h3><p>One of the most elegant aspects of this solution is how it handles cluster-specific configurations. Let’s visualize this with a concrete example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/619/1*jZy9N6wStOwipqnH5Os6ag.png" /></figure><p>Consider how this plays out with Datadog configuration. The system allows you to define baseline monitoring settings while enabling per-cluster customization. For example:</p><pre># Default configuration (values/addons-values/defaults.yaml)<br>datadog:<br>  logLevel: INFO<br>  logs:<br>    enabled: &#39;true&#39;<br>    containerCollectAll: &#39;true&#39;<br><br># Cluster-specific override (values/addons-values/clusters/cluster-1/datadog.yaml)<br>datadog:<br>  containerIncludeLogs: &#39;kube_namespace:cluster-1&#39;</pre><p>This layered approach ensures consistent baseline configuration while allowing for cluster-specific requirements. It’s particularly powerful when managing clusters across different environments or with different requirements.</p><h3>Version Management: Flexibility Meets Control</h3><p>The solution implements a sophisticated version management system that provides flexibility without sacrificing control. This is implemented through a combination of ApplicationSet configuration and cluster labels:</p><pre># In values/clusters.yaml<br>clusters:<br>  - name: cluster-1<br>    labels:<br>      datadog-version: &quot;3.70.7&quot;  # Cluster-specific version<br><br># In ApplicationSet template<br>targetRevision: &#39;{{`{{default .values.revision (index .metadata.labels (print .values.app &quot;-version&quot;))}}`}}&#39;</pre><p>This approach provides several benefits:</p><ol><li>Default versions can be set at the environment level</li><li>Individual clusters can run different versions when needed</li><li>Version updates can be managed through GitOps workflows</li></ol><h3>Understanding Secret Management in Multi-Cluster Environments</h3><p>One of the most crucial aspects of managing multiple Kubernetes clusters is handling secrets securely. Our solution implements two complementary approaches for secret management, each serving different needs. Let’s explore how these work together to create a robust and secure system.</p><h4>The Two Pillars of Secret Management</h4><p>Before diving into the technical details, it’s important to understand that secret management in Kubernetes typically faces two distinct challenges:</p><ol><li><strong>Continuous Secret Synchronization</strong>: Some secrets need to be constantly monitored and updated across clusters. Think of these like a synchronized password system — when the source changes, all copies need to update automatically.</li><li><strong>Deployment-Time Secret Injection</strong>: Other times, we need to inject secrets into our resources during deployment, similar to filling in sensitive information in a template right before it’s used.</li></ol><p>Our solution addresses these challenges using two powerful tools: External Secrets Operator (ESO) and ArgoCD Vault Plugin (AVP). Let’s examine each one in detail.</p><h4>External Secrets Operator: The Continuous Synchronizer</h4><p>Think of ESO as a diligent secretary who constantly monitors and updates your important documents. Here’s how it works in our solution:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rNWdYFRD1FmvEVZX9P4tEA.png" /></figure><p>The workflow begins when ArgoCD pulls and applies an ExternalSecret manifest from our Git repository. This manifest is like a set of instructions telling ESO what secrets to manage. For example:</p><pre>apiVersion: external-secrets.io/v1beta1<br>kind: ExternalSecret<br>metadata:<br>  name: {{ .name }}<br>  namespace: argocd<br>spec:<br>  refreshInterval: &quot;1h&quot;  # Check for updates every hour</pre><p>Once this manifest is applied, ESO begins its continuous monitoring cycle:</p><ol><li>It watches for ExternalSecret resources in the cluster</li><li>When it finds one, it connects to AWS Secrets Manager to fetch the specified secrets</li><li>It creates or updates Kubernetes secrets with these values</li><li>This cycle repeats based on the refresh interval</li></ol><p>This is particularly valuable for cluster credentials because:</p><ul><li>They need to stay current at all times</li><li>Multiple components might need access to them</li><li>Changes need to propagate automatically</li><li>We want a reliable audit trail</li></ul><h4>ArgoCD Vault Plugin: The Deployment-Time Injector</h4><p>While ESO handles continuous synchronization, AVP serves a different but equally important role. Think of it as a document processor that fills in sensitive information right before a document is finalized.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3IRZ_l9yTzAbFcPZcbBDBQ.png" /></figure><p>Let’s walk through how AVP works:</p><ol><li>It starts when ArgoCD pulls a manifest containing special placeholder tags:</li></ol><pre>apiVersion: v1<br>kind: Secret<br>metadata:<br>  name: datadog-tags<br>  annotations:<br>    avp.kubernetes.io/path: {{ .Values.clusterName }}<br>stringData:<br>  DD_TAGS: &lt;dd_tags&gt;  # This placeholder will be replaced</pre><ol><li>During the ArgoCD sync process:</li></ol><ul><li>AVP recognizes these special tags</li><li>It fetches the corresponding values from AWS Secrets Manager</li><li>It injects these values into the manifest</li><li>The completed manifest is then applied to the cluster</li></ul><p>The plugin is configured in ArgoCD through custom tooling:</p><pre>configManagementPlugins: |<br>  - name: argocd-vault-plugin-helm<br>    init:<br>      command: [&quot;/bin/sh&quot;, &quot;-c&quot;]<br>      args: [&quot;helm dependency build&quot;]<br>    generate:<br>      command: [&quot;sh&quot;, &quot;-c&quot;]<br>      args: [&quot;helm template $ARGOCD_APP_NAME ${HELM_VALUES} . | argocd-vault-plugin generate -&quot;]</pre><p>This enables:</p><ol><li>Secure secret management without storing sensitive data in Git</li><li>Dynamic secret injection during the deployment process</li><li>Integration with existing secret management systems</li></ol><h4>How They Work Together: A Real-World Example</h4><p>To understand how these two mechanisms complement each other, let’s look at how they work together in our cluster addon management solution:</p><p>ESO handles:</p><ul><li>Cluster credentials for ArgoCD</li><li>Long-lived API keys</li><li>Service account tokens</li></ul><p>AVP manages:</p><ul><li>Configuration values during deployment</li><li>Application-specific secrets</li><li>Dynamic tags and labels</li></ul><p>This separation of concerns provides several benefits:</p><ol><li>Clear distinction between continuously synchronized secrets and deployment-time secrets</li><li>Different refresh patterns for different types of secrets</li><li>Appropriate handling mechanisms for each secret type</li><li>Enhanced security through specialized tools</li></ol><h3>Bringing It All Together</h3><p>The power of this solution lies in how these components work together to create a robust, secure, and scalable system for managing cluster addons. Let’s look at a typical workflow:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*R8ltYUNSCCg8508RFZMAYw.png" /></figure><h3>Conclusion and Looking Forward</h3><p>This architecture provides a robust foundation for managing cluster add-ons at scale. The combination of ApplicationSets, sophisticated configuration management, and secure secret handling creates a system that is both powerful and maintainable.</p><p>In Part 3, we’ll dive into the practical aspects of implementing this solution, including:</p><ul><li>Step-by-step setup guide</li><li>Common challenges and solutions</li><li>Best practices for day-2 operations</li><li>Advanced use cases and customizations</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=aeb6c3908c6d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Multi-Cluster Addon Management with ArgoCD: A GitOps Approach — Part 1]]></title>
            <link>https://medium.com/@moran.weissman/multi-cluster-addon-management-with-argocd-a-gitops-approach-part-1-497f9290534d?source=rss-fe44884b7c71------2</link>
            <guid isPermaLink="false">https://medium.com/p/497f9290534d</guid>
            <dc:creator><![CDATA[Moran Weissman]]></dc:creator>
            <pubDate>Sat, 25 Jan 2025 23:18:51 GMT</pubDate>
            <atom:updated>2025-01-25T23:18:51.513Z</atom:updated>
            <content:encoded><![CDATA[<h3>Multi-Cluster Addon Management with ArgoCD: A GitOps Approach — Part 1</h3><p>Managing add-ons across multiple Kubernetes clusters is a complex challenge many organizations face. While running multiple clusters offers benefits like environment isolation and fault tolerance, it introduces significant operational complexity. In this series, we’ll explore a GitOps-based solution using ArgoCD to manage this complexity effectively.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fOtSLYGQL2aAzdRxQmZS4Q.png" /></figure><h3>The Multi-Cluster Management Challenge</h3><p>Let’s dive into the real challenges DevOps teams face when managing multiple Kubernetes clusters, especially when it comes to add-on management.</p><h3>1. Version Control and Consistency</h3><p>One of the biggest challenges is the variety of tools and methods different teams use to deploy the same add-ons. Here’s what this often looks like in practice:</p><ul><li>Team A: helm install datadog-operator --namespace datadog</li><li>Team B: CI pipeline with custom kubectl apply commands</li><li>Team C: Terraform Helm provider</li><li>Team E: Custom shell scripts wrapping kubectl commands</li></ul><p>This diversity leads to inconsistent deployments across clusters, making troubleshooting and updates a nightmare.</p><h3>2. Operational Overhead</h3><p>Consider a common scenario: upgrading the Datadog agent across 10 clusters. Here’s what it typically looks like:</p><pre># Traditional approach - repeated manually for each cluster<br>helm upgrade datadog datadog/datadog \<br>  --namespace datadog \<br>  --set datadog.apiKey=$DD_API_KEY \<br>  --values custom-values.yaml</pre><p>This approach brings several problems:</p><ul><li>Manual execution in each cluster</li><li>Different value files per environment</li><li>Inconsistent secret management (AWS Secrets Manager, Vault, environment variables)</li><li>No standardized rollback procedure</li></ul><p>[Second image: Traditional vs GitOps Flow Comparison diagram]</p><h3>3. Configuration Management</h3><p>Here’s a common anti-pattern we see in many organizations:</p><pre># Dev team values.yaml<br>datadog:<br>  logLevel: DEBUG<br>  env: development<br>  <br># Prod team values.yaml<br>datadog:<br>  log_level: INFO  # Inconsistent naming<br>  environment: prod  # Different key names</pre><p>These inconsistencies make it difficult to:</p><ul><li>Track configuration changes</li><li>Troubleshoot issues</li><li>Maintain configuration parity between environments</li><li>Ensure security compliance</li></ul><h3>The Solution: GitOps with ArgoCD</h3><p>[Third image: solution-architecture.svg]</p><p>Our solution combines several powerful tools and patterns to address these challenges:</p><h3>1. ArgoCD</h3><ul><li>Core GitOps engine providing declarative configuration</li><li>Automated sync and drift detection</li><li>Single source of truth for all cluster configurations</li></ul><h3>2. ApplicationSets</h3><ul><li>Dynamic application generation based on cluster metadata</li><li>Template-based configuration management</li><li>Centralized add-on version control</li></ul><h3>3. External Secrets Operator (ESO)</h3><ul><li>Secure secret management</li><li>Integration with AWS Secrets Manager</li><li>Consistent secret handling across clusters</li></ul><h3>4. ArgoCD Vault Plugin (AVP)</h3><ul><li>Secrets injection during the sync process</li><li>Support for multiple secret backends</li><li>Template-based secret management</li></ul><h3>5. Helm</h3><ul><li>Configuration templating and management</li><li>Environment-specific value overrides</li><li>Reusable chart patterns</li></ul><h3>What’s Coming Next</h3><p>In Part 2, we’ll dive into the technical implementation of this solution, showing you how to:</p><ul><li>Set up the basic infrastructure</li><li>Configure ArgoCD for multi-cluster management</li><li>Implement secure secret handling</li><li>Create your first addon deployment</li></ul><p>In the next part where we’ll get our hands dirty with the actual implementation!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=497f9290534d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Practical DevOps Interview Guide]]></title>
            <link>https://medium.com/@moran.weissman/a-practical-devops-interview-guide-d3cd0e1e86dc?source=rss-fe44884b7c71------2</link>
            <guid isPermaLink="false">https://medium.com/p/d3cd0e1e86dc</guid>
            <category><![CDATA[ci-cd-pipeline]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[interviews-and-insights]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Moran Weissman]]></dc:creator>
            <pubDate>Mon, 30 Dec 2024 14:02:37 GMT</pubDate>
            <atom:updated>2024-12-30T16:52:16.961Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*wo-3Y7MnLhKgIzre" /></figure><p>This article is a collection of common questions and scenarios you might encounter in a DevOps interview, along with key points to help you prepare thoughtful and structured answers. It’s not an exhaustive guide to all things DevOps, but a set of topics based on real-world challenges and practical personal experience. Whether you’re an interviewer looking for ideas or a candidate brushing up on your next opportunity, this guide is meant to be a helpful resource.</p><p>If you’re looking for more in-depth examples or want to contribute your own questions, check out the companion GitHub repository: <a href="https://github.com/MoranWeissman/devops-interview-questions">DevOps Interview Questions.</a></p><h3>Table of Contents</h3><blockquote>· <a href="#2b7a"><strong>Continuous Integration (CI)</strong></a><strong><br>· </strong><a href="#53ad"><strong>Continuous Deployment (CD)</strong></a><strong><br>· </strong><a href="#6c66"><strong>GitOps &amp; ArgoCD</strong></a><strong><br>· </strong><a href="#c18b"><strong>Docker</strong></a><strong><br>· </strong><a href="#ff00"><strong>Kubernetes</strong></a><strong><br>· </strong><a href="#064a"><strong>Helm</strong></a><strong><br>· </strong><a href="#6031"><strong>Secrets</strong></a><strong><br>· </strong><a href="#f69c"><strong>Security</strong></a><strong><br>· </strong><a href="#a94e"><strong>Shift Left</strong></a><strong><br>· </strong><a href="#fdc4"><strong>General / Architecture</strong></a><strong><br>· </strong><a href="#9cf7"><strong>Conclusion</strong></a><strong><br>· </strong><a href="#8b52"><strong>Additional Resources</strong></a></blockquote><h3>Continuous Integration (CI)</h3><blockquote>Focuses on building code, versioning, branching strategies, and ensuring every commit is validated properly.</blockquote><h4>Clean Build Environments</h4><p>Your builds sometimes fail because leftover files from previous runs cause unexpected results.</p><ul><li>How would you make sure each build runs in a clean, consistent environment?</li></ul><p><strong>Key Points</strong></p><ul><li><strong>Containerized</strong> or <strong>ephemeral</strong> build agents ensure no leftover files or dependencies.</li><li>Reproducible environment: each run starts fresh.</li><li>Avoid “works on my machine” issues by isolating dependencies.</li></ul><h4>Git &amp; Branch Policies</h4><p>When setting up a new Git repository for a service, how do you make sure collaboration and merging tomainare done in a controlled, high-quality way?</p><p><strong>Key Points</strong></p><ul><li>Require <strong>pull requests</strong> to merge into main.</li><li>Use <strong>branch protections</strong>: code reviews, mandatory checks, or tests.</li><li>Possibly adopt a branching strategy (Git Flow, OneFlow, or trunk-based).</li></ul><h4>Testing from Dev/Personal Branches (Version Bump Avoidance)</h4><p>You want developers to build and test code on a personal branch without bumping the official app version.</p><ul><li>How do you do that?</li></ul><p><strong>Key Points</strong></p><ul><li>Only bump the version if building from main (after merging).</li><li>Use <strong>ephemeral tags</strong> (like dev-&lt;commit-hash&gt;) for personal branches.</li><li>This prevents cluttering your production semver with test builds.</li></ul><h4>Code Validation in CI</h4><p>What kind of checks do you usually include in a CI pipeline to keep code quality high?</p><p><strong>Key Points</strong></p><ul><li><strong>Linting</strong> and <strong>static analysis</strong> for code style and potential security flaws.</li><li><strong>Unit tests</strong>, code coverage, and maybe integration tests.</li></ul><h4>Git Flow vs OneFlow</h4><p>Explain the main differences between Git Flow and OneFlow, and give an example of when you might pick one over the other.</p><p><strong>Key Points</strong></p><ul><li><strong>Git Flow:</strong> Has a develop branch and separate release/ branches.</li><li><strong>Git Flow</strong>: Good for scheduled releases or big features that need isolation.</li><li><strong>OneFlow</strong>: Fewer branches; merges features straight into main.</li><li><strong>OneFlow</strong>: Easier for continuous delivery or smaller teams who release often.</li></ul><h4>Unit / Integration / E2E Tests</h4><p>A new developer asks how different types of tests fit into the CI pipeline.</p><ul><li>Can you explain the roles of unit, integration, and end-to-end tests?</li></ul><p><strong>Key Points</strong></p><ul><li><strong>Unit tests</strong>: Check individual pieces of code in isolation.</li><li><strong>Integration tests</strong>: Validate how services or components interact (e.g., API to DB).</li><li><strong>E2E tests</strong>: Full user flow from start to finish, mirroring real production usage.</li></ul><h4>Scenario: Slow Build Times</h4><p>The CI builds are getting slower and slower, which annoys developers. What could you do to speed things up?</p><p><strong>Key Points</strong></p><ul><li><strong>Caching</strong> dependencies so you don’t rebuild or re-download everything on each run.</li><li>Splitting large monolithic builds into smaller jobs or microservices.</li><li>Using multi-stage Docker builds or ephemeral agents to reduce overhead.</li></ul><h4>Scenario: <strong>Docker Build Time Optimization in CI</strong></h4><p>Your Docker builds in CI/CD are taking longer than expected, especially for a .NET or Java app with large dependencies.</p><ul><li>What can you do to improve the build time?</li></ul><p><strong>Key Points</strong></p><ul><li>Optimize your Dockerfile structure: place static dependency installation steps at the top for caching.</li><li>Use multi-stage builds to keep final images small.</li><li>Possibly store a “build cache” or layer cache between runs so you don’t rebuild everything from scratch each time.</li></ul><h3>Continuous Deployment (CD)</h3><blockquote>Automating deployments, promotions across environments, and using different strategies like rolling or canary.</blockquote><h4>Promotion Across Environments</h4><p>You have dev, staging, and prod environments. How would you automate moving a service from dev to staging, then on to prod, with proper checks along the way?</p><p><strong>Key Points</strong></p><ul><li>Multi-stage pipeline with approvals or gating (manual or automated).</li><li>Same config, but different overrides for each environment (Helm or plain YAML).</li><li>Possibly a final manual step for production if your org requires it.</li></ul><h4>K8s Deployment Strategies</h4><p>You want minimal downtime and easy rollback. Compare rolling updates, blue/green deployments, and canary releases.</p><p><strong>Key Points</strong></p><ul><li><strong>Rolling</strong>: Replaces pods one by one; simpler to set up but partial downtime if something goes wrong mid-roll.</li><li><strong>Blue/Green</strong>: Parallel environments; easy rollback by flipping traffic back.</li><li><strong>Canary</strong>: Gradual traffic shift, letting you observe performance in real time with a fraction of traffic.</li></ul><h4>Examples of Blue/Green &amp; Canary</h4><p>Your new microservice is critical, and you’re debating between blue/green and canary. Give a quick example of each strategy to show how you’d roll out a new version.</p><ul><li><strong>Blue/Green</strong>: Deploy the new version side-by-side, test it, then switch over.</li><li><strong>Canary</strong>: Send maybe 5% of traffic to the new version, monitor, then increase if no issues arise.</li></ul><h4>Bonus: Canary/Blue-Green in Kubernetes</h4><p>Can you do canary or blue-green deployments directly in Kubernetes without extra tools?</p><p><strong>Key Points</strong></p><ul><li>Yes, but you have to handle traffic splits, labels, or separate services yourself.</li><li>Tools like <strong>Argo Rollouts</strong> or a service mesh (Istio) make it easier.</li></ul><h4>Argo Rollouts Explanation</h4><p>What is Argo Rollouts, and how does it help with advanced deployment strategies?</p><p><strong>Key Points</strong></p><ul><li><strong>Kubernetes controller</strong> that replaces Deployments with CRDs.</li><li>Supports canary, blue-green, progressive rollouts with health checks.</li><li>Integrates with ingress controllers or service meshes for traffic shaping.</li></ul><h4>Scenario: Slow Canary Deployments with Argo Rollouts</h4><p>You set up Argo Rollouts for canary deployments, but traffic shifting is slower than expected, delaying the rollout.</p><ul><li>What steps do you take to debug this behavior?</li><li>How can you ensure canary traffic shifts happen faster?</li></ul><p><strong>Key Points</strong></p><ul><li>Check the <strong>Rollout object</strong> for correct strategy, weights, and health checks.</li><li>Make sure the ingress or service mesh is applying traffic splits properly.</li><li>Validate that health checks or success criteria aren’t too strict, causing slow or paused progress</li><li>Monitor logs or metrics to see if pods are failing readiness or not meeting thresholds.</li></ul><h3>GitOps &amp; ArgoCD</h3><blockquote>Focuses on GitOps concepts, ArgoCD usage, the app-of-apps pattern, and ApplicationSets.</blockquote><h4>What Is GitOps?</h4><p>Your manager says, “We should do GitOps!” How do you explain the core idea of GitOps?</p><p><strong>Key Points</strong></p><ul><li>Infrastructure and app config in Git as the single source of truth.</li><li>A tool (ArgoCD) automatically updates the cluster to match Git.</li><li>All changes go through PRs, so everything is tracked and auditable.</li></ul><h4>ArgoCD as a GitOps Tool</h4><p>Why is ArgoCD considered a GitOps solution and not just another deployment tool?</p><p><strong>Key Points</strong></p><ul><li>Watches a Git repo for changes, automatically syncing them into the cluster.</li><li>Declarative approach: no manual clicks or imperative commands.</li><li>Integrates with Helm, Kustomize, or plain YAML.</li></ul><h4>App-of-Apps Pattern</h4><p>You have a bunch of microservices, each with its own repo or chart. You want a single place to manage them.</p><ul><li>How would you do this in ArgoCD?</li></ul><p><strong>Key Points</strong></p><ul><li>Root “umbrella” application referencing multiple child apps.</li><li>Each child is a separate Helm chart or folder.</li><li>App-of-apps keeps everything organized under one main config.</li></ul><h4>Adding a New App Under a Root “Umbrella”</h4><p>If you already have a root “app-of-apps” that manages several services, how do you add a brand-new microservice?</p><p><strong>Key Points</strong></p><ul><li>Create a new ArgoCD “child” Application in the same or separate repo.</li></ul><h4>ApplicationSet</h4><p>What is an ApplicationSet in ArgoCD, and when might you use it? Name at least one generator type.</p><p><strong>Key Points</strong></p><ul><li><strong>ApplicationSet</strong> is a CRD that can create multiple ArgoCD Applications automatically.</li><li>Used for deploying the same app across multiple clusters or generating apps for each folder/branch.</li><li>Generators: List, Git directory, Cluster, or SCM provider.</li></ul><h4>Scenario: Deprecated APIs Causing ArgoCD Sync Failures</h4><p>After upgrading Kubernetes, an ArgoCD application fails to sync due to deprecated APIs in its Helm chart (e.g., extensions/v1beta1 → apps/v1).</p><ul><li>How do you identify which APIs are outdated?</li><li>How do you fix them?</li></ul><p><strong>Key Points</strong></p><ul><li>Check chart templates for old API references.</li><li>Replace them with newer versions (apps/v1).</li><li>Tools like Pluto or helm template can highlight deprecated APIs.</li><li>Keep charts updated and test in a staging cluster before upgrading production.</li></ul><p>Two ArgoCD apps share a namespace and deploy conflicting resources. Give each app its own namespace or rename the overlapping resources.</p><h3>Docker</h3><blockquote>Discusses container basics, Dockerfiles, and best practices.</blockquote><h4>Docker vs Container</h4><p>What’s the difference between Docker as a tool and the concept of a container?</p><p><strong>Key Points</strong></p><ul><li><strong>Docker</strong> is a platform for building/managing containers.</li><li>A <strong>container</strong> is an isolated environment bundling the app and dependencies.</li><li>Docker is popular, but other runtimes exist (Podman, Containerd).</li></ul><h4>Dockerfile: ENTRYPOINT vs CMD</h4><p>A coworker is confused about ENTRYPOINT and CMD in a Dockerfile. How do you explain the difference?</p><p><strong>Key Points</strong></p><ul><li><strong>ENTRYPOINT</strong>: The main command the container will always run.</li><li><strong>CMD</strong>: Default arguments that can be overridden at runtime.</li><li>Typically, you set ENTRYPOINT to the main process and use CMD for optional flags.</li></ul><h4>Image Size Optimization</h4><p>You have a huge Docker image based on a full JDK. You want to reduce its size.</p><ul><li>How do you do it, and why does size matter?</li></ul><p><strong>Key Points</strong></p><ul><li>Use <strong>multi-stage builds</strong> or switch to a smaller base (JRE or Alpine).</li><li>Clean up leftover artifacts (logs, caches).</li><li>Smaller images pull faster, less storage overhead, fewer security risks.</li></ul><h4>Build-time vs Runtime Secrets</h4><p>You need to use some private tokens or credentials during the build.</p><ul><li>How do you add them without exposing them in the final image?</li></ul><p><strong>Key Points</strong></p><ul><li><strong>ARG</strong> for build-time secrets (not present in the final image).</li><li>Inject secrets at runtime via environment variables or secret managers.</li><li>Don’t store secrets in plain text in the Dockerfile or version control.</li></ul><h4>Reducing Docker Build Time</h4><p>Your Docker builds take too long, especially for .NET or Java projects.</p><ul><li>How can you make them faster?</li></ul><p><strong>Key Points</strong></p><ul><li>Reorder Dockerfile steps to install dependencies first, so you can cache them.</li><li>Use multi-stage builds to keep final images small.</li><li>Possibly store or share a build cache between CI runs.</li></ul><h4>Docker-in-Docker or Alternatives</h4><p>Sometimes we build Docker images inside a container during CI.</p><ul><li>Why do we do that, and what are some alternatives?</li></ul><p><strong>Key Points</strong></p><ul><li><strong>Docker-in-Docker</strong>: runs a Docker daemon inside a container, but can be less secure.</li><li>Alternatives: <strong>Kaniko, Buildah, Podman</strong> to build without a Docker daemon.</li><li>Reduces the need for privileged mode in CI.</li></ul><h4>Other Container Build Tools</h4><p>If Docker wasn’t an option, what else could you use to build and run containers?</p><p><strong>Key Points</strong></p><ul><li><strong>Podman, Buildah, Containerd</strong>.</li><li>For Java, <strong>Jib</strong> can build containers without a Docker daemon.</li><li>Some environments rely on containerd or rkt (less common nowadays).</li></ul><h4>Scenario: Kaniko Image Build Fails</h4><p>ou are using Kaniko to build and push Docker images in a Kubernetes Pod, but the build fails.</p><ul><li>What steps would you take to troubleshoot Kaniko’s failure?</li><li>How can you configure it to build images without a Docker daemon?</li></ul><p><strong>Key Points</strong></p><ul><li>Verify the Kaniko Pod has access to your Dockerfile, context, and registry credentials.</li><li>Kaniko doesn’t need a Docker daemon; pass --context and --destination parameters correctly.</li><li>Ensure you have the right RBAC permissions if it needs to create or manage certain resources in K8s.</li></ul><h3>Kubernetes</h3><blockquote>Core K8s topics: pods, services, ingress, CRDs, and debugging.</blockquote><h4>Deploying a New Application</h4><p>If you have a new microservice to run in Kubernetes, what resources do you usually set up?</p><p><strong>Key Points</strong></p><ul><li>Usually a <strong>Deployment</strong> (or StatefulSet if stateful) and a <strong>Service</strong>.</li><li>Possibly an Ingress or LoadBalancer if external access is required.</li><li>Might use Helm for templating.</li></ul><h4>Services vs Ingress</h4><p>What does a Service do in Kubernetes, and how is it different from an Ingress?</p><p><strong>Key Points</strong></p><ul><li><strong>Service</strong>: Exposes pods at a stable address, can be ClusterIP, NodePort, or LoadBalancer.</li><li><strong>Ingress</strong>: Defines routing rules for HTTP/HTTPS traffic to one or more Services.</li></ul><h4>CRDs &amp; Operators</h4><p>How do CRDs (Custom Resource Definitions) and Operators help you extend Kubernetes beyond its default features?</p><p><strong>Key Points</strong></p><ul><li>A <strong>CRD</strong> adds a new type of object (like “MyDatabase”) to the cluster.</li><li>An <strong>Operator</strong> watches these CRDs and automates tasks (install, upgrade, manage).</li><li>Good for complex/stateful apps so K8s can handle them more natively.</li></ul><h4>Logs &amp; Crash Troubleshooting</h4><p>Your app keeps crashing after a few minutes in Kubernetes.</p><ul><li>How would you check what’s going on?</li></ul><p><strong>Key Points</strong></p><ul><li>Inspect logs from the pod/container.</li><li>Check events or error messages for the pod.</li><li>See if it’s an OOM kill, code exception, or config problem.</li></ul><h4>Resource in “Terminating” State</h4><p>Sometimes a pod or other resource is stuck “Terminating” for a long time. *</p><ul><li>Why could that happen, and what might you do?</li></ul><p><strong>Key Points</strong></p><ul><li><strong>Finalizers</strong> might be blocking deletion.</li><li>The app might not handle termination signals well, so it never exits.</li><li>You can remove the finalizer or do a force delete if absolutely needed.</li></ul><h4>Debugging Inside a Container</h4><p>You need to run commands inside a container for debugging.</p><ul><li>How do you do that in a Kubernetes environment?</li></ul><p><strong>Key Points</strong></p><ul><li>Typically use a CLI to exec into the container.</li><li>If multiple containers, specify which container.</li><li>Make sure you have the right RBAC privileges.</li></ul><h4>Editing a Resource Live</h4><p>You spot a small config mistake in a live resource.</p><ul><li>How would you fix it right away in the cluster?</li><li>What risks might that cause?</li></ul><p><strong>Key Points</strong></p><ul><li>You can <strong>edit</strong> the resource in place with the CLI, but that can cause drift from Git or Helm config.</li><li>If you’re using GitOps, the next sync might overwrite your manual fix.</li><li>Best practice: fix it in your config repo or chart too.</li></ul><h4>Service-to-Service Communication</h4><p>How do different services within the same cluster talk to each other?</p><p><strong>Key Points</strong></p><ul><li><strong>Cluster DNS:</strong> &lt;service-name&gt;.&lt;namespace&gt;.svc.cluster.local.</li><li>A Service provides a stable endpoint, even if pod IPs change.</li></ul><h4>Sidecar/Init Containers</h4><p>What are sidecar containers and init containers, and why might you use them?</p><p><strong>Key Points</strong></p><ul><li><strong>Init</strong> containers run first to do setup tasks (migrations, config).</li><li><strong>Sidecar</strong> containers run alongside the main app for logging, proxying, etc.</li><li>Helps separate concerns in a single pod.</li></ul><h4>Bonus: Resource Name Limits</h4><p>Is there a name length limit or other format rule for K8s resources?</p><p><strong>Key Points</strong></p><ul><li>Usually follows DNS label rules (lowercase, up to 63 chars, alphanumeric + dashes).</li><li>Some resource types might vary slightly, but typically the same constraints apply.</li></ul><h4>Scenario: RBAC Permissions for Kaniko</h4><p>You’re running an Azure DevOps agent in Kubernetes, which uses Kaniko to build and push Docker images. It’s failing because it can’t create needed resources.</p><ul><li>How would you troubleshoot the missing permissions?</li><li>How can RBAC be configured to give Kaniko the required access?</li></ul><p><strong>Hints</strong></p><ul><li>Check the Pod logs for permission errors (e.g., “forbidden”).</li><li>Assign a <strong>ServiceAccount</strong> with an appropriate Role/RoleBinding that allows creating ConfigMaps, Pods, etc.</li><li>Verify that the agent is using this ServiceAccount when building.</li></ul><h4>Scenario: Kubernetes Pod Logs Lost After Crash</h4><p>A Kubernetes Pod crashes unexpectedly, and its logs are lost because the container restarts too quickly.</p><ul><li>How would you recover logs from a previously crashed container?</li><li>How can you ensure logs are always accessible?</li></ul><p><strong>Key Points</strong></p><ul><li>Use the CLI to get logs from the <strong>previous</strong> container instance (-p option), if still available.</li><li>Centralize logs in an external system like ELK, Loki, or FluentD.</li><li>Ensure your app flushes logs frequently so they aren’t lost on crash.</li></ul><h4>Scenario: Resource Exhaustion (OOMKills) in Pods</h4><p>A Kubernetes Pod crashes intermittently and is marked as OOMKilled.</p><ul><li>How would you identify the cause of the memory spikes?</li><li>How do you stop the Pod from running out of memory in the future?</li></ul><p><strong>Key Points</strong></p><ul><li>Check resource usage with kubectl top or a monitoring tool.</li><li>Increase the memory limit if the app truly needs more, or find memory leaks.</li><li>Monitor usage over time, maybe use VPA (Vertical Pod Autoscaler) if appropriate.</li></ul><h3>Helm</h3><blockquote>Helm chart usage, templating, hooks, etc.</blockquote><h4>Chart Resource Requests/Limits</h4><p>Why do we set resource requests and limits in a Helm chart, and how do we manage them across environments?</p><p><strong>Key Points</strong></p><ul><li>Ensures pods have enough CPU/memory, prevents resource hogging.</li><li>Helm values.yaml can differ for dev vs prod.</li><li>Good for cost control and stability.</li></ul><h4>Requests vs Limits Differences</h4><p>What’s the difference between resource requests and limits in Kubernetes, and how does Helm simplify handling them?</p><p><strong>Key Points</strong></p><ul><li><strong>Requests</strong>: minimum guaranteed resources.</li><li><strong>Limits</strong>: maximum allowed before throttling or OOMKill.</li><li>Helm: store these in values.yaml for easy environment overrides.</li></ul><h4>Multiple Jobs Order</h4><p>If you have multiple jobs that need to run in a certain sequence, how can Helm handle that?</p><p><strong>Key Points</strong></p><ul><li>Hooks with <strong>weights</strong> (lower weight runs first).</li><li>Or a single job that does tasks in order.</li><li>Sometimes separate subcharts if they’re truly independent.</li></ul><h4>Dynamic Resource Generation</h4><p>You want to create several similar resources from a list in values.yaml. How do you do that with Helm?</p><p><strong>Key Points</strong></p><ul><li>Use {{- range .Values.myItems }} in the template.</li><li>Each item in the list gets its own resource.</li><li>_helpers.tpl can keep repeated logic DRY.</li></ul><h4>TPL Helper File</h4><p>What is the tpl function in Helm, and why might you use it?</p><p><strong>Key Points</strong></p><ul><li>tpl parses a string as a Helm template at runtime.</li><li>Good for user-provided or nested templates in values.yaml.</li><li>Keep advanced logic or partials in _helpers.tpl.</li></ul><h4>Sequence of Operations (Hooks)</h4><p>You want a job to run before your main app starts.</p><ul><li>How do you do that in Helm?</li></ul><p><strong>Key Points</strong></p><ul><li>Use <strong>Helm hooks</strong> (pre-install, post-install) on that job.</li><li>The job runs first; if it succeeds, Helm proceeds to install the rest.</li><li>Weights can fine-tune the order of multiple hooks.</li></ul><h4>Handling API Deprecations</h4><p>A new Kubernetes version might deprecate older APIs.</p><ul><li>How do you update your Helm charts to handle that?</li></ul><p><strong>Key Points</strong></p><ul><li>Replace old references (e.g., extensions/v1beta1) with apps/v1.</li><li>Tools like pluto can scan for deprecated usage.</li><li>Test in a lower environment or staging cluster first.</li></ul><h4>Scenario: <strong>Deprecated APIs Causing Sync Failures w/ Helm</strong><br>After upgrading the cluster, your Helm chart can’t sync in ArgoCD because it references deprecated APIs.</h4><ul><li>How do you find which APIs are outdated?</li><li>How do you fix them in your chart?</li></ul><p><strong>Key Points</strong></p><ul><li>Look at the chart’s templates for older API versions (e.g., extensions/v1beta1).</li><li>Update them to the newer equivalents (e.g., apps/v1).</li><li><strong>Helm 3.12+</strong> offers a <strong>server-side dry run</strong> via:</li></ul><pre>helm upgrade --dry-run=server ...</pre><p>This checks the manifests against the actual cluster APIs, catching potential deprecations or validation issues before you apply them.</p><ul><li>You can also use tools like <strong>Pluto</strong> to scan for deprecated or removed APIs.</li><li>Test in a non-production cluster to confirm everything works with the new APIs.</li></ul><h3>Secrets</h3><blockquote>Deals with handling sensitive data like passwords, tokens, and keys in DevOps.</blockquote><h4>What Are Secrets &amp; Why</h4><p>Organizations always talk about storing things like passwords or tokens as “secrets.” What’s the big deal?</p><p><strong>Key Points</strong></p><ul><li>It keeps sensitive data out of plain text in code or config.</li><li>Minimizes risk if repos or logs get exposed.</li><li>In Kubernetes, a Secret is base64-encoded, but a real secrets manager provides better security.</li></ul><h4>Storing &amp; Managing Secrets</h4><p>Your team has many credentials for different microservices.</p><ul><li>How can you safely store and use them without embedding them in code?</li></ul><p><strong>Key Points</strong></p><ul><li>Use a <strong>secrets manager</strong> (like HashiCorp Vault, AWS Secrets Manager, Azure Key Vault).</li><li>Give each service controlled access.</li><li>Automate secret rotation and auditing, especially if you handle sensitive data.</li></ul><h3>Security</h3><blockquote>Focuses on adding security checks to CI/CD, avoiding hardcoded secrets, and building secure images.</blockquote><h4>Pipeline Security</h4><p>You want to catch vulnerabilities early.</p><ul><li>How can you add security steps to your CI/CD pipeline?</li></ul><p><strong>Key Points</strong></p><ul><li><strong>Static code analysis</strong> (SAST) to look for known flaws.</li><li><strong>Image scanning</strong> for Docker containers.</li><li>Dependency checks to flag libraries with known CVEs.</li></ul><h4>Avoiding Hardcoded Secrets</h4><p>You found actual passwords in your pipeline scripts.</p><ul><li>How do you clean that up and prevent it from happening again?</li></ul><p><strong>Key Points</strong></p><ul><li>Store secrets in a secure variable store or a secrets manager.</li><li>Don’t commit them to Git.</li><li>Use environment variables or injected secrets at runtime.</li></ul><h4>Secure Docker Builds</h4><p>Your production app runs in Docker containers.</p><ul><li>What steps can you take to make sure those containers are secure?</li></ul><p><strong>Key Points</strong></p><ul><li>Use <strong>minimal base images</strong>, patch them regularly.</li><li>Don’t run as root if you can avoid it.</li><li>Scan images for vulnerabilities before deploying.</li></ul><h3>Shift Left</h3><blockquote>Idea of doing testing and security earlier in the development process.</blockquote><h4>Shift Left Concept</h4><p>What does “Shift Left” mean in software development, and why is it important?</p><p><strong>Hint</strong></p><ul><li>Do QA/testing/security as early as possible, not at the end.</li><li>Helps catch problems sooner, which is cheaper to fix.</li><li>Encourages developers to take ownership of quality from the start.</li></ul><h4>Why Shift Left</h4><p>Your testers always find big problems right before release.</p><ul><li>How could a Shift Left approach help?</li></ul><p><strong>Solution</strong></p><ul><li>If devs run tests and checks on each commit, issues are spotted earlier.</li><li>Fewer surprises at the end.</li><li>Faster feedback loops and more stable releases</li></ul><h4>CI/CD Tools for Shift Left</h4><p>How do tools like Jenkins, GitHub Actions, or Azure Pipelines help a Shift Left approach?</p><p><strong>Key Points</strong></p><ul><li>They let you run automated tests and scans on every push or pull request.</li><li>If something fails, devs see it right away.</li><li>Can even spin up test environments automatically for quick integration checks.</li></ul><h3>General / Architecture</h3><blockquote>Covers broader DevOps or architecture topics.</blockquote><h4>Stateless vs Stateful</h4><p>What’s the main difference between stateless and stateful apps, and how does this affect scaling?</p><p><strong>Key Points</strong></p><ul><li><strong>Stateless</strong>: Doesn’t keep data in memory across requests; easier to scale horizontally.</li><li><strong>Stateful</strong>: Maintains sessions or data that might need external storage. Harder to scale.</li></ul><h4>DevOps Culture</h4><p>How would you describe “DevOps culture,” and what does it change compared to older dev-and-ops silos?</p><p><strong>Key Points</strong></p><ul><li>Focus on <strong>collaboration</strong> and <strong>automation</strong>.</li><li>Shared responsibility for stability, performance, and delivery.</li><li>Closer feedback loops, continuous integration, continuous delivery.</li></ul><h4>Scenario: Handling a Critical Outage</h4><p>Production goes down during peak hours. Describe how you’d handle the incident, from detecting the problem to resolving it.</p><p><strong>Key Points</strong></p><ul><li>Quick triage: check logs, monitoring, recent changes, and alerts.</li><li>Possibly roll back the last deployment if that caused it.</li><li>Communicate clearly with stakeholders.</li><li>After fixing it, do a post-mortem to learn from it.</li></ul><h3>Conclusion</h3><p>This <strong>Practical DevOps Interview Guide</strong> aims to:</p><ol><li>Give <strong>interviewers</strong> a structured way to discuss everything from CI pipelines to advanced Kubernetes setups.</li><li>Help <strong>candidates</strong> understand the range of DevOps topics and focus on the “how” and “why” — not just memorizing answers.</li></ol><p>If you want even more details and scenario expansions, check out the <a href="https://github.com/MoranWeissman/devops-interview-questions">original DevOps Interview Questions repo on GitHub</a>. Hopefully, this collection leads to better, more real-world DevOps interviews.</p><p><strong>Good luck, and happy interviewing!</strong></p><h3>Additional Resources</h3><ul><li><strong>GitHub Repo</strong>: <a href="https://github.com/MoranWeissman/devops-interview-questions">DevOps Interview Questions</a></li><li><strong>Official Helm</strong>: <a href="https://helm.sh/">helm.sh</a></li><li><strong>ArgoCD</strong>: <a href="https://argo-cd.readthedocs.io/">argo-cd.readthedocs.io</a></li><li><strong>Kubernetes</strong>: kubernetes.io/docs</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d3cd0e1e86dc" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Part III: Building Docker images in Kubernetes with Kaniko and Azure DevOps]]></title>
            <link>https://medium.com/@moran.weissman/part-iii-building-and-pushing-a-docker-image-using-kaniko-to-ecr-6eeaeb0c9f6f?source=rss-fe44884b7c71------2</link>
            <guid isPermaLink="false">https://medium.com/p/6eeaeb0c9f6f</guid>
            <category><![CDATA[azure-pipelines]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[kaniko]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[azure-devops-pipeline]]></category>
            <dc:creator><![CDATA[Moran Weissman]]></dc:creator>
            <pubDate>Wed, 01 Mar 2023 17:56:46 GMT</pubDate>
            <atom:updated>2023-03-09T09:20:46.392Z</atom:updated>
            <content:encoded><![CDATA[<p>In this article, we will continue from where we left off in the <a href="https://medium.com/@moran.weissman/part-ii-deploying-azure-devops-agents-to-eks-via-argocd-74403409a08a">previous article</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/508/1*20vpZARuDkRQYDR1w9M90Q.jpeg" /></figure><p>We are not limited by the kind of CI pipelines we can run on our self-managed Kubernetes Agents, to most things anyway.</p><p>We can build Python, Go, Dotnet, or run automation tests using Robot Framework.. but what about our Docker images CI pipelines? We can’t just install a Docker on our Agent and perform a docker build because the Agent is running inside a container.</p><p>So the first option that came to my mind is implementing DinD(Docker in Docker), but security-wise it’s not recommended. We can discuss why not to, but let’s talk about one of the alternatives I chose to work with — Kaniko.</p><p><strong>Kaniko: in a nutshell</strong></p><p>Kaniko is an open-source tool for building container images inside a container without needing a Docker daemon or privileged access, making it ideal for use in Kubernetes or other cloud-native environments.</p><p><strong>Kaniko vs. DinD</strong></p><p>Kaniko and DinD are used to build Docker images within a Kubernetes cluster, but their approach differs. Kaniko is a containerized build tool within a pod, while DinD runs a Docker engine within a pod to build images. Kaniko is generally considered more secure as it does not require privileged access to the host machine, while DinD may have more flexibility and support for Docker-specific features.</p><p>For this article, I have prepared a simple dotnet application and a Dockerfile that builds it. We will build this application using Azure Pipelines, of course, and Kaniko. We will publish it to our ECR. Inside my GitHub repository, you will find all that is needed:</p><ul><li>Application code</li><li>Dockerfile</li><li>Azure Pipeline</li></ul><p>clone or fork it and make it available on your Azure Git.</p><p><a href="https://github.com/MoranWeissman/docker-demo">GitHub - MoranWeissman/docker-demo: An simple dotnet application that counts numbers</a></p><p>Before we start working with Kaniko in our pipelines, we must understand that there is no native way to run Kaniko for Azure DevOps. as opposed to other CI systems. Hence, using Kaniko requires an understanding of Azure DevOps and Kubernetes.</p><p>Without diving too much into the details, I’ll mention that when using Kaniko, we need to provide the location of the build context (code and Dockerfile). Currently, there’s support only for a local directory, local Tar Gz, StdIn, GCS bucket, Azure Blob, S3 Bucket, and Git repository (not Azure Git).</p><p>This means we need to pass the code somehow to Kaniko, and we will do this by running an init container before the Kaniko container starts that will clone our dotnet repository for Kaniko to use.</p><p>Let’s dive in.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-QlfW0TMm0-M_k570Jo5WQ.png" /></figure><p>In this pipeline, I’m going to implement the use of templates in Azure DevOps. One of the best practices is to use templates whenever possible to avoid code duplication. In the case of a common type of pipeline, such as one that builds Docker images, it’s advised to write a template. I suggest managing the templates on a separate repository. But this is usually very specific to the needs and use-case of the company.</p><p>You can take the following files or even use the repository as is:</p><p><a href="https://github.com/MoranWeissman/azure-pipeline-templates">GitHub - MoranWeissman/azure-pipeline-templates: This repository contains examples for various Azure DevOps custom templates</a></p><p>and make sure you have these templates available on a separate repository because I’m referencing them from within the Kaniko pipeline. You can put them wherever you like as long as you update them accordingly in the pipeline resources section.</p><p>Here is a brief explanation of the files:</p><ul><li><strong><em>build-docker-image-via-kaniko-job.yml</em></strong>: This is a main template, the only Job in our pipeline. So that you know, variables and logic are firmly based on the previous solution. If you only need the Kaniko pipeline, you may have to adjust to your specific needs.</li><li><strong><em>create-kaniko-pod-task.yml: </em></strong>This task will create a YAML called ‘deploy.yaml’ and create the Kaniko pod and ConfigMap needed for Kaniko. in the <strong>initContainer</strong> section, we will need to make sure the <strong>azure-devops-git-cloner</strong> image version is correct, I’ll elaborate about it up ahead.</li><li><strong><em>apply-and-monitor-kaniko-pod-steps.yml: </em></strong>This is the exciting part. In this template, we have several tasks: create the Kaniko pod, and monitor the progress of the Kaniko pod (both initContainer and main-container) while printing the output of the Docker build that is being done. Once finished, it deletes all build resources and will exit with an error in case it fails, with relevant logs printed.</li></ul><p><strong>Prerequisites:</strong></p><ul><li>AWS Access Keys with permissions to read from and push to ECR</li><li>A Kubernetes secret that contains the AWS keys and region (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and AWS_REGION) for allowing Kaniko to authenticate using ecr-login. Because I’m deploying my demo app on the same cluster that my agents are running on, I will update my already existing secret in AWS Secret Manager, and the External Secrets Operator will automatically update my Kubernetes secret that already exists, with the newly added keys. We need this as a Kubernetes secret because, in the Kaniko YAML definition file, we are mounting the access keys into environment variables.</li><li>You can read some more information about pushing to ECR via Kaniko at the following URL:</li></ul><p><a href="https://github.com/GoogleContainerTools/kaniko#pushing-to-amazon-ecr">GitHub - GoogleContainerTools/kaniko: Build Container Images In Kubernetes</a></p><ul><li>A custom tool called <strong>azure-devops-git-cloner</strong> Docker image needs to be available in your ECR to be pulled by Kaniko’s init-container to do the git clone from Azure Git.</li></ul><p><strong>azure-devops-git-cloner:</strong></p><p>To use Kaniko with Azure DevOps Agent, we need to add an init-container to clone our Azure Git repository and pass the code context to Kaniko. Since there is no native way of doing so, the alternatives are either writing a custom bash script within the init-container definition or, as I preferred of doing — using prepared code as a Docker image that will clone the repository for us by using the pipeline’s running token as credentials to git clone the repository via HTTPS (thus, no need to save the PAT anywhere).</p><p>I have prepared a repository with all the needed files: the code itself, Dockerfile, and even an Azure DevOps YAML pipeline ready to be created.</p><p>The pipeline expects to get the AWS credentials keys from a library group, and we can use the same group as I did in the first article. If you don’t have then create one that contains the keys. Alternatively, you can modify the pipeline to use a service connection instead. This is a personal preference decision; feel free to use it as needed.</p><p><a href="https://github.com/MoranWeissman/azure-devops-git-cloner">GitHub - MoranWeissman/azure-devops-git-cloner: code for simple cloning an azure devops repo over HTTPS using PAT</a></p><p>Before creating the pipeline, don’t forget to ensure you have a repository in ECR called azure-devops-git-cloner to push the Docker image.</p><p>Once the pipeline is created, run it and expect the following results:</p><blockquote>(The version of the Docker will be updated in the VERSION file only if the pipeline will run as part of CI and not manually, just a note)</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/864/1*Negl_NuUp93OCcCJLHM2ZA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/759/1*ITNROHO1fS3dKpvE7qmTeg.png" /></figure><p>We are creating a Kaniko pod because running the binary itself is currently not supported, and the only supported method is running it as a container. Hence, the pipeline will create a pod that will run the Kaniko container for us, meaning doing a “Docker run” without actually using a Docker. But for this to happen, we must allow our Agent pod the proper permissions to create a Pod in the cluster and a configMap since it’s part of the requirements.</p><p>Luckily, if you have followed the previous steps, I’ve already prepared the configuration in our GitOps repository (azdo-agents-gitops). I’ll remind you that in the Argo branch (eks-azdo-agents), under the folder “agents/templates/configurations/cluster” we have two files:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/357/1*mOrihK4CdezPlS3hWZMhYw.png" /></figure><p>These files contain the permissions level, which I found is needed for the agent to create the necessary resources in the cluster and bind the role to the relevant service accounts that will require the permissions to perform the actions.</p><p>Ensure proper permissions for the Builder user to commit the version change into the code. I write about it in chapter I, and these are the permissions required:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/1*15zUHMG5MGa7-Cqtk7FhUQ.png" /></figure><p>Let’s create the pipeline and run it! But before, ensure that the repository for the Docker image exists in ECR.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/743/1*sgeiPQJLN4bGAJpt2DFw-A.png" /></figure><p>After the pipeline finishes running, the expected result would be something like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GphzE61qv3EbHlBIKjnYRQ.png" /></figure><p>And now we have a Docker image in our ECR that was built with a containerized agent and published to ECR.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/746/1*4i7Yllli0g4Myl6CPmzS5w.png" /></figure><p>I hope this solution answers your needs or that you’ve learned something new :)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6eeaeb0c9f6f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Part II: Using GitOps with ArgoCD to Automate Azure DevOps Agent Deployment to EKS]]></title>
            <link>https://medium.com/@moran.weissman/part-ii-deploying-azure-devops-agents-to-eks-via-argocd-74403409a08a?source=rss-fe44884b7c71------2</link>
            <guid isPermaLink="false">https://medium.com/p/74403409a08a</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[argo-cd]]></category>
            <category><![CDATA[azure-devops]]></category>
            <category><![CDATA[azure-pipelines]]></category>
            <category><![CDATA[ci-cd-pipeline]]></category>
            <dc:creator><![CDATA[Moran Weissman]]></dc:creator>
            <pubDate>Mon, 27 Feb 2023 21:32:46 GMT</pubDate>
            <atom:updated>2023-03-09T08:32:57.433Z</atom:updated>
            <content:encoded><![CDATA[<p>In this article, we will continue from where we left off in the <a href="https://medium.com/@moran.weissman/part-i-building-and-deploying-azure-devops-agent-docker-images-to-ecr-5f3ec8838a43">previous article</a>.</p><p>It’s Argo time! To deploy Azure DevOps agents to EKS using the solution provided here, you must install ArgoCD on your cluster. While there are many ways to deploy it, we won’t dive into the specifics here. However, you can refer to this simple guide as a starting point:</p><p><a href="https://github.com/argoproj/argo-helm/tree/main/charts/argo-cd#installing-the-chart.">argo-helm/charts/argo-cd at main · argoproj/argo-helm</a></p><p>In addition to ArgoCD, we’ll be using several other tools, including:</p><ul><li><a href="https://helm.sh/docs/topics/library_charts/">Helm Library Charts</a></li><li><a href="https://github.com/kedacore/keda">KEDA</a></li><li><a href="https://github.com/external-secrets/external-secrets">External Secrets Operator</a></li></ul><p>By the end of the process, we’ll have an Azure DevOps agents running on our EKS, which will be connected to a dedicated agent pool in Azure DevOps.</p><p><strong>Connecting our CI with our ArgoCD</strong></p><p>Our main goal is to do GitOps whenever a new agent version is updated on one of our agent’s Helm charts.</p><p>To make the CI/CD as automatic as possible, there are several steps in our current Docker build pipeline that takes the new Docker tag version and updates it in the agent Helm chart values file.</p><p>Once ArgoCD identifies a change (reconciliation is every 3 minutes by default), it will change the state of our agent’s root application (App of Apps) to “Out of Sync,” which will allow us to click on Sync and update all the agents to the latest version. This also can be done automatically, and auto-sync can be enabled, but each use-case and its needs</p><p>First, Create a personal access token (PAT) in Azure DevOps.</p><p><a href="https://learn.microsoft.com/en-us/azure/devops/organizations/accounts/use-personal-access-tokens-to-authenticate?view=azure-devops&amp;tabs=Windows">Use personal access tokens - Azure DevOps</a></p><p>Our agent will need the following environment variable to be passed as a secret from Kubernetes:</p><ul><li>AZP_TOKEN: The personal access token (PAT) used for the agent to authenticate with the Azure DevOps organization. This is a secret and will be stored in our secret manager.</li></ul><p>When it comes to what level of permissions the token needs to have, the principle of least privilege is recommended, which means granting only the necessary permissions to perform the required actions. Just to remind you, these recommendations are general guidelines, and you should assess your specific use case to determine which permissions are needed.</p><p>Here are the permissions that require more than “Read” access, along with some recommendations:</p><ul><li>Agent Pools: Read &amp; manage</li><li>Build: Read &amp; execute</li><li>Code: Read &amp; write</li><li>Connected server: Connected server</li><li>Deployment Groups: Read &amp; manage</li><li>Environment: Read &amp; manage</li><li>Packaging: Read &amp; write</li><li>Pipeline Resources: Use</li><li>Pull Request Threads: Read &amp; write</li><li>Service Connections: Read &amp; query</li><li>Test Management: Read &amp; write</li><li>Tokens: Read &amp; manage</li><li>Wiki: Read &amp; write</li></ul><p>It’s important to remember that Personal Access Tokens are a form of credential that grants access to your Azure DevOps account or organization. Thus, they are secrets.<br> The environment variables that are not secrets will be passed via the ArgoCD application using the agent Helm chart.</p><p>I will store PAT as a secret in my AWS Secret Manager for this article. But it can be stored in whatever secrets providers you are using, as long as it is accessible to the Kubernetes cluster.</p><p>Let’s start configuring our agents before we deploy them using ArgoCD.</p><p>The agent configuration can be found in the eks-azdo-agents branch, which is in the azdo-agents-gitops repository.</p><p>To deploy the agents, we need to make sure the values files for both app-of-apps that we have, are correctly configured.</p><p>The following environment variables are not secrets and will be configured directly from the agent values file:</p><ul><li>AZP_URL: The URL of the Azure DevOps organization. <em>(i.e. https://dev.azure.com/&lt;organization&gt;)</em>. This is not a secret and will be passed via the Helm chart.</li><li>AZP_AGENT_NAME: The name of the agent. If this variable is not set, the hostname will be used as the agent’s name. (the Pod’s name)</li><li>AZP_POOL: The name of the agent pool to use. The “Default” pool will be used if this variable is not set.</li></ul><p>Before we continue, you need to ensure you have a dedicated agent pool in your Azure DevOps account. KEDA will listen to its queue.</p><p><a href="https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/pools-queues?view=azure-devops&amp;tabs=yaml%2Cbrowser">Create and manage agent pools - Azure Pipelines</a></p><p><strong>External Secrets Operator permissions</strong></p><p>Before deploying our agent, we must ensure the External Secrets Operator can fetch secrets from AWS Secret Manager by creating a proper IAM Role.</p><p>The following code exemplifies the permissions it needs and the trust relationship policy.</p><p>You can find instructions about how to create an IAM role in the link below:</p><p><a href="https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-user.html">Creating a role to delegate permissions to an IAM user</a></p><p>IAM Role Permissions example:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/440fcae6d05cce229530f8a2254fb763/href">https://medium.com/media/440fcae6d05cce229530f8a2254fb763/href</a></iframe><p>Trust relationship policy:</p><p>For this, you’ll have to provide the OIDC provider of your EKS.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0175ae1d78802bba7547081200d1bca6/href">https://medium.com/media/0175ae1d78802bba7547081200d1bca6/href</a></iframe><p><strong>Note: Keep your IAM Role ARN, we’ll need to configure it in the tools App of Apps values file.</strong></p><p><strong>Installing ArgoCD and deploying our agent</strong></p><p>Since this article is not about installing ArgoCD, I won’t dive into it and will assume you already have it installed.</p><p>If you don’t, I suggest installing it via Helm chart and later setting ArgoCD to manage itself via an independent application</p><p>You can find instructions for installing ArgoCD in the link below.</p><p><a href="https://argo-cd.readthedocs.io/en/stable/operator-manual/installation/">Installation - Argo CD - Declarative GitOps CD for Kubernetes</a></p><p><strong>So how do we set up and run our agent via GitOps?</strong></p><p>First, we need to connect our GitOps repository (azdo-agents-gitops) to ArgoCD, by adding it as a repository, and there are several ways to do it:</p><ul><li>Using the UI</li><li>Declaratively using a K8s Secret YAML file</li><li>Imperatively via the ArgoCD CLI</li></ul><p>Each way has its pros and cons, but we won’t dive into them in this article.</p><p>I will add the repository via the UI to keep it simple, but I encourage you to work as much GitOps as possible and <a href="https://argo-cd.readthedocs.io/en/stable/operator-manual/declarative-setup/#repositories">store the repositories on your Git</a>.</p><p><a href="https://argo-cd.readthedocs.io/en/stable/operator-manual/declarative-setup/#repositories">Declarative Setup - Argo CD - Declarative GitOps CD for Kubernetes</a></p><p>As for the SSH private keys, you can use <a href="https://github.com/bitnami-labs/sealed-secrets">Sealed Secrets</a> to encrypt them and, thus, not have them exposed in your Git.</p><p><strong>Prerequisites for connecting Azure DevOps Git Repository:</strong></p><ul><li>Login with a read only permissions user account to Azure DevOps</li><li>Create a private key for ArgoCD and store it in a secret vault</li><li><a href="https://learn.microsoft.com/en-us/azure/devops/repos/git/use-ssh-keys-to-authenticate?view=azure-devops">Add the public key to Azure DevOps on your read only user account</a></li></ul><p><a href="https://learn.microsoft.com/en-us/azure/devops/repos/git/use-ssh-keys-to-authenticate?view=azure-devops">Use SSH key authentication - Azure Repos</a></p><p><strong>To connect the GitOps repository:</strong></p><ol><li>Log in to the ArgoCD UI and navigate to the “Settings” page.</li><li>Click on the “Repositories” tab.</li><li>Click the “Connect repo using SSH” button.</li><li>In the “Name” field, provide a name for the repository.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/344/1*3qx0UK7QSroY6WFwzUn1gA.png" /></figure><p>5. In the “Project” field, select the project to which the repository belongs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/251/1*2Pvupk2Jb5-7gTU5FB6N0g.png" /></figure><p>6. In the “Repository URL” field, provide the SSH URL for the repository you want to connect.</p><p>7. In the “SSH private key data” field, paste in the contents of your SSH private key.</p><p>8. Click the “Connect” button to connect the repository to ArgoCD.</p><p>The outcome should be a connected repository:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*7oAazbKlgQs1kYETZpE92w.png" /></figure><p><strong>Deploying Azure DevOps agent to our EKS via ArgoCD</strong></p><p>And now, to the fun part, make some Argo magic.</p><p>We have our two App of Apps root applications ready to be deployed, but before we deploy them, we need to ensure the order in which they will be deployed.</p><p>We need KEDA and ESO (External Secrets Operator) to be deployed before the Agents because they depend on the following:</p><ul><li>An already fetched Kubernetes secrets from AWS Secret Manager by ESO</li><li>ScaledObject resource that is part of KEDA APIs</li></ul><p>Hence, we will remember that the first root application that needs to be synced is the tools app-of-apps.</p><p><strong><em>Note: there is a way of making all of that being done automatically, but it’s out of this article scope, and I’ll write an article about it.</em></strong></p><p><strong>To deploy our app-of-apps:</strong></p><ol><li>Open your terminal and open your cloned GitOps repository (from you Azure Git)</li><li>Git switch to your cluster branch, which contains ArgoCD YAML files.</li><li>Make sure you have kubectl installed and a connection to your EKS</li><li>Set your Kubernetes cluster context to the cluster you want to use</li><li>Create the tools app-of-apps on the namespace that argocd is installed on, usually argocd.</li></ol><pre>helm template tools/ | kubectl apply -f - -n argocd</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/584/1*jNTGiKtkvt7YadyYBr6MTg.png" /></figure><p>Once finished, open your ArgoCD UI and see that External Secrets Operator and KEDA were deployed and the root application that manages them.</p><p>This type of installation of ArgoCD applications via kubectl should happen only the first time you want to create the App of Apps, after it, the applications state will be managed by the Git.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*nCo8AVGYosMqOhq95OJZXA.png" /></figure><p>After clicking SYNC to the tools-root:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*8xvgSvrEZf1QesluE9I8vA.png" /></figure><p>6. Repeat the previous step, but instead of creating the tools app-of-apps, create the agents app-of-apps.</p><pre>helm template agents/ | kubectl apply -f - -n argocd</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/524/1*W-ossfBLvMxEMLTsmBa9iA.png" /></figure><p>Once the root application is created, we’ll need to sync it so that it will make the relevant AppProjects as well.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/439/1*DDJ-9Xji6iBU1mdJpZ7Y7A.png" /></figure><p>After syncing:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/613/1*6_qC0nyNTa3IVJ21wZbjpQ.png" /></figure><p>In the ArgoCD UI, filter the projects to show only the agent’s project:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/226/1*pvvAyhJTyvR4mbUpNR1eCg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*wCUHhH-CJvlOIvdoUzBpGQ.png" /></figure><p>Now, our agents are deployed and managed via ArgoCD in our managed Kubernetes (EKS).</p><p>In Azure DevOps, let’s see if our agents are registered to our Agents pool that we’ve created:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*Skx9FQYbMMC58aI5RFPn1Q.png" /></figure><p>So far, we’ve covered the following topics:</p><ul><li><strong>Building and deploying Azure DevOps agent Docker images to ECR</strong></li><li><strong>Deploying Azure DevOps agents to EKS via Azure DevOps pipelines and ArgoCD</strong></li></ul><p>The following is a flow chart of the final architecture as it would be:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_KawJT_W0yN1RyTqdzzycQ.png" /></figure><p>But before we continue to the next chapter, here is a question that needs to be asked: do you think these two App-of-Apps should exist? If there is a dependency between the two of them, and currently there is no native support for putting waves on ArgoCD applications.. maybe it’s better to use just regular apps for the tools?</p><p>Let me know what you think in the comments below :)</p><p>Next Chapter: <a href="https://medium.com/@moran.weissman/part-iii-building-and-pushing-a-docker-image-using-kaniko-to-ecr-6eeaeb0c9f6f">Building Docker images in Kubernetes with Kaniko and Azure DevOps</a></p><p><a href="https://medium.com/@moran.weissman/part-iii-building-and-pushing-a-docker-image-using-kaniko-to-ecr-6eeaeb0c9f6f">Part III: Building and pushing a Docker Image Using Kaniko to ECR</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=74403409a08a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Part I: How I Built and Deployed Azure DevOps Agents to ECR]]></title>
            <link>https://medium.com/@moran.weissman/part-i-building-and-deploying-azure-devops-agent-docker-images-to-ecr-5f3ec8838a43?source=rss-fe44884b7c71------2</link>
            <guid isPermaLink="false">https://medium.com/p/5f3ec8838a43</guid>
            <category><![CDATA[azure-pipelines]]></category>
            <category><![CDATA[argo-cd]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[keda]]></category>
            <category><![CDATA[azure-devops]]></category>
            <dc:creator><![CDATA[Moran Weissman]]></dc:creator>
            <pubDate>Mon, 27 Feb 2023 21:06:36 GMT</pubDate>
            <atom:updated>2023-03-09T09:24:47.941Z</atom:updated>
            <content:encoded><![CDATA[<p>In this article, we will continue from where we left off in the <a href="https://medium.com/@moran.weissman/overcoming-challenges-with-azure-devops-pipelines-how-i-deployed-self-managed-agents-on-ae30c5556291">previous article</a>.</p><p>To get started, you must create two new repositories in your <a href="https://learn.microsoft.com/en-us/azure/devops/repos/get-started/what-is-repos?view=azure-devops">Azure Repos Git</a> that will hold the Agent Dockerfiles, script provided by Microsoft and the CI pipeline YAML file, and ArgoCD manifests.</p><p><a href="https://learn.microsoft.com/en-us/azure/devops/repos/git/create-new-repo?view=azure-devops">Create a new Git repo in your project - Azure Repos</a></p><p>To start, you can clone the following repositories from my personal GitHub, which contains all that is needed (azdo-docker-agents, azdo-agents-gitops), and use them or push them to your repositories.</p><ul><li><a href="https://github.com/MoranWeissman/azdo-agents-gitops">GitHub - MoranWeissman/azdo-agents-gitops: GitOps repository for deploying Azure DevOps agents on Kubernetes</a></li><li><a href="https://github.com/MoranWeissman/azdo-docker-agents">GitHub - MoranWeissman/azdo-docker-agents: Repository for creating Azure DevOps Docker agents</a></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zVIjt5PO57Ivkn2r8JWa6g.png" /></figure><p>While the <strong>azdo-docker-agents</strong> repository has a standard structure for repositories that build Docker images, this is not the case in <strong>azdo-agents-gitops</strong>, which is more unique (and one of many possible ways) that I chose to implement GitOps.</p><p><strong>The main branch will act as the GitOps branch and will be monitored by ArgoCD.</strong></p><p><strong>It will have the following:</strong></p><ul><li>Library Helm Chart for the Agents contains what is needed to deploy an agent on Kubernetes.</li><li>Agent’s Helm charts that use the Library Chart as a dependency</li></ul><p><strong>On a branch representing our ArgoCD applications, called eks-azdo-agents, we’ll have two App-of-Apps:</strong></p><p><strong>(you can read more about App of Apps </strong><a href="https://argo-cd.readthedocs.io/en/stable/operator-manual/cluster-bootstrapping/#app-of-apps-pattern"><strong>here</strong></a><strong>)</strong></p><p><strong>agents/</strong></p><ul><li>An <strong>applications</strong> folder for the ArgoCD agent’s applications, configurations, and secrets</li><li>An <strong>eso-secrets</strong> folder containing the External Secrets Operator CRDs, which in turn contain the secrets to be fetched from the AWS secret manager.</li><li>A <strong>configuration</strong> folder for the argo-cd-cm, argocd-rbac-cm, cluster-role-agent, and cluster-role-binding-agent. The cluster role and role binding are necessary to allow the agent to create a Kaniko pod during the pipeline for building and deploying Docker images. We will talk about it later.</li><li>A <strong>projects</strong> folder that contains the ArgoCD appProjects</li><li>The Agents root application that manages the rest of the applications</li></ul><p><strong>tools/</strong></p><ul><li>An <strong>applications</strong> folder for the External Secrets Operator (ESO), ESO Secrets and KEDA</li><li>A <strong>projects</strong> folder that contains the ArgoCD appProjects for configurations and tools.</li><li>The tools root application that manages the rest of the applications</li></ul><p>Once the repositories are created, we can create the CI pipeline to build and deploy the Docker images to ECR.</p><p>The CI pipeline which exists in the <strong>azdo-docker-agents</strong> repository will perform the following actions:</p><ul><li>Build Docker images for generic use and Python 3.8.</li><li>Use the generic Docker image as a base for other framework agents such as Python, Java, and Dotnet.</li><li>Update all relevant Dockerfiles with the recent whenever a new base image is built.</li></ul><p>However, before creating the pipeline, we must secure our AWS credentials (Access and Secret access keys) by saving them in a secret management system.</p><p>For this article, we will use Azure DevOps Library Groups and seal the variables that contain the credentials so that they are only accessible to the pipeline to consume.</p><p>The variable names can be changed to suit your needs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*SlK6ne5A8UKpeQZ_scQFbw.png" /></figure><p>An essential part of the pipeline is to increment the Docker image version each build and update the</p><p>the VERSION file (both in the root and the pyhon38 directory), the Dockerfile base image of specific frameworks, and the Docker image tag in the Helm chart of the Agent, which is in the azdo-agents-gitops repository using a commit &amp; push.</p><p>To push the commit to the Agents and GitOps repositories, you must provide the necessary permissions for the Azure DevOps builder user.</p><p>Follow these steps: (on both repositories)</p><ol><li>Select your repository from your Azure DevOps project.</li><li>Click on the “Branches” tab and select the main branch.</li><li>Click on the three-dot menu and select “Branch security.”</li><li>Under “Users and groups,” select the builder user.</li><li>Ensure the following permissions are enabled in the “Permissions” section.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/1*15zUHMG5MGa7-Cqtk7FhUQ.png" /></figure><p>After providing the necessary permissions, you can create the pipeline and point it to the “azdo-docker-agents” repository. To create the pipeline, follow the instructions in this link:</p><p><a href="https://learn.microsoft.com/en-us/azure/devops/pipelines/create-first-pipeline?view=azure-devops&amp;tabs=java%2Ctfs-2018-2%2Cbrowser#create-your-first-pipeline-1">Create your first pipeline</a></p><p>And use the <a href="https://github.com/MoranWeissman/azdo-docker-agents/blob/main/azure-pipelines.yaml">azure-pipelines.yaml</a> which can be found in the agents Docker repository.</p><p>Once you finish reviewing the pipeline YAML, click on “Save.”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*PhEfZyyRucUNcQYEdR8MOg.png" /></figure><p>Before running the pipeline, you need to make some adjustments to the variables YAML file, which is in the folder .azure-devops/azure-pipeline-variables.yaml</p><p>You can play with the values as needed, but the main change will be in the AWS Account ID and the region.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9503ad5c53d50ae2afd29d7700403821/href">https://medium.com/media/9503ad5c53d50ae2afd29d7700403821/href</a></iframe><p>Now decided to build our first generic Agent, which will also be the base image for the Python agent.</p><p>Please create a corresponding ECR repository so the image can be pushed. Ensure that you have a corresponding ECR repository already created so that the image can</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/549/1*nETyseUHTESWEzZIY9tadg.png" /></figure><p>If you decided to store your credentials in Azure DevOps library groups, after triggering the pipeline, it will request permission to access it.</p><p>Finally, you should see a screen that shows the pipeline has been completed successfully.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/858/1*0pOlAxSJ3G3nFfa7pP3vAg.png" /></figure><p>Upon successfully running the pipeline, the Azure DevOps agent Docker image has been deployed to our ECR repository.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/786/1*78nS-kAhM4p_RWV9W48VUg.png" /></figure><p>Let’s continue to the Python agent. The process is similar. Remember that the repository in ECR needs to exist before the pipeline runs; otherwise, you will get a 404 when the pipeline tries to push the image.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/792/1*rXcVAl0d6IrbG67WOfLtiA.png" /></figure><p>Next Chapter: <a href="https://medium.com/@moran.weissman/part-ii-deploying-azure-devops-agents-to-eks-via-argocd-74403409a08a">Automating Azure DevOps Agent Deployment to EKS with ArgoCD and GitOps</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5f3ec8838a43" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Unleashing the Power of Scalability: Deploying Self-Managed Agents on K8s for Azure DevOps…]]></title>
            <link>https://medium.com/@moran.weissman/overcoming-challenges-with-azure-devops-pipelines-how-i-deployed-self-managed-agents-on-ae30c5556291?source=rss-fe44884b7c71------2</link>
            <guid isPermaLink="false">https://medium.com/p/ae30c5556291</guid>
            <dc:creator><![CDATA[Moran Weissman]]></dc:creator>
            <pubDate>Sun, 26 Feb 2023 19:33:44 GMT</pubDate>
            <atom:updated>2023-03-09T08:12:11.927Z</atom:updated>
            <content:encoded><![CDATA[<h3>Unleashing the Power of Scalability: Deploying Self-Managed Agents on K8s for Azure DevOps Pipelines</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ftnGiX3gFJBDF0Ff6ejOMQ.jpeg" /></figure><p>As a developer or part of a DevOps team, you may have encountered challenges when running pipelines on virtual machines or managed VMs provided by Microsoft Azure. These challenges can include the following:</p><ul><li>Maintenance concerns, such as patching and upgrading, can be time-consuming and may cause downtime. Additionally, managing security for multiple VMs can be challenging.</li><li>Storage space is limited and may run out, especially when working with large datasets or multiple projects.</li><li>Reliability on local IT infrastructure, such as on-premises servers, can be limiting and may cause issues if the infrastructure goes down or needs maintenance.</li><li>Limited run-time for Microsoft-hosted agents, up to 60 minutes for private projects.</li></ul><p>To overcome these challenges, I recommend using self-managed Azure DevOps agents running on Kubernetes.</p><h3>What are Self-Managed Azure DevOps Agents?</h3><p>Self-managed Azure DevOps agents are containers you can deploy and manage on a Kubernetes cluster. Using self-managed agents, you can use Kubernetes benefits, including auto-scaling, self-healing, and storage capacity. In this article, I’ll cover how to build, deploy, and use Azure DevOps agents as containers on the Kubernetes cluster using various services, including <a href="https://argo-cd.readthedocs.io/en/stable/">ArgoCD</a>, Helm, <a href="https://keda.sh/">KEDA</a>, <a href="https://github.com/GoogleContainerTools/kaniko">Kaniko</a>, and External Secrets Operator (ESO).</p><figure><img alt="Azure DevOps Agents on Kubernetes solution overview" src="https://cdn-images-1.medium.com/max/1024/1*xFSQOO5GYinfEbfsF2QAAw.png" /></figure><h3>Solution Overview</h3><p>Using self-managed agents on Kubernetes, I could guarantee that my pipelines had the necessary resources to run seamlessly and efficiently. The ability to spin up new agents on demand and unlimited storage space allowed me to build and deploy docker images without delay.</p><p>Azure DevOps, as of today, does not have a native solution for running Agents on containers, they have provided a Dockerfile and a script that needs to be implemented in it, and that’s it. This solution tackles the challenge of running the Agents on Kubernetes, as well as:</p><ul><li>Implementing <a href="https://www.weave.works/technologies/gitops/">GitOps </a>for easier maintenance.</li><li>Deploying agents via ArgoCD</li><li>Using KEDA for scaling the Agents as needed by listening to the Agents pool queue</li><li>Providing the ability to build Docker images using Kaniko</li></ul><p>And in general, giving a creative and scalable automation solution.</p><p>In addition, I will provide several different frameworks agents and a pipeline example for building a Docker image within the Agent using Kaniko.</p><p>I will give you the technical expertise needed to implement a similar solution in your environment and achieve the same results as I did.</p><h3>Prerequisites</h3><p>Before diving into the technical details, here are some prerequisites you’ll need to have in place:</p><ul><li>An AWS account</li><li>Amazon Elastic Kubernetes Service (EKS)</li><li>ArgoCD installed</li><li>Azure DevOps</li></ul><p>also, I highly recommend that you will have prior knowledge in Azure Pipelines, ArgoCD, Kubernetes, and Docker to understand and implement this solution.</p><h3>Implementation Steps</h3><p>To make this technical guide easier to navigate, I’ve broken it up into the following parts:</p><p><a href="https://medium.com/@moran.weissman/part-i-building-and-deploying-azure-devops-agent-docker-images-to-ecr-5f3ec8838a43"><strong>Part I: How I Built and Deployed Azure DevOps Agents to ECR:</strong></a> First, you’ll need to build the Docker images for the agents and push them to Amazon ECR. This is where your images will be stored and accessed by Kubernetes.</p><p><a href="https://medium.com/@moran.weissman/part-ii-deploying-azure-devops-agents-to-eks-via-argocd-74403409a08a"><strong>Part II: Automating Azure DevOps Agent Deployment to EKS with ArgoCD and GitOps:</strong></a> Once you’ve built the images, you’ll deploy them to Amazon EKS using ArgoCD. This will ensure that your agents run on the Kubernetes cluster and can be used in your pipelines.</p><p><a href="https://medium.com/@moran.weissman/part-iii-building-and-pushing-a-docker-image-using-kaniko-to-ecr-6eeaeb0c9f6f"><strong>Part III: Building Docker images in Kubernetes with Kaniko and Azure DevOps</strong></a><strong>: </strong>Finally, you’ll build a Docker image using Kaniko. Kaniko is a tool that allows you to build Docker images in Kubernetes without needing privileged access to the host.</p><h3>Disclaimer</h3><p>While I recommend using the specific technologies and tools outlined in this article, it’s important to note that there are always alternatives. The same result can be achieved with other tools as well.</p><p>For example, Helm library charts are not mandatory, and you can use multiple Helm charts. However, if you have dozens of different agents, managing changes that will need to occur for all of them could be challenging.</p><p>Although I chose to use the <a href="https://external-secrets.io/v0.7.2/">External Secrets Operator</a> (ESO) to fetch secrets, it’s important to note that ESO is not mandatory when working with Azure DevOps pipelines. While ESO is a well-accepted tool in the CNCF and recommended by ArgoCD, other options are available for fetching and storing secrets, such as <a href="https://github.com/bitnami-labs/sealed-secrets">sealed secrets</a> or <a href="https://secrets-store-csi-driver.sigs.k8s.io/">Secrets Store CSI Driver</a>.</p><p>Next Chapter: <a href="https://medium.com/@moran.weissman/part-i-building-and-deploying-azure-devops-agent-docker-images-to-ecr-5f3ec8838a43">How I Built and Deployed Azure DevOps Agents to ECR</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ae30c5556291" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>